CS505: Natural Language Processing

How can we get computers to understand and generate human language? This is among the most challenging—and currently, the most quickly advancing—approaches in contemporary artificial intelligence. Natural language systems are deployed in the world in increasingly many forms: chatbots, code assistants, web agents, among others. This course provides an introduction to the engineering and science that underlies current NLP systems.

Prerequisites

Probability
Proficiency in Python programming
Linear algebra
Calculus

Highly recommended prerequisites: Not required, but it will be very useful to have taken a machine learning course before taking this one. Check out these resources to help get you acquainted with the basics:

The official PyTorch tutorial
A more brief PyTorch tutorial
The Illustrated Transformer
This short online book about neural networks and deep learning (Chapters 1, 2, and 5), and this video series by 3Blue1Brown on neural networks and backpropagation (Chapters 1 through 4)
The HuggingFace course

Learning objectives
Students will:

Gain exposure to foundational ideas in NLP.
Understand the theory underlying current NLP ideas.
Learn how to implement each element of the contemporary NLP pipeline.

Logistics

Instructor: Aaron Mueller (amueller@bu.edu)
Teaching Fellow: Ge Gao (ggao02@bu.edu)

New: Some lab/discussion sessions will be used for homework help; these are entirely optional! See schedule below (rows highlighted in pink); these will only happen on weeks where there is an assignment due. Come with questions, or just to co-work on the assignments.
Lectures: Tuesdays and Thursdays 3:30pm - 4:45pm, CAS 522
Office hours:
- Tue., 5:00pm - 6:30pm - Aaron (CDS 806)
- Thu., 11:00am - 1:00pm - Ge (Blue collaborative space near CDS 727)
- Thu., 5:00pm - 6:00pm - Ge (On Zoom; see "Helpful links" pinned post on Piazza for link)
- Fri., 2:00pm - 3:30pm - Aaron (CDS 806)
News and announcements: All news and announcements will be made in class and posted on this page.
Changes: The instructor reserves the right to make changes to the syllabus or project due dates. These changes will be announced as early as possible.

For questions, join the Piazza! Entry code: 5uzfqw5vwqd
For uploading homeworks/projects, use Gradescope. Entry code: WNNEKG

News

Feb. 3: Homework 1 has been released! Code and data are linked here (and in the PDF). Start early: this assignment will have you implement a BPE tokenizer, an n-gram language model, and an RNN language model, and then answer a few conceptual questions. This should be doable, as we've provided a great deal of code and have covered (or will soon cover) each of these concepts in class. But it may take some time to wrap your head around some of the implementation details. This will be due Feb. 19, in just over two weeks.
New homework help sessions! On Mondays during the weeks when homeworks are due, we'll hold optional homework help sessions. These will happen at your lab sections. Come with questions or to co-work—these will work a lot like office hours!
Jan. 22: Homework 0 is out! Code and data are linked here (and in the PDF). This is meant to be an introduction to programming with PyTorch, and will cover the topics from our lectures on text classification and (to a much smaller degree) tokenization and n-grams. Use the PyTorch tutorials above, and use the tips on the class slides for text classification!

Readings

We will be loosely following this book throughout the course: Jurafsky & Manning (J&M), available online for free. You may find it helpful to do these readings to prepare questions for lecture or to review the content in more depth after lecture. Content in the J&M book may appear on the exam; however, you will not be expected to know the topics in the textbook that are not covered in lecture.

I will also provide links to optional readings and resources related to the content we cover in class. These can be found in the schedule below. These are to supplement the course material for those interested in reading further.

Course Schedule

Note: we will almost definitely alter this schedule! Order may also change depending on the availability of guests.

Date	Topic	Homework	Readings
Jan 20, 2026	Course introduction What is NLP? Overview of course topics [Slides]	Prereq review released (ungraded) Prereq review solutions	No readings.
Jan 22, 2026	Text classification Logistic regression Gradient descent Features Machine learning basics [Slides]	HW0 released [Code and data]	J&M: Chapter 4 Optional: Review of linear algebra: Kolter (2017)
Jan 27, 2026	Introduction to language modeling Tokenization The type-token distinction Feature engineering [Slides]		J&M: Chapter 2 Optional: BPE: Sennrich et al. (2016)
Jan 29, 2026	Sequence modeling Review of probability theory N-grams [Slides]		J&M: Chapter 3 J&M: Appendix C (mistakenly called B in the text) Optional: Review of probability: Maleki & Do
Feb 2, 2026	Homework 0 help session: Text classification
Feb 3, 2026	Neural sequence modeling I Feed-forward neural networks Backpropagation Embeddings [Slides]	HW0 due HW1 released [Code and data]	J&M: Chapter 5 J&M: Chapter 6 Optional: Backpropagation: Rumelhart et al. (1986) word2vec: Mikolov et al. (2013) GloVe: Pennington et al. (2014)
Feb 5, 2026	Neural sequence modeling II Recurrent neural networks LSTMs This lecture is pre-recorded. You can watch it [here]. [Slides]		J&M: Chapter 13 Optional: Great blog post on RNN language models: Karpathy (2015) RNN cheatsheet: Stanford CS 230 RNN LMs:Mikolov et al. (2010) LSTMs: Hochreiter & Schmidhuber (1997)
Feb 10, 2026	Attention Transformers Parallel processing		J&M: Chapter 8 Optional: The Illustrated Transformer: Alammar (2018) Transformers: Vaswani et al. (2017) The first big attention paper: Bahdanau et al. (2014) More attention: Luong et al. (2015)
Feb 12, 2026	Large language models I Pre-training Autoregressive language models: GPT Scaling laws Generation methods	HW2 released	J&M: Chapter 7 Optional: GPT-2: Radford, Wu et al. (2019) GPT-3: Brown et al. (2020) Scaling laws: Kaplan et al. (2020) Nucleus sampling: Holtzman et al. (2019)
Feb 17, 2026	Homework 1 help session: Language modeling Note: This is a Monday schedule!
Feb 19, 2026	Large language models II Masked language models: BERT Sequence-to-sequence models: T5	HW1 due	J&M: Chapter 10 Optional: BERT: Devlin et al. (2018) T5: Raffel et al. (2019)
Feb 24, 2026	Language model use and adaptation Prompting In-context learning Fine-tuning Continued pre-training Low-rank adaptation (LoRA)		Optional: A good continued pre-training method: Gururangan et al. (2020) Why do demonstrations help? Min et al. (2022) Chain of thought: Wei et al. (2022) LoRA: Hu et al. (2021)
Feb 26, 2026	Machine translation Linguistic typology Beam search Evaluation: BLEU, human rating, embedding-based methods		J&M: Chapter 12 Optional: Statistical phrase-based machine translation: Koehn et al. (2003) Learning to translate by prompting an LM with a grammar book: Tanzer et al. (2023) BLEU: Papineni et al. (2002) The first neural MT approach to outperform statistical approaches: Bahdanau et al. (2014) A recent survey of MT research: Ataman et al. (2025)
Mar 2, 2026	Homework 2 help session: Transformers
Mar 3, 2026	Post-training Instruction tuning Reinforcement learning from human feedback (RLHF) Direct preference optimization (DPO)	Final project proposal released	J&M: Chapter 9 Optional, but highly recommended: Instruction tuning: Wei et al. (2021) RLHF: Ouyang et al. (2022) Quick guide to key concepts in reinforcement learning: OpenAI post DPO: Rafailov et al. (2024)
Mar 5, 2026	Morphology and syntax I Intro to morphology and syntax Sequence tagging POS tagging Hidden Markov models (HMMs)	HW2 due HW3 released	J&M: Chapter 17 J&M: Appendix A
Mar 10, 2026	Spring break - no class
Mar 12, 2026	Spring break - no class
Mar 16, 2026	Homework 3 help session: Syntax and parsing
Mar 17, 2026	Morphology and syntax II (Probabilistic) context-free grammars Constituency parsing CKY		J&M: Chapter 18 J&M: Appendix E (through E.2)
Mar 19, 2026	Morphology and syntax III Dependency grammars Dependency parsing Shift-reduce	HW3 due Friday, Mar. 20	J&M: Chapter 19
Mar 24, 2026	Exam
Mar 26, 2026	Semantics Argument structure Semantic role labeling		J&M: Chapter 21
Mar 31, 2026	Review of exam Discourse and pragmatics The structure of conversation Coreference resolution	Final project proposal due	J&M: Chapter 23
Apr 2, 2026	NLP applications and benchmarking The NLP community Paraphrase detection Natural language inference Multiple-choice question answering Open-ended question answering		Optional: GLUE: Wang et al. (2018) MMLU: Hendrycks et al. (2020) SQuAD: Rajpurkar et al. (2016) Dynabench: Kiela et al. (2021)
Apr 7, 2026	Interpretability and evaluation I Best practices in evaluation Out-of-distribution generalization		Optional: A survey of generalization research in NLP: Hupkes et al. (2023) Syntactic generalization in Transformers: Ahuja et al. (2024)
Apr 9, 2026	Interpretability and evaluation II (Mechanistic) interpretability Model editing		Optional: Targeted model editing: Meng et al. (2022) Distributed alignment search: Geiger et al. (2023) A guide to saliency map methods: Molnar (2019) A survey of probing papers: Belinkov (2022) A survey of mechanistic interpretability: Mueller et al. (2025)
Apr 14, 2026	Retrieval and tool use		Optional: Retrieval-augmented generation (RAG): Lewis et al. (2020) Toolformer: Schick et al. (2023)
Apr 16, 2026	LLM bias, safety, and fairness	Midway report due	Optional: Implicit biases in LLMs: Bai et al. (2025) A survey on bias and fairness in LLMs: Gallegos et al. (2024) Debiasing may not always work as intended: Gonen & Goldberg (2019) Controlling hallucinations and toxicity in LLMs: Chen et al. (2025)
Apr 21, 2026	Multimodal NLP Language grounding Vision language models		Optional: Winoground: Thrush et al. (2022) LLaVA: Liu et al. (2023) CLIP: Radford et al. (2021)
Apr 23, 2026	Human language processing Surprisal theory Language acquisition Developmentally plausible pre-training		Optional: Surprisal theory: Levy (2008) How many words do children hear? Gilkerson et al. (2017) BabyLM: Warstadt, Mueller et al. (2023)
Apr 28, 2026	Guest lecture
Apr 30, 2026	Final project help session
TBD		Final report due

List of Topics

By the end of this course, you should be familiar with each of the following topics. Items with an asterisk* may be on the exam.

The history of natural language processing:
- Claude Shannon and the invention of language models
- The Turing test
- ELIZA
- Statistical methods
- Neural methods
Tokenization:
- The type–token distinction*
- Words*
- Characters*
- Bytes*
- Unicode*
- UTF-8*
- Byte-pair encoding (BPE)*
Classification:
- Binary logistic regression*
- Multinomial logistic regression*
Nonlinear activation functions:
- Sigmoid*
- Softmax*
- ReLU*
Machine learning basics:
- Loss functions*, including:
- Gradients*
- Gradient descent*
- Stochastic gradient descent*
- Learning rates*
- Mini-batching*
- Train-test splits*
Neural network architectures:
- Feedforward layers*
- Recurrent neural networks*
- Long short-term memory networks (LSTMs)*
- Transformers*
Language model architectures:
- N-gram language models*
- Recurrent neural networks language models*
- Transformer language models*
- Autoregressive language models*
- Masked language models*
- Sequence-to-sequence language models*
Language modeling methods:
- Next-token prediction*
- Pre-training*
NLP tasks:
- Sentiment classification*
- Language identification*
- Spam detection*
- Toxicity detection
- Machine translation*
- Question answering (QA):
Generation methods:
- Greedy decoding*
- Top-k sampling*
- Nucleus sampling*
- Beam search*
Language model training:
- Pre-training*
- Scaling laws*

Knowing thy data:
- Corpora and their construction*
- Documents vs. paragraphs vs. sentences*
- Domain shift
- Temporal shift
Prompting:
- In-context learning*
- Chain-of-thought prompting*
Fine-tuning and adaptation:
- Continued pre-training*
- Fine-tuning*
- Low-rank adapters*
Evaluation:
- Precision, recall, and F1*
- Accuracy*
- Surprisal and perplexity*
- Out-of-distribution (OOD) generalization
Reinforcement learning basics:
- Policy*
- Reward functions*
Post-training methods:
- Reinforcement learning from human feedback (RLHF)*
- Direct preference optimization (DPO)*
- Reinforcement learning with verifiable rewards (RLVR)*
Syntax and parsing:
- Constituencies*
- Dependencies*
- Context-free grammars*
- CKY*
- Shift-reduce*
- The context-sensitivity of natural language
Morphology:
- Morphemes*
- Analytic vs. synthetic languages
- Fusional vs. agglutinative languages
Semantics:
- Semantic role labeling
- Theta roles
Discourse:
- Coreference resolution
- Pragmatic inference
Multilingual NLP:
- High- vs. low-resource languages
- Linguistic typology
- Language imbalance*
Interpretability:
- Saliency maps
- Circuits
- Shortcuts and spurious correlations
- Causation
Retrieval:
- Retrieval-augmented generation (RAG)
- Dense passage retrievers (DPR)
Agents
- Language models as agents
- Tool use
- Multi-agent frameworks
Social impacts and considerations:
- AI safety
- Bias, fairness, and toxicity
- Factuality and hallucinations

Grading

The course is graded out of 100 total points.

Homeworks: 15 points

Homework 0: 4 points
Homework 1: 4 points
Homework 2: 4 points
Homework 3: 3 points

The homeworks are largely for your benefit as study tools. You may use AI in any way you wish to complete the homeworks, but you will find studying for the exam much easier if you understand the methods you'll be implementing in the homeworks. Regardless of whether you decide to use AI tools, you (the student) are fully responsible for what you submit.

Exam: 35 points

There will be one exam about 2/3 of the way through the course. See the list of topics above for a guide to what the exam will cover. You may not use any electronic resources for the exam; this includes the textbook, AI tools, the internet, text messages, among other items.

This will be an open-note exam! If you bring notes, they must be on one physical piece of paper (front and back).

Final project: 50 points

This is an open-ended project where you will review and pursue an NLP topic of your choosing.

Project proposal: 5 points
Perform a literature review on your topic of choice. Also briefly describe your planned project, including the task you'll be focusing on, your data, methods, baselines, and evaluation.
Midway report: 15 points
Report your progress on the final project thus far. By this point, you should have run some experiments and obtained some preliminary results. Outline your plan for the rest of the project.
Final report: 30 points
Describe your experiments, present your results, and report your findings in the style of a typical NLP paper.

Grading of the final project will be based on the following:

Proposal: Well-motivated idea with a concrete experimental plan. Creativity is encouraged more than bulletproof-but-incremental ideas!
Midway report:
- Clear problem definition and motivation
- The most essential related work is present
- Reasonable set of initial experiments
- Clear presentation of methods and evaluation protocol
- Clear presentation and description of preliminary results
- Well-reasoned discussion about your initial experiments. If successful, what do the experiments tell us so far? If they didn't turn out how you expected, why do you think this was?
- Concrete plan for the rest of the project. It's ok to pivot from your original plan if needed!
Final report: Much like the midway report, but in addition:
- The related work should be reasonably complete and well-integrated throughout the report
- More complete methods section
- More complete results section
- Rigorous evaluation
- Discussion and conclusion composed of well-formulated arguments, grounded in your experimental findings and the broader literature
- Novelty and creativity

Can we publish our final project? It is feasible to convert a course project into an academic publication, but it can take a lot of work! I encourage those interested to discuss this with me at the end of the semester.

Policies and Conduct

Outside Resources & AI Policy

AI tools are completely allowed for the homeworks. I recommend doing the assignments on your own as exam preparation, but for the purpose of grading, you can complete the assignments completely with AI if you so choose. It is the student's responsibility to verify any submitted content.

AI tools are allowed for the final project. Our policy here is more nuanced: you may use AI as a tool, but do not use AI as a crutch or replacement for thinking. What's the difference? AI as a tool includes:

Using AI to help you workshop your initial ideas
Using AI to refine your writing, including spell-checking, reorganization, among other copyediting-like actions
Using AI to help you debug code

AI as a crutch/replacement includes:

Using AI to come up with your final project idea for you from scratch
Using AI to write all of your code
Using AI to generate your reports

The line between tool and crutch can be fuzzy, so if you're unsure, I recommend asking ahead of time! I promise not to judge if you ask before you turn in the assignment. :)

No AI tools are allowed during exams. These will be hand-written in class.

I strongly encourage you to use any outside source at your disposal when doing the homework and your final project. Your reports and code should be original, but you may take inspiration from existing papers as long as you give them proper credit. When doing your project, feel free to base your implementations on publicly available code as well (as long as you make significant modifications to accommodate your original idea), but be sure to give proper credit in your report and your GitHub README if you do so.

For the final project, failing to properly cite an outside source is equivalent to taking credit for ideas that are not your own, which is plagiarism.

Academic Integrity

Read through BU's Academic Conduct Code. All students are expected to abide by these guidelines. In the context of this class, it's particularly important that you cite the source of your ideas, facts, and/or methods, and do not claim someone else's work as your own.

You are allowed to work in groups to do the homeworks, but you should upload your homework individually.

Absence and Late Work Policy

Attendance will not be taken. Attend lectures as you wish.

You have 6 free late days that you can use however you wish with no excuse necessary. Using a late day means that you can still receive full credit for the assignment with no late penalty. Turning in an assignment late after using all your late days will incur a 10% drop in the score for each late day. 5 days after an assignment's due date, the assignment can no longer be turned in, regardless of whether you use your free late days. This applies to all homeworks. It also applies to the proposal and midway report for the final project—but not the final report, which must be turned in on time. Note that a late day is a step function: turning in a homework 5 minutes late is equivalent to turning it in 23 hours late, so if you know you'll be late, we recommend taking the extra time to verify your understanding of the material.

Extensions can be negotiated in cases of medical emergency or other sudden pressing circumstances. Students should contact the course staff ASAP and negotiate before the assignment's original due date. If this applies to the first homework, please come talk to us the first day of class.

Exams cannot easily be made up. If you know you cannot make an exam day, you must notify us at least 14 days in advance so that we can make alternate arrangements.

Accommodations

Boston University's policy is to provide reasonable accommodations to students with qualifying disabilities who are enrolled in Boston University courses. Students seeking accommodations must engage in an interactive process with, and provide appropriate documentation of their disability to, Disability & Access Services (DAS). If this applies, please get in touch with me as soon as possible to discuss accommodations; note that students are not required to disclose information regarding their disability, if applicable, but should request approval for such accommodations through DAS beforehand.

Religious Observance

Students are permitted to be absent from class, including classes involving examinations, labs, excursions, and other special events, for purposes of religious observance. In-class, take-home and lab assignments, and other work shall be made up in consultation with the student's instructors. More details on BU's religious observance policy are available here.

Acknowledgments

Much of the content in this course was inspired by NLP courses taught by Greg Durrett, John Hewitt, and Jason Eisner. Please do check out their syllabi (available online) if you're interested in getting a new perspective on many of these topics!