CS505: Natural Language Processing

How can we get computers to understand and generate human language? This is among the most challenging—and currently, the most quickly advancing—approaches in contemporary artificial intelligence. Natural language systems are deployed in the world in increasingly many forms: chatbots, code assistants, web agents, among others. This course provides an introduction to the engineering and science that underlies current NLP systems.

Prerequisites

Probability
Proficiency in Python programming
Linear algebra
Calculus

Highly recommended prerequisites: Not required, but it will be very useful to have taken a machine learning course before taking this one.

The Illustrated Transformer
This short online book about neural networks and deep learning (Chapters 1, 2, and 5), and this video series by 3Blue1Brown on neural networks and backpropagation (Chapters 1 through 4)

Learning objectives
Students will:

Gain exposure to foundational ideas in NLP.
Understand the theory underlying current NLP ideas.
Learn how to implement each element of the contemporary NLP pipeline.

Logistics

Classes: Tuesdays and Thursdays 3:30pm - 4:45pm, CDS 801
Instructor: Aaron Mueller (amueller@bu.edu)
Teaching Fellows: Ge Gao (ggao02@bu.edu)
Office hours: TBD
News and announcements: All news and announcements will be made in class and posted on this page.
Changes: The instructor reserves the right to make changes to the syllabus or project due dates. These changes will be announced as early as possible.

News

Watch this section for homework and project updates!

Book

We will be loosely following this book throughout the course: Jurafsky & Manning, available online for free.

Course Schedule

Note: we will almost definitely alter this schedule! Order may also change depending on the availability of guests.

Date	Topic	Homework	Readings
Jan 20, 2025	Course introduction What is NLP? Overview of course topics	HW-1 released (ungraded)
Jan 22, 2025	Text classification Logistic regression Gradient descent Features Machine learning basics	HW0 released	J&M (2025): Chapter 2
Jan 27, 2025	Introduction to language modeling Tokenization The type-token distinction Feature engineering		J&M (2025): Chapter 3
Jan 29, 2025	Sequence modeling Review of probability theory N-grams		J&M (2025): Chapter 4
Feb 3, 2025	Neural sequence modeling I Feed-forward neural networks Backpropagation Embeddings	HW0 due HW1 released	J&M: Chapter 5 J&M: Chapter 6
Feb 5, 2025	Neural sequence modeling II Recurrent neural networks LSTMs		J&M: Chapter 13
Feb 10, 2025	Attention Transformers Parallel processing		J&M: Chapter 8
Feb 12, 2025	Large language models I Pre-training Autoregressive language models: GPT	HW1 due HW2 released	J&M: Chapter 7
Feb 17, 2025	Large language models II Masked language models: BERT Sequence-to-sequence models: T5
Feb 19, 2025	Evaluating language models
Feb 24, 2025	NLP tasks I Prompting In-context learning
Feb 26, 2025	NLP tasks II Fine-tuning Low-rank adapters
Mar 3, 2025	Post-training I Intro to reinforcement learning Reinforcement learning from human feedback (RLHF)	HW2 due Final project proposal released	J&M: Chapter 9
Mar 5, 2025	Post-training II Direct preference optimization (DPO) Reinforcement learning with verifiable rewards (RLVR)		J&M: Chapter 9
Mar 10, 2025	Spring break - no class
Mar 12, 2025	Spring break - no class
Mar 17, 2025	Morphology and syntax I Intro to morphology and syntax POS tagging Hidden Markov models (HMMs)	HW3 released
Mar 19, 2025	Morphology and syntax II (Probabilistic) context-free grammars Constituency parsing CKY
Mar 24, 2025	Morphology and syntax III Dependency parsing Shift-reduce
Mar 26, 2025	Semantics Models of meaning Semantic role labeling Semantic parsing	HW3 due
Mar 31, 2025	Exam
Apr 2, 2025	Review of exam Discourse and pragmatics The structure of conversation Coreference resolution	Final project proposal due
Apr 7, 2025	NLP tasks I: Classification Paraphrase detection Natural language inference Multiple-choice question answering
Apr 9, 2025	NLP tasks II: Generation Machine translation Open-ended question answering
Apr 14, 2025	Multilingual NLP Multilingual pre-training Low-resource languages
Apr 16, 2025	Interpretability and evaluation I Best practices in evaluation Out-of-distribution generalization
Apr 21, 2025	Interpretability and evaluation II (Mechanistic) interpretability Model editing	Midway report due
Apr 23, 2025	Retrieval and tool use
Apr 28, 2025	Guest lecture
Apr 30, 2025	Final project help session
TBD	Final report due

List of Topics

By the end of this course, you should be familiar with each of the following topics. Items with an asterisk* may be on the exam.

The history of natural language processing:
- Claude Shannon and the invention of language models
- The Turing test
- ELIZA
- Statistical methods
- Neural methods
Tokenization:
- The type–token distinction*
- Words*
- Characters*
- Bytes*
- Unicode*
- UTF-8*
- Byte-pair encoding (BPE)*
Classification:
- Binary Logistic regression*
- Multinomial logistic regression*
Nonlinear activation functions:
- Sigmoid*
- Softmax*
Machine learning basics:
- Loss functions*, including:
- Gradients*
- Gradient descent*
- Stochastic gradient descent*
- Learning rates*
- Mini-batching*
- Train-test splits*
Neural network architectures:
- Feedforward layers*
- Recurrent neural networks*
- Long short-term memory networks (LSTMs)*
- Transformers*
Language model architectures:
- N-gram language models*
- Recurrent neural networks language models*
- Transformer language models*
- Autoregressive language models*
- Masked language models*
- Sequence-to-sequence language models*
Language modeling methods:
- Next-token prediction*
- Pre-training*
Language model architectures:
- N-gram language models*
- Recurrent neural networks language models*
- Transformer langiuage models*
NLP tasks:
- Sentiment classification
- Language identification
- Spam detection
- Toxicity detection
- Machine translation
- Question answering (QA):
- Machine translation

Knowing thy data:
- Corpora and their construction*
- Documents vs. paragraphs vs. sentences*
- Domain shift*
- Temporal shift*
Prompting:
- In-context learning*
- Chain-of-thought prompting*
Fine-tuning and adaptation:
- Continued pre-training*
- Fine-tuning*
- Low-rank adapters*
Evaluation:
- Precision, recall, and F1*
- Accuracy*
- Surprisal and perplexity*
- Out-of-distribution (OOD) generalization
Reinforcement learning basics:
- Policy*
- Reward functions*
Post-training methods:
- Reinforcement learning from human feedback (RLHF)*
- Direct preference optimization (DPO)*
- Reinforcement learning with verifiable rewards (RLVR)*
Syntax and parsing:
- Constituencies*
- Dependencies*
- Context-free grammars*
- CKY*
- Shift-reduce*
- The context-sensitivity of natural language
Morphology:
- Morphemes*
- Analytic vs. synthetic languages
- Fusional vs. agglutinative languages
Semantics:
- Semantic role labeling
- Theta roles
Discourse:
- Coreference resolution
- Pragmatic inference
Multilingual NLP:
- High- vs. low-resource languages
- Linguistic typology
- Language imbalance
Interpretability:
- Saliency maps
- Circuits
- Shortcuts and spurious correlations
- Causation
Retrieval:
- Retrieval-augmented generation (RAG)
- Dense passage retrievers (DPR)
Agents
- Language models as agents
- Tool use
- Multi-agent frameworks
Agents
- Tool use
- Multi-agent systems
Social impacts and considerations:
- AI safety
- Bias, fairness, and toxicity
- Factuality and hallucinations

Grading

The course is graded out of 100 total points.

Homeworks: 15 points

Homework 0: 4 points
Homework 1: 4 points
Homework 2: 4 points
Homework 3: 3 points

The homeworks are largely for your benefit as study tools. You may use AI in any way you wish to complete the homeworks. Regardless of whether you decide to use AI tools, you are fully responsible for what you submit.

Exam: 35 points

There will be one exam about 2/3 of the way through the course. See the list of topics above for a guide to what the exam will cover. You may not use any electronic resources for the exam; this includes the textbook, AI tools, the internet, text messages, among other items.
This will be an open-note exam! If you bring notes, they must be on one physical piece of paper. Electronic notes will not be allowed.

Final project: 50 points

This is an open-ended project where you will review and pursue an NLP topic of your choosing.

Project proposal: 5 points
Perform a literature review on your topic of choice. Also briefly describe your planned project, including the task you'll be focusing on, your data, methods, baselines, and evaluation.
Midway report: 10 points
Report your progress on the final project thus far. By this point, you should have run some experiments and obtained some preliminary results. Outline your plan for the rest of the project.
Final report: 25 points
Describe your experiments, present your results, and report your findings in the style of a typical NLP paper.

Grading of the final project will be based on the following:

Proposal: Well-motivated idea with a concrete experimental plan. Creativity is encouraged more than bulletproof-but-incremental ideas!
Midway report:
- Clear problem definition and motivation
- The most essential related work is present
- Reasonable set of initial experiments
- Clear presentation of methods and evaluation protocol
- Clear presentation and description of preliminary results
- Well-reasoned discussion about your initial experiments. If successful, what do the experiments tell us so far? If they didn't turn out how you expected, why do you think this was?
- Concrete plan for the rest of the project. It's ok to pivot from your original plan if needed!
Final report: Much like the midway report, but in addition:
- The related work should be reasonably complete and well-integrated throughout the report
- More complete methods section
- More complete results section
- Rigorous evaluation
- Discussion and conclusion composed of well-formulated arguments, grounded in your experimental findings and the broader literature
- Novelty and creativity

Can we publish our final project? It is feasible to convert a course project into an academic publication, but it can take a lot of work! I encourage those interested to discuss this with me at the end of the semester.

Policies and Conduct

Outside Resources & AI Policy

AI tools are completely allowed for the homeworks. I recommend doing the assignments on your own as exam preparation, but for the purpose of grading, you can complete the assignments completely with AI if you so choose. It is the student's responsibility to verify any submitted content.

AI tools are allowed for the final project. Our policy here is more nuanced: you may use AI as a tool, but do not use AI as a crutch or replacement for thinking. What's the difference? AI as a tool includes:

Using AI to help you workshop your initial ideas
Using AI to refine your writing, including spell-checking, reorganization, among other copyediting-like actions
Using AI to help you debug code

AI as a crutch/replacement includes:

Using AI to come up with your final project idea for you from scratch
Using AI to write all of your code
Using AI to generate your reports

The line between tool and crutch can be fuzzy, so if you're unsure, I recommend asking ahead of time! I promise not to judge if you ask before you turn in the assignment. :)

No AI tools are allowed during exams. These will be hand-written in class.

I strongly encourage you to use any outside source at your disposal when doing the homework and your final project. Your reports and code should be original, but you may take inspiration from existing papers as long as you give them proper credit. When doing your project, feel free to base your implementations on publicly available code as well (as long as you make significant modifications to accommodate your original idea), but be sure to give proper credit in your report and your GitHub README if you do so.

For the final project, failing to properly cite an outside source is equivalent to taking credit for ideas that are not your own, which is plagiarism.

Academic Integrity

Read through BU's Academic Conduct Code. All students are expected to abide by these guidelines. In the context of this class, it's particularly important that you cite the source of your ideas, facts, and/or methods, and do not claim someone else's work as your own.

Absence and Late Work Policy

Attendance will not be taken. Attend lectures as you wish.

You have 6 free late days that you can use however you wish with no excuse necessary. Using a late day means that you can still receive full credit for the assignment with no late penalty. Turning in an assignment late after using all your late days will incur a 10% drop in the score for each late day. 5 days after an assignment's due date, the assignment can no longer be turned in, regardless of whether you use your free late days. This applies to all homeworks. It also applies to the proposal and midway report for the final project—but not the final report, which must be turned in on time. Note that a late day is a step function: turning in a homework 5 minutes late is equivalent to turning it in 23 hours late, so if you know you'll be late, we recommend taking the extra time to verify your understanding of the material.

Extensions can be negotiated in cases of medical emergency or other sudden pressing circumstances. Students should contact the course staff ASAP and negotiate before the assignment's original due date. If this applies to the first homework, please come talk to us the first day of class.

Exams cannot easily be made up. If you know you cannot make an exam day, you must notify us at least 14 days in advance so that we can make alternate arrangements.

Accommodations

Boston University's policy is to provide reasonable accommodations to students with qualifying disabilities who are enrolled in Boston University courses. Students seeking accommodations must engage in an interactive process with, and provide appropriate documentation of their disability to, Disability & Access Services (DAS). If this applies, please get in touch with me as soon as possible to discuss accommodations; note that students are not required to disclose information regarding their disability, if applicable, but should request approval for such accommodations through DAS beforehand.

Religious Observance

Students are permitted to be absent from class, including classes involving examinations, labs, excursions, and other special events, for purposes of religious observance. In-class, take-home and lab assignments, and other work shall be made up in consultation with the student's instructors. More details on BU's religious observance policy are available here.