CS599 B1: Advanced Natural Language Processing

This is a course aimed toward students interested in pursuing research in natural language processing. We will focus on current methods (largely based on Transformers and LLMs). While it is not possible to cover the entirety of the NLP research landscape in a single course, the content of this course will give you the core concepts and skills needed to understand and extend the ideas of many papers being published in NLP (and related) venues today.

Prerequisites

• Probability
• Proficiency in Python programming
• Linear algebra
• Calculus
• A passing familiarity with supervised machine learning methods

Highly recommended prerequisites: While not strictly required, it will be very useful to have a good grasp of machine learning. If you have taken at least one of CS542, CS505, CS585, or a related course and performed well, you should be in good shape. Also check out these resources to help get you acquainted with the basics of machine learning, programming in PyTorch, and Transformers:

• The official PyTorch tutorial
• A more brief PyTorch tutorial
• The Illustrated Transformer
• This short online book about neural networks and deep learning (Chapters 1, 2, and 5), and this video series by 3Blue1Brown on neural networks and backpropagation (Chapters 1 through 4)
• This HuggingFace course
• Slides and scheduled readings from Intro to NLP (Spring 2026)

Learning objectives
Students will:

1. Gain exposure to recent methods, systems, frameworks, and datasets in NLP
2. Understand the concepts and implementation details underlying the success of state-of-the-art NLP methods
3. Gain experience in performing original NLP research, and communicating one's findings

Logistics

Instructor: Aaron Mueller (amueller@bu.edu)

• Lectures: Tuesdays and Thursdays 12:30pm - 1:45pm, MCS (111 Cummington Mall) B33
• Office hours:
  • Tue., 2:00pm - 3:30pm (CDS 806)
  • Wed., 4:00pm - 5:30pm (CDS 806)
• News and announcements: All news and announcements will be made in class and posted on this page.
• Changes: The instructor reserves the right to make changes to the syllabus or project due dates. These changes will be announced as early as possible.

• For questions, use our class Piazza. Entry code in the slides from the first class.
• For uploading homeworks/projects, we'll use Gradescope. Entry code in the slides from the first class.

News

• Sep. 3: Homework 1 has been released! You can download the instructions here, and the code here. Start early; this assignment has many parts, and functions as a self-contained crash course into the fundamentals of how Transformer language models work. This will likely be the most time-intensive assignment of the semester.
• Sep. 3: Homework 0 (prerequisites review) has been released. You can download the PDF here: HW0 Instructions. The solutions are here: HW0 Solutions.

Readings

In the course schedule below, you can find the required readings for class. These will usually be research papers. You may find these readings helpful for coming up with questions before class, reinforcing your understanding, and/or preparing for the quizzes. When we have quizzes, core concepts covered in the required readings before that test is fair game (especially if they were also covered in lecture).

I will also provide links to optional readings and resources related to the content we cover in class. These can be found in the schedule below. These are to supplement the course material for those interested in reading further.

Course Schedule

Note: we will almost definitely alter this schedule! Order may also change depending on the availability of guests.

Date	Topic	Homework	Readings	Quiz?
Sep. 3, 2026	Course overview; Review of language models [Slides]	HW1 Released [Instructions] [Code]
Sep. 8, 2026	Transformers [Slides]		Transformers: Vaswani et al. (2017) GPT-2: Radford et al. (2019) Optional: The Illustrated Transformer: Alammar (2018)
Sep. 10, 2026	Tokenization and positional embeddings [Slides]		BPE: Sennrich et al. (2012) RoPE: Su et al. (2021) Optional: NoPE: Kazemnejad et al. (2023)
Sep. 15, 2026	Optimizers and training [Slides]		Adam: Kingma & Ba (2015) AdamW: Loshchilov & Hutter (2019)
Sep. 17, 2026	Generating and evaluating text [Slides]		Nucleus sampling: Holtzman et al. (2020)
Sep. 22, 2026	Scaling laws, emergence, and in-context learning		Scaling laws: Kaplan et al. (2020) In-context learning/GPT-3: Brown et al. (2020)
Sep. 24, 2026	Fine-tuning, distillation, and instruction tuning	HW1 Due, HW2 Released	Olmo 3: Olmo 3 Team (2025) Instruction tuning: Wei et al. (2022)
Sep. 29, 2026	Post-training/Reinforcement Learning I		RLHF: Ouyang et al. (2022)	Quiz I - Transformers
Oct. 1, 2026	Post-training/Reinforcement Learning II	Project proposal released	DPO: Rafailov et al. (2024) RLVR: DeepSeek R1 paper (2025)
Oct. 6, 2026	Post-training/Reinforcement Learning III; Agents I		ReAct: Yao et al. (2022) Lee et al. (2026)
Oct. 8, 2026	Agents II		Multi-agent debate: Du et al. (2023)
Oct. 13, 2026	Monday schedule - no class
Oct. 15, 2026	Long-context reasoning	HW2 Due	Evaluating long-context reasoning: Liu et al. (2024)
Oct. 20, 2026	Mixture-of-experts models	Project proposal due	Mixture-of-experts: Shazeer et al. (2017)	Quiz II - RL and Agents
Oct. 22, 2026	Quantization		8-bit quantization: Dettmers et al. (2022) Optional: 4-bit quantization: Dettmers & Zettlemoyer (2023)
Oct. 27, 2026	Guest lecture - Uncertainty and confidence estimation		Tian et al. (2023)
Oct. 29, 2026	No class - EMNLP
Nov. 3, 2026	AI safety and fairness		BlackHat Talk on the OpenAI/HuggingFace incident Chain-of-thought Monitoring: Baker et al. (2025) Blog post about the recent history of chain-of-thought monitoring: Hase & Potts (2026)
Nov. 5, 2026	Interpretability I		Causal mediation analysis: Vig et al. (2020) DAS: Wu et al. (2024) Optional: Survey on interpretability: Mueller et al. (2026)	Quiz III - Calibration and Efficiency
Nov. 10, 2026	Interpretability II		Linear representation hypothesis: Park et al. (2023) Manifold steering: Wurgaft et al. (2026)
Nov. 12, 2026	Training dynamics I	Midway report due	Intervening during training: Chen et al. (2024)
Nov. 17, 2026	Training dynamics II		Rational analysis of learning: Wurgaft*, Lubana* et al. (2025)
Nov. 19, 2026	Diffusion LMs	Peer reviews due Friday, Nov. 20	Diffusion LLMs: Nie et al. (2025)
Nov. 24, 2026	Thanksgiving - no class
Nov. 26, 2026	Thanksgiving - no class
Dec. 1, 2026	State space models		Mamba: Gu & Dao (2024)
Dec. 3, 2026	Class choice	Poster PDF due Friday, Dec. 4		Quiz IV - Interpretability and alternative architectures
Dec. 8, 2026	Guest lecture
Dec. 10, 2026	Poster session
TBD		Final report due

List of Topics

By the end of this course, you should be familiar with each of the following topics.

Grading

The course is graded out of 100 total points.

Homeworks: 20 points

• Homework 1: 12 points
• Homework 2: 8 points

The homeworks are largely for your benefit as study tools. You may use AI to complete the homeworks, but you will find studying for the quizzes much easier if you understand the methods you'll be implementing in the homeworks. Regardless of whether you decide to use AI tools, you (the student) are fully responsible for what you submit.

Quizzes: 30 points

There will be 3-4 in-class quizzes, each worth 10 points (your lowest quiz grade will be dropped). See the course schedule (rows highlighted in blue) for quiz dates. Quizzes will happen at the start of class. These will consist of a mixture of multiple-choice and short-answer questions. These will cover topics that were covered in lectures preceding the quiz (and homeworks if applicable). Quizzes cannot be easily made up without significant advanced notice.

Final project: 50 points

This is an open-ended project where you will pursue an original NLP project of your own design. You must pursue your project in groups of 2 to 4 people.

• Project proposal: 5 points

  Perform a literature review on your topic of choice. Also briefly describe your planned project, including the task you'll be focusing on, your data, methods, baselines, and evaluation.

• Midway report: 15 points

  You will submit an intermediate version of your project to a submission portal. You will receive 2 reviews from your classmates, as well as feedback from the instructor.

• Peer review: 5 points

  You will write reviews for 2 of your classmates' midway reports.

• Final report: 25 points

  Describe your experiments, present your results, and report your findings in the style of a typical NLP paper.

Grading of the final project will be based on the following:

• Proposal: Well-motivated idea with a concrete experimental plan. Creativity is encouraged more than bulletproof-but-incremental ideas!
• Midway report:
  • Clear problem definition and motivation
  • The most essential related work is present
  • Reasonable set of initial experiments
  • Clear presentation of methods and evaluation protocol
  • Clear presentation and description of preliminary results
  • Well-reasoned discussion about your initial experiments. If successful, what do the experiments tell us so far? If they didn't turn out how you expected, why do you think this was?
  • Concrete plan for the rest of the project. It's ok to pivot from your original plan if needed!
• Final report: Much like the midway report, but in addition:
  • The related work should be reasonably complete and well-integrated throughout the report
  • More complete methods section
  • More complete results section
  • Rigorous evaluation
  • Discussion and conclusion composed of well-formulated arguments, grounded in your experimental findings and the broader literature
  • Novelty and creativity
• Peer reviews:
  • Accurate summary
  • Fair, balanced, and constructive feedback
  • Well-reasoned justifications for your feedback
  • Rating follows from the justification

Can we publish our final project? It is feasible to convert a course project into an academic publication, but it can take a lot of work! I encourage those interested to discuss this with me at the end of the semester.

Policies and Conduct

Outside Resources & AI Policy

AI tools are allowed for the homeworks. I recommend doing the assignments on your own as quiz preparation, but for the purpose of grading, you can complete the assignments completely with AI if you so choose. It is the student's responsibility to verify any submitted content.

AI tools are partially allowed for the final project. Our policy here is more nuanced: you may use AI as a tool, but do not use AI as a crutch or replacement for thinking. What's the difference? AI as a tool includes:

• Using AI to help you workshop your initial ideas
• Using AI to refine your writing, including spell-checking, reorganization, among other copyediting-like actions
• Using AI to help you write/debug code

AI as a crutch/replacement includes:

• Using AI to come up with your final project idea for you from scratch
• Using AI to write all of your code
• Using AI to generate your reports

A good heuristic is that "AI as a crutch/replacement" includes anything where you no longer understand core detail(s) of what is in your code or writing. The line between tool and crutch can sometimes be fuzzy, so if you're unsure, I recommend asking ahead of time! I promise not to judge if you ask before you turn in the assignment. :)

Note that points will be deducted if we cannot understand the content of the report (due to, for example, an overly dense writing style featuring many undefined technical terms, hyphenated compounds, or excessive numeric detail throughout). Pangram scores will not be used to determine scores directly, but may be used as part of the justification for removing points if the clarity of the presentation is a concern first, or if any references are found to be hallucinated, among other reasons.

The most recent version of Pangram has been found to only very rarely yield false positives. Even if you use AI to completely rewrite text that you wrote first, it will typically return 100% AI-edited, rather than 100% AI-generated (which is still not ideal, but won't be penalized directly). If you lose points due to this policy, you may come to office hours and explain your terms, logic, and reasoning through your methods and results more clearly. If the flow of ideas is still not particularly logical or you cannot answer questions about the content or implementation, the deduction will stand.

No AI tools are allowed during quizzes. These will be hand-written in class.

I strongly encourage you to use any outside source at your disposal when doing the homework and your final project. Your reports and code should be original, but you may take inspiration from existing papers as long as you give them proper credit. When doing your project, feel free to base your implementations on publicly available code as well (as long as you make significant modifications to accommodate your original idea), but be sure to give proper credit in your report and your GitHub README if you do so.

For the final project, failing to properly cite an outside source is equivalent to taking credit for ideas that are not your own, which is plagiarism.

Academic Integrity

Read through BU's Academic Conduct Code. All students are expected to abide by these guidelines. In the context of this class, it's particularly important that you cite the source of your ideas, facts, and/or methods, and do not claim someone else's work as your own.

You are allowed to work in groups to do the homeworks, but you should upload your homework individually.

Absence and Late Work Policy

Attendance will not be taken. Attend lectures as you wish.

You have 5 free late days that you can use however you wish with no excuse necessary. Using a late day means that you can still receive full credit for the assignment with no late penalty. Turning in an assignment late after using all your late days will incur a 10% drop in the score for each late day. 5 days after an assignment's due date, the assignment can no longer be turned in, regardless of whether you use your free late days. This applies to all homeworks. It also applies to the proposal and midway report for the final project—but not the final report, which must be turned in on time. Note that a late day is a step function: turning in a homework 5 minutes late is equivalent to turning it in 23 hours late, so if you know you'll be late, we recommend taking the extra time to verify your understanding of the material.

Extensions can be negotiated in cases of medical emergency or other sudden pressing circumstances. Students should contact the course staff ASAP and negotiate before the assignment's original due date. If this applies to the first homework, please come talk to us the first day of class.

Quizzes cannot easily be made up. If you know you cannot make a quiz day, you must notify us at least 14 days in advance so that we can make alternate arrangements.

Accommodations

Boston University's policy is to provide reasonable accommodations to students with qualifying disabilities who are enrolled in Boston University courses. Students seeking accommodations must engage in an interactive process with, and provide appropriate documentation of their disability to, Disability & Access Services (DAS). If this applies, please get in touch with me as soon as possible to discuss accommodations; note that students are not required to disclose information regarding their disability, if applicable, but should request approval for such accommodations through DAS beforehand.

Religious Observance

Students are permitted to be absent from class, including classes involving examinations, labs, excursions, and other special events, for purposes of religious observance. In-class, take-home and lab assignments, and other work shall be made up in consultation with the student's instructors. More details on BU's religious observance policy are available here.

Acknowledgments

Much of the content in this course was inspired by NLP courses taught by Tatsunori Hashimoto, Percy Liang, Sean Welleck, and Greg Durrett. Please do check out their syllabi (available online) if you're interested in getting a new perspective on many of these topics!