Syllabus (not maintained in 2026 fall)
The syllabus is not maintained in 2026 fall. Lecture notes are from prior years and are subject to change. Updated lecture notes will be posted after the lecture.
Lecture notes
Readings (=optional)
- How Language Shapes the Way We Think. Lera Boroditsky.
- Meaning and Intelligence in Language Models (KDD 2025 Keynote). Chris Manning.
Links (=optional)
- NLP Highlights podcast. Various Allen Institute AI2 Researchers.
Lecture notes
Links (=optional)
- Python tidbits for NLP.
- Natural Language Corpus Data: Beautiful Data. Peter Norvig.
- A Mathematical Theory of Communication. Claude Shannon.
- Prediction and Entropy of Printed English. Claude Shannon.
Lecture notes
Readings (=optional)
- Words and Tokens (J+M chapter 2). Dan Jurafsky and James H. Martin.
Links (=optional)
- BPE tutorial at Huggingface. Huggingface.
Lecture notes
- Language models
- Exercise: n-gram Language Models
- The unreasonable effectiveness of Character-level Language Models (Yoav Goldberg)
Readings (=optional)
- N-Gram Language Models (J+M chapter 3). Dan Jurafsky and James H. Martin.
- Language models. Michael Collins.
Links (=optional)
- A Bit of Progress in Language Modeling. Stanley Chen and Joshua Goodman.
- One Billion Word Benchmark for Measuring Progress in Statistical Language Modeling. Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, and Thorsten Brants.
Lecture notes
- Naive Bayes
- Logistic regression
- Evaluation
- Python Notebook: Naive Bayes Classifiers
- Python Notebook: Log-linear Models
- Exercise: Log linear models
- Exercise: Linear combinations
Readings (=optional)
- Naive Bayes and Text Classification (J+M Appendix B). Dan Jurafsky and James H. Martin.
- Logistic Regression and Text Classification (J+M chapter 4). Dan Jurafsky and James H. Martin.
- Log-linear models, MEMMs and CRFs. Michael Collins.
Lecture notes
- Neural networks and classifications review
- Python Notebook: Feedforward networks and XOR
- Exercise: Nothing but Net(works) (Tom McCoy)
Readings (=optional)
- Neural Networks (J+M chapter 6). Dan Jurafsky and James H. Martin.
- Feedforward neural networks. Michael Collins.
- A Primer on Neural Network Models for Natural Language Processing. Yoav Goldberg.
Links (=optional)
- A Review of the Neural History of Natural Language Processing. Sebastian Ruder.
- Natural Language Understanding with Distributed Representation. Kyunghyun Cho.
- Efficient Backprop. LeCun, Bottou, Orr and Muller.
- What Every Computer Scientist Should Know About Floating-Point Arithmetic. D. Goldberg.
- Gradient checks and parameter updates. Andrej Karpathy.
- Computational Graphs, and Backpropagation. Michael Collins.
Lecture notes
- Word vectors (vector space models)
- Word2Vec
- Python Notebook: Singular Value Decomposition for NLP
- Exercise: word2vec
Readings (=optional)
- Embeddings (J+M chapter 5). Dan Jurafsky and James H. Martin.
- Improving Distributional Similarity with Lessons Learned from Word Embeddings. Omer Levy, Yoav Goldberg, and Ido Dagan.
Lecture notes
Readings (=optional)
- Hidden Markov Models (J+M Appendix A.1-A.4). Dan Jurafsky and James H. Martin.
Links (=optional)
- Introduction to Hidden Markov Models. Anoop Sarkar.
- N-grams versus HMMs. Anoop Sarkar.
- Parsing with HMMs. Anoop Sarkar.
- Viterbi algorithm for HMMs. Anoop Sarkar.
- Language models and HMMs. Anoop Sarkar.
- Supervised learning of HMMs. Anoop Sarkar.
- Lagrange multipliers for HMM parameter updates. Anoop Sarkar.
- Hidden Markov Models. Michael Collins.
- Forward-backward algorithm. Michael Collins.
Lecture notes
- Recurrent Neural Networks
- LSTMs and GRUs
- RNN based Neural Language model (experiments) (Tomas Mikolov)
- Python Notebook: RNN Language model
Readings (=optional)
- RNNs and LSTMs (J+M chapter 13). Dan Jurafsky and James H. Martin.
- Understanding LSTM Networks. Christopher Olah.
Links (=optional)
- How to implement a recurrent neural network. peterroelants.github.io.
- Statistical language models based on neural networks. Tomas Mikolov.
Lecture notes
Lecture notes
Readings (=optional)
- Transformers (J+M chapter 8). Dan Jurafsky and James H. Martin.
- Attention Is All You Need. Vaswani et al. 2017.
- The Annotated Transformer. Alexander Rush.
Links (=optional)
- Illustrated Transformer. Jay Alammar.
Lecture notes
Readings (=optional)
- Masked Language Models (J+M chapter 10). Dan Jurafsky and James H. Martin.
- Illustrated BERT. Jay Alammar.
- Contextual Word Representations: A Contextual Introduction. Noah A. Smith.
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Devlin et al. 2018.
- Emergent linguistic structure in artificial neural networks trained by self-supervision. Manning et al. 2020.
- A Primer in BERTology: What We Know About How BERT Works. Rogers, Kovaleva, and Rumshisky.
Lecture notes
Readings (=optional)
- Large Language Models (J+M chapter 7). Dan Jurafsky and James H. Martin.
- Masked Language Models (J+M chapter 10). Dan Jurafsky and James H. Martin.
- Illustrated BERT. Jay Alammar.
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Devlin et al. 2018.
- Improving Language Understanding by Generative Pre-Training. Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever.
- Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J. Liu.
Links (=optional)
- Semi-supervised Sequence Learning. Andrew M. Dai, Quoc V. Le.
- Deep contextualized word representations. Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, Luke Zettlemoyer.
- RoBERTa: A Robustly Optimized BERT Pretraining Approach. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, Veselin Stoyanov.
- Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context. Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V. Le, Ruslan Salakhutdinov.
- ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators. Kevin Clark, Minh-Thang Luong, Quoc V. Le, Christopher D. Manning.
- ALBERT: A Lite BERT for Self-supervised Learning of Language Representations. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, Radu Soricut.
Lecture notes
- Lecture notes
- Intro to Generation with LLMs (Huggingface)
- Generation Strategies (Huggingface)
- Contrastive Search (Tian Lan)
- Categorical Reparameterization with Gumbel-Softmax (Eric Jang, Shixiang Gu, Ben Poole)
Links (=optional)
- A Contrastive Framework for Neural Text Generation. Yixuan Su, Tian Lan, Yan Wang, Dani Yogatama, Lingpeng Kong, Nigel Collier.
Lecture notes
Readings (=optional)
- HuggingFace PEFT: Parameter-Efficient Fine-Tuning of Billion-Scale Models on Low-Resource Hardware. Sourab Mangrulkar, Sayak Paul".
- Prefix-Tuning: Optimizing Continuous Prompts for Generation. Xiang Lisa Li, Percy Liang.
- AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuning. Yaqing Wang, Sahaj Agarwal, Subhabrata Mukherjee, Xiaodong Liu, Jing Gao, Ahmed Hassan Awadallah, Jianfeng Gao.
- LoRA: Low-Rank Adaptation of Large Language Models. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen.
- Adapter methods. docs.adapterhub.ml.
Links (=optional)
- AdapterHub: A Framework for Adapting Transformers. Jonas Pfeiffer, Andreas Rücklé, Clifton Poth, Aishwarya Kamath, Ivan Vulić, Sebastian Ruder, Kyunghyun Cho, Iryna Gurevych.
- Parameter-Efficient Transfer Learning for NLP. Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, Sylvain Gelly.
- Simple, Scalable Adaptation for Neural Machine Translation. Ankur Bapna, Naveen Arivazhagan, Orhan Firat.
- AdapterFusion: Non-Destructive Task Composition for Transfer Learning. Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, Iryna Gurevych.
- Parameter-Efficient Tuning with Special Token Adaptation. Xiaocong Yang, James Y. Huang, Wenxuan Zhou, Muhao Chen.
Lecture notes
Readings (=optional)
- Language Models are Unsupervised Multitask Learners. Open AI.
- Language Models are Few-Shot Learners. Open AI.
- GPT-4 Technical Report. Open AI.
- Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?. Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, Luke Zettlemoyer.
- In-Context Learning Learns Label Relationships but Is Not Conventional Learning. Jannik Kossen, Yarin Gal, Tom Rainforth.
Links (=optional)
- In-context Examples Selection for Machine Translation. Sweta Agrawal, Chunting Zhou, Mike Lewis, Luke Zettlemoyer, Marjan Ghazvininejad.
- How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation. Amr Hendy, Mohamed Abdelrehim, Amr Sharaf, Vikas Raunak, Mohamed Gabr, Hitokazu Matsushita, Young Jin Kim, Mohamed Afify, Hany Hassan Awadalla.
Lecture notes
Links (=optional)
- Post-training (J+M chapter 9.1). Dan Jurafsky and James H. Martin.
- Scaling Instruction-Finetuned Language Models. Google.
Lecture notes
- Lecture notes
- Direct Preference Optimization: Your Language Model is Secretly a Reward Model (Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, Chelsea Finn)
Links (=optional)
- Post-training (J+M chapter 9.2). Dan Jurafsky and James H. Martin.
- HuggingFace Blog on RLHF.
- Introduction to RL.
- Proximal Policy Optimization Algorithms. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov.
- LIMA: Less Is More for Alignment. Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, Omer Levy.
Lecture notes
- Lecture notes
- Scaling Laws for Neural Language Models (Open AI)
- Training Compute-Optimal Large Language Models (Google Deepmind)
- GLaM: Efficient Scaling of Language Models with Mixture-of-Experts (Google)
- PaLM: Scaling Language Modeling with Pathways (Google)
Links (=optional)
- AI and Compute. OpenAI.
- Scaling laws notebook. Andrej Karpathy.
- The FLOPs Calculus of Language Model Training. Dzmitry Bahdanau.
- The AI Brick Wall – A Practical Limit For Scaling Dense Transformer Models, and How GPT 4 Will Break Past It. Dylan Patel.
Lecture notes
Readings (=optional)
- Constituency Grammars (J+M chapter 18). Dan Jurafsky and James H. Martin.
- Dependency Parsing (J+M chapter 19). Dan Jurafsky and James H. Martin.