Syllabus
The syllabus is preliminary and subject to change.
Lecture notes
Links (=optional)
- The Bitter Lesson. Richard Sutton.
- SHRDLU.
- IBM Watson competes in Jeopardy!.
- Speech Recognition Breakthrough for the Spoken, Translated Word by Rick Rashid. Rick Rashid.
- NLP Highlights podcast. Various Allen Institute AI2 Researchers.
- The Thesis Review. Sean Welleck.
- Humans of AI: Stories, Not Stats. Devi Parikh and Dhruv Batra.
Lecture notes
- Probability Models for NLP
- Probability models and language
- Language models
- Exercise: n-gram Language Models
- The unreasonable effectiveness of Character-level Language Models (Yoav Goldberg)
Links (=optional)
- Python tidbits for NLP.
- Natural Language Corpus Data: Beautiful Data. Peter Norvig.
- A Mathematical Theory of Communication. Claude Shannon.
- Prediction and Entropy of Printed English. Claude Shannon.
- Language models. Michael Collins.
- A Bit of Progress in Language Modeling. Stanley Chen and Joshua Goodman.
- One Billion Word Benchmark for Measuring Progress in Statistical Language Modeling. Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, and Thorsten Brants.
Lecture notes
- Sequence prediction
- Feedforward neural networks
- Multi-layer Perceptrons for n-gram LMs
- Intro to Feedforward networks
- Feedforward networks and XOR
Links (=optional)
- Feedforward neural networks. Michael Collins.
- Pytorch Internals. Edward Z. Yang.
- Exercise: Nothing but Net(works). Tom McCoy.
- Efficient Backprop. LeCun, Bottou, Orr and Muller.
- What Every Computer Scientist Should Know About Floating-Point Arithmetic. D. Goldberg.
- Gradient checks and parameter updates. Andrej Karpathy.
- Revisiting Simple Neural Probabilistic Language Models. Simeng Sun, Mohit Iyyer.
- Introduction to the CoNLL-2000 Shared Task: Chunking. Erik F. Tjong Kim Sang and Sabine Buchholz.
Lecture notes
- Lecture notes
- Alignment Exercise (Kevin Knight)
- Exercise: Philae Obelisk
- Neural Machine Translation by Jointly Learning to Align and Translate (Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio)
Links (=optional)
- Word Alignment and the Expectation-Maximization Algorithm. Adam Lopez.
- A Statistical MT Tutorial Workbook. Kevin Knight.
- Sequence to Sequence Learning with Neural Networks. Ilya Sutskever, Oriol Vinyals, Quoc V. Le.
- Effective Approaches to Attention-based Neural Machine Translation. Minh-Thang Luong, Hieu Pham, Christopher D. Manning.
Lecture notes
- Lecture notes
- Transformer code tutorial
- Attention is all you need (Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin)
- Transformer: A Novel Neural Network Architecture for Language Understanding (posted by Jakob Uszkoreit)
- Stanford cs224n lecture notes on self attention (John Hewitt)
- The Annotated Transformer (Austin Huang, Suraj Subramanian, Jonathan Sum, Khalid Almubarak, and Stella Biderman. Original by Sasha Rush)
- The Illustrated Transformer (Jay Alammar)
Links (=optional)
- RASPy: Think like a Transformer. Sasha Rush.
- Convolutional Sequence to Sequence Learning. Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, Yann N. Dauphin.
- Layer Normalization. Jimmy Lei Ba, Jamie Ryan Kiros, Geoffrey E. Hinton.
- Dropout: A Simple Way to Prevent Neural Networks from Overfitting. Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, Ruslan Salakhutdinov.
- Improving Transformer Optimization Through Better Initialization. Xiao Shi Huang, Felipe Perez, Jimmy Ba, Maksims Volkovs.
- A Mathematical Framework for Transformer Circuits. Anthropic.
- Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned. Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, Ivan Titov.
Lecture notes
- Lecture notes
- Intro to Generation with LLMs (Huggingface)
- Generation Strategies (Huggingface)
- Contrastive Search (Tian Lan)
- Categorical Reparameterization with Gumbel-Softmax (Eric Jang, Shixiang Gu, Ben Poole)
Links (=optional)
- A Contrastive Framework for Neural Text Generation. Yixuan Su, Tian Lan, Yan Wang, Dani Yogatama, Lingpeng Kong, Nigel Collier.
Lecture notes
- Lecture notes
- Measuring Massive Multitask Language Understanding (Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, Jacob Steinhardt)
- MTEB: Massive Text Embedding Benchmark (Niklas Muennighoff, Nouamane Tazi, Loïc Magne, Nils Reimers)
- AI and the Everything in the Whole Wide World Benchmark (Deborah Raji, Emily Denton, Emily M. Bender, Alex Hanna, Amandalynne Paullada)
- Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data (Emily M. Bender, Alexander Koller)
Links (=optional)
Lecture notes
- Lecture notes
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova)
- Improving language understanding with unsupervised learning (Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever)
Links (=optional)
- Semi-supervised Sequence Learning. Andrew M. Dai, Quoc V. Le.
- Deep contextualized word representations. Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, Luke Zettlemoyer.
- RoBERTa: A Robustly Optimized BERT Pretraining Approach. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, Veselin Stoyanov.
- Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context. Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V. Le, Ruslan Salakhutdinov.
- ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators. Kevin Clark, Minh-Thang Luong, Quoc V. Le, Christopher D. Manning.
- Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J. Liu.
- ALBERT: A Lite BERT for Self-supervised Learning of Language Representations. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, Radu Soricut.
Lecture notes
- Lecture notes
- Prefix-Tuning: Optimizing Continuous Prompts for Generation (Xiang Lisa Li, Percy Liang)
- AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuning (Yaqing Wang, Sahaj Agarwal, Subhabrata Mukherjee, Xiaodong Liu, Jing Gao, Ahmed Hassan Awadallah, Jianfeng Gao)
- LoRA: Low-Rank Adaptation of Large Language Models (Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen)
- Adapter methods (docs.adapterhub.ml)
Links (=optional)
- AdapterHub: A Framework for Adapting Transformers. Jonas Pfeiffer, Andreas Rücklé, Clifton Poth, Aishwarya Kamath, Ivan Vulić, Sebastian Ruder, Kyunghyun Cho, Iryna Gurevych.
- Parameter-Efficient Transfer Learning for NLP. Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, Sylvain Gelly.
- Simple, Scalable Adaptation for Neural Machine Translation. Ankur Bapna, Naveen Arivazhagan, Orhan Firat.
- AdapterFusion: Non-Destructive Task Composition for Transfer Learning. Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, Iryna Gurevych.
- Parameter-Efficient Tuning with Special Token Adaptation. Xiaocong Yang, James Y. Huang, Wenxuan Zhou, Muhao Chen.
Lecture notes
- Lecture notes
- Language Models are Unsupervised Multitask Learners (Open AI)
- Language Models are Few-Shot Learners (Open AI)
- GPT-4 Technical Report (Open AI)
- Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? (Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, Luke Zettlemoyer)
- In-Context Learning Learns Label Relationships but Is Not Conventional Learning (Jannik Kossen, Yarin Gal, Tom Rainforth)
- Metaprompting (Anthropic.com)
Links (=optional)
- In-context Examples Selection for Machine Translation. Sweta Agrawal, Chunting Zhou, Mike Lewis, Luke Zettlemoyer, Marjan Ghazvininejad.
- How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation. Amr Hendy, Mohamed Abdelrehim, Amr Sharaf, Vikas Raunak, Mohamed Gabr, Hitokazu Matsushita, Young Jin Kim, Mohamed Afify, Hany Hassan Awadalla.
Lecture notes
- Lecture notes
- ORPO: Monolithic Preference Optimization without Reference Model (Jiwoo Hong, Noah Lee, James Thorne)
- Direct Preference Optimization: Your Language Model is Secretly a Reward Model (Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, Chelsea Finn)
- Training Verifiers to Solve Math Word Problems (Cobbe et al. (OpenAI))
Links (=optional)
- Scaling Instruction-Finetuned Language Models. Google.
- Proximal Policy Optimization Algorithms. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov.
- LIMA: Less Is More for Alignment. Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, Omer Levy.
Lecture notes
- Lecture notes
- Generalization through Memorization: Nearest Neighbor Language Models (Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, Mike Lewis)
- Improving language models by retrieving from trillions of tokens (Deepmind)
Links (=optional)
- Nearest Neighbor Knowledge Distillation for Neural Machine Translation. Zhixian Yang, Renliang Sun, Xiaojun Wan.
Lecture notes
- Lecture notes
- Distilling the Knowledge in a Neural Network (Geoffrey Hinton, Oriol Vinyals, Jeff Dean)
Links (=optional)
- Sequence-Level Knowledge Distillation. Yoon Kim, Alexander Rush.
- Dark Knowledge. Geoffrey Hinton.
- DistilBERT. Huggingface.
- Compressing Large-Scale Transformer-Based Models: A Case Study on BERT. Prakhar Ganesh, Yao Chen, Xin Lou, Mohammad Ali Khan, Yin Yang, Hassan Sajjad, Preslav Nakov, Deming Chen, Marianne Winslett.
Lecture notes
- Lecture notes
- Scaling Laws for Neural Language Models (Open AI)
- Training Compute-Optimal Large Language Models (Google Deepmind)
- GLaM: Efficient Scaling of Language Models with Mixture-of-Experts (Google)
- Scaling Scaling Laws with Board Games (Andy L. Jones)
- PaLM: Scaling Language Modeling with Pathways (Google)
Links (=optional)
- AI and Compute. OpenAI.
- Scaling laws notebook. Andrej Karpathy.
- The FLOPs Calculus of Language Model Training. Dzmitry Bahdanau.
- The AI Brick Wall – A Practical Limit For Scaling Dense Transformer Models, and How GPT 4 Will Break Past It. Dylan Patel.
Lecture notes
- Do We Need to Create Big Datasets to Learn a Task? (Swaroop Mishra, Bhavdeep Singh Sachdeva)
- Shortformer: Better Language Modeling using Shorter Inputs (Ofir Press, Noah A. Smith, Mike Lewis)
- Active Learning for BERT: An Empirical Study (Liat Ein-Dor, Alon Halfon, Ariel Gera, Eyal Shnarch, Lena Dankin, Leshem Choshen, Marina Danilevsky, Ranit Aharonov, Yoav Katz, Noam Slonim)
Lecture notes
- Lecture notes
- Efficient Transformers: A Survey (Yi Tay, Mostafa Dehghani, Dara Bahri, Donald Metzler)
- Reformer: The Efficient Transformer (Nikita Kitaev, Lukasz Kaiser, Anselm Levskaya)
- Rethinking Attention with Performers (Krzysztof Marcin Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Quincy Davis, Afroz Mohiuddin, Lukasz Kaiser, David Benjamin Belanger, Lucy J Colwell, Adrian Weller)
- Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation (Ofir Press, Noah A. Smith, Mike Lewis)
- Transformer-XL: Attentive Language Models beyond a Fixed-Length Context (Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, Ruslan Salakhutdinov)
- ∞-former: Infinite Memory Transformer (Pedro Henrique Martins, Zita Marinho, Andre Martins)
- FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness (Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, Christopher Ré)
Lecture notes
- Lecture notes
- Tutorial - What is a variational autoencoder? (Jaan Altosaar)
- Optimus: Organizing Sentences via Pre-trained Modeling of a Latent Space (Chunyuan Li, Xiang Gao, Yuan Li, Baolin Peng, Xiujun Li, Yizhe Zhang, Jianfeng Gao)
- Finetuning Pretrained Transformers into Variational Autoencoders (Seongmin Park, Jihwa Lee)
Links (=optional)
- Neural Variational Inference for Text Processing. Yishu Miao, Lei Yu, Phil Blunsom.
- Improved Variational Autoencoders for Text Modeling using Dilated Convolutions. Zichao Yang, Zhiting Hu, Ruslan Salakhutdinov, Taylor Berg-Kirkpatrick.
- Glow: Better reversible generative models. Prafulla Dhariwal, Durk Kingma.
- Glow: Generative Flow with Invertible 1x1 Convolutions. Prafulla Dhariwal, Durk Kingma.
- Digit Fantasies by a Deep Generative Model. Durk Kingma.
Lecture notes
- Efficient Methods for Natural Language Processing: A Survey (Marcos Treviso, Ji-Ung Lee, Tianchu Ji, Betty van Aken, Qingqing Cao, Manuel R. Ciosici, Michael Hassid, Kenneth Heafield, Sara Hooker, Colin Raffel, Pedro H. Martins, André F. T. Martins, Jessica Zosa Forde, Peter Milder, Edwin Simpson, Noam Slonim, Jesse Dodge, Emma Strubell, Niranjan Balasubramanian, Leon Derczynski, Iryna Gurevych, Roy Schwartz)
- Efficient Transformers: A Survey (Yi Tay, Mostafa Dehghani, Dara Bahri, Donald Metzler)