Syllabus
The syllabus is preliminary and subject to change.
Lecture notes
- Alignment Exercise (Kevin Knight)
- Language Models
Links (=optional)
- NLP Highlights podcast. Various Allen AI researchers.
Lecture notes
- Lecture notes
- Neural Machine Translation by Jointly Learning to Align and Translate (Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio)
Links (=optional)
- Sequence to Sequence Learning with Neural Networks. Ilya Sutskever, Oriol Vinyals, Quoc V. Le.
- Effective Approaches to Attention-based Neural Machine Translation. Minh-Thang Luong, Hieu Pham, Christopher D. Manning.
Lecture notes
- Lecture notes
- Attention is all you need (Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin)
- Transformer: A Novel Neural Network Architecture for Language Understanding (posted by Jakob Uszkoreit)
- Stanford cs224n lecture notes on self attention (John Hewitt)
- The Annotated Transformer (Austin Huang, Suraj Subramanian, Jonathan Sum, Khalid Almubarak, and Stella Biderman. Original by Sasha Rush)
- The Illustrated Transformer (Jay Alammar)
Links (=optional)
- Convolutional Sequence to Sequence Learning. Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, Yann N. Dauphin.
- Layer Normalization. Jimmy Lei Ba, Jamie Ryan Kiros, Geoffrey E. Hinton.
- Dropout: A Simple Way to Prevent Neural Networks from Overfitting. Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, Ruslan Salakhutdinov.
- Improving Transformer Optimization Through Better Initialization. Xiao Shi Huang, Felipe Perez, Jimmy Ba, Maksims Volkovs.
- A Mathematical Framework for Transformer Circuits. Anthropic.
- Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned. Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, Ivan Titov.
Lecture notes
- Lecture notes
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova)
- Improving language understanding with unsupervised learning (Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever)
Links (=optional)
- Semi-supervised Sequence Learning. Andrew M. Dai, Quoc V. Le.
- Deep contextualized word representations. Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, Luke Zettlemoyer.
- RoBERTa: A Robustly Optimized BERT Pretraining Approach. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, Veselin Stoyanov.
- Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context. Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V. Le, Ruslan Salakhutdinov.
- ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators. Kevin Clark, Minh-Thang Luong, Quoc V. Le, Christopher D. Manning.
- Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J. Liu.
- ALBERT: A Lite BERT for Self-supervised Learning of Language Representations. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, Radu Soricut.
Lecture notes
- Lecture notes
- AI and the Everything in the Whole Wide World Benchmark (Deborah Raji, Emily Denton, Emily M. Bender, Alex Hanna, Amandalynne Paullada)
- Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data (Emily M. Bender, Alexander Koller)
Links (=optional)
Lecture notes
- Lecture notes
- Distilling the Knowledge in a Neural Network (Geoffrey Hinton, Oriol Vinyals, Jeff Dean)
Links (=optional)
- Sequence-Level Knowledge Distillation. Yoon Kim, Alexander Rush.
- Dark Knowledge. Geoffrey Hinton.
- DistilBERT. Huggingface.
- Compressing Large-Scale Transformer-Based Models: A Case Study on BERT. Prakhar Ganesh, Yao Chen, Xin Lou, Mohammad Ali Khan, Yin Yang, Hassan Sajjad, Preslav Nakov, Deming Chen, Marianne Winslett.
Lecture notes
- Lecture notes
- Prefix-Tuning: Optimizing Continuous Prompts for Generation (Xiang Lisa Li, Percy Liang)
- AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuning (Yaqing Wang, Sahaj Agarwal, Subhabrata Mukherjee, Xiaodong Liu, Jing Gao, Ahmed Hassan Awadallah, Jianfeng Gao)
- LoRA: Low-Rank Adaptation of Large Language Models (Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen)
- Adapter methods (docs.adapterhub.ml)
Links (=optional)
- AdapterHub: A Framework for Adapting Transformers. Jonas Pfeiffer, Andreas Rücklé, Clifton Poth, Aishwarya Kamath, Ivan Vulić, Sebastian Ruder, Kyunghyun Cho, Iryna Gurevych.
- Parameter-Efficient Transfer Learning for NLP. Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, Sylvain Gelly.
- Simple, Scalable Adaptation for Neural Machine Translation. Ankur Bapna, Naveen Arivazhagan, Orhan Firat.
- AdapterFusion: Non-Destructive Task Composition for Transfer Learning. Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, Iryna Gurevych.
- Parameter-Efficient Tuning with Special Token Adaptation. Xiaocong Yang, James Y. Huang, Wenxuan Zhou, Muhao Chen.
Lecture notes
- Lecture notes
- Scaling Laws for Neural Language Models (Open AI)
- Training Compute-Optimal Large Language Models (DeepMind)
- Language Models are Unsupervised Multitask Learners (Open AI)
- Language Models are Few-Shot Learners (Open AI)
- GPT-4 Technical Report (Open AI)
Links (=optional)
- The AI Brick Wall – A Practical Limit For Scaling Dense Transformer Models, and How GPT 4 Will Break Past It. Dylan Patel.
- Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?. Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, Luke Zettlemoyer.
- Proximal Policy Optimization Algorithms. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov.
- LIMA: Less Is More for Alignment. Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, Omer Levy.
Lecture notes
- Lecture notes
- Generalization through Memorization: Nearest Neighbor Language Models (Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, Mike Lewis)
- Improving language models by retrieving from trillions of tokens (Deepmind)
Links (=optional)
- Nearest Neighbor Knowledge Distillation for Neural Machine Translation. Zhixian Yang, Renliang Sun, Xiaojun Wan.
Lecture notes
- Lecture notes
- Tutorial - What is a variational autoencoder? (Jaan Altosaar)
- Optimus: Organizing Sentences via Pre-trained Modeling of a Latent Space (Chunyuan Li, Xiang Gao, Yuan Li, Baolin Peng, Xiujun Li, Yizhe Zhang, Jianfeng Gao)
- Finetuning Pretrained Transformers into Variational Autoencoders (Seongmin Park, Jihwa Lee)
Links (=optional)
- Neural Variational Inference for Text Processing. Yishu Miao, Lei Yu, Phil Blunsom.
- Improved Variational Autoencoders for Text Modeling using Dilated Convolutions. Zichao Yang, Zhiting Hu, Ruslan Salakhutdinov, Taylor Berg-Kirkpatrick.
- Glow: Better reversible generative models. Prafulla Dhariwal, Durk Kingma.
- Glow: Generative Flow with Invertible 1x1 Convolutions. Prafulla Dhariwal, Durk Kingma.
- Digit Fantasies by a Deep Generative Model. Durk Kingma.
Lecture notes
- Efficient Transformers: A Survey (Yi Tay, Mostafa Dehghani, Dara Bahri, Donald Metzler)
- Reformer: The Efficient Transformer (Nikita Kitaev, Lukasz Kaiser, Anselm Levskaya)
- Rethinking Attention with Performers (Krzysztof Marcin Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Quincy Davis, Afroz Mohiuddin, Lukasz Kaiser, David Benjamin Belanger, Lucy J Colwell, Adrian Weller)
- Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation (Ofir Press, Noah A. Smith, Mike Lewis)
- Transformer-XL: Attentive Language Models beyond a Fixed-Length Context (Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, Ruslan Salakhutdinov)
- ∞-former: Infinite Memory Transformer (Pedro Henrique Martins, Zita Marinho, Andre Martins)
- FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness (Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, Christopher Ré)
Lecture notes
- Do We Need to Create Big Datasets to Learn a Task? (Swaroop Mishra, Bhavdeep Singh Sachdeva)
- Shortformer: Better Language Modeling using Shorter Inputs (Ofir Press, Noah A. Smith, Mike Lewis)
- Active Learning for BERT: An Empirical Study (Liat Ein-Dor, Alon Halfon, Ariel Gera, Eyal Shnarch, Lena Dankin, Leshem Choshen, Marina Danilevsky, Ranit Aharonov, Yoav Katz, Noam Slonim)
Lecture notes
- Efficient Methods for Natural Language Processing: A Survey (Marcos Treviso, Ji-Ung Lee, Tianchu Ji, Betty van Aken, Qingqing Cao, Manuel R. Ciosici, Michael Hassid, Kenneth Heafield, Sara Hooker, Colin Raffel, Pedro H. Martins, André F. T. Martins, Jessica Zosa Forde, Peter Milder, Edwin Simpson, Noam Slonim, Jesse Dodge, Emma Strubell, Niranjan Balasubramanian, Leon Derczynski, Iryna Gurevych, Roy Schwartz)
- Efficient Transformers: A Survey (Yi Tay, Mostafa Dehghani, Dara Bahri, Donald Metzler)