-
Natural Language Processing with Transformers, Lewis Tunstall, Leandro von Werra, Thomas Wolf, 2022
github repo with source code: https://github.com/nlp-with-transformers/notebooks
-
Adam: A Method for Stochasitc Optimization, D. Kingma et al, 2014
-
Attention Is All You Need, Vaswani et al, Google Brain, 2017
-
The Annotated Transformer - delving into Vaswani's paper "Attention Is All You Need", 2018
-
Attention in Natural Language Processing, Galassi et al., 2020
-
Deriving Machine Attention from Human Rationales, Y. Bao et al, 2018
-
HyperAttention: Long-context Attention in Near-Linear Time, Insu Han et al, 2023
-
FAIRSEQ: A Fast, Extensible Toolkit for Sequence Modeling, Ott et al., 2019
-
Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling, Chung et al., 2014
-
Bidirectional Recurrent Neural Networks, Mike Schuster, Kuldip Paliwal, 1997
-
Translation Modeling with Bidirectional Recurrent Neural Networks, M. Sundermeyer, et al, 2014
-
A Decomposable Attention Model for Natural Language Inference, Parikh et al., Google Research, 2016
-
Sequence to Sequence Learning with Neural Networks, Sutskever et al, Google Research, 2014
-
Transforming Auto-encoders, G. Hinton, A. Krizhevsky, et al., 2011
-
A Neural Probabilistic Language Model, Y. Bengio et al, 2003
-
LSTM Can Solve Hard Long Time Lag Problems, Sepp Hochreiter, Juergen Schmidthhuber, NIPS, 1996
-
Recurrent Continuous Translation Models, Nal Kalchbrenner, Phil Blunsom, Oxford U, 2013
-
Understanding LSTM: a tutorial into Long Short-Term Memory, R. Staudemeyer et al., 2019
-
Generating Sequences with Recurrent Neural Networks, Alex Graves, UofToronto, 2014
-
DeLighT: Deep and Light-weight Transformer, S. Mehta et al, 2020
-
Meta-Transformer: A Unified Framework for Multimodal Learning, Zhang, Y, et al, 2023
-
Small-scale proxies for large-scale Transformer training instabilities, M. Wortsman et al, 2023
-
Formal Algorithms for Transformers, M. Phuong et al, DeepMind, 2022
-
Boolformer: Symbolic Regression of Logic Functions with Transformers, d'Ascoli et al, 2023
-
Transformer-Based Direct Hidden Markov Model for Machine Translation, W. Wang et al, Aachen U, 2021
-
Introduction to Transformers: an NLP Perspective, T. Xiao et al, 2023
-
xLSTM: Extended Long Short-Term Memory, Maximilian Beck et al, 2024
-
FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning, Tri Dao , 2023
-
A Mathematical Framework for Transformer Circuits, Nelson Elhage et al, Anthropic, 2021
the full article online: here
-
Understanding Transformer Reasoning Capabilities via Graph Algorithms, Clayton Sanford et al, 2024
-
Understanding Transformers via N-gram Statistics, Timothy Nguyen, Google DeepMind, 2024
-
When Can Transformers Count to n? G. Yehudai et al, NYU, DeepMind, 2024
-
Efficient Estimation of Word Representations in Vector Space, Tomas Mikolov et al, Google, 2013
-
Is Cosine-Similarity of Embeddings Really About Similarity? Harald Steck, Chaitanya Ekanadham, 2024
-
Hierarchical Attention Networks for Document Classification, Z. Yang et al, CMU, 2016
-
Why does an attention layer in a transformer learn context?, DataScience.StackExchange, 2020
-
In-context Learning and Induction Heads, Catherine Olsson et al, Anthropic, 2023
-
Transformers Learn In-Context by Gradient Descent, Johannes von Oswald et al, 2023
-
Transformers as Algorithms: Generalization and Stability in In-context Learning, Y. Li et al, 2023
-
What Can Transformers Learn In-Context? A Case Study of Simple Function Classes, S. Garg, 2023
related repo: https://github.com/dtsip/in-context-learning
related youtube presentation: https://www.youtube.com/watch?v=DiJsg93zQDc
-
The Transient Nature of Emergent In-Context Learning in Transformers, A. Singh et al, UCL, 2023
-
In-context Learning in Transformers - SLT Seminar 46, youtube video
-
related repo: https://qtransformer.github.io/
... More articles on Transformers
-
Visualizing Attention in Vision Transformer with Aryan Jadon
-
Vision Transformers, Explained, Skylar Jean Callis, Toward Data Science, Feb, 2024
-
Comparison of Convolutional Neural Networks and Vision Transformers (ViTs) with Illas Papastratis
-
Do Vision Transformers See Like Convolutional Neural Networks? M. Raghu, Google Brain, 2022
-
An Image is Worth 16X16 Wwords: Transformers for Image Recognition at Scale, A. Dosovitskiy, 2021
-
Transformers for Image Recognition at Scale, Nel Houlsby and Dirk Weissenborn, Dec 2020, blog
-
Vision Transformers (ViT) in Image Recognition – 2024 Guide, Gaudenz Boesch, viso.ai blog
-
xLSTM: Extended Long Short-Term Memory, Maximilian Beck et al, 2024
-
LST Can Solve Hard Long Time Lag Problems, Sepp Hochreiter et al, 1996
-
Multi-Head State Space Model for Speech Recognition, Yassir Fathullah et al, U of Cambridge, 2023
-
HiPPO: Recurrent Memory with Optimal Polynomial Projections, A. Gu et al, Stanford U., 2020
-
Diagonal State Spaces are as Effective as Structured State Spaces, A. Gupta, A. Gu et al, 2022
-
It’s Raw! Audio Generation with State-Space Models, K. Goel, A. Gu, et al, 2022
-
Efficiently Modeling Long Sequences with Structured State Spaces, K. Goel, A. Gu et al, 2022
-
Hungry Hungry Hippos: Towards Language Modeling with State Space Models, D. Fu, T. Dao, 2023
-
Mamba: Linear-Time Sequence Modeling with Selective State Spaces, A. Gu, T. Dao, CMU, 2023
Mamba repo: https://github.com/state-spaces/mamba -
Mamba: Can it replace Transformers? Vishal Rajput, Medium Jan 8, 2024
as a pdf file: here -
Why Mamba was rejected? Joe El Khoury, Medium, Feb 28, 2024
as a pdf file: here -
Zamba: A Compact 7B SSM Hybrid Model, Paolo Glorioso et al, 2024
related repo: https://github.com/kyegomez/Zamba
-
relevant paper: iTransformer: Inverted Transformers Are Effective for Time Series Forecasting, Yong Liu et al, 2023
-
relevant repo: https://github.com/google-research/text-to-text-transfer-transformer
-
relevant paper: Lag-Llama: Towards Foundation Models for Probabilistic Time Series Forecasting, K. Rasul et al, 2023
relevant paper: A decoder-only foundation model for time-series forecasting, A. Das et al, 2023
relevant paper: Chronos: Learning the Language of Time Series, AF Ansari et al, 2024
relevant paper: Unified Training of Universal Time Series Forecasting Transformers, Woo, G et al, 2024
-
The A-Z of Transformers: Everything You Need to Know with François Porcher
related paper: Neural Machine Translation by Jointly Learning to Align and Translate, D. Bahdanau et al, 2015
related repo: Transformers from Scratch
-
Transformers — Intuitively and Exhaustively Explained with Daniel Warfield
-
De-coded: Transformers explained in plain English with Chris Hughes
-
Transformers Explained Visually — Not Just How, but Why They Work So Well, Ketan Doshi, Jun 2, 2021
-
Transformers Explained Visually (Part 1): Overview of Functionality with Ketan DOshi, Dec 13, 2020
-
Transformers Explained Visually (Part 2): How it works, step-by-step with Ketan Doshi, Jan 2, 2021
-
Transformers Explained Visually (Part 3): Multi-head Attention, deep dive, Jan 16, 2020
-
Explainable AI: Visualizing Attention in Transformers with Abby Morgan
-
How to Take Advantage of the New Disruptive AI Technology Called Transformers with Jordi Torres
-
Transfer Learning: The Democratization of Transformers with Jori Torres
-
Visualizing Attention in Vision Transformer with Aryan Jadon
-
Vision Transformers, Explained, Skylar Jean Callis, Toward Data Science, Feb, 2024
-
The Transformer Architecture of GPT Models with Beatriz Stollniz
-
Learning Transformers Code First Part 1 - The Setup with Lily Hughs-Robinson
-
Learning Transformers Code First Part 2 - GPT Up Close and Personal with Lily Hughs-Robinson
-
Were Abstract Painters The First Encoders with Wouter van Heeswijk
-
Comparison of Convolutional Neural Networks and Vision Transformers (ViTs) with Illas Papastratis
related paper: Do Vision Transformers See Like Convolutional Neural Networks? M. Raghu, Google Brain, 2022
related paper: How Do Vision Transformers Work? N. Park et al, 2022
related paper: An Image is Worth 16X16 Wwords: Transformers for Image Recognition at Scale, A. Dosovitskiy, 2021
related blog: https://blog.research.google/2020/12/transformers-for-image-recognition-at.html
related blog: https://becominghuman.ai/transformers-in-vision-e2e87b739feb
related blog: https://viso.ai/deep-learning/vision-transformer-vit/
-
Understanding Temporal Fusion Transformer with Mouna Labiadh
related article: Temporal Fusion Transformer for Interpretable Multi-horizon Time Series Forecasting
-
Forecasting book sales with Temporal Fusion Transformer with Mouna Labiadh
-
Personalized Recommendations with Transformers with Enis Teper
-
Implementing Seq2Seq Models for Efficient Time Series Forecasting with Max Brenner
-
Fine-Tune Smaller Transformer Models: Text Classification, Ida Silfverskiöld, 2024
Stanford CS 25 Transformers United: 25 Lectures Set, youtube playlist
Stanford CS 25: Lecture 1 Transformers United: DL Models that have revolutionized NLP, CV, RL
Stanford CS 25: Lecture 2 Transformers in Language: The development of GPT Models, GPT3
Stanford CS 25: Lecture 3 Transformers in Vision: Tackling problems in Computer Vision
Stanford CS 25: Lecture 4 Decision Transformer: Reinforcement Learning via Sequence Modeling
Stanford CS 25: Lecture 5 Mixture of Experts (MoE) paradigm and the Switch Transformer
Stanford CS 25: Lecture 6 DeepMind's Perceiver and Perceiver IO: new data family architecture
Stanford CS 25: Lecture 7 Self Attention and Non-parametric transformers (NPTs)
Stanford CS 25: Lecture 8 Transformer Circuits, Induction Heads, In-Context Learning
Stanford CS 25: Lecture 9 Audio Research: Transformers for Applications in Audio, Speech, Music
Stanford CS 25: Lecture 10 Represent part-whole hierarchies in a neural network, Geoff Hinton
Stanford CS 25: Lecture 11 Introduction to Transformers w/ Andrej Karpathy
Stanford CS 25: Lecture 12 Language and Human Alignment
Stanford CS 25: Lecture 13 Emergent Abilities and Scaling in LLMs
Stanford CS 25: Lecture 14 Strategic Games
Stanford CS 25: Lecture 15 Robotics and Imitation Learning
Stanford CS 25: Lecture 16 Common Sense Reasoning
Stanford CS 25: Lecture 17 Biomedical Transformers
Stanford CS 25: Lecture 18 Neuroscience-Inspired Artificial Intelligence
Stanford CS 25: Lecture 19 Low-level Embodied Intelligence w/ Foundation Models
Stanford CS 25: Lecture 20 Generalist Agents in Open-Ended Worlds
Stanford CS 25: Lecture 21 How I Learned to Stop Worrying and Love the Transformer
Stanford CS 25: Lecture 22 Recipe for Training Helpful Chatbots
Stanford CS 25: Lecture 23 No Language Left Behind: Scaling Human-Centered Machine Translation
Stanford CS 25: Lecture 25 Retrieval Augmented Language Models
GPT in 60 Lines of code: https://jaykmody.com/blog/gpt-from-scratch/