-
LLM Pruning and Distillation in Practice: The Minitron Approach, ST Sreenivas et al, NVidia, 2024
-
To Believe or Not to Believe Your LLM, Yasin Abbasi Yadkori et al, Google DeepMind, 2024
-
Evaluating Reward Models for Language Modeling, N. Lambert et al, U. Washington, 2024
-
Jamba: AHybrid Transformer-Mamba Language Model, Opher Lieber et al, 2024
-
Better & Faster Large Language Models via Multi-token Prediction, Fabian Gloeckle et al, 2024
-
FACtual enTailment fOr hallucInation Detection, Vipula Rawte et al, 2024
-
ReALM:Reference Resolution As Language Modeling, Joel Ruben Antony Moniz et al, Apple, 2024
-
A Neural Probabilistic Language Model, Y. Bengio et al, 2003
-
Unfamiliar Finetuning Examples Control How Language Models Hallucinate, Katie Kang et al, 2024
-
MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training, B. McKinzie et al, Apple, 2024
-
Self-Discover: Large Language Models Self-Compose Reasoning Structures, P. Zhou et al, 2024
-
Solving olympiad geometry without human demonstrations, TH Trinh et al, DeepMind, 2023
-
Self-Rewarding Language Models, W. Yuan et al, Meta, NYU, 2024
-
The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits, S. Ma et al , 2024
-
related repo: https://sites.google.com/view/chain-of-code
-
Llama 2: Open Foundation and Fine-Tuned Chat Models, Hugo Touvron, Louis Martin, et al, 2023
-
The Dawn of LMMs: Preliminary Explorations with GPT-4Vision, Yang et al, Microsoft, 2023
-
Sparks of artificial general intelligence: early experiments with GPT-4, Microsoft Research
-
Large Language Models can learn rules, Zhu et al, DeepMind, 2023
-
LLaMA: Open and Efficient Foundation Language Models, Meta AI
-
Physics of Large Language Models (Part 1), Context-free Grammar, Meta FAIR Labs, 2023
-
Understanding LLMs: A Comprehensive Overview from Training to Inference, Liu et al, 2024
-
Multimodality and Large Multimodal Models (LMMs), Chip Huyen, 2023, online article
-
Introduction to Transformers: an NLP Perspective, T. Xiao et al, 2023
-
Transformers Learn In-Context by Gradient Descent, Oswald et al, 2023
-
Transformers as Algorithms: Generalization and Stability in In-context Learning, Li et al., 2023
-
Hyena Hierarchy: Towards Larger Convolutional Language Models, Poli et al., 2023
-
Toward Understanding Why Adam Converges faster Than SGD for Transformers, Pan et al., CMU, 2023
-
Can GPT-3 Perform Statutory Reasoning?, Blair-Stanek, A et al., 2023
-
An Explanation of In-context Learning as Implicity Bayesian Inference, Xie et al., Stanford, 2022
-
Efficient Transformers: A Survey, Tay et al., Google Research, 2022
-
Emergent Abilities of Large Language Models, Wei et al., Google Research, 2022
-
A Path Towards Autonomous Machine Intelligence, Yann LeCun, 2022
-
Holisitc Evaluation of Language Models, Center for Research on Foundation Models, Stanford, 2022
-
A Systematic Evaluation of Large Language Models of Code, Xu et al., CMU, 2022
-
Evaluating Large Language Models Trained on Code, Chen et al., OpenAI, 2021
-
Language Models are Few-Shot Learners, Brown et al., OpenAI, 2020
-
Adam: A Method for Stochasitc Optimization, D. Kingma et al, 2014
-
Attention Is All You Need, Vaswani et al, Google Brain, 2017
-
HyperAttention: Long-context Attention in Near-Linear Time, Insu Han et al, 2023
-
Attention in Natural Language Processing, Galassi et al., 2020
-
Vision Language Transformers: A Survey, Clayton Fields, Casey Kennington, 2023
-
FAIRSEQ: A Fast, Extensible Toolkit for Sequence Modeling, Ott et al., 2019
-
Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling, Chung et al., 2014
-
A Decomposable Attention Model for Natural Language Inference, Parikh et al., Google Research, 2016
-
Sequence to Sequence Learning with Neural Networks, Sutskever et al, Google Research, 2014
-
Transforming Auto-encoders, G. Hinton, A. Krizhevsky, et al., 2011
-
Understanding LSTM: a tutorial into Long Short-Term Memory, R. Staudemeyer et al., 2019
-
Toward Understanding Why Adam Converges Faster Than SGD for Transformers, Pan, Y, et al, 2023
-
How ChatGPT Behavior is Changing Over Time?, Chen, L, Stanford, UC Berkeley, 2023
-
What Is ChatGPT Doing and Why Does It Work?, S. Wolfram, Feb 2023, online article
-
Meta-Transformer: A Unifed Framework for Multi-Modal Learning, Zhang, Y., et al, 2023
-
Improving Language Understanding by Generative Pre-Training, A. Redford, Open AI, 2018
-
Llama 2: Open Foundation and Fine-Tuned Chat Models, H. Touvron, Meta, 2023
-
Scaling Language Models - Methods, Analysis and Insights from Training Gopher,JW Rae, DeepMind, 2021
-
Pengi: An Audio Language Model for Audio Tasks, S. Deshmukh et al, 2024
-
A Comprehensive Overview of Large Language Models, H. Naveed et al, 2024
-
Are Long-LLMs A Necessity For Long-Context Tasks?, H. Qian et al, 2024
-
Solving Olympiad Geometry without Human Demonstrations, TH Trinh et al, Google DeepMind, 2023
-
LeanDojo: Theorem Proving with Retrieval-Augmented Language Models, K. Yang et al, 2023
-
NeurIPS Tutorial on Machine Learning for Theorem Proving, video
-
DeepMath - Deep Sequence Models for Premise Selection, Alexander Alemi, Francois Chollet et al, 2016
-
Let's build the GPT Tokenizer with Andrej Karpathy (February 2024)
-
Language Models are Unsupervised Multitask Learners, Alec Radford et al, 2018
-
Neural Machine Translation of Rare Words with Subword Units, Rico Senrich et al, 2016
-
Towards infinite LLM context windows, Towards Data Science, Krzysztof K. Zdeb, 2024
-
RoFormer: Enhanced Transformer with Rotary Position Embedding, Jianlin Su et al, 2023
-
YaRN: Efficient Context Window Extension of Large Language Models, B. Peng et al, U of Geneva, 2023
-
LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens, Y. Ding et al, 2024
-
relevant paper: iTransformer: Inverted Transformers Are Effective for Time Series Forecasting, Yong Liu et al, 2023
-
relevant repo: https://github.com/google-research/text-to-text-transfer-transformer
-
relevant paper: Lag-Llama: Towards Foundation Models for Probabilistic Time Series Forecasting, K. Rasul et al, 2023
relevant paper: A decoder-only foundation model for time-series forecasting, A. Das et al, 2023
relevant paper: Chronos: Learning the Language of Time Series, AF Ansari et al, 2024
relevant paper: Unified Training of Universal Time Series Forecasting Transformers, Woo, G et al, 2024
-
relevant paper: Chronos: Learning the Language of Time Series, AF Ansari et al, 2024
-
TimeGPT-1, Azul Garza, Max Mergenthaler-Canseco, Nixtla, 2023
-
TTM model and source code: https://huggingface.co/ibm/TTM, https://github.com/IBM/tsfm/tree/main/tsfm_public/models/tinytimemixer
-
Retrieval-Augmented Generation for Large Language Models: A Survey, Y. gao et al, 2024
-
relevant repo: https://github.com/microsoft/graphrag
relevant online article: https://www.microsoft.com/en-us/research/blog/graphrag-new-tool-for-complex-data-discovery-now-on-github/
-
Searching for Best Practices in Retrieval-Augmented Generation, X. Wang et al, 2024
-
Graph Retrieval-Augmented Generation: A Survey, B. Peng et al, 2024
-
RAFT: A new way to teach LLMs to be better at RAG, Cedric Vidal, 2024
-
RAFT: Adapting Language Model to Domain Specific RAG, T. Zhang et al, 2024 (blog)
-
RAFT: Adapting Language Model to Domain Specific RAG, T. Zhang et al, 2024 (paper)
relevant repos:
-
Contextual Position Encoding: Learning to Count What's Important, O. Golovneva et al, Meta, 2024
-
Augmenting Language Models with Long Term Memory, W. Wang et al, UC Santa Barbara, 2023
-
huggingface repo: LLM compiler
-
LLM Evaluation Metrics: The Ultimate LLM Evaluation Guide, Jeffrey Ip, online article, 2024
-
confident-ai
repo for LLM evaluation: https://github.com/confident-ai/deepeval -
Unveiling LLM Evaluation Focused on Metrics: Challenges and Solutions, T. Hu et al, 2024
-
GPT in 60 lines of NumPy code with Jay Mody, blog, (January, 2023)
-
Let's build GPT: from scratch, in code, spelled out with Andrej Karpathy (February 2023)
-
Let's build the GPT Tokenizer with Andrej Karpathy (February 2024)
The resource below attempts to visualize what is happening in LLM under the hood and is a helpful tool to comprehend the work of decoder-only Transformer-based LLMs. The author Brendan Bycroft has made an interesting attempt to visualize these structures and clarify how they operate. This webpage in the link below provides visualization for a family of GPT models, presented in 3D animations with walkthrough. The tool provides a step-by-step guide for single-token inference, coupled with interactive elements for a hands-on experience.
The blog: https://thegenerality.com/agi/
some of the articles:
-
The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits, S. Ma et al , 2024
-
BitNet: Scaling 1-bit Transformers for Large Language Models
-
Retentive Network: A Successor to Transformer for Large Language Models, Sun et al, 2023
-
Large Language Model for Science: A Study on P vs. NP, Q. Dong et al, 2023
-
Augmenting Language Models with Long-Term Memory, W. Wang et al, 2023
-
LONGNET: Scaling Transformers to 1,000,000,000 Tokens, J. Ding et al, 2023
-
The Transformer Architecture of GPT Models with Beatriz Stollniz
-
Learning Transformers Code First Part 1 - The Setup with Lily Hughs-Robinson
-
Learning Transformers Code First Part 2 - GPT Up Close and Personal with Lily Hughs-Robinson
-
Understanding Large Language Models: The Physics of ChatGPT and BERT with Tim Lou
-
Transformer Architectures and the Rise of BERT, GPT, and T5: A Beginner's Guide with Manas Joshi
-
Inside GPT - I: Understanding the text generation with Fatih Demirci
-
Platypus: Quick, Cheap and Powerful LLM with Salvatore Raieli
-
Configuring Nemo-Guardrails Your Way: An Alternative Method for LLM with Masatake Hirono
-
Fine-Tuning Large Language Models (LLMs) with Shawhin Talebi
-
Conversations as Directed Graphs with LangChain with Daniel Warfield
-
Self-Supervised Learning Using Projection Heads with Daniel Warfield
-
Summing Coin Values in Images using Lang-SAM and Deep Learning with Gamze Zorlubas
-
‘Talk’ to Your SQL Database Using LangChain and Azure OpenAI with Satwiki De
-
RLHF: Reinforcement Learning from Human Feedback with Ms Aerin
related paper: Training language models to follow instructions with human feedback, Ouyang et al, OpenAI, 2022
related code: Implementation of RLHF (Reinforcement Learning with Human Feedback) on top of the PaLM architecture
-
How to Convert Any Text Into a Graph of Concepts with Rahul Nayak
related repo: Knowledge_Graph
-
LLMs for Everyone: Running LangChain and a MistralAI 7B Model in Google Colab with Dmitrii Eliuseev
-
LLMs for Everyone: Running the LLaMA-13B model and LangChain in Google Colab with Dmitrii Eliuseev
related repo: https://github.com/ggerganov/llama.cpp
related repo: https://github.com/langchain-ai/langchain
related repo: https://colab.research.google.com/
-
Is Mamba the End of ChatGPT As We Know It? Igancio de Gregorio
related paper: Mamba: Linear-Time Sequence Modeling with Selective State Spaces, A. Gu et al, CMU, 2024
-
RLAIF: Reinforcement Learning from AI Feedback with Cameron R. Wolfe, Jan, 2024
related paper: RLAIF: Scaling Reinforcement Learning from Human Feedback with AI Feedback, Harrison Lee et al, 2023
related paper: Constitutional AI: Harmlessness from AI Feedback, Y. Bai, 2022
related paper: PaLM: Scaling Language Modeling with Pathways, A. Chowdhery et al, 2022
related paper: PaLM 2 Technical Report, Google, 2023
related paper: Chain-of-Thought Prompting Elicits Reasoning in Large Language Models, Wei et al, Google Research, 2022
related paper: Self-Consistency Improves Chain of Thought Reasoning in Language Models, Wang et al, Google Research, ICLR 2023
related paper: Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback, Bai et al, Anthropic, 2022
related paper: A General Language Assistant as a Laboratory for Alignment, A. Askell et al, Anthropic, 2021
related paper: Learning to summarize from human feedback, N. Stiennon et al, OpenAI, 2022
-
Mistral AI vs. Meta: Comparing Top Open-source LLMs with Luis Roque, Jan 2024
-
Text Embeddings, Classification, and Semantic Search, Shaw Talebi, March 2024
-
How to Build a Local Open-Source LLM Chatbot With RAG, Dr. Leon Eversberg, April, 2024