PyTorch implementations of interesting papers and ideas in Machine Learning along with my own ideas and experiments.
The aim is generally to be clear, interpretable and communicative rather than terse and hyper-performant. I aim to who the main ideas on a read-through.
Main areas of focus are:
- Sparse Mixture of Experts models e.g. Switch Transformers, Expert Choice Routing
- Optimisers like Sophia
- Comparing Norms and Activations in different settings
- GPT sampling