A byte-level Byte Pair Encoding (BPE) algorithm for tokenization in large language models (LLMs), similar to those used in GPT, Llama, and Mistral.

This is project for sequence to sequence NLP task. We developed a custom model to understand the process of task using PyTorch. We also fine tuned pre-trained transformer models to improve the performance of translation task.

An implementation of the GPT(generative pretrained transformer) model, from scratch, which produces Shakespearean text by training on the dialogues written by Shakespeare along with the GPT Encoder.

Morphologically biased byte-pair encoding

A Visualizer to check how BPE Tokenizer in an LLM Works

Code repo for the paper "AutoGO: Automated Computation Graph Optimization for Neural Network Evolution", accepted to NeurIPS 2023.

Kotlin multiplatform BPE tokenizer library for OpenAI models

Order-agnostic lossless compressor using BPE and Huffman Coding.

Byte-Pair Encoding tokenizer for training large language models on huge datasets

Byte-level byte pair encoding (BPE) in Haskell

This repository houses my assignments completed during the Deep Learning course as part of my Master's in Data Analytics program. Explore diverse projects showcasing hands-on applications of advanced neural networks and machine learning techniques.

Byte pair encoding tokenizer as used in some large language models.

Byte-pair encoding implementation in Python.

High performance unsupervised text tokenization for Ruby

an efficient ranked retrieval system for English corpora, optimised with VBE and BPE.

An Introduction to Natural Language Processing (NLP)

Go BPE tokenizer (Encoder+Decoder) for GPT2 and GPT3

This is a tool that encrypts a sequence of words (or pieces of texts) using the AES-256 algorithm and encodes the encrypted result into a PNG image by linking each byte value to a specific color. It also decodes the before image to get back the original sequence of words

R package for Byte Pair Encoding based on YouTokenToMe

Modern Eager TensorFlow implementation of Attention Is All You Need