Named entity recognition in Malayalam using BiLSTM and TENER (Transformer Encoder)
-
Updated
Jul 13, 2023 - Jupyter Notebook
Named entity recognition in Malayalam using BiLSTM and TENER (Transformer Encoder)
This is project for sequence to sequence NLP task. We developed a custom model to understand the process of task using PyTorch. We also fine tuned pre-trained transformer models to improve the performance of translation task.
This is a tool that encrypts a sequence of words (or pieces of texts) using the AES-256 algorithm and encodes the encrypted result into a PNG image by linking each byte value to a specific color. It also decodes the before image to get back the original sequence of words
Feature extraction from sequential data
Order-agnostic lossless compressor using BPE and Huffman Coding.
An Introduction to Natural Language Processing (NLP)
an efficient ranked retrieval system for English corpora, optimised with VBE and BPE.
Code repo for the paper "AutoGO: Automated Computation Graph Optimization for Neural Network Evolution", accepted to NeurIPS 2023.
R package for Byte Pair Encoding based on YouTokenToMe
Modern Eager TensorFlow implementation of Attention Is All You Need
A byte-level Byte Pair Encoding (BPE) algorithm for tokenization in large language models (LLMs), similar to those used in GPT, Llama, and Mistral.
Byte-Pair Encoding (BPE) (subword-based tokenization) algorithm implementaions from scratch with python
Byte-pair encoding implementation in Python.
Byte pair encoding tokenizer as used in some large language models.
This repository houses my assignments completed during the Deep Learning course as part of my Master's in Data Analytics program. Explore diverse projects showcasing hands-on applications of advanced neural networks and machine learning techniques.
Morphologically biased byte-pair encoding
A Visualizer to check how BPE Tokenizer in an LLM Works
This project aims to implement word-based, character-based and subword-based tokenization techniques.
Add a description, image, and links to the byte-pair-encoding topic page so that developers can more easily learn about it.
To associate your repository with the byte-pair-encoding topic, visit your repo's landing page and select "manage topics."