NLPToolkit

Introduction

nlptoolkit is a toolkit for NLP(Natural Language Processing) and LLM(Large Language Models) using Pytorch. nlptoolkit has implemented many language models and data preprocessing methods. More importantly, it provides a lot of examples that can run end-to-end.

Tokenizer

• BaseTokenizer
• JiebaTokenizer
• SentencePieceTokenizer
• BytePairEncoding(BPE)Tokenizer
• BertTokenizer

Support Models

Supported Language Models:

• RNNLM
• CNNLM
• Ngram
• SkipGram
• CBOW
• Glove
• CoVe
• ELMO
• ULMFiT
• Seq2Seq | Attention Seq2Seq

Supported Transformer Models:

• Transformer
• Bert
• XLNet
• GPT
• GPT2
• RoBERTa
• T5

Dependencies

• Python 3.5+
• Pytorch 1.5.0+

Reference:

• https://zh.d2l.ai/
  • Dive into Deep Learning，D2L.ai
• https://github.com/dmlc/gluon-nlp/
  • GluonNLP: NLP made easy
• https://github.com/huggingface/tokenizers
  • Provides an implementation of today's most used tokenizers, with a focus on performance and versatility.
• https://github.com/The-AI-Summer/self-attention-cv
  • Self-attention building blocks for computer vision applications in PyTorch
• 自然语言处理：基于预训练模型的方法（作者：车万翔、郭江、崔一鸣）

License

NLPToolkit is released under the Apache 2.0 license.

Citation

Please cite the repo if you use the data or code in this repo.

@misc{NLPToolkit,
  author = {jianzhnie},
  title = {NLPToolkit: NLPToolkit is a toolkit for NLP and LLMs using Pytorch},
  year = {2023},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/jianzhnie/NLP-Toolkit}},
}