Skip to content

napoler/BulidDataset

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

BulidDataset

数据集预处理

包含lm,孪生,seq2seq,文本分类,示例参考dataDome目录下

默认使用Bert 21128分词方案,如果想要修改自己的分词可以修改config下的词典方案。

Getting Started

Download links:

SSH clone URL: ssh://git@git.jetbrains.space/terrychanorg/yuxunlianlm-bert/BulidDataset.git

HTTPS clone URL: https://git.jetbrains.space/terrychanorg/yuxunlianlm-bert/BulidDataset.git

These instructions will get you a copy of the project up and running on your local machine for development and testing purposes.

Prerequisites

What things you need to install the software and how to install them.

Examples

Deployment

Add additional notes about how to deploy this on a production system.

Resources

Add links to external resources for this project, such as CI server, bug tracker, etc.

关于tkitDatasetEx

tkitDatasetEx各种函数