Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

我用json格式的数据做大模型微调,但是数据集过大,在训练时爆内存,请问支持jsonl形式的格式吗,怎么改成jsonl格式呢数据 #579

Open
258863 opened this issue Jun 21, 2024 · 4 comments
Assignees

Comments

@258863
Copy link

258863 commented Jun 21, 2024

No description provided.

@258863
Copy link
Author

258863 commented Jun 21, 2024

image
原来是这样写的

@JunnYu
Copy link
Member

JunnYu commented Jun 24, 2024

你好,图中的那个是支持训练的时候支持json传入参数,例如 python run.py argument.json
argument.json如下

{
    "batch_size": 1,
    "learning_rate": 1e-4
}

@258863
Copy link
Author

258863 commented Jun 26, 2024

你好,图中的那个是支持训练的时候支持json传入参数,例如 python run.py argument.json argument.json如下

{
    "batch_size": 1,
    "learning_rate": 1e-4
}

但是数据大概40g一次性加载过程中会爆内存,预估加载需要300多g内存,所以除了json数据集还支持其他格式吗比如jsonl

@LokeZhou
Copy link
Collaborator

请问是哪个模型?是否开启了mixtoken,如果开启了,可以关掉试试

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants