We read every piece of feedback, and take your input very seriously.
To see all available qualifiers, see our documentation.
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
在使用多机多卡时,要提前使用tokenized_path参数在单节点服务器上提前处理好数据集 但是在后续多节点训练的时候,会使用data.utils.split_dataset函数,会对数据集打乱或者切分,这时候依旧会使用磁盘进行缓存数据. 然后就会导致多节点同时操作同一个磁盘目录.会导致相应异常. 因此建议能否将data.utils.split_dataset中的切分和随机逻辑迁移到data.loader.get_dataset中
如果有其它更好的方法,请大佬指教.
No response
The text was updated successfully, but these errors were encountered:
No branches or pull requests
Reminder
System Info
Reproduction
在使用多机多卡时,要提前使用tokenized_path参数在单节点服务器上提前处理好数据集
但是在后续多节点训练的时候,会使用data.utils.split_dataset函数,会对数据集打乱或者切分,这时候依旧会使用磁盘进行缓存数据.
然后就会导致多节点同时操作同一个磁盘目录.会导致相应异常.
因此建议能否将data.utils.split_dataset中的切分和随机逻辑迁移到data.loader.get_dataset中
如果有其它更好的方法,请大佬指教.
Expected behavior
No response
Others
No response
The text was updated successfully, but these errors were encountered: