Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

关于pretraining时denoising的问题 #71

Open
tu2022 opened this issue Jun 28, 2023 · 3 comments
Open

关于pretraining时denoising的问题 #71

tu2022 opened this issue Jun 28, 2023 · 3 comments

Comments

@tu2022
Copy link

tu2022 commented Jun 28, 2023

我看bart的论文在pretraining的时候会有五种denoising的方法，在bart_dataset.py中我看insert_ratio和rotate_ratio是设为0，似乎不能将其设为大于0的数，是否意味着不能进行text infilling和rotation？
@choosewhatulike
Copy link
Member

是的，denoising我们follow了BART的设置，只使用text infilling，没有加入insert和rotate。BART论文中表示这样效果最好

@tu2022
Copy link
Author

tu2022 commented Jul 3, 2023

还有个问题想问一下，你们这个预训练时，每个iteration时训练global batch size条数据吗？训练的每一条数据是截止至1024长度的文章，还是一整篇文章，文章被切割成一句一句，每一句padding到1024？

@choosewhatulike
Copy link
Member

是第一种，太长的文章会被分成多个1024。短的会padding到1024

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@choosewhatulike @tu2022