Skip to content
This repository has been archived by the owner on Apr 23, 2024. It is now read-only.

compatibility of youtokentome.BPE outputs with the official implementation #93

Open
l-k-11235 opened this issue Jan 11, 2022 · 0 comments
Open

compatibility of youtokentome.BPE outputs with the official implementation #93

l-k-11235 opened this issue Jan 11, 2022 · 0 comments

Comments

@l-k-11235
Copy link

Hello !
Is it possible to make youtokentome.BPE outputs compatible with official implementation of bpe_dropout ? (https://github.com/rsennrich/subword-nmt#how-implementation-differs-from-sennrich-et-al-2016)
The matter is that when building the merge operations table we have the space symbol "▁" and not the end-of-word token “”.
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@l-k-11235