Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

PreprocessingMetadata enhancement #2

Open
hlibbabii opened this issue Feb 27, 2020 · 1 comment
Open

PreprocessingMetadata enhancement #2

hlibbabii opened this issue Feb 27, 2020 · 1 comment
Labels
enhancement New feature or request
Milestone
Codeprep v2

Comments

@hlibbabii
Copy link
Member

  • Rename PreprocessingMetadata -> PreppedTokenMetadata
  • Represent word_boundaries field as a list of the number of subtoken in each token, e.g
    [1, 3, 1, 2] instead of [0, 1, 4, 5, 7]
  • Remove non-processible tokens filed. Return non-processible tokens as a separate object
  • Provide a method for returning the metadata for the last tokens:
>>> metadata.for_last_tokens(n: int)
@hlibbabii hlibbabii added this to the Codeprep v2 milestone Feb 27, 2020
@hlibbabii
Copy link
Member Author

This enhancement is useful for easier implementation of the calculation of the context statistics in giganticode-langmodels

hlibbabii added a commit that referenced this issue Feb 27, 2020
@hlibbabii
#2: Remove non-processible tokens filed. Return non-processible token…
eb1ef76 
…s as a separate object
hlibbabii added a commit that referenced this issue Feb 27, 2020
@hlibbabii
#2: rename PreprocessingMetadata -> PreppedTokenMetadata
8c9a061
hlibbabii added a commit that referenced this issue Feb 27, 2020
@hlibbabii
#2: Represent word_boundaries field as a list of the number of subtok…
8ed5685 
…en in each token, e.g [1, 3, 1, 2] instead of [0, 1, 4, 5, 7]
hlibbabii added a commit that referenced this issue Feb 27, 2020
@hlibbabii
#2: make PreppedTokenMetadata a dataclass
62e0bc0
hlibbabii added a commit that referenced this issue Feb 27, 2020
@hlibbabii
#2: Provide a method which truncates old metadata
fa4460a
hlibbabii added a commit that referenced this issue Feb 28, 2020
@hlibbabii
#2: make changes to iterators
5fa14e9
@hlibbabii hlibbabii added the enhancement New feature or request label Mar 18, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
Codeprep v2
Development

No branches or pull requests
1 participant
@hlibbabii