add poseval #195
Merged
add poseval #195
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
The documentation is not available anymore as the PR was closed or merged. |
sashavor
approved these changes
Jul 22, 2022
metrics/poseval/README.md
Outdated
|
|
||
| ## Limitations and bias | ||
|
|
||
| In contrast to seqeval the poseval metric treats each token independantly and computes the classification report overal all concatenated sequences. |
There was a problem hiding this comment.
maybe a link to Seqeval here?
also, a few minor typos: "In contrast to seqeval, the poseval metric treats each token independently and computes the classification report over all concatenated sequences."
metrics/poseval/poseval.py
Outdated
| @@ -0,0 +1,113 @@ | |||
| # Copyright 2020 The HuggingFace Evaluate Authors. | |||
There was a problem hiding this comment.
A bit unrelated, but should we just massively update all of our code from 2020 to 2022? 😂
mathemakitten
pushed a commit
that referenced
this pull request
Aug 3, 2022
* add poseval * fix style * add suggestion from review
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds the Poseval metric. Since Seqeval does not work well with data that is not in IOB format (see here) and since POS usually assigns one label per word with poseval each token is evaluated independently (as suggested here).
cc @fxmarty @lewtun since that's useful for the
evalautorandEval on the hub.