Adding regard measurement #271
Conversation
|
The documentation is not available anymore as the PR was closed or merged. |
Co-authored-by: helen <31600291+mathemakitten@users.noreply.github.com>
Co-authored-by: helen <31600291+mathemakitten@users.noreply.github.com>
Co-authored-by: helen <31600291+mathemakitten@users.noreply.github.com>
There was a problem hiding this comment.
I like the comparison between groups! Once the example TODOs and the README/formatting get updated I think it should be pretty much good to go, pinging @lvwerra for any further thoughts
Co-authored-by: helen <31600291+mathemakitten@users.noreply.github.com>
There was a problem hiding this comment.
Thanks for adding this! Looks good overall! Two comments:
- If we want to keep this a measurement (which are primarily applied to datasets) I suggest switching
prediction->datato comply with the other measurements. - I left a few suggestions to simplify a few things with list comprehensions.
changing `predictions` to `data`
changing predictions to data
Co-authored-by: Leandro von Werra <lvwerra@users.noreply.github.com>
Co-authored-by: Leandro von Werra <lvwerra@users.noreply.github.com>
|
Changed |
sashavor
changed the title
|
|
||
| @evaluate.utils.file_utils.add_start_docstrings(_DESCRIPTION, _KWARGS_DESCRIPTION) | ||
| class Regard(evaluate.Measurement): | ||
| def _info(self): |
There was a problem hiding this comment.
I would do a check if the config name is either compare or default. checkout the glue metrics for an example of how to do this.
There was a problem hiding this comment.
otherwise if you mistype compare you will silently get the default.
There was a problem hiding this comment.
like this? (line 121)
This code is pretty clunky, so I would love thoughts about how to get the list comprehension to work better, @mathemakitten !
Also, one of the initial use cases for regard was to compare the values for several populations (e.g. men and women, for instance), so would it make sense to have either the default or optional behavior to be something like:
and then the result would be the difference in regard between the two groups?