-
Notifications
You must be signed in to change notification settings - Fork 237
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add a fingerprint for each EvaluationModule #206
Conversation
The documentation is not available anymore as the PR was closed or merged. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM! Also interested to hear what @lhoestq thinks. We plan to use this to cache evaluator computations.
Please wait with merging - want to do a minor release first.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In datasets
we use fingerprint
to identify data, while we use hash
to identify dataset scripts. For example a DatasetBuilder
has a hash
that identifies the code of the dataset script it is going to run.
Not a strong opinion but I think you can also name it hash
for consistency
Sounds good to me - also have no strong opinion :) |
Renamed, thanks for the clarification on hash vs. fingerprint! |
* Add fingerprint for Hub modules * Rename evaluation module fingerprint to _hash * fix typo Co-authored-by: helen <helen@huggingface.co>
In order to support #126 we need to fingerprint whichever EvaluationModule (metric, measurement, etc) we're using for later reproducibility.
This extracts the already-computed hash from each EvaluationModule and makes it easy to access in the
evaluation_cls
viamodule._fingerprint
, similar to how indatasets
you can dods._fingerprint
.Test via