Eval metrics #587
Merged
Eval metrics #587
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Member
lilacheden
commented
Feb 20, 2024
lilacheden
commented
- Standard api for disabling/enabling confidence interval computation of metric/metricpipeline
- make kendalltau and roc_auc metrics and not metric pipelines (to temporarily overcome the multiple metric pipeline issue)
Codecov ReportAttention:
Additional details and impacted files@@ Coverage Diff @@
## main #587 +/- ##
==========================================
+ Coverage 87.78% 87.97% +0.18%
==========================================
Files 85 85
Lines 7170 7208 +38
==========================================
+ Hits 6294 6341 +47
+ Misses 876 867 -9 ☔ View full report in Codecov by Sentry. |
matanor
approved these changes
Feb 20, 2024
src/unitxt/metrics.py
Outdated
| @@ -80,6 +80,14 @@ def new_random_generator(): | |||
| def disable_confidence_interval_calculation(self): | |||
| self.n_resamples = None | |||
|
|
|||
| def disable_confidence_interval_calculation_return_n_resamples(self): | |||
There was a problem hiding this comment.
maybe just return self.n_resamples from the disable_confidence_interval_calculation method?
that shouldn't affect the existing code at all, no?
src/unitxt/metrics.py
Outdated
| kendall_results = self.kendalltau(references, predictions, variant=self.variant) | ||
| corr = kendall_results.correlation | ||
| return { | ||
| self.main_score: corr, | ||
| "p_val": kendall_results.pvalue, | ||
| "kendalltau_p_val": kendall_results.pvalue, |
There was a problem hiding this comment.
Suggested change
| "kendalltau_p_val": kendall_results.pvalue, | |
| f"{self.main_score}_p_val": kendall_results.pvalue, |
There was a problem hiding this comment.
the current main score is with a "_b" suffix, so perhaps they should be consistent?
main_score = "kendalltau_b"
| metric.metric, MetricWithConfidenceInterval | ||
| ): | ||
| metric.metric.disable_confidence_interval_calculation() | ||
| metric.disable_confidence_interval_calculation() |
There was a problem hiding this comment.
do we also need an abstract disable_confidence_interval_calculation() in Metric?
There was a problem hiding this comment.
currently IIUC only the "overrides" were defined ..
lilacheden
force-pushed
the
eval_metrics
branch
2 times, most recently
from
c0c1cf2
to
afc8030
Compare
Signed-off-by: lilacheden <lilach.edel@gmail.com>
…es issue) Signed-off-by: lilacheden <lilach.edel@gmail.com>
Signed-off-by: lilacheden <lilach.edel@gmail.com>
Signed-off-by: lilacheden <lilach.edel@gmail.com>
…fix set_n_resamples Signed-off-by: lilacheden <lilach.edel@gmail.com>
Signed-off-by: lilacheden <lilach.edel@gmail.com>
Signed-off-by: lilacheden <lilach.edel@gmail.com>
lilacheden
force-pushed
the
eval_metrics
branch
from
d57af26
to
6fba78a
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.