-
Notifications
You must be signed in to change notification settings - Fork 39
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
add LlamaIndex faithfulness metric #971
Conversation
src/unitxt/metrics.py
Outdated
@@ -2254,6 +2251,53 @@ def compute( | |||
} | |||
|
|||
|
|||
class LlamaIndexFaithfulness(LlamaIndexCorrectness): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There is a lot of duplicated code between the metrics. Can you please create a common base class (e.g. LlamaIndexMetric), and more the code there?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
One change requested regarding refactoring common code - see details in the comment.
Signed-off-by: Ariel Gera <ariel.gera1@ibm.com>
Signed-off-by: Ariel Gera <ariel.gera1@ibm.com>
src/unitxt/metrics.py
Outdated
"""LlamaIndex based metric class for evaluating faithfulness.""" | ||
|
||
metric_name_prefix = "faithfulness" | ||
reference_field = "contexts" # metric doesn't require reference answers |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is this needed? I don't see it being used.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As this is a referenceless metric, this is there to prevent failures of InstanceMetric
where no "references"
field was provided
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
(
Line 877 in 1242b02
if self.reference_field == "references": |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the quick replies on the CR comments. I see. But a "references" are always passed in unitxt - even for reference-less metrics. How did you get a failure?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Right, that is a different discussion that is outside the scope of this pull request... So if you want I can remove this
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes. For now, please do (it will avoid confusion).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done. I will say on a more general note that as unitxt provides the possibility for users to use just specific parts of the library (in this case, maybe I want to use a unitxt metric without using the full unitxt pipeline), these sorts of questions might need more thought (cc @elronbandel)
Signed-off-by: Ariel Gera <ariel.gera1@ibm.com>
Signed-off-by: Ariel Gera <ariel.gera1@ibm.com>
I approved - waiting for test to pass and auto merge. |
* add LlamaIndex faithfulness metric Signed-off-by: Ariel Gera <ariel.gera1@ibm.com> * share code between LlamaIndex metrics Signed-off-by: Ariel Gera <ariel.gera1@ibm.com> * use existing 'score_prefix' field Signed-off-by: Ariel Gera <ariel.gera1@ibm.com> * remove unused field Signed-off-by: Ariel Gera <ariel.gera1@ibm.com> --------- Signed-off-by: Ariel Gera <ariel.gera1@ibm.com>
No description provided.