-
Notifications
You must be signed in to change notification settings - Fork 182
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add Vectara Hallucination Detection Model #950
Conversation
Check out this pull request on See visual diffs & provide feedback on Jupyter Notebooks. Powered by ReviewNB |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
❌ Changes requested.
- Reviewed the entire pull request up to 4f3353e
- Looked at
61
lines of code in1
files - Took 1 minute and 3 seconds to review
More info
- Skipped
4
files when reviewing. - Skipped posting
1
additional comments because they didn't meet confidence threshold of50%
.
1. trulens_eval/trulens_eval/feedback/provider/hugs.py:485
:
- Assessed confidence :
100%
- Grade:
40%
- Comment:
The method hallucination_evaluator does not handle the case when the response from the API is not a list and not a proper HTTP response. This could lead to unexpected behavior. Consider adding an else clause to handle this case. - Reasoning:
The new method hallucination_evaluator is not handling the case when the response from the API is not a list and not a proper HTTP response. This could lead to unexpected behavior.
Workflow ID: wflow_I5ako30SCD4tU8DH
Want Ellipsis to fix these issues? Tag @ellipsis-dev
in a comment. We'll respond in a few minutes. Learn more here.
@@ -0,0 +1,300 @@ | |||
{ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you remove the .ipynb checkpoints from this change? Also can you elaborate on why there are two separate notebooks to show this capability? |
@@ -0,0 +1,300 @@ | |||
{ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
since this notebook is focused on the Vectara HHEM evaluator, can you rename the notebook to reflect that?
Suggestion: vectara_hallucination_evaluator.ipynb
Reply via ReviewNB
@@ -0,0 +1,300 @@ | |||
{ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It'd be useful here to show usage of this evaluator as part of a recorded app, e.g. as shown in https://www.trulens.org/trulens_eval/langchain_quickstart/
Reply via ReviewNB
@@ -0,0 +1,260 @@ | |||
{ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
hi @MN-Noor would you like to take the last reviews above ? @joshreini1 , thanks for the comments, should we keep both notebooks renamed or remove one or the other? sorry to bother you but i'll be able to wrap this up :-) |
@Josephrp keep the one in |
@joshreini1 thanks ! @MN-Noor wrapped it up nicely , hope that's us done - until next time ! |
ha! i sure do , here's me . hope that's okay :-) |
* added vectra hallucination feedback function in hugs.py * rag notebook to evaluate responces using vectra HHEM * Create hugging__Face_.ipynb with all my thanks ! * formated hhem notebook * formatting * moved hhem notebook * handling incorrect responce formats * added vectara hhem evaluate * documentataion added * remove hugging face.file --------- Co-authored-by: MN-Noor <mn05052002@gmail.com> Co-authored-by: tonic <josephp@tuta.io> Co-authored-by: MN-Noor <108609236+MN-Noor@users.noreply.github.com> Co-authored-by: Josh Reini <60949774+joshreini1@users.noreply.github.com>
* added vectra hallucination feedback function in hugs.py * rag notebook to evaluate responces using vectra HHEM * Create hugging__Face_.ipynb with all my thanks ! * formated hhem notebook * formatting * moved hhem notebook * handling incorrect responce formats * added vectara hhem evaluate * documentataion added * remove hugging face.file --------- Co-authored-by: MN-Noor <mn05052002@gmail.com> Co-authored-by: tonic <josephp@tuta.io> Co-authored-by: MN-Noor <108609236+MN-Noor@users.noreply.github.com> Co-authored-by: Josh Reini <60949774+joshreini1@users.noreply.github.com>
Added vectara hallucination detection model to the huggingface class
adding an exposition/model example using huggingface models end to end for demo
this is a draft PR , just need to add some text to explain the examples we have chosen
Summary:
The PR adds a new method
hallucination_evaluator
to theHuggingface
class inhugs.py
for evaluating the hallucination score of a combined input of two statements using the Huggingface hallucination evaluation model.Key points:
hallucination_evaluator
method toHuggingface
class inhugs.py
.HUGS_HALLUCINATION_API_URL
endpoint for the hallucination evaluation model from Huggingface.model_output
andretrieved_text_chunks
, combines them, and sends a POST request to the API.Generated with ❤️ by ellipsis.dev