-
Notifications
You must be signed in to change notification settings - Fork 182
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Langchain Quickstart notebook does not work #1179
Comments
Hey @mkhammoud! 👋 I'm here to help you with any bugs, questions, or contributions you have in mind. Let's squash those bugs together! To resolve the issue with the Langchain Quickstart notebook where you receive the error Here is the corrected code snippet for initializing the feedback functions: from trulens_eval.feedback.provider import OpenAI
from trulens_eval import Feedback
import numpy as np
# Initialize provider class
provider = OpenAI()
# select context to be used in feedback. the location of context is app specific.
from trulens_eval.app import App
context = App.select_context(rag_chain)
# Define a groundedness feedback function
f_groundedness = (
Feedback(provider.groundedness_measure_with_cot_reasons)
.on(context.collect()) # collect context chunks into a list
.on_output()
)
# Question/answer relevance between overall question and answer.
f_answer_relevance = (
Feedback(provider.relevance)
.on_input_output()
)
# Question/statement relevance between question and each context chunk.
f_context_relevance = (
Feedback(provider.context_relevance_with_cot_reasons)
.on_input()
.on(context)
.aggregate(np.mean)
) Ensure that the Additionally, you can use the from trulens_eval.app import App
from trulens_eval.schema import record as mod_record_schema
# Assuming you have an instance of your app and a record
app_instance = ... # Your app instance
record_instance = mod_record_schema.Record(...) # Your record instance
# Check selectors
f_groundedness.check_selectors(app=app_instance, record=record_instance)
f_answer_relevance.check_selectors(app=app_instance, record=record_instance)
f_context_relevance.check_selectors(app=app_instance, record=record_instance) This will help ensure that the selectors are valid and exist in the source data, preventing the Refer to the |
Solved by #1187 (merged to main). You can install it from github:
|
Let me know if you have any issues with this! |
Bug Description
Whenever I try to run the langchain quick start, I get the following error: trulens_eval.feedback.feedback.InvalidSelector: Selector record.app.first.steps__.context.first.get_relevant_documents.rets does not exist in source data.
The RAG itself is perfectly working fine and I get the UI and I can see the evaluation of the relevance metric only. While nothing for context relevance or groundedness. Also in the UI I can see the retrieved context
To Reproduce
Running the code without any single change and I still get this error.
Expected behavior
Expecting to have groundedness and qs / context relevance results.
Relevant Logs/Tracebacks
Exception in thread Thread-6 (_future_target_wrapper):
Traceback (most recent call last):
File "C:\Users\mohkh\AppData\Local\Programs\Python\Python311\Lib\site-packages\trulens_eval\feedback\feedback.py", line 1071, in _extract_selection
arg_vals[k] = list(q.get(source_data))
^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\mohkh\AppData\Local\Programs\Python\Python311\Lib\site-packages\trulens_eval\utils\serial.py", line 1034, in get
for start_selection in start_items:
File "C:\Users\mohkh\AppData\Local\Programs\Python\Python311\Lib\site-packages\trulens_eval\utils\serial.py", line 1035, in get
for last_selection in last_step.get(start_selection):
File "C:\Users\mohkh\AppData\Local\Programs\Python\Python311\Lib\site-packages\trulens_eval\utils\serial.py", line 446, in get
raise KeyError(
KeyError: 'Key not in dictionary: get_relevant_documents'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "C:\Users\mohkh\AppData\Local\Programs\Python\Python311\Lib\threading.py", line 1045, in _bootstrap_inner
self.run()
File "C:\Users\mohkh\AppData\Local\Programs\Python\Python311\Lib\site-packages\ipykernel\ipkernel.py", line 761, in run_closure
_threading_Thread_run(self)
File "C:\Users\mohkh\AppData\Local\Programs\Python\Python311\Lib\threading.py", line 982, in run
self._target(*self._args, **self._kwargs)
File "C:\Users\mohkh\AppData\Local\Programs\Python\Python311\Lib\site-packages\trulens_eval\utils\python.py", line 475, in _future_target_wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\mohkh\AppData\Local\Programs\Python\Python311\Lib\site-packages\trulens_eval\app.py", line 614, in _manage_pending_feedback_results
record.wait_for_feedback_results()
File "C:\Users\mohkh\AppData\Local\Programs\Python\Python311\Lib\site-packages\trulens_eval\schema\record.py", line 186, in wait_for_feedback_results
feedback_result = future_result.result()
^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\mohkh\AppData\Local\Programs\Python\Python311\Lib\concurrent\futures_base.py", line 449, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "C:\Users\mohkh\AppData\Local\Programs\Python\Python311\Lib\concurrent\futures_base.py", line 401, in __get_result
raise self._exception
File "C:\Users\mohkh\AppData\Local\Programs\Python\Python311\Lib\concurrent\futures\thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\mohkh\AppData\Local\Programs\Python\Python311\Lib\site-packages\trulens_eval\utils\python.py", line 475, in _future_target_wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\mohkh\AppData\Local\Programs\Python\Python311\Lib\site-packages\trulens_eval\utils\threading.py", line 174, in _run_with_timeout
raise e
File "C:\Users\mohkh\AppData\Local\Programs\Python\Python311\Lib\site-packages\trulens_eval\utils\threading.py", line 159, in _run_with_timeout
res: T = fut.result(timeout=timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\mohkh\AppData\Local\Programs\Python\Python311\Lib\concurrent\futures_base.py", line 456, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "C:\Users\mohkh\AppData\Local\Programs\Python\Python311\Lib\concurrent\futures_base.py", line 401, in __get_result
raise self.exception
File "C:\Users\mohkh\AppData\Local\Programs\Python\Python311\Lib\concurrent\futures\thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\mohkh\AppData\Local\Programs\Python\Python311\Lib\site-packages\trulens_eval\tru.py", line 435, in run_and_call_callback
temp = ffunc.run(app=app, record=record)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\mohkh\AppData\Local\Programs\Python\Python311\Lib\site-packages\trulens_eval\feedback\feedback.py", line 839, in run
raise e
File "C:\Users\mohkh\AppData\Local\Programs\Python\Python311\Lib\site-packages\trulens_eval\feedback\feedback.py", line 825, in run
input_combinations = list(
^^^^^
File "C:\Users\mohkh\AppData\Local\Programs\Python\Python311\Lib\site-packages\trulens_eval\feedback\feedback.py", line 1073, in extract_selection
raise InvalidSelector(
trulens_eval.feedback.feedback.InvalidSelector: Selector record.app.first.steps.context.first.get_relevant_documents.rets does not exist in source data.
Environment:
The text was updated successfully, but these errors were encountered: