Releases: truera/trulens
TruLens-Eval v0.33.0
What's Changed
- timeouts for wait_for_feedback_results by @sfc-gh-pmardziel in #1267
- TruLens Streamlit components by @sfc-gh-jreini in #1224
- Run the dashboard on an unused port by default by @sfc-gh-jreini in #1280 and @sfc-gh-jreini in #1275
Documentation Updates
- Reflect Snowflake SQLAlchemy Release in "Connect to Snowflake" Docs by @sfc-gh-jreini in #1281
- Update guardrails examples by @sfc-gh-jreini in #1275
Bug Fixes
- Remove duplicated tests by @sfc-gh-dkurokawa in #1283
- fix LlamaIndex streaming response import by @sfc-gh-chu in #1276
Full Changelog: trulens-eval-0.32.0...trulens-eval-0.33.0
TruLens-Eval v0.32.0
What's Changed
- Context filtering guardrails by @sfc-gh-jreini in #1192
- Query optimizations for TruLens dashboard resulting in 4-32x benchmarked speedups by @sfc-gh-chu in #1216
- Logging in Snowflake database by @sfc-gh-chu in #1216
- Snowflake Cortex feedback provider by @sfc-gh-dhuang in #1202
- improve langchain prompting using native messages by @nicoloboschi in #1194
- fix groundedness with no supporting evidence by @nicoloboschi in #1193
- Improve Microsecond support by @sfc-gh-gtokernliang in #1195
- SkipEval exception by @sfc-gh-pmardziel in #1200
- Update pull_request_template.md by @sfc-gh-jreini in #1234
- Use rounding instead of flooring in feedback score extraction by @sfc-gh-dhuang in #1244
Documentation
- Benchmarking Snowflake arctic-instruct feedback function of groundedness by @sfc-gh-dhuang in #1185
- Evaluation Benchmarks Page by @sfc-gh-jreini in #1190
- Documentation for snowflake sqlalchemy implementation by @sfc-gh-chu in #1216*
- Documentation for logging in snowflake database by @sfc-gh-chu in #1216
- Documentation for cortex provider by @sfc-gh-dhuang in #1202
Examples
- Context filtering guardrails added to quickstarts by @sfc-gh-jreini in #1192
- Update Arctic model notebook to use new Cortex provider by @sfc-gh-dhuang in #1202
- New example showing cortex finetuning by @sfc-gh-dhuang in #1202
- show how to add cost/latency/usage details in virtual records by @sfc-gh-jreini in #1197
Bug Fixes
- Enable formatting during PR build. Also format code that wasn't formatted. by @sfc-gh-dkurokawa in #1212
- Fix test cases generation - normalization step for SummEval score by @sfc-gh-dhuang in #1217
- Enable regex to extract floats in score generation by @sfc-gh-dhuang in #1223
- Fix cost tracking in OpenAI and LiteLLM endpoints by @sfc-gh-dhuang in #1228
- remove deprecated legacy caching by @sfc-gh-jreini in #1233
- Remove remaining streamlit legacy caching by @JushBJJ in #1246
trulens-eval-0.31.0
What's Changed
- Parallelize groundedness LLM calls for speedup by @sfc-gh-dhuang in #1180
- Option for quieter deferred evaluation by @epinzur in #1178
- Support for langchain >=0.2.x retrievers via instrumenting the
invoke
method by @nicoloboschi in #1187
Examples
- ❄️ Snowflake Arctic quickstart by @joshreini1 in #1156
Bug fixes
- Fix a few more old groundedness references + llamaindex agent toolspec import by @daniel-huang-1230 in #1161
- Very minor fix of print statement by @sfc-gh-dhuang in #1173
- Fix sidebar logo formatting by @sfc-gh-chu in https://github.com/truera/trulens/pull/1169\
- [bugfix] prevent stack overflow in jsonify by @piotrm0 in #1176
Full Changelog: trulens-eval-0.30.1...trulens-eval-0.31.0
trulens-eval-0.30.1
What's Changed
- update comprehensiveness by @daniel-huang-1230 and @joshreini1 in #1064
- glossary additions by @piotrm0 in #1144
Bug Fixes
- Add langchain-community to optional requirements by @joshreini1 in #1146
- Checks for use of openai endpoint by @piotrm0 in #1154
Full Changelog: trulens-eval-0.29.0...trulens-eval-0.30.1
TruLens Eval v0.29.0
Breaking Changes
In this release, we re-aligned the groundedness feedback function with other LLM-based feedback functions. It's now faster and easier to define a groundedness feedback function, and can be done with a standard LLM provider rather than importing groundedness on its own. In addition, the custom groundedness aggregation required is now done by default.
Before:
from trulens_eval.feedback.provider.openai import OpenAI
from trulens_eval.feedback import Groundedness
provider = OpenAI() # or any other LLM-based provider
grounded = Groundedness(groundedness_provider=provider)
f_groundedness = (
Feedback(grounded.groundedness_measure_with_cot_reasons, name = "Groundedness")
.on(Select.RecordCalls.retrieve.rets.collect())
.on_output()
.aggregate(grounded.grounded_statements_aggregator)
)
After:
provider = OpenAI()
f_groundedness = (
Feedback(provider.groundedness_measure_with_cot_reasons, name = "Groundedness")
.on(Select.RecordCalls.retrieve.rets.collect())
.on_output()
)
This change also applies to the NLI-based groundedness feedback function available from the Huggingface provider.
Before:
from trulens_eval.feedback.provider.openai import Huggingface
from trulens_eval.feedback import Groundedness
from trulens_eval.feedback.provider import Huggingface
huggingface_provider = Huggingface()
grounded = Groundedness(groundedness_provider=huggingface_provider)
f_groundedness = (
Feedback(grounded.groundedness_measure_with_cot_reasons, name = "Groundedness")
.on(Select.RecordCalls.retrieve.rets.collect())
.on_output()
.aggregate(grounded.grounded_statements_aggregator)
)
After:
from trulens_eval.feedback import Feedback
from trulens_eval.feedback.provider.hugs = Huggingface
huggingface_provider = Huggingface()
f_groundedness = (
Feedback(huggingface_provider.groundedness_measure_with_nli, name = "Groundedness")
.on(Select.RecordCalls.retrieve.rets.collect())
.on_output()
)
In addition to the change described above, below you can find the full release description.
What's Changed
- update groundedness prompt by @bpmcgough in #1112
- Default names for rag triad utility by @joshreini1 in #1122
- Unify groundedness interface by @joshreini1 in #1135
Bug Fixes
- Fixed bug with trace view initialization when no feedback functions exist by @walnutdust in #1108
- Remove references to running moderation endpoint on AzureOpenAI by @joshreini1 in #1116
- swap rag utility (qs)relevance by @piotrm0 in #1120
- Fix Link in Readme by @timbmg in #1128
- chore: remove unused code cell by @stokedout in #1113
- trurails: update to getattr by @joshreini1 in #1130
- Fix typo in README.md by @eltociear in #1136
- fix rag triad and awaitable calls by @piotrm0 in #1110
- Remove placeholder feedback for asynchronous responses by @arn-tru in #1127
- Stop iteration streams in openai cost tracking by @piotrm0 in #1138
Examples
- Show OSS models (and tracking) in LiteLLM application by @joshreini1 in #1109
New Contributors
- @stokedout made their first contribution in #1113
- @timbmg made their first contribution in #1128
- @bpmcgough made their first contribution in #1112
- @eltociear made their first contribution in #1136
Full Changelog: trulens-eval-0.28.0...trulens-eval-0.29.0
TruLens Eval v0.28.0
What's Changed
- Meta-eval / feedback functions benchmarking notebooks, ranking-based eval utils, and docs update by @daniel-huang-1230 in #991
- App delete functionality added by @arn-tru in #1061
- Added test coverage to langchain provider by @arn-tru in #1062
- Configurable table prefix by @piotrm0 in #971
- Add example systemd service file by @piotrm0 in #1072
Bug fixes
- Queue fixed for python version lower than 3.9 by @arn-tru in #1066
- Fix test-tru by @piotrm0 in #1070
- Removed broken tests by @arn-tru in #1076
- Fix legacy db missing abstract method by @piotrm0 in #1077
- Release test fixes by @piotrm0 in #1078
- Docs fixes by @piotrm0 in #1075
Examples
- MongoDB Atlas quickstart by @joshreini1 in #1056
- OpenAI Assistants API (quickstart) by @joshreini1 in #1041
Full Changelog: trulens-eval-0.27.2...trulens-eval-0.28.0
trulens-eval-0.27.2
Bug Fix
- add missing pprint import by @joshreini1 in #1054
Full Changelog: trulens-eval-0.27.1...trulens-eval-0.27.2
trulens-eval-0.27.1
What's changed
Documentation updates
- pipelines readme by @piotrm0 in #1030
- docs | standards on proper names by @markdavidmc0 in #997
- docs glossary by @piotrm0 in #1029
- Fix TruLens docs link in hybrid retriever notebook by @daniel-huang-1230 in #1035
- docs README by @joshreini1 in #1034
- docs: fix typo by @nicoloboschi in #1036
- more pipelines docs by @piotrm0 in #1033
- Fix azure docs pipeline by @joshreini1 in #1037
- Docs updates for feedback, instrumentation apis, examples by @joshreini1 in #1032
- Proper names and glossary expansion in docs by @piotrm0 in #1042
Bug fixes
- Import improvements, fix version conflicts by @joshreini1 in #1047
- Fix import and favicon by @arn-tru in #1049
- remove pkg_resources and distutils by @piotrm0 in #1052
- pin streamlit-aggrid version by @piotrm0 in #1043
New Contributors
- @markdavidmc0 made their first contribution in #997
- @nicoloboschi made their first contribution in #1036
Full Changelog: trulens-eval-0.27.0...trulens-eval-0.27.1
trulens-eval-0.27.0
What's Changed
- Python 3.12 support by @joshreini1 in #1012
- Design guidelines for contributors @piotrm0 in #1015
- Pull request template by @piotrm0 in #1021
- Handle utf8 encoding issues in trulens database @arn-tru in #1023
- Split system and user prompts for feedback functions by @joshreini1 in #1018
- Enable Meta and Mistral models on AWS Bedrock by @joshreini1 in #1018
- Added support for Langchain MultiQueryRetriever by @sayedsohan in #1014
- Add Vectara Hallucination Detection Model by @Josephrp in #950
- Parametrize temperature for create chat completion by @daniel-huang-1230 in #1026
Examples
- Update LiteLLM quickstart to show TogetherAI model usage by @joshreini1 in #1018
- Add Claude-3 as a feedback provider example by @joshreini1 in #1018
- Notebook to show evaluation of Langchain MultiQueryRetriever by @sayedsohan in #1014
- Added example to show usage of Vectara Hallucination Detection Model by @Josephrp in #950
New Contributors
- @arn-tru made their first contribution in #1023
- @sayedsohan made their first contribution in #1014
- @Josephrp made their first contribution in #950
Full Changelog: trulens-eval-0.26.0...trulens-eval-0.27.0
trulens-eval-0.26.0
What's Changed
- QS Relevance -> Context Relevance by @joshreini1 in #977
- Verify feedback selectors on recorder init by @piotrm0 in #961
- Relax llama version by @joshreini1 in #985
- Allow VirtualRecords to have multiple calls to the same component. by @piotrm0 in #988
- Allow Feedback.run with args even if they had selectors specified. by @piotrm0 in #1003
Documentation
- update doc-building requirements by @piotrm0 in #990
- docs updates/additions by @piotrm0 in #996
- Update feedback docs by @joshreini1 in #999
- doc usage formatting by @piotrm0 in #1002
Examples
- Existing data quickstart by @joshreini1 in #976
- Adds Azure Quickstart for LangChain by @ingridstevens in #984
Bug Fixes
- fix more docs links by @piotrm0 in #987
- Fix broken colab links by @joshreini1 in #994
Full Changelog: trulens-eval-0.25.1...trulens-eval-0.26.0