Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[DRAFT] lamini provider and endpoint improvements #970

Draft
wants to merge 500 commits into
base: main
Choose a base branch
from
Draft

[DRAFT] lamini provider and endpoint improvements #970

wants to merge 500 commits into from

Conversation

piotrm0
Copy link
Contributor

@piotrm0 piotrm0 commented Mar 8, 2024 •
edited
Loading

  • Added WithOutputType mixin for LLMProviders to indicate their completion generation can accept output_type.
  • Added Lamini provider with the above mixin.
  • Used output_type in said provider to force completion to generate integer scores.
  • Use output_type to force correct reasons outputs.
  • Added EndpointDelayError to be raised when an endpoint request produces a result suggesting a need to wait. Catching this in Endpoint.run_in_pace to incorporate the suggested delay.
  • Updated trubot example to new pinecone and langchain usage.
  • Added pace argument to Endpoint constructor to be able to set the two pacing parameters instead of just rpm.
  • Added option to trubot to use a specified LLMProvider instead of openai.
  • Added "with_cot_reasons" to feedback.rag_triad to use the COT reasons variant of groundedness.
  • Tests.
  • Example notebooks.

DavidKurokawa and others added 30 commits November 8, 2023 14:57
@DavidKurokawa
Fix "the the" typo. (#551)
1f7535c
@joshreini1
readme-links-fix (#553)
5514d90 
* readme-links-fix

* Update gh_top_intro.md

* Update use_cases_any.md
@github-actions @joshreini1
Automated File Generation from Docs Notebook Changes (#554)
8fb63d2 
Co-authored-by: joshreini1 <joshreini1@users.noreply.github.com>
@joshreini1
Josh/llama moved webpage reader (#557)
2508cea 
* fix import for webpagereader

* webpagereader move

* webpage reader move

* webpagereader move async

* move webpagereader qp

* move wbpr qr
@joshreini1
remove colab_ prefixed notebooks from combine_nb script (#559)
ff40d5a 
* remove colab_ prefixed notebooks from combine_nb script

* Delete trulens_eval/examples/quickstart/colab directory

* remove colab_dependencies.ipynb
@piotrm0 @grit-app @joshreini1
OpenAI v1.x migraation + robust hasattr for instrumentation issues (#555
ddb2841 
)

* [bot] migrate files

* bump openai versions to >=1.1.1

* Update text2text_quickstart.py

* Update keys.py

* fix format check

* first

* fix imports

* format

* remove openai key setting bc it is now instantiated with client

* remove extra import

* more keys migration

* convert pydantic model resposnes to dict

* update endpoint

* key in client for endpoint

* instrumen the client

* moderation response to dict

* response hadnling

* migrate moderation

* remove old key setting in azure

* logger bug

* remove logger

* remove other loggers

* remove dependency on llama service context

* undo embeddings change

* response handling

* more updates

* instrument client.completions instance, debugging

* update to openai 1.x

* Reverting to instrument module

* update versions

* old bug in appui

* don't use safe_* in Lens

* bug fix and dev notes

* dev notes

* more notes

* bug fixes

* more devnotes

* remove extra prints, convert others to logger.info

* remove unneeded instrument_instance

* remove extra client instantiation, openai imports

* client treatment in openai.py

* Fix openai client
Make it a member of openAI endpoint

* fix self-harm moderation

* pin llama_index

---------

Co-authored-by: grit-app[bot] <grit-app[bot]@users.noreply.github.com>
Co-authored-by: Josh Reini <joshreini1@gmail.com>
Co-authored-by: Josh Reini <60949774+joshreini1@users.noreply.github.com>
Co-authored-by: Shayak Sen <shayak@truera.com>
@joshreini1
vertex quickstart (#563)
66a17c6 
* vertex quickstart

* pin versions to b4 openai

* clear outputs

* colab name
@joshreini1
add seed parameter (#560)
555ebad
@shayaks
Release 0.18.0 readiness (#564)
8f06579 
Co-authored-by: Shayak Sen <shayak@truera.com>
@shayaks
Releases/rc trulens eval 0.18.0 (#566)
833d4c9 
* Release 0.18.0 readiness

* Rationalize llama_index langchain

---------

Co-authored-by: Shayak Sen <shayak@truera.com>
@shayaks
Fix colab links (#568)
41d5677 
* Release 0.18.0 readiness

* Rationalize llama_index langchain

* Fix colab links

---------

Co-authored-by: Shayak Sen <shayak@truera.com>
@github-actions @shayaks @joshreini1
Automated File Generation from Docs Notebook Changes (#567)
e7b9fc4 
Co-authored-by: shayaks <shayaks@users.noreply.github.com>
Co-authored-by: Josh Reini <60949774+joshreini1@users.noreply.github.com>
@joshreini1 @piotrm0
@grit-app @shayaks @github-actions
Docs + Examples rewrite (#571)
a50b0b4 
* if groundedness output is not list, set as list so agg functions properly

* fix 0 resolving to null then -1 bug

* new demo flow

* data + eval questions

* jailbreak question

* flow updates

* eval questions

* honest harmless evals

* Add helpful evals

* update demo notebook

* use cases folder for examples

* slim down quickstarts

* move lc async to exp

* move lc retrieval agent to exp

* lc rag example

* core concept images

* add rag triad core concept

* core concepts work

* core concepts ff

* core concepts in docs

* small changes to 3h core concpet

* new quickstart, rewrite readmes

* small changes to reamde

* add new quickstart to tests

* fix use case paths

* use case paths

* OpenAI v1.x migraation + robust hasattr for instrumentation issues (#555)

* [bot] migrate files

* bump openai versions to >=1.1.1

* Update text2text_quickstart.py

* Update keys.py

* fix format check

* first

* fix imports

* format

* remove openai key setting bc it is now instantiated with client

* remove extra import

* more keys migration

* convert pydantic model resposnes to dict

* update endpoint

* key in client for endpoint

* instrumen the client

* moderation response to dict

* response hadnling

* migrate moderation

* remove old key setting in azure

* logger bug

* remove logger

* remove other loggers

* remove dependency on llama service context

* undo embeddings change

* response handling

* more updates

* instrument client.completions instance, debugging

* update to openai 1.x

* Reverting to instrument module

* update versions

* old bug in appui

* don't use safe_* in Lens

* bug fix and dev notes

* dev notes

* more notes

* bug fixes

* more devnotes

* remove extra prints, convert others to logger.info

* remove unneeded instrument_instance

* remove extra client instantiation, openai imports

* client treatment in openai.py

* Fix openai client
Make it a member of openAI endpoint

* fix self-harm moderation

* pin llama_index

---------

Co-authored-by: grit-app[bot] <grit-app[bot]@users.noreply.github.com>
Co-authored-by: Josh Reini <joshreini1@gmail.com>
Co-authored-by: Josh Reini <60949774+joshreini1@users.noreply.github.com>
Co-authored-by: Shayak Sen <shayak@truera.com>

* vertex quickstart (#563)

* vertex quickstart

* pin versions to b4 openai

* clear outputs

* colab name

* add seed parameter (#560)

* Release 0.18.0 retqadiness (#564)

Co-authored-by: Shayak Sen <shayak@truera.com>

* Releases/rc trulens eval 0.18.0 (#566)

* Release 0.18.0 readiness

* Rationalize llama_index langchain

---------

Co-authored-by: Shayak Sen <shayak@truera.com>

* Fix colab links (#568)

* Release 0.18.0 readiness

* Rationalize llama_index langchain

* Fix colab links

---------

Co-authored-by: Shayak Sen <shayak@truera.com>

* Automated File Generation from Docs Notebook Changes (#567)

Co-authored-by: shayaks <shayaks@users.noreply.github.com>
Co-authored-by: Josh Reini <60949774+joshreini1@users.noreply.github.com>

* fix conflict

* fix readme conflicts

* small changes to reamde

* symlink new quickstart

* core concepts

* layout, links

* add missing headers to quickstart

* remove langchain rag quickstart

* hf notebook

* prototype evals

* add prototype and hf to testing

* move dashboard app ui to experimental

* remove extra tru.reset_database

* ground truth evals quickstart

* ground truth evals to testing

* symlink groundtruthevals to docs

* notebook links in core concepts ff

* add link to rag triad core concept

* clean up integration quickstarts

* quickstart small changes

* groundedness collect for quickstarts

* collect for groundness exp examples

---------

Co-authored-by: Piotr Mardziel <piotrm@gmail.com>
Co-authored-by: grit-app[bot] <grit-app[bot]@users.noreply.github.com>
Co-authored-by: Shayak Sen <shayak@truera.com>
Co-authored-by: shayaks <47965099+shayaks@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: shayaks <shayaks@users.noreply.github.com>
@github-actions @joshreini1
Automated File Generation from Docs Notebook Changes (#573)
a82bce6 
Co-authored-by: joshreini1 <joshreini1@users.noreply.github.com>
@joshreini1
add new quickstarts to nav (#574)
df3399e 
* add new quickstarts to nav

* rename t2t quickstart in nav

* text2text format nits, hf colab button

* fix extra stuff in vector store specification
@github-actions @joshreini1
Automated File Generation from Docs Notebook Changes (#576)
d57b0c8 
Co-authored-by: joshreini1 <joshreini1@users.noreply.github.com>
@piotrm0
async updates/bugfixes (#569)
d316ae2 
* clean langchain async

* note

* fix async detection for wrapping
@joshreini1
fix leaderboard display for human feedback (#575)
be622f2
@walnutdust @joshreini1
fix(app selector): Fix missing . in app selectors for the timeline (#579
189db47 
)

Co-authored-by: Josh Reini <60949774+joshreini1@users.noreply.github.com>
@walnutdust @joshreini1
fix: [MLNN-1094] Fix lists in streamlit not being displayed correctly. (
54e3c5e 
#578)



---------

Co-authored-by: Josh Reini <60949774+joshreini1@users.noreply.github.com>
@joshreini1
higher is better key error fix (#581)
099444b
@joshreini1
Iterating on RAGs (#580)
0f23b0e 
* if groundedness output is not list, set as list so agg functions properly

* fix 0 resolving to null then -1 bug

* new demo flow

* data + eval questions

* jailbreak question

* flow updates

* eval questions

* honest harmless evals

* Add helpful evals

* update demo notebook

* split notebooks

* header md

* symlinc iterating rag nbs to docs

* remove errant number

* add notebook links to 3h core concept

* text updates to hhh evals

* make runnable on colab

* remove files no longer needed bc examples are self-contained

* remove dups

* undo spacing change on evaluations.py
@shayaks
Prep for release (#585)
e3094d3 
Co-authored-by: Shayak Sen <shayak@truera.com>
@shayaks
Releases/rc trulens eval 0.18.1 (#587)
1f2f9b6 
* Prep for release

* Fix chromadb issues

---------

Co-authored-by: Shayak Sen <shayak@truera.com>
@shayaks
Releases/rc trulens eval 0.18.1 (#588)
a87b9ba 
* Prep for release

* Fix chromadb issues

* Update test notebooks

---------

Co-authored-by: Shayak Sen <shayak@truera.com>
@shayaks
Releases/rc trulens eval 0.18.1 (#589)
a69d39a 
* Prep for release

* Fix chromadb issues

* Update test notebooks

* fix llama_index_instrumentation.ipynb

---------

Co-authored-by: Shayak Sen <shayak@truera.com>
@joshreini1
Josh/fix readme links (#582)
45520be 
* fix links in readme

* fix readme links
@github-actions @joshreini1
Merge generated docs and test files into main (#591)
88ee2b3 
* Automated File Generation from Docs Notebook Changes

* Update README.md

* Update README.md

fix readme links

* Update README.md

---------

Co-authored-by: joshreini1 <joshreini1@users.noreply.github.com>
Co-authored-by: Josh Reini <60949774+joshreini1@users.noreply.github.com>
@piotrm0 @joshreini1
unpin 2 reqs (#590)
fbf6710 
Co-authored-by: Josh Reini <60949774+joshreini1@users.noreply.github.com>
@piotrm0
serialize openai client (#595)
e33f14b 
* first

* working version, maybe

* note

* fixes

* note

* nit
@piotrm0 piotrm0 changed the title lamini provider lamini provider and endpoint improvements Mar 8, 2024
@piotrm0 piotrm0 changed the title lamini provider and endpoint improvements [DRAFT] lamini provider and endpoint improvements Mar 8, 2024
ingridstevens and others added 27 commits March 8, 2024 06:28
@ingridstevens
Update typo in pip install on azure_openai.ipynb (#973)
9654f2b
@piotrm0 @joshreini1
readd redirects (#972)
dafabc6 
* uncomment redirects and add required plugin to docs_requirements

* move redirects map

* redirects for quickstart links

* redirects for quickstart links

* hack for ipynb old links

* update old cc links

* remove extra cells, add ref to nemo in overview

* add missing tiktoken requirement in lc quickstart

---------

Co-authored-by: Josh Reini <60949774+joshreini1@users.noreply.github.com>
Co-authored-by: Josh Reini <joshreini1@gmail.com>
@github-actions @joshreini1
Automated File Generation from Docs Notebook Changes (#975)
f6984f1 
Co-authored-by: joshreini1 <joshreini1@users.noreply.github.com>
@joshreini1
fix colab link - langchain ensemble notebook (#980)
c0a0cf3 
* fix colab link - langchain ensemble notebook

* fix faiss install

* latest openai
@joshreini1
allow deserialization for faiss example (#978)
1d0c0fc
@joshreini1
fix truchain (#974)
0a2e5b5
@joshreini1
patch (#982)
f77e7ee
@joshreini1
QS Relevance -> Context Relevance (#977)
1c2a646 
* qs relevance -> context relevance

* update quickstart notebooks

* update langchain quickstart

* update missed prompt change
@github-actions @joshreini1
Automated File Generation from Docs Notebook Changes (#983)
15a19f7 
Co-authored-by: joshreini1 <joshreini1@users.noreply.github.com>
@joshreini1
version bump (#981)
b25575e
@joshreini1
existing data quickstart (#976)
40f78bb 
* existing data quickstart

* symlink to docs

* mkdocs

* nb title
@ingridstevens
Adds Azure Quickstart for LangChain (#984)
1058a5d 
* Create azure_openai_langchain.ipynb

Adds langchain quickstart for azure

* renames llama index quickstart to differentiate between langchain and llama index.

* updates trulens-eval to 0.25.1
@piotrm0
fix more docs links (#987)
4398e5c 
* fix links

* typo

* more typos
@piotrm0
update (#990)
d92e79f
@piotrm0
verify feedback selectors on recorder init (#961)
5927f66 
* checkpoint verify selectors implementation

* improve warning message for bad selectors

* adding markdown output for selector hints and creating more accurate dummy records to check against

* remove debug notes

* ignore selector check failures beyond rets or 1 deeper than known args.

* fix 2 more docs links
@joshreini1
relax llama version (#985)
759d035
@piotrm0 @joshreini1
Allow VirtualRecords to have multiple calls to the same component. (#988
83223c4 
)

* example and updates to virtual records

* adjust note

* add combinations option to Feedback.aggregate

* disabled selector prechecks on tru virtual

* nit

* add env listing to eval pipeline after testing packages are installed

* use attr instead of str to check combination parameters

* remove unneeded pass

* add verbose to pip list

* add verbose to pip installs in pipeline

* remove alias use in main __init__

* one more

* rename schema in import

* adjust llama index version test

* pydantic issue debugging

* add notes about weird import bug

* added skipped feedback status and used to mark feedbacks skipped due to ifexists

---------

Co-authored-by: Josh Reini <60949774+joshreini1@users.noreply.github.com>
@joshreini1
Fix broken colab links (#994)
76e28ff 
* fix link

* update langchain async colab link

* remove obsolete alpaca notebook

* anthropic quickstart link

* bedrock colab link

* Update pinecone_quickstart.ipynb
@piotrm0 @joshreini1
docs updates/additions (#996)
c5353c4 
* all

* import fixes and proper names standards

* rebuild for annotations

* more doc updates

* more nits

---------

Co-authored-by: Josh Reini <60949774+joshreini1@users.noreply.github.com>
@joshreini1
add install redirect in docs (#995)
bb11c2d 
* add install redirect in docs

* typo
@joshreini1
Update feedback docs (#999)
5c1d2ff 
* update feedback selector docs

* guide updates

* add aggregation to anatomy

* mkdocs

* base provider class docs

* reorganization, copy

* reorganization, etc

* organizational updates
@piotrm0 @joshreini1
doc usage formatting (#1002)
0c6ba29 
* Fixing docs.

* satisfy abc for deprecated db

* indent fix

* few more formatting fixes

* more format fixes

---------

Co-authored-by: Josh Reini <60949774+joshreini1@users.noreply.github.com>
@piotrm0 @joshreini1
Allow Feedback.run with args even if they had selectors specified. (#…
eaa10f2 
…1003)

* All

* update litellm req

---------

Co-authored-by: Josh Reini <60949774+joshreini1@users.noreply.github.com>
@piotrm0
Merge remote-tracking branch 'origin/main' into piotrm/lamini_provider
433c04f
@piotrm0
fixing
2457e18
@piotrm0
fixing
a59e5a8
@piotrm0
class for storing prompt and data types in one
fc153a2
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
dependencies Pull requests that update a dependency file size:XXL This PR changes 1000+ lines, ignoring generated files.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.