Add evaluation section to LLMOps guide #2614
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…-section-rag-guide
…-section-rag-guide
|
Important Auto Review SkippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the You can disable this status message by setting the Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media? TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (invoked as PR comments)
Additionally, you can add CodeRabbit Configration File (
|
…ml-io/zenml into doc/add-evaluation-section-rag-guide
htahir1
requested changes
Apr 16, 2024
|
Images automagically compressed by Calibre's image-actions ✨ Compression reduced images by 22.7%, saving 77.76 KB.
255 images did not require optimisation. Update required: Update image-actions configuration to the latest version before 1/1/21. See README for instructions. |
…ml-io/zenml into doc/add-evaluation-section-rag-guide
|
Images automagically compressed by Calibre's image-actions ✨ Compression reduced images by 2%, saving 5.09 KB.
253 images did not require optimisation. Update required: Update image-actions configuration to the latest version before 1/1/21. See README for instructions. |
…ml-io/zenml into doc/add-evaluation-section-rag-guide
htahir1
approved these changes
Apr 17, 2024
|
Images automagically compressed by Calibre's image-actions ✨ Compression reduced images by 32.5%, saving 29.64 KB.
254 images did not require optimisation. Update required: Update image-actions configuration to the latest version before 1/1/21. See README for instructions. |
strickvl
added a commit
that referenced
this pull request
Apr 17, 2024
* fix missing word in docs * add eval section structure and first texts * more retrieval eval * complete retrieval eval section * formatting and adding code example * add motivation section for eval * generation eval updates * add automated eval section * finalise generation eval guide section * add final code example reference * generation final additions * add eval in practice section (and small addition to generation docs) * add eval in practice section * update TOCs * fix toc * add missing table of contents back * fix locations of files * apply evaluation edits * add extra 'next steps' section for retrieval * convert and use table * try html wrapping * try another wrapping option * and another wrapping option * try the simple route * simple option part 2 * final wrapping * add image of embeddings questions dataset * update language on external tools * Optimised images with calibre/image-actions * add dummy illustrations for zuri * Optimised images with calibre/image-actions * add eval in 65 LOC implementation and cleanup * update TOC * Optimised images with calibre/image-actions * add instructions on cloning and running the pipeline * update embedding viz images * Optimised images with calibre/image-actions * update embeddings viz text in rag guide * add 65 loc eval guide * add link for jaccard * switch order * update embeddings code * add link * update embeddings code in docs * replace supabase image with uptodate one * wrap inference command * update inference code * Optimised images with calibre/image-actions * update further inference code * evaluation small edits * small retrieval doc edits * add table for basic examples * add links and small fixes * replace images * Optimised images with calibre/image-actions * fix diagrams * smaller diagrams * Optimised images with calibre/image-actions --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> (cherry picked from commit a619b82)
strickvl
added a commit
that referenced
this pull request
Apr 17, 2024
* fix missing word in docs * add eval section structure and first texts * more retrieval eval * complete retrieval eval section * formatting and adding code example * add motivation section for eval * generation eval updates * add automated eval section * finalise generation eval guide section * add final code example reference * generation final additions * add eval in practice section (and small addition to generation docs) * add eval in practice section * update TOCs * fix toc * add missing table of contents back * fix locations of files * apply evaluation edits * add extra 'next steps' section for retrieval * convert and use table * try html wrapping * try another wrapping option * and another wrapping option * try the simple route * simple option part 2 * final wrapping * add image of embeddings questions dataset * update language on external tools * Optimised images with calibre/image-actions * add dummy illustrations for zuri * Optimised images with calibre/image-actions * add eval in 65 LOC implementation and cleanup * update TOC * Optimised images with calibre/image-actions * add instructions on cloning and running the pipeline * update embedding viz images * Optimised images with calibre/image-actions * update embeddings viz text in rag guide * add 65 loc eval guide * add link for jaccard * switch order * update embeddings code * add link * update embeddings code in docs * replace supabase image with uptodate one * wrap inference command * update inference code * Optimised images with calibre/image-actions * update further inference code * evaluation small edits * small retrieval doc edits * add table for basic examples * add links and small fixes * replace images * Optimised images with calibre/image-actions * fix diagrams * smaller diagrams * Optimised images with calibre/image-actions --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> (cherry picked from commit a619b82)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This pull request adds a new section to the documentation that explains how to evaluate the performance of the RAG system in practice. It covers topics such as separating evaluation from the main pipeline, using a local LLM judge for faster iteration, the limitations of automated evaluation, and determining the frequency and depth of evaluation.
Gitbook preview link