Ports over LongRAGPack, Corrective RAG Pack, and Self-Discover Pack to Workflows #15160

jonathanhliu21 · 2024-08-06T01:32:13Z

Description

Changes the following LlamaPacks to use the new workflow feature, and adds examples under the workflow folder showcasing the usage of these workflows in the context of implementing the LlamaPacks.

LongRAGPack
Corrective RAG
Self-Discover

New Package?

Not a new package.

Version Bump?

Did I bump the version in the pyproject.toml file of the package I am updating? (Except for the llama-index-core package)

Yes
No

Type of Change

Please delete options that are not relevant.

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
This change requires a documentation update

How Has This Been Tested?

Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration

Added new unit/integration tests
Added new notebook (that tests end-to-end)
I stared at the code and made sure it makes sense

Suggested Checklist:

I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have added Google Colab support for the newly added notebooks.
My changes generate no new warnings
I have added tests that prove my fix is effective or that my feature works
New and existing unit tests pass locally with my changes
I ran make format; make lint to appease the lint gods

review-notebook-app · 2024-08-06T01:32:17Z

Check out this pull request on

See visual diffs & provide feedback on Jupyter Notebooks.

Powered by ReviewNB

logan-markewich · 2024-08-06T01:36:35Z

Awesome @jonathanhliu21 !

I'm curious if you had any overall feedback (good or bad) after trying out the workflows syntax? 👀

jonathanhliu21 · 2024-08-06T02:00:15Z

Awesome @jonathanhliu21 !

I'm curious if you had any overall feedback (good or bad) after trying out the workflows syntax? 👀

Overall, I think that structuring workflows is pretty intuitive. In terms of feedback, I think being able to specify which step to run when run() is called at the beginning could be helpful when there are multiple steps that can take in a StartEvent. Additionally, defining event classes was somewhat tedious in the self-discover case, as all of the variables in the event class was the same type. However, I wasn't completely sure if this is the right way to use workflows, so please let me know if there's a cleaner way to define events.

masci · 2024-08-06T09:35:31Z

I think being able to specify which step to run when run() is called at the beginning could be helpful when there are multiple steps that can take in a StartEvent.

That's tricky because ideally the workflow shouldn't depend on the order of the processed events. In other words, if two steps wait for a StartEvent, which one goes first shouldn't matter. If it matters, the step that needs to go after shouldn't probably take a StartEvent.

For context, what you ask is definitely doable with the current asyncio implementation, but might not be the case the moment we offer a different workflows "runtime".

logan-markewich · 2024-08-06T22:07:18Z

llama-index-packs/llama-index-packs-corrective-rag/llama_index/packs/corrective_rag/base.py

-        self, retrieved_nodes: List[Document], query_str: str
-    ) -> List[str]:
+        query_str = ev.get("query_str")
+        retriever_kwargs = ev.get("retriever_kwargs")


might be safer to do retriever_kwargs = ev.get("retriever_kwargs", {})

logan-markewich · 2024-08-06T22:08:20Z

llama-index-packs/llama-index-packs-corrective-rag/llama_index/packs/corrective_rag/base.py


-    def retrieve_nodes(self, query_str: str, **kwargs: Any) -> List[NodeWithScore]:
+    @step(pass_context=True)
+    async def retrieve(self, ctx: Context, ev: StartEvent) -> Optional[RetrieveEvent]:


I guess this event assumes the index is already constructed. Should we raise an error if not? Or make the index / llm / tavilly tool part of the workflow constructor?

let's add a check?

logan-markewich · 2024-08-06T22:10:10Z

llama-index-packs/llama-index-packs-corrective-rag/llama_index/packs/corrective_rag/base.py

+        self._wf = CorrectiveRAGWorkflow()
+
+        asyncio.run(
+            self._wf.run(documents=documents, tavily_ai_apikey=tavily_ai_apikey)


ah I see, we have this in the constructor of the rag pack itself

logan-markewich · 2024-08-06T22:10:59Z

llama-index-packs/llama-index-packs-corrective-rag/llama_index/packs/corrective_rag/base.py

-            return self.get_result(relevant_text, "", query_str)
+    def run(self, query_str: str, **kwargs: Any) -> Any:
+        """Run the pipeline."""
+        return asyncio.run(self._wf.run(query_str=query_str, retriever_kwargs=kwargs))


small nit: slightly safer to use our util instead

from llama_index.core.async_utils import asyncio_run ... return asyncio_run(...)

logan-markewich · 2024-08-06T22:11:24Z

llama-index-packs/llama-index-packs-corrective-rag/pyproject.toml

@@ -29,7 +29,7 @@ license = "MIT"
 maintainers = ["ravi-theja"]
 name = "llama-index-packs-corrective-rag"
 readme = "README.md"
-version = "0.1.1"
+version = "0.1.2"


we should bump the llama-index-core dep to be v0.10.61 as well

logan-markewich · 2024-08-06T22:16:28Z

llama-index-packs/llama-index-packs-self-discover/llama_index/packs/self_discover/base.py

+        """Get modules step."""
+        # get input data, store llm into ctx
+        task = ev.get("task")
+        llm: LLM = ev.get("llm")


For workflows with a single entry point, might be more friendly to confirm these are actually properly passed in, and raise an appropriate error if not

logan-markewich

Looks good to me! Mostly just comments about updating asyncio.run() -> asyncio_run() and bumping the version of the llama-index-core dep

jerryjliu · 2024-08-16T06:58:08Z

llama-index-packs/llama-index-packs-self-discover/llama_index/packs/self_discover/base.py

+        llm: LLM = ev.get("llm")
+        ctx.data["llm"] = llm
+
+        if task is None or llm is None:


nit: should have this check before assigning llm to ctx.data

jerryjliu · 2024-08-16T06:58:35Z

llama-index-packs/llama-index-packs-corrective-rag/llama_index/packs/corrective_rag/base.py

        llm = OpenAI(model="gpt-4")
-        self.relevancy_pipeline = QueryPipeline(
+        ctx.data["relevancy_pipeline"] = QueryPipeline(


this...is probably ok for now until we have hierarchical workflows? @logan-markewich thoughts?

I mean, we could reimplement the pipeline to be a workflow now too

and update with the slightly nicer api when it lands

@logan-markewich unrelated but one of the original motivations for QPs were the syntatic sugar for common chains e.g. (prompt -> LLM -> output parsing)

should we do that for workflows?

jerryjliu · 2024-08-16T06:58:59Z

llama-index-packs/llama-index-packs-corrective-rag/llama_index/packs/corrective_rag/base.py


-    def retrieve_nodes(self, query_str: str, **kwargs: Any) -> List[NodeWithScore]:
+    @step(pass_context=True)
+    async def retrieve(self, ctx: Context, ev: StartEvent) -> Optional[RetrieveEvent]:


let's add a check?

…o Workflows (#15160)

* update to v2 * update deps, fix __modify_schema__ * model_validator instead of root_validator * field_validator over validator * TypeAdapter over parse_obj_as * wip model_dump_json over json * wip * wip * move private vars after super init * move private vars after super init * populate_by_name over allow_population_by_field_name * add str(v) in legacy_json_to_doc of docstore/utils module * add Annotated types * use annotation * transformations passing * pydantic annotations module * more callback fixes * update lock * dict shouldn't use by_alias * add default callables * add default callables * wip llms * annotations for LLMs to add class name in model_dump * add model_validator to LLM * update __pydantic_private__ * make callback manager optional for llms * add field_serialization * add field_serialization * in favor of definitions * fix pydantic validation errors * fix dict interface * pants * try typing_extensions * fix optimum * fix optimum * 15275 feature request make all events accessible like mappings (#15310) * add DictLikeEvent * remove test event * cr * cr * feat: make send_event assign the target step (#15259) * feat: make send_event assign the target step * fix: change exception type and add else block * use __private_attributes__ * bug: Semantic double merging splitter creates chunks larger thank chunk size (#15188) fix bug with semantic double merging splitter creating chunks larger than max_chunk_size Co-authored-by: Konrad Rucińsk <konrad.rucinski@bitpeak.pl> * Update code_hierarchy.py adding php support (#15145) * Update code_hierarchy.py adding php support * Update code_hierarchy.py fixing linting error * Update pyproject.toml bumping version * test core only * revert back integration changes * fix annotations * impl __get_pydantic_core_schema__ and __get_pydantic_json_schema__ for BaseOutputParser * impl __get_pydantic_core_schema__ and __get_pydantic_json_schema__ for CallbackManager * use CallbackManager instead of Annotation * cleanup * remove pydantic_annotations module * remove deprecated BaseConfig from pydantic bridge * chore: bump deprecated actions (#15331) bump deprecated actions * chore: let dependabot monitor Github actions (#15360) let dependabot monitor Github actions * chore(deps): bump github/codeql-action from 2 to 3 (#15369) Bumps [github/codeql-action](https://github.com/github/codeql-action) from 2 to 3. - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](github/codeql-action@v2...v3) --- updated-dependencies: - dependency-name: github/codeql-action dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(deps): bump JRubics/poetry-publish from 1.17 to 2.0 (#15370) * Add citation query engine with workflows (#15372) * Add citation query engine with workflows * Add descriotion, cookbook link, google colab * decouple ingest from workflows * Update description * Add docstring for create_citations_nodes function * GCSReader: Implementing ResourcesReaderMixin and FileSystemReaderMixin (#15365) * fix bug when caling llama-index-postprocessor-dashscope-rerank (#15358) * update HF's completion_to_prompt (#15354) * Feat: Wrapper for SambaNova (Sambaverse and SambaStudio) with Llama-index (#15220) * v0.10.66 (#15403) * Add JSONAlyze Query Engine using workflows cookbook (#15408) * Add JSONAlyze Query Engine * Update with review * Use backticks to escape label (#15324) * update pydantic dep * Fixed issue #15414 and added ability to do partial matchfor Neptune Analytics (#15415) * escape json in structured llm (#15404) * update completion_to_prompt of HF pipeline (#15437) * Fix missing 'self' in example notebook. (#15429) * Add workaround for Handling MySQL Protocol Limitations in TiDB Vector with insert_batch_size (#15428) * Cleanlab's cookbook (#15352) * When building a vector store with use_async=True it doesn't set a TTL on records (#15333) * Vertexai embedding enhancement (#15367) * feat: add tool calling for cohere (#15144) * Add get/set API to the Context and make it coroutine-safe (#15152) * make the Context coroutine-safe * remove parent property * change API * docs * use context manager and add unit tests * Fix LangChainDeprecationWarning (#15397) * Update langchain.py fix LangChainDeprecationWarning of ChatMessageHistory * Fix LangChainDeprecationWarning Handling different versions * Fix LangChainDeprecationWarning fix changes * fix linting * Ports over LongRAGPack, Corrective RAG Pack, and Self-Discover Pack to Workflows (#15160) * settings, secret, and start using SerializeAsAny * wip * serialize as any T * wip, SerializeAsAny * wip, SerializeAsAny * wip, SerializeAsAny * chore: remove leftover occurences of `GradientFinetuneEngine` (#15456) * Exposed NeptuneQueryException and added additional debug information (#15448) * docs: promote openllmetry to one-click integration (#15443) * feat: Integrating Azure OpenAI Finetuning (#15297) * avoid nltk 3.9 since its broken (#15473) * Add GigaChat embedding (#15278) * feat: support to configure the base url of jina.ai (#15431) * Enhance PreprocessReader (#15302) * v0.10.67 (#15474) * wip, SerializeAsAny * GenericModel, ValidationError import, model_rebuild * handle warning of protected prefix on fields * wip ConfigDict * use ollama client for embeddings (#15478) * wip ConfigDict * gte, lte * Bug-15158: Fixing the issues in loading file paths (#15311) * Bug-15158: Fixing the issues in loading file paths * Bump up the version in pyproject.toml * Add llama-index-graph-stores-neo4j dependency in GraphRAG v2 (#15479) Signed-off-by: Hollow Man <hollowman@opensuse.org> * wip model_dump over dict * model_dump over dict * llm_rerank conflict * model_json_dump over dict and model_copy over copy * model_validate over parse_obj * model_validate_json over parse_raw * model_dump_json over json * wip * revert to see if breaking * add back previous changes with fixed BUILD * make default_id_func serializable * remove type param from Field * make CallbackManager not optional * implement model_serializer to override model_dump for nested classes * implement model_serializer to override model_dump for nested classes --------- Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: Hollow Man <hollowman@opensuse.org> Co-authored-by: Lin Chia Cheng <74391293+Tonyrj3268@users.noreply.github.com> Co-authored-by: Konrad Ruciński <46794180+rucinsk1@users.noreply.github.com> Co-authored-by: Konrad Rucińsk <konrad.rucinski@bitpeak.pl> Co-authored-by: Joe Huss <detain@interserver.net> Co-authored-by: Massimiliano Pippi <mpippi@gmail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ravi Theja <ravi03071991@gmail.com> Co-authored-by: Chetan Choudhary <chetanchoudhary975@gmail.com> Co-authored-by: afalf <101537385+afalf@users.noreply.github.com> Co-authored-by: Ethan Yang <ethan.yang@intel.com> Co-authored-by: pradeep-suresh2002 <124234409+pradeep-suresh2002@users.noreply.github.com> Co-authored-by: Logan <logan.markewich@live.com> Co-authored-by: Ian Robinson <iansrobinson@gmail.com> Co-authored-by: bechbd <bechbd@users.noreply.github.com> Co-authored-by: Jerry Liu <jerryjliu98@gmail.com> Co-authored-by: Ali Abbasi <abbasi.ali.tab@gmail.com> Co-authored-by: Ian <ArGregoryIan@gmail.com> Co-authored-by: Ashish Sardana <ashishsardana21@gmail.com> Co-authored-by: mvaquero-atlasti <147490466+mvaquero-atlasti@users.noreply.github.com> Co-authored-by: Sagar Jadhav <108457149+sagarjadhavcalsoft@users.noreply.github.com> Co-authored-by: Anirudh31415926535 <anirudh@cohere.com> Co-authored-by: guodong <songoodong@163.com> Co-authored-by: Jonathan Liu <81734282+jonathanhliu21@users.noreply.github.com> Co-authored-by: Nir Gazit <nirga@users.noreply.github.com> Co-authored-by: Dongwoo Jeong <dongwoo.jeong@lge.com> Co-authored-by: Kirill <58888049+KirillKukharev@users.noreply.github.com> Co-authored-by: Cheese <qizhi.wang@pingcap.com> Co-authored-by: preprocess-co <137915090+preprocess-co@users.noreply.github.com> Co-authored-by: ReviewBuddy <91291176+pitchdarkdata@users.noreply.github.com> Co-authored-by: ℍ𝕠𝕝𝕝𝕠𝕨 𝕄𝕒𝕟 <hollowman@opensuse.org>

jonathanhliu21 added 8 commits August 2, 2024 11:49

change longragpack to use workflows

39ec07d

adds longrag example notebook for workflows

c5b0a5b

use workflows on corrective RAG

3c8398b

Merge branch 'main' into llama-pack-workflow

769740c

adds corrective rag example nb

a753eff

change self discover rag to workflows

9fc6868

adds self discover workflow notebook

5282052

vbump

b2b5e22

dosubot bot added the size:XXL This PR changes 1000+ lines, ignoring generated files. label Aug 6, 2024

jonathanhliu21 added 2 commits August 5, 2024 18:47

change typing

fd7e5c7

change typing from "| None" to optional

51c0d7c

logan-markewich reviewed Aug 6, 2024

View reviewed changes

logan-markewich self-assigned this Aug 6, 2024

jonathanhliu21 added 2 commits August 6, 2024 17:04

replace asyncio.run, make run() friendlier

e56627e

Merge branch 'main' into llama-pack-workflow

a808025

jonathanhliu21 requested a review from logan-markewich August 9, 2024 02:43

jerryjliu approved these changes Aug 16, 2024

View reviewed changes

dosubot bot added the lgtm This PR has been approved by a maintainer label Aug 16, 2024

add checks

6270817

jerryjliu merged commit 71c2cfd into run-llama:main Aug 17, 2024
8 checks passed

nerdai pushed a commit that referenced this pull request Aug 17, 2024

Ports over LongRAGPack, Corrective RAG Pack, and Self-Discover Pack t…

4d8c9ab

…o Workflows (#15160)

nerdai pushed a commit that referenced this pull request Aug 17, 2024

Ports over LongRAGPack, Corrective RAG Pack, and Self-Discover Pack t…

1ce290d

…o Workflows (#15160)

nerdai pushed a commit that referenced this pull request Aug 17, 2024

Ports over LongRAGPack, Corrective RAG Pack, and Self-Discover Pack t…

68fc685

…o Workflows (#15160)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Ports over LongRAGPack, Corrective RAG Pack, and Self-Discover Pack to Workflows #15160

Ports over LongRAGPack, Corrective RAG Pack, and Self-Discover Pack to Workflows #15160

jonathanhliu21 commented Aug 6, 2024

review-notebook-app bot commented Aug 6, 2024

logan-markewich commented Aug 6, 2024

jonathanhliu21 commented Aug 6, 2024

masci commented Aug 6, 2024

logan-markewich Aug 6, 2024

logan-markewich Aug 6, 2024

jerryjliu Aug 16, 2024

logan-markewich Aug 6, 2024

logan-markewich Aug 6, 2024

logan-markewich Aug 6, 2024

logan-markewich Aug 6, 2024

logan-markewich left a comment

jerryjliu Aug 16, 2024

jerryjliu Aug 16, 2024

logan-markewich Aug 16, 2024

logan-markewich Aug 16, 2024

jerryjliu Aug 16, 2024

jerryjliu Aug 16, 2024

Ports over LongRAGPack, Corrective RAG Pack, and Self-Discover Pack to Workflows #15160

Ports over LongRAGPack, Corrective RAG Pack, and Self-Discover Pack to Workflows #15160

Conversation

jonathanhliu21 commented Aug 6, 2024

Description

New Package?

Version Bump?

Type of Change

How Has This Been Tested?

Suggested Checklist:

review-notebook-app bot commented Aug 6, 2024

logan-markewich commented Aug 6, 2024

jonathanhliu21 commented Aug 6, 2024

masci commented Aug 6, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

logan-markewich left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment