community: retrievers: added capability for using Product Quantization as one of the retriever.

[Vishnunkumar]

• Community: "Retrievers: Product Quantization"

  • This PR adds Product Quantization feature to the retrievers to the Langchain Community. PQ is one of the fastest retrieval methods if the embeddings are rich enough in context due to the concepts of quantization and representation through centroids
  • Description: Adding PQ as one of the retrievers
  • Dependencies: using the package nanopq for this PR
  • Twitter handle: vishnunkumar_

• Add tests and docs: If you're adding a new integration, please include

  • Added unit tests for the same in the retrievers.
  • [] Will add an example notebook subsequently

• Lint and test: Run make format, make lint and make test from the root of the package(s) you've modified. See contribution guidelines for more: https://python.langchain.com/docs/contributing/ - done the same

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
langchain	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Jul 24, 2024 1:52pm

[baskaryan]

could we add a docs page to docs/docs/integrations/retrievers/?

[baskaryan]

could we call this something like NanoPQRetriever to make clear what dependency is being used?

[Vishnunkumar]

Felt like the algorithm's name should be given more importance than the package. Thought similar on the lines of TF-IDF, SVM or KNN. However, let me if package based naming would make sense

[baskaryan]

think since this package is less well known/used than sklearn its more important to make it obvious whats being used

[Vishnunkumar]

Understood, will update on this.

[baskaryan]

could we also add a docs page to /docs/docs/integrations/retrievers

[Vishnunkumar]

could we also add a docs page to /docs/docs/integrations/retrievers

Hi @baskaryan , will be adding the same

[Vishnunkumar]

Hi @baskaryan , hope you are well. I have the following queries.

• I am able to see the following error in the , Feel like this can be resolved only after NanoPq gets merged with the mainline, let me know if there is any misunderstanding from my side.
• Also, any reference to how we can add docs in the API Reference for NanoPQRetreiver.

[Vishnunkumar]

Hi @baskaryan , hope you are well. I have the following queries.

• I am able to see the following error in the , Feel like this can be resolved only after NanoPq gets merged with the mainline, let me know if there is any misunderstanding from my side.
• Also, any reference to how we can add docs in the API Reference for NanoPQRetreiver.

@baskaryan , @ccurme @efriis . Any suggestion on these. Would love to hear and work on this to get this merged.

[ccurme]

Hi @Vishnunkumar, is your local langchain-community installed in your environment? e.g.,

cd /path/to/your/langchain/libs/community/
pip install -e .

[Vishnunkumar]

Hi @Vishnunkumar, is your local langchain-community installed in your environment? e.g.,

cd /path/to/your/langchain/libs/community/
pip install -e .

Hi @ccurme , thanks will look into it and update.

[Vishnunkumar]

Hi @Vishnunkumar, is your local langchain-community installed in your environment? e.g.,

cd /path/to/your/langchain/libs/community/
pip install -e .

Hi @ccurme , thanks will look into it and update.

Hi @ccurme , Have resolved this, let me know of the next steps.