langchain[minor],docs[minor]: Add MatryoshkaRetriever
#4458
Conversation
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
dosubot
bot
added
size:L
|
Does this need to be in experimental? |
…to brace/addaptive-retrieval
No, can move --just seemed right at the time but ig it goes both ways. |
bracesproul
changed the title
docs/core_docs/docs/modules/experimental/retrievers/matryoshka_retrieval.mdx
Outdated
Show resolved
Hide resolved
| @@ -0,0 +1,128 @@ | |||
| import { CallbackManagerForRetrieverRun } from "@langchain/core/callbacks/manager"; | |||
There was a problem hiding this comment.
I think this should live in langchain and not community since it's more of an algorithm?
| constructor(fields: MatryoshkaRetrievalFields<Store>) { | ||
| super({ | ||
| ...fields, | ||
| vectorStore: fields.smallStore, |
There was a problem hiding this comment.
Why override this name?
| * | ||
| * The MatryoshkaRetrieval retriever performs two semantic searches. The first uses the smaller | ||
| * store which returns many results, but is less accurate. Then, using the documents returned | ||
| * from the smaller store, the larger store is used to perform a more accurate search over the |
There was a problem hiding this comment.
Could you go into a bit more detail as to how this works? What does "more accurate search" mean?
| query, | ||
| this.largeK, | ||
| (doc: DocumentInterface) => | ||
| smallResultIds.includes(doc.metadata[this.idKey]), |
There was a problem hiding this comment.
This filter will not always be a function will it?
| const smallResultIds = smallResults.map( | ||
| (result) => result.metadata[this.idKey] | ||
| ); | ||
| const largeResults = await this.largeStore.similaritySearch( |
There was a problem hiding this comment.
Wait lmao this is just parent document isn't it?
bracesproul
changed the title
MatryoshkaRetrieval to lc experimental
bracesproul
changed the title
MatryoshkaRetrieval to lc experimentalMatryoshkaRetrieval
bracesproul
added
lgtm
![dosubot[bot]](https://github.com/avatars/in/324583?s=60&v=4){kind=small}
| @@ -3646,6 +3650,15 @@ | |||
| "import": "./retrievers/vespa.js", | |||
There was a problem hiding this comment.
Hey there! I noticed that this PR introduces a new dependency for "./retrievers/matryoshka_retrieval" in the package.json file, potentially impacting the project's dependencies. This comment is to flag the change for maintainers to review the type of dependency change (peer/dev/hard). Keep up the great work!
bracesproul
changed the title
MatryoshkaRetrieval MatryoshkaRetriever
cc @nfcampos