Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support grouping of vector data in HNSW #3227

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

heemin32
Copy link

@heemin32 heemin32 commented Jan 30, 2024

Add IDGrouper in SearchParameter so that it can be used to dedupe search result based on its group id.

Issue: #3087

@heemin32
Copy link
Author

heemin32 commented Feb 7, 2024

@mdouze , would you mind taking a look?

@mdouze
Copy link
Contributor

mdouze commented Feb 9, 2024

Sorry for being unresponsive.
The reason I'm reluctant with this solution is that IMO the new ResultHandler API makes it possible to move the whole de-duplication logic outside of Faiss. In the calling code, just set up a ResultHandler class that does the deduplication and call hnsw.search on that.

@heemin32
Copy link
Author

heemin32 commented Feb 9, 2024

I search HNSW through IndexIDMap. Do you have a plan to expand the ResultHandler to IndexIDMap? I need a way to access id_map data to dedupe result properly.

@mdouze
Copy link
Contributor

mdouze commented Jul 8, 2024

I still have to write an example with a custom ResultHandler.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants