-
Notifications
You must be signed in to change notification settings - Fork 19
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Feature Request] Ability to search and index documents with other metadata #9
Comments
Hello, dear metalhead :) I want to add most of the features you mentioned in the next few months. Currently, I am working on adding ANN search, KNN search, and semantic re-ranking. Search only works as you described (OR) for now. I will notify you when new features are released. |
Hi @regstuff, I added several new features in |
Hey @AmenRa what's the status on filtering by metadata? I would have used this library in at least 2-3 more papers over the past few months if it had:
|
Hi, sorry for the delay. |
Honestly, even just being able to pass in a subset of indexes and getting them scored would be sufficient... i can do the date filtering and metadata filtering on my side. Instead of now, where I would potentially need to score the whole index in relation to a query term and then filter afterwards, which is wasteful/unworkable... |
I have an implementation for this. I'm not 100% familiar with numba and numba's approach to typing so there may be a few errors there, but it works for the toy example:
prints:
|
Hi, yes that's one way to do it. |
would you like me to open a PR? |
I truly appreciate your help and feedback, but I am building a separate searcher with metadata and doc ids filtering as I prefer to keep it separate from the SparseRetriever for the moment. |
Hi, I released an experimental retriever with filtering functionalities. |
great!! thanks so much man, I'll check it out. Actually just needed this. Ahhh i see it only supports SparseRetriever right now. I will take a look |
I took a look, i haven't fully parsed what you're doing in I can see why you didn't do the After a ton of headbanging and reading import faiss
from retriv.dense_retriever.dense_retriever import DenseRetriever
import random
# load index
dr = DenseRetriever.load(index_name='full-index__2023-08-17') ## pretrained index
encoded_query = dr.encoder('test') ## test query
encoded_query = encoded_query.reshape(1, len(encoded_query))
## baseline_query
res_0 = dr.ann_searcher.faiss_index.search(encoded_query, 100) ## results when we don't pass in a subset of the ids
## subset
ids_list = [
527938, 447029, 144546, 657363, 278523, 168930, 210273, 643925,
668591, 113593, 234311, 893935, 233993, 421327, 749351, 282167,
531984, 234300, 31035, 571879, 29422, 29645, 103771, 791342,
234419, 560528, 838081
]
## randomly sort them
ids_list = sorted(ids_list, key=lambda x: random.random()) ## shuffle the ids in the `subset` that we wish to restrict the search to. let's see if faiss search maintains the order of the desired subset
## next three lines are the recipe for how to filter faiss_index to a subset of ids
sel = faiss.IDSelectorBatch(ids_list)
search_params = faiss.SearchParametersHNSW(sel=sel, efSearch=526)
res_1 = dr.ann_searcher.faiss_index.search(encoded_query, 100, params=search_params)
## randomly sort them again
ids_list = sorted(ids_list, key=lambda x: random.random()) ## shuffle the ids in the `subset` that we wish to restrict the search to.
sel = faiss.IDSelectorBatch(ids_list) # this `id_list` has been shuffled, but we want to test if the faiss ordering is the same.
search_params = faiss.SearchParametersHNSW(sel=sel, efSearch=526)
res_2 = dr.ann_searcher.faiss_index.search(encoded_query, 100, params=search_params)
## sanity check tests
## 1️⃣1️⃣1️⃣1️⃣ test 1: does subsetted FAISS return all the ids in our subset
scores, doc_ids = res_1
doc_ids = list(filter(lambda x: x != -1, doc_ids[0]))
print(sorted(doc_ids) == sorted(ids_list))
## 2️⃣2️⃣2️⃣2️⃣ test 2: are the two shuffled faiss lists equal to each other?
print((res_1[1] == res_2[1]).all())
## print metadata about our index.
print(open('~/.retriv/collections/full-index__2023-08-17/faiss_index_infos.json').read()) This returns:
Note the This also changes for different kinds of indexes. For example, if we force from autofaiss import build_index
import numpy as np
## force the index to be not HNSW
embeddings = np.float32(np.random.rand(10000, 512))
index, index_infos = build_index(embeddings, save_on_disk=False, should_be_memory_mappable=True)
## query
q = np.random.rand(1, 512)
full_search_res = index.search(q, k=10)
subset_ids = full_search_res[1][0][3:5].tolist()
# this is how we restrict search to a subset of IDs when `autofaiss` uses the IVF index.
subset_res = index.search(
q,
k=10,
params=faiss.SearchParametersIVF(sel=faiss.IDSelectorBatch(subset_ids), nprobe=512)
)
# sanity check
### 1️⃣1️⃣1️⃣1️⃣ test 1: does FAISS return search over the all of the correct ids?
print(subset_ids == list(filter(lambda x: x != -1 , subset_res[1][0])))
## here is info about our index
print(index_infos) This returns:
So I don't know enough about these indices and how Anyway, I'm happy to do a PR if this helps. |
@alex2awesome can you add comments to your code, please? I do not understand what you are doing / trying to achieve. |
ok i updated... All I was doing was showing a recipe for how to use the The code I pasted here is showing how to do it, and running some sanity checks. |
Hi,
Nice choice with War Pigs in the example. :)
Been looking for a pure-python based search engine, ever since Whoosh stopped being actively developed.
Realize this library is just getting started out, but was wondering if it is possible to add the ability to search and filter by metadata as well.
For example
I might want to search for all lines where the album has the word "pigs" for eg.
Also, is the search OR by default, as in find ANY of the words in the query. Can we search with AND and other BOOLEAN operators, as well as proximity and phrase search? Lucene has these features.
Any plan to combine knn search with the text search?
The text was updated successfully, but these errors were encountered: