-
Notifications
You must be signed in to change notification settings - Fork 1.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Feature Request]: Retrieve A List Of Metadata Or A List Of Collections #2516
Comments
@Alexie-Z-Yevich, thanks for your explanation. Let me ask a clarifying question. If your app is multi-tenant, e.g., each user owns their collection or DB, why do you need to query across all users' knowledgebases? Is that some admin functionality? |
Not all data uploaded by users is queried. Of course, your viewpoint also reminds me of another possibility: the company's managers can upload some public information for reference. |
@Alexie-Z-Yevich, the simplest solution to your problem can be document categories which can be implemented in the following ways: Metadata, single collection per user: each time a user uploads a document they specify what caregory it falls into. You chunk the document (if necessary) and to each chunk, you attach metadata with the corresponding category. You query data using: collection.query(query_texts=[user_query_input], where={"category":user_query_category} The list of those categories you can keep either outside of Chroma in a separate relational DB or within Chroma - 1) have a metadata document that gets updated each time a new category is added or removed (this is hacky, but can work) and 2) Query for all categories each time, which is what you do in right now I believe. Doing Collection per user per category: Each type of document goes into a separate collection. Each collection has metadata associated with the category it represents. You can create collections like so: client.get_or_create_collection(f"{user_id}-{category}", metadata={"category":category}) # make sure to sanitize user_id and category Then, when you present the list of categories to the user: user_categories = [f"{c.metadata['category']}" for c in client.list_collections() if c.name.startswith(f"{user}")] The |
@tazarov It has effectively solved my problem. Thank you very much! ❤❤❤ |
@Alexie-Z-Yevich, glad to be able to help. If you have any further feel free to jump into Discord. We have a great community there and live interactions will make for speedy answers :) |
@tazarov Sorry, I still have a question. Can't multiple collections be searched simultaneously? Although category distinguishes major categories, each user will have many collections, which makes it very difficult to recall if there are two texts in different collections.💦💦 |
I'm sorry I have so many questions🙏🙏🙏 |
@Alexie-Z-Yevich, you can totally do concurrent queries across multiple collections. You can even do them in parallel/simultaneously if you'd like to reduce user latency. I get your point that there can be an overlap between categories, and that does not necessarily need to translate into redundant copies. Let's take my first approach from above - Example: collection.query(query_texts=[user_input_query],where={"medical":True,"game":True} The key to the above query is that for each document, you store all the categories it belongs to as boolean metadata fields (this makes for speedier filters). The above example can also be applied for the multi-collection approach where you can pick to store the document in only one collection, but add metadata with the other category it also matches against then at query time you can add those additional categories as above example. Let me know if the above makes sense in your context. Regarding the admin role question - I guess there is more than one way to achieve this. With the single collection approach, you could add extra categories to search that are only related to corporate documents. For the multi-collection one, you can conversely search either a single collection with corporate docs and filter based on category or have multiple collections (e.g. one per major category) of corporate docs that you search in parallel to the user-selected collections. |
@tazarov I don't know if I understand your meaning correctly. As you can see, whether I use keys or values as labels, metadata returns not a list, but the key I last stored. chroma_client = chromadb.PersistentClient(path="../vector_store/admin/")
collections = chroma_client.get_or_create_collection(user_name)
print(collections.get())
sources = collections.metadata # or collections.metadata['A1']
print(sources) |
Describe the problem
May I ask if there is a function in Chroma that can retrieve a list of metadata or a list of collections.
I encountered an issue when developing a large language model knowledge base using ChromaDB in multi-user mode, where I couldn't distinguish between multiple documents in multi-user mode. I initially used a solution where one user has one ChromaDB and one document uses one collection, but I found that there seems to be no relevant interface implementation for querying multiple collections together. Afterwards, a single ChromaDB was used for all users, with each user divided by a collection and different documents distinguished by metadata under the collection. The problem encountered was that metadata had to be set as key value pairs, and there was no direct function that could return a list of metadata to me.
Because I am a newcomer who has just started practical development and only studied in school before, I cannot be sure if my requirements follow the design team's design ideas for Chroma. My current solution is to store documents in metadata as {'source ': file_name} and use set ([metadata ['source'] for metadata in collections. get() ['mata s']) to obtain a list of documents.
But doing so would require using get() to request all the data each time, which would be very costly. Therefore, I hope to receive some advice from the Chroma team: does Chroma already have a mechanism that meets my needs; If not, will we consider resolving this issue in the future?
Describe the proposed solution
Add an interface for one click retrieval of lists related to collections or metadata, or provide examples of how different users can choose to use certain documents in Chroma.
Alternatives considered
No response
Importance
would make my life easier
Additional Information
No response
The text was updated successfully, but these errors were encountered: