[ENH] Get vectors orchestrator #2348
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Reviewer ChecklistPlease leverage this checklist to ensure your code review is thorough before approving Testing, Bugs, Errors, Logs, Documentation
System Compatibility
Quality
|
HammadB
changed the title
HammadB
commented
Jun 16, 2024
HammadB
commented
Jun 16, 2024
HammadB
commented
Jun 16, 2024
sanketkedia
reviewed
Jun 16, 2024
sanketkedia
reviewed
Jun 16, 2024
sanketkedia
approved these changes
Jun 17, 2024
Anush008
pushed a commit
to Anush008/chroma
that referenced
this pull request
Jun 27, 2024
This PR adds the Get Vectors() RPC for query nodes. 1. Adds the GetVectors instrumented rpc call and associated types 2. Adds a GetVectorsOrchestrator for managing these queries 3. Adds the GetVectorsOperator for reading from the log and record segment to respond to get_vectors 4. Moves common orchestrator functions into a /common and refactors hnsw orchestrator to use this instead (DRY) 5. Adds the `ChromaError` Trait for ChannelError in Sender - needed to use that as Box<dyn ChromaError> elsewhere. I explicitly make the choice to not have to materialize in this operator since it doesn't need to for any reason. Tasks - [x] Stub out - [x] Types - [x] Implement return - [x] Server input parse - [x] Server output parse - [x] Init segments - [x] Pull log + filter - [x] Operator for reading - [x] Add test (Will handle in separate PR) - [x] Validate that the get() semantics for missing ids is correct (I have validated that the semantics are the same - we just ignore invalid entries in the output)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds the Get Vectors() RPC for query nodes.
ChromaErrorTrait for ChannelError in Sender - needed to use that as Box elsewhere.I explicitly make the choice to not have to materialize in this operator since it doesn't need to for any reason.
Tasks