Tiny collection of utilities to help you managed ChromaDB indices.
WARNING: These tools rely on internal ChromaDB APIs and may break in the future.
Before you use these tools make sure your ChromaDB persistent dir, on which you intend to run these tools, is backed up.
pip install chromadb-opsGather general information about your persistent Chroma instance. This command is useful to understand what's going on internally in Chroma and to get recommendations or support from the team by providing the output.
chops info /path/to/persist_dirSupported options are:
--skip-collection-names(-s) - to skip specific collections--privacy-mode(-p) - privacy mode hides paths and collection names so that the output can be shared without exposing sensitive information
When sharing larger outputs consider storing the output in a file:
chops info /path/to/persist_dir -p > chroma_info.txtSample output:
General Info
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Property ┃ Value ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ Chroma Version │ 0.5.5 │
│ Number of Collection │ 1 │
│ Persist Directory │ /tmp/tmp9l3ceuvp │
│ Persist Directory Size │ 142.2MiB │
│ SystemDB size: │ 81.6MiB (/tmp/tmp9l3ceuvp/chroma.sqlite3) │
│ Orphan HNSW Directories │ [] │
└─────────────────────────────┴────────────────────────────────────────────────┘
───────────────────────────────── Collections ──────────────────────────────────
───────────────────────────────────── test ─────────────────────────────────────
'test' Collection Data
┏━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Table Data ┃ Value ┃
┡━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ ID │ 9e80e4fd-fd4b-47b8-810c-e8ffa57c1912 │
│ Name │ test │
│ Metadata │ None │
│ Dimension │ 1536 │
│ Tenant │ default_tenant │
│ Database │ default_database │
│ Records │ 10,000 │
│ WAL Entries │ 10,000 │
└────────────────────┴─────────────────────────────────────────────────────────┘
─────────────────────────────────── Segments ───────────────────────────────────
Metadata Segment (test)
┏━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Property ┃ Value ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ Segment ID │ 832fa2cd-6c40-4eee-ad7d-35f260acaaaa │
│ Type │ urn:chroma:segment/metadata/sqlite │
│ Scope │ METADATA │
│ SysDB Max Seq ID │ 10,000 │
└─────────────────────────┴────────────────────────────────────────────────────┘
HNSW Segment (test)
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Property ┃ Value ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ Segment ID │ 13609103-d317-4556-a744-008c96229b72 │
│ Type │ urn:chroma:segment/vector/hnsw-local-persist… │
│ Scope │ VECTOR │
│ Path │ /tmp/tmp9l3ceuvp/13609103-d317-4556-a744-008… │
│ SysDB Max Seq ID │ 0 │
│ HNSW Dir Size │ 60.6MiB │
│ HNSW Metadata Max Seq ID │ 10,000 │
│ HNSW Metadata Total Labels │ 10,000 │
│ WAL Gap │ 0 │
│ HNSW Raw Total Active Labels │ 10,000 │
│ HNSW Raw Allocated Labels │ 10,000 │
│ HNSW Orphan Labels │ set() │
│ Fragmentation Level │ 0.0 │
└──────────────────────────────┴───────────────────────────────────────────────┘- Fragmentation Level - the higher the value the more unnecessary memory and performance hits your HNSW index suffers. It needs to be rebuilt.
- Orphan HNSW Directories - these are directories that are not associated with any collection. They can be safely deleted.
- WAL Entries - high values usually means that you need prune your WAL. Use either this tool or the official Chroma CLI.
- HNSW Orphan Labels - this must always be empty set, if you see anything else report it in Discord.
This command ensures your WAL is committed to binary vector index (HNSW).
chops commit-wal /path/to/persist_dirNote: You can skip certain collections by running
chops commit-wal /path/to/persist_dir --skip <collection_name>
This command cleans up the committed portion of the WAL and VACUUMs the database.
chops clean-wal /path/to/persist_dirThis commands exports the WAL to a jsonl file. The command can be useful in taking backups of the WAL.
chops export-wal /path/to/persist_dir --out /path/to/export.jsonlNote: If --out or -o is not specified the command will print the output to stdout.
This command rebuilds the full-text search index.
Note: Why is this needed? Users have reported broken FTS indices that result in a error of this kind:
no such table: embedding_fulltext_search
chops rebuild-fts /path/to/persist_dirNote: You have to mount your persist directory into the container for the commands to work.
Building the image:
docker build -t chops .docker run -it --rm -v ./persist_dir:/chroma-data ghcr.io/amikos-tech/chromadb-ops/chops:latest commit-wal /chroma-datadocker run -it --rm -v ./persist_dir:/chroma-data ghcr.io/amikos-tech/chromadb-ops/chops:latest clean-wal /chroma-datadocker run -it --rm -v ./persist_dir:/chroma-data -v ./backup:/backup ghcr.io/amikos-tech/chromadb-ops/chops:latest export-wal /chroma-data --out /backup/export.jsonldocker run -it --rm -v ./persist_dir:/chroma-data ghcr.io/amikos-tech/chromadb-ops/chops:latest rebuild-fts /chroma-data