HnswDensevector SafeTensor Generator #2515

Panizghi · 2024-06-02T05:06:59Z

Linked issue : castorini/ura-projects#31 (comment)
@17Melissa will provide the flow command below :)

17Melissa · 2024-06-02T05:27:14Z

Setup for NFCorpus Indexing with Safetensors

To efficiently perform NFCorpus indexing using Safetensors, follow this setup workflow:

Download and Unzip Collections
- Begin by downloading the necessary collections and unzipping them. For instance:
  wget https://rgw.cs.uwaterloo.ca/pyserini/data/beir-v1.0.0-bge-base-en-v1.5.tar -P collections/tar xvf collections/beir-v1.0.0-bge-base-en-v1.5.tar -C collections/
Prepare the Environment
- Navigate to the Safetensors directory within the Anserini project
  cd /anserini/src/main/python/safetensors
- Install the required Python packages:
  pip install -r requirements.txt
- Activate the virtual environment
  python3 -m venv venv
  source venv/bin/activate
Convert JSON to Safetensors Format
- Use the provided script to convert JSON files to Safetensors format
  python3 -m json_to_bin
- the script will create the following files in the target directory
  - Saved vectors to ../../../../target/safetensors/vectors/part00_vectors.safetensors
  - Saved docids to ../../../../target/safetensors/docids/part00_docids.safetensors
  - Saved docid_to_idx mapping to ../../../../target/safetensors/docid_to_idx/part00_docid_to_idx.json

Indexing Procedure

To build HNSWSafetensors indexes, use the following sample command:

bin/run.sh io.anserini.index.SafeTensorsIndexCollection \
  -collection JsonDenseVectorCollection \
  -input collections/beir-v1.0.0/bge-base-en-v1.5/nfcorpus  \
  -index indexes/beir-v1.0.0/bge-base-en-v1.5/nfcorpus/ \
  -generator HnswJsonWithSafeTensorsDenseVectorDocumentGenerator \
  -threads 9 -storePositions -storeDocvectors -storeRaw \
  -vectorsDirectory target\safetesnors\vectors \
  -docidsDirectory  target\safetesnors\docids \
  -docidToIdxDirectory  target\safetesnors\docid_to_idx \
>& logs/log.beir-v1.0.0-bge-base-en-v1.5 &

Ensure all paths and parameters are adjusted according to your setup and directory structure.

lintool · 2024-06-02T11:45:00Z

Can you make the safetensors collection go into collections/beir-v1.0.0/bge-base-en-v1.5.safetensors/, alongside the original? So all files should go into collections/beir-v1.0.0/bge-base-en-v1.5.safetensors/nfcorpus/.

We also shouldn't need a new indexer. The indexing command should be similar to https://github.com/castorini/anserini/blob/master/docs/regressions/regressions-beir-v1.0.0-nfcorpus-bge-base-en-v1.5-hnsw.md

e.g.,

bin/run.sh io.anserini.index.IndexHnswDenseVectors \
  -collection JsonDenseVectorCollection \
  -input /path/to/beir-v1.0.0-bge-base-en-v1.5 \
  -generator HnswJsonWithSafeTensorsDenseVectorDocumentGenerator \
  -index indexes/lucene-hnsw.beir-v1.0.0-nfcorpus-bge-base-en-v1.5/ \
  -threads 16 -M 16 -efC 100 -memoryBuffer 65536 -noMerge \
  >& logs/log.beir-v1.0.0-bge-base-en-v1.5 &

With the only exception being a different -generator.

17Melissa · 2024-06-15T15:44:11Z

Updated Workflow for Safetensors Conversion and Indexing Process

Create Directory: Create the safetensors folder collections/beir-v1.0.0/bge-base-en-v1.5.safetensors/nfcorpus
Run Conversion Script: Execute the python script json_to_bin.py from the root directory using the command:
python src/main/python/safetensors/json_to_bin.py
Execute Indexing Command: Following the indexing command below, which you will run after the conversion script completes

bin/run.sh io.anserini.index.IndexHnswDenseVectors -collection JsonDenseVectorCollection -input collections/beir-v1.0.0/bge-base-en-v1.5/nfcorpus  -generator HnswJsonWithSafeTensorsDenseVectorDocumentGenerator -index indexes/beir-v1.0.0/bge-base-en-v1.5/nfcorpus/ -threads 16 -M 16 -efC 100 -memoryBuffer 65536 -noMerge >& logs/log.beir-v1.0.0-nq.bge-base-en-v1.5 &

...in/java/io/anserini/index/generator/HnswJsonWithSafeTensorsDenseVectorDocumentGenerator.java

src/main/python/safetensors/json_to_bin.py

Panizghi · 2024-07-09T04:43:49Z

Updates

Removed hardcoded path.
Removed indexer arguments and updated the path hierarchy.
Internal mapping of the docid and vectors.
Updated argument for Python input and output.

Updated commands

Python

python src/main/python/safetensors/json_to_bin.py --input collections/beir-v1.0.0/bge-base-en-v1.5/nfcorpus/vectors.part00.jsonl --output collections/beir-v1.0.0/bge-base-en-v1.5.safetensors/nfcorpus

Java

bin/run.sh io.anserini.index.IndexHnswDenseVectors -collection JsonDenseVectorCollection -input collections/beir-v1.0.0/bge-base-en-v1.5/nfcorpus -generator HnswJsonWithSafeTensorsDenseVectorDocumentGenerator -index indexes/beir-v1.0.0/bge-base-en-v1.5/nfcorpus/ -threads 16 -M 16 -efC 100 -memoryBuffer 65536 -noMerge >& logs/log.beir-v1.0.0-nq.bge-base-en-v1.112 &

Melissa

Clean fix branch

lintool · 2024-07-09T11:27:53Z

Looking at this command:

bin/run.sh io.anserini.index.SafeTensorsIndexCollection \
  -collection JsonDenseVectorCollection \
  -input collections/beir-v1.0.0/bge-base-en-v1.5/nfcorpus  \
  -index indexes/beir-v1.0.0/bge-base-en-v1.5/nfcorpus/ \
  -generator HnswJsonWithSafeTensorsDenseVectorDocumentGenerator \
  -threads 9 -storePositions -storeDocvectors -storeRaw \
  -vectorsDirectory target\safetesnors\vectors \
  -docidsDirectory  target\safetesnors\docids \
  -docidToIdxDirectory  target\safetesnors\docid_to_idx \
>& logs/log.beir-v1.0.0-bge-base-en-v1.5 &

What are these three options doing?

  -vectorsDirectory target\safetesnors\vectors \
  -docidsDirectory  target\safetesnors\docids \
  -docidToIdxDirectory  target\safetesnors\docid_to_idx \

And why are these the same?

  -input collections/beir-v1.0.0/bge-base-en-v1.5/nfcorpus  \
  -index indexes/beir-v1.0.0/bge-base-en-v1.5/nfcorpus/ \

I would expect -index to specify the location of the index?

Panizghi · 2024-07-09T13:32:50Z

I think you are looking at the older command this is the updated one

bin/run.sh io.anserini.index.IndexHnswDenseVectors  \
-collection JsonDenseVectorCollection \
-input collections/beir-v1.0.0/bge-base-en-v1.5/nfcorpus \
-generator HnswJsonWithSafeTensorsDenseVectorDocumentGenerator \ 
-index indexes/beir-v1.0.0/bge-base-en-v1.5/nfcorpus/ 
-threads 16 -M 16 -efC 100 -memoryBuffer 65536 -noMerge >& logs/log.beir-v1.0.0-nq.bge-base-en-v1.112 &

lintool · 2024-07-09T13:35:56Z

I think you are looking at the older command this is the updated one

bin/run.sh io.anserini.index.IndexHnswDenseVectors  \
-collection JsonDenseVectorCollection \
-input collections/beir-v1.0.0/bge-base-en-v1.5/nfcorpus \
-generator HnswJsonWithSafeTensorsDenseVectorDocumentGenerator \ 
-index indexes/beir-v1.0.0/bge-base-en-v1.5/nfcorpus/ 
-threads 16 -M 16 -efC 100 -memoryBuffer 65536 -noMerge >& logs/log.beir-v1.0.0-nq.bge-base-en-v1.112 &

Ah, please update to keep up to date?

Panizghi · 2024-07-09T13:49:19Z

My apologies it got lost within all the commits : )
is right here #2515 (comment)

Python

python src/main/python/safetensors/json_to_bin.py 
--input collections/beir-v1.0.0/bge-base-en-v1.5/nfcorpus/vectors.part00.jsonl 
--output collections/beir-v1.0.0/bge-base-en-v1.5.safetensors/nfcorpus

Java

bin/run.sh io.anserini.index.IndexHnswDenseVectors 
-collection JsonDenseVectorCollection 
-input collections/beir-v1.0.0/bge-base-en-v1.5/nfcorpus 
-generator HnswJsonWithSafeTensorsDenseVectorDocumentGenerator 
-index indexes/beir-v1.0.0/bge-base-en-v1.5/nfcorpus/ 
-threads 16 -M 16 -efC 100 -memoryBuffer 65536 -noMerge >& logs/log.beir-v1.0.0-nq.bge-base-en-v1.5 &

src/main/java/io/anserini/index/AbstractIndexer.java

src/main/java/io/anserini/index/generator/DenseVectorDocumentGenerator.java

lintool · 2024-07-11T11:46:33Z

Sorry, I'm confused again:

bin/run.sh io.anserini.index.IndexHnswDenseVectors 
-collection JsonDenseVectorCollection 
-input collections/beir-v1.0.0/bge-base-en-v1.5/nfcorpus 
-generator HnswJsonWithSafeTensorsDenseVectorDocumentGenerator 
-index indexes/beir-v1.0.0/bge-base-en-v1.5/nfcorpus/ 
-threads 16 -M 16 -efC 100 -memoryBuffer 65536 -noMerge >& logs/log.beir-v1.0.0-nq.bge-base-en-v1.5 &

Why would -collection be JsonDenseVectorCollection now? Currently, DenseVectorDocumentGenerator reads from Json: https://github.com/castorini/anserini/blob/master/src/main/java/io/anserini/index/generator/DenseVectorDocumentGenerator.java

So we'd have something like SafeTensorsDenseVectorCollection that reasons from SafeTenors?

lintool · 2024-07-11T11:56:30Z

I'm not getting your logic, but I think you need to implement two classes:

SafeTensorsDenseVectorCollection
SafeTensorsDenseVectorDocumentGenerator

And your command would be something like -collection SafeTensorsDenseVectorCollection ... -generator SafeTensorsDenseVectorDocumentGenerator.

And you'd "wire everything together".

Add to onboarding reproduction logs (castorini#2546)

Panizghi · 2024-07-15T07:19:44Z

updated command :

bin/run.sh io.anserini.index.IndexHnswDenseVectors 
-collection SafeTensorsDenseVectorCollection 
-input collections/beir-v1.0.0/bge-base-en-v1.5.safetensors/nfcorpus   
-generator SafeTensorsDenseVectorDocumentGenerator 
-index indexes/beir-v1.0.0/bge-base-en-v1.5/nfcorpus/ 
-threads 16 -M 16 -efC 100 -memoryBuffer 65536 -noMerge  >& logs/log.beir-v1.0.0-nq.bge-base-en-v1.5 &

Panizghi added 6 commits June 1, 2024 01:33

not issue

85fe2e0

update

e0a7937

test working

b486073

fix paths

b1b4ef0

fix

07f777c

delete extra files

2d84a38

Panizghi mentioned this pull request Jun 2, 2024

Anserini: replace verbose json-based vector format with more compact binary encoding castorini/ura-projects#31

Open

file ext fix

03dfdde

r4meng and others added 3 commits June 11, 2024 00:15

refactor code based on feedback

9e40ec3

python fix

7187238

hardcoded path

bbc8a20

lintool requested changes Jun 15, 2024

View reviewed changes

Panizghi added 2 commits June 16, 2024 15:53

indent fix

008864e

path python fix

23c90b8

Panizghi force-pushed the master branch from 8935d28 to 086ef9c Compare June 16, 2024 22:24

java

59165a6

Panizghi force-pushed the master branch 2 times, most recently from 62cd3c7 to ff75047 Compare July 9, 2024 05:01

Panizghi closed this Jul 9, 2024

Panizghi force-pushed the master branch from 257edd6 to 3885b5c Compare July 9, 2024 05:10

Panizghi added 4 commits July 9, 2024 00:19

Merge pull request #1 from Panizghi/Melissa

699f42e

Melissa

Update HnswJsonWithSafeTensorsDenseVectorDocumentGenerator.java

9a8bbff

Update AbstractIndexer.java

2094e24

Update json_to_bin.py

4868d00

Panizghi reopened this Jul 9, 2024

Delete src/main/java/io/anserini/index/SafeTensorsIndexCollection.java

bca9754

Panizghi and others added 3 commits July 9, 2024 01:47

Fixed paths and updated indexing commands

01a642e

Merge branch 'master' into clean-fix-branch

8403c48

Merge pull request #2 from Panizghi/clean-fix-branch

9ef81c7

Clean fix branch

fix eof

2aa4435

Panizghi added 2 commits July 11, 2024 02:11

revert indexhnsw

af44889

abstarct ndexer update

5e80648

lintool requested changes Jul 11, 2024

View reviewed changes

src/main/java/io/anserini/index/AbstractIndexer.java Outdated Show resolved Hide resolved

src/main/java/io/anserini/index/generator/DenseVectorDocumentGenerator.java Outdated Show resolved Hide resolved

XKTZ and others added 11 commits July 14, 2024 23:37

Add to onboarding reproduction logs (castorini#2546)

507866c

file

7d7ce98

fix eof

7d3726a

abstarct ndexer update

24e9bb8

Merge pull request #3 from castorini/master

5f1d8c8

Add to onboarding reproduction logs (castorini#2546)

test

76cf238

test

ae1044a

test

d1c3a3c

test

f804c6e

test

92987dd

fixed collection and indexer

01b1c51

Panizghi added 2 commits July 16, 2024 23:30

update

1fcd094

added comments

897fac7

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HnswDensevector SafeTensor Generator #2515

HnswDensevector SafeTensor Generator #2515

Panizghi commented Jun 2, 2024 •

edited

Loading

17Melissa commented Jun 2, 2024

lintool commented Jun 2, 2024

17Melissa commented Jun 15, 2024

Panizghi commented Jul 9, 2024

lintool commented Jul 9, 2024

Panizghi commented Jul 9, 2024

lintool commented Jul 9, 2024

Panizghi commented Jul 9, 2024 •

edited

Loading

lintool commented Jul 11, 2024

lintool commented Jul 11, 2024

Panizghi commented Jul 15, 2024

HnswDensevector SafeTensor Generator #2515

Are you sure you want to change the base?

HnswDensevector SafeTensor Generator #2515

Conversation

Panizghi commented Jun 2, 2024 • edited Loading

17Melissa commented Jun 2, 2024

Setup for NFCorpus Indexing with Safetensors

Indexing Procedure

lintool commented Jun 2, 2024

17Melissa commented Jun 15, 2024

Updated Workflow for Safetensors Conversion and Indexing Process

Panizghi commented Jul 9, 2024

Updates

Updated commands

Python

Java

lintool commented Jul 9, 2024

Panizghi commented Jul 9, 2024

lintool commented Jul 9, 2024

Panizghi commented Jul 9, 2024 • edited Loading

Python

Java

lintool commented Jul 11, 2024

lintool commented Jul 11, 2024

Panizghi commented Jul 15, 2024

Panizghi commented Jun 2, 2024 •

edited

Loading

Panizghi commented Jul 9, 2024 •

edited

Loading