Skip to content

Latest commit

 

History

History
451 lines (387 loc) · 28 KB

fatjar-regressions-v0.36.0.md

File metadata and controls

451 lines (387 loc) · 28 KB
Raw

Anserini Fatjar Regresions (v0.36.0)

Fetch the fatjar:

wget https://repo1.maven.org/maven2/io/anserini/anserini/0.36.0/anserini-0.36.0-fatjar.jar

Note that prebuilt indexes will be downloaded to ~/.cache/pyserini/indexes/. Currently, this path is hard-coded (see Anserini #2322). If you want to change the download location, the current workaround is to use symlinks, i.e., symlink ~/.cache/pyserini/indexes/ to the actual path you desire.

Let's start out by setting the ANSERINI_JAR and the OUTPUT_DIR:

export ANSERINI_JAR="anserini-0.36.0-fatjar.jar"
export OUTPUT_DIR="."

TREC 2024 RAG

❗ Beware, you need lots of space to run these experiments. The msmarco-v2.1-doc prebuilt index is 63 GB uncompressed. The msmarco-v2.1-doc-segmented prebuilt index is 84 GB uncompressed. Both indexes will be downloaded automatically.

Here are the instructions for reproducing runs on the MS MARCO V2.1 document corpus with prebuilt indexes (adjust number of threads based on available resources):

TOPICS=(msmarco-v2-doc-dev msmarco-v2-doc-dev2 trec2021-dl trec2022-dl trec2023-dl rag24-raggy-dev); for t in "${TOPICS[@]}"
do
    java -cp $ANSERINI_JAR io.anserini.search.SearchCollection -index msmarco-v2.1-doc -topics $t -output $OUTPUT_DIR/run.msmarco-v2.1-doc.bm25.${t}.txt -threads 16 -bm25
done
Evaluation

Run these commands for evaluation:

java -cp $ANSERINI_JAR trec_eval -c -M 100 -m recip_rank msmarco-v2.1-doc.dev $OUTPUT_DIR/run.msmarco-v2.1-doc.bm25.msmarco-v2-doc-dev.txt
java -cp $ANSERINI_JAR trec_eval -c -M 100 -m recip_rank msmarco-v2.1-doc.dev2 $OUTPUT_DIR/run.msmarco-v2.1-doc.bm25.msmarco-v2-doc-dev2.txt
echo ''
java -cp $ANSERINI_JAR trec_eval -c -M 100 -m map dl21-doc-msmarco-v2.1 $OUTPUT_DIR/run.msmarco-v2.1-doc.bm25.trec2021-dl.txt
java -cp $ANSERINI_JAR trec_eval -c -M 100 -m recip_rank -c -m ndcg_cut.10 dl21-doc-msmarco-v2.1 $OUTPUT_DIR/run.msmarco-v2.1-doc.bm25.trec2021-dl.txt
java -cp $ANSERINI_JAR trec_eval -c -m recall.100 dl21-doc-msmarco-v2.1 $OUTPUT_DIR/run.msmarco-v2.1-doc.bm25.trec2021-dl.txt
java -cp $ANSERINI_JAR trec_eval -c -m recall.1000 dl21-doc-msmarco-v2.1 $OUTPUT_DIR/run.msmarco-v2.1-doc.bm25.trec2021-dl.txt
echo ''
java -cp $ANSERINI_JAR trec_eval -c -M 100 -m map dl22-doc-msmarco-v2.1 $OUTPUT_DIR/run.msmarco-v2.1-doc.bm25.trec2022-dl.txt
java -cp $ANSERINI_JAR trec_eval -c -M 100 -m recip_rank -c -m ndcg_cut.10 dl22-doc-msmarco-v2.1 $OUTPUT_DIR/run.msmarco-v2.1-doc.bm25.trec2022-dl.txt
java -cp $ANSERINI_JAR trec_eval -c -m recall.100 dl22-doc-msmarco-v2.1 $OUTPUT_DIR/run.msmarco-v2.1-doc.bm25.trec2022-dl.txt
java -cp $ANSERINI_JAR trec_eval -c -m recall.1000 dl22-doc-msmarco-v2.1 $OUTPUT_DIR/run.msmarco-v2.1-doc.bm25.trec2022-dl.txt
echo ''
java -cp $ANSERINI_JAR trec_eval -c -M 100 -m map dl23-doc-msmarco-v2.1 $OUTPUT_DIR/run.msmarco-v2.1-doc.bm25.trec2023-dl.txt
java -cp $ANSERINI_JAR trec_eval -c -M 100 -m recip_rank -c -m ndcg_cut.10 dl23-doc-msmarco-v2.1 $OUTPUT_DIR/run.msmarco-v2.1-doc.bm25.trec2023-dl.txt
java -cp $ANSERINI_JAR trec_eval -c -m recall.100 dl23-doc-msmarco-v2.1 $OUTPUT_DIR/run.msmarco-v2.1-doc.bm25.trec2023-dl.txt
java -cp $ANSERINI_JAR trec_eval -c -m recall.1000 dl23-doc-msmarco-v2.1 $OUTPUT_DIR/run.msmarco-v2.1-doc.bm25.trec2023-dl.txt
echo ''
java -cp $ANSERINI_JAR trec_eval -c -M 100 -m map rag24.raggy-dev $OUTPUT_DIR/run.msmarco-v2.1-doc.bm25.rag24-raggy-dev.txt
java -cp $ANSERINI_JAR trec_eval -c -M 100 -m recip_rank -c -m ndcg_cut.10 rag24.raggy-dev $OUTPUT_DIR/run.msmarco-v2.1-doc.bm25.rag24-raggy-dev.txt
java -cp $ANSERINI_JAR trec_eval -c -m recall.100 rag24.raggy-dev $OUTPUT_DIR/run.msmarco-v2.1-doc.bm25.rag24-raggy-dev.txt
java -cp $ANSERINI_JAR trec_eval -c -m recall.1000 rag24.raggy-dev $OUTPUT_DIR/run.msmarco-v2.1-doc.bm25.rag24-raggy-dev.txt

And these are the expected scores:

recip_rank            	all	0.1654
recip_rank            	all	0.1732

map                   	all	0.2281
recip_rank            	all	0.8466
ndcg_cut_10           	all	0.5183
recall_100            	all	0.3502
recall_1000           	all	0.6915

map                   	all	0.0841
recip_rank            	all	0.6623
ndcg_cut_10           	all	0.2991
recall_100            	all	0.1866
recall_1000           	all	0.4254

map                   	all	0.1089
recip_rank            	all	0.5783
ndcg_cut_10           	all	0.2914
recall_100            	all	0.2604
recall_1000           	all	0.5383

map                   	all	0.1251
recip_rank            	all	0.7060
ndcg_cut_10           	all	0.3631
recall_100            	all	0.2433
recall_1000           	all	0.5317

Here are the instructions for reproducing runs on the MS MARCO V2.1 segmented document corpus with prebuilt indexes (adjust number of threads based on available resources):

TOPICS=(msmarco-v2-doc-dev msmarco-v2-doc-dev2 trec2021-dl trec2022-dl trec2023-dl rag24-raggy-dev); for t in "${TOPICS[@]}"
do
    java -cp $ANSERINI_JAR io.anserini.search.SearchCollection -index msmarco-v2.1-doc-segmented -topics $t -output $OUTPUT_DIR/run.msmarco-v2.1-doc-segmented.bm25.${t}.txt -threads 16 -bm25 -hits 10000 -selectMaxPassage -selectMaxPassage.delimiter "#" -selectMaxPassage.hits 1000
done
Evaluation

Run these commands for evaluation:

java -cp $ANSERINI_JAR trec_eval -c -M 100 -m recip_rank msmarco-v2.1-doc.dev $OUTPUT_DIR/run.msmarco-v2.1-doc-segmented.bm25.msmarco-v2-doc-dev.txt
java -cp $ANSERINI_JAR trec_eval -c -M 100 -m recip_rank msmarco-v2.1-doc.dev2 $OUTPUT_DIR/run.msmarco-v2.1-doc-segmented.bm25.msmarco-v2-doc-dev2.txt
echo ''
java -cp $ANSERINI_JAR trec_eval -c -M 100 -m map dl21-doc-msmarco-v2.1 $OUTPUT_DIR/run.msmarco-v2.1-doc-segmented.bm25.trec2021-dl.txt
java -cp $ANSERINI_JAR trec_eval -c -M 100 -m recip_rank -c -m ndcg_cut.10 dl21-doc-msmarco-v2.1 $OUTPUT_DIR/run.msmarco-v2.1-doc-segmented.bm25.trec2021-dl.txt
java -cp $ANSERINI_JAR trec_eval -c -m recall.100 dl21-doc-msmarco-v2.1 $OUTPUT_DIR/run.msmarco-v2.1-doc-segmented.bm25.trec2021-dl.txt
java -cp $ANSERINI_JAR trec_eval -c -m recall.1000 dl21-doc-msmarco-v2.1 $OUTPUT_DIR/run.msmarco-v2.1-doc-segmented.bm25.trec2021-dl.txt
echo ''
java -cp $ANSERINI_JAR trec_eval -c -M 100 -m map dl22-doc-msmarco-v2.1 $OUTPUT_DIR/run.msmarco-v2.1-doc-segmented.bm25.trec2022-dl.txt
java -cp $ANSERINI_JAR trec_eval -c -M 100 -m recip_rank -c -m ndcg_cut.10 dl22-doc-msmarco-v2.1 $OUTPUT_DIR/run.msmarco-v2.1-doc-segmented.bm25.trec2022-dl.txt
java -cp $ANSERINI_JAR trec_eval -c -m recall.100 dl22-doc-msmarco-v2.1 $OUTPUT_DIR/run.msmarco-v2.1-doc-segmented.bm25.trec2022-dl.txt
java -cp $ANSERINI_JAR trec_eval -c -m recall.1000 dl22-doc-msmarco-v2.1 $OUTPUT_DIR/run.msmarco-v2.1-doc-segmented.bm25.trec2022-dl.txt
echo ''
java -cp $ANSERINI_JAR trec_eval -c -M 100 -m map dl23-doc-msmarco-v2.1 $OUTPUT_DIR/run.msmarco-v2.1-doc-segmented.bm25.trec2023-dl.txt
java -cp $ANSERINI_JAR trec_eval -c -M 100 -m recip_rank -c -m ndcg_cut.10 dl23-doc-msmarco-v2.1 $OUTPUT_DIR/run.msmarco-v2.1-doc-segmented.bm25.trec2023-dl.txt
java -cp $ANSERINI_JAR trec_eval -c -m recall.100 dl23-doc-msmarco-v2.1 $OUTPUT_DIR/run.msmarco-v2.1-doc-segmented.bm25.trec2023-dl.txt
java -cp $ANSERINI_JAR trec_eval -c -m recall.1000 dl23-doc-msmarco-v2.1 $OUTPUT_DIR/run.msmarco-v2.1-doc-segmented.bm25.trec2023-dl.txt
echo ''
java -cp $ANSERINI_JAR trec_eval -c -M 100 -m map rag24.raggy-dev $OUTPUT_DIR/run.msmarco-v2.1-doc-segmented.bm25.rag24-raggy-dev.txt
java -cp $ANSERINI_JAR trec_eval -c -M 100 -m recip_rank -c -m ndcg_cut.10 rag24.raggy-dev $OUTPUT_DIR/run.msmarco-v2.1-doc-segmented.bm25.rag24-raggy-dev.txt
java -cp $ANSERINI_JAR trec_eval -c -m recall.100 rag24.raggy-dev $OUTPUT_DIR/run.msmarco-v2.1-doc-segmented.bm25.rag24-raggy-dev.txt
java -cp $ANSERINI_JAR trec_eval -c -m recall.1000 rag24.raggy-dev $OUTPUT_DIR/run.msmarco-v2.1-doc-segmented.bm25.rag24-raggy-dev.txt

And these are the expected scores:

recip_rank            	all	0.1973
recip_rank            	all	0.2000

map                   	all	0.2609
recip_rank            	all	0.9026
ndcg_cut_10           	all	0.5778
recall_100            	all	0.3811
recall_1000           	all	0.7115

map                   	all	0.1079
recip_rank            	all	0.7213
ndcg_cut_10           	all	0.3576
recall_100            	all	0.2330
recall_1000           	all	0.4790

map                   	all	0.1391
recip_rank            	all	0.6519
ndcg_cut_10           	all	0.3356
recall_100            	all	0.3049
recall_1000           	all	0.5852

map                   	all	0.1561
recip_rank            	all	0.7465
ndcg_cut_10           	all	0.4227
recall_100            	all	0.2807
recall_1000           	all	0.5745

To generate jsonl output containing the raw documents that can be reranked and further processed, use the -outputRerankerRequests option to specify an output file. For example:

java -cp $ANSERINI_JAR io.anserini.search.SearchCollection \
  -index msmarco-v2.1-doc \
  -topics trec2023-dl \
  -output $OUTPUT_DIR/run.msmarco-v2.1-doc.bm25.trec2023-dl.txt \
  -bm25 -hits 20 \
  -outputRerankerRequests $OUTPUT_DIR/results.msmarco-v2.1-doc.bm25.trec2023-dl.jsonl

And the output looks something like:

$ head -n 1 $OUTPUT_DIR/results.msmarco-v2.1-doc.bm25.trec2023-dl.jsonl | jq 
{
  "query": {
    "text": "How does the process of digestion and metabolism of carbohydrates start",
    "qid": 2000138
  },
  "candidates": [
    {
      "docid": "msmarco_v2.1_doc_15_390497775",
      "score": 14.3364,
      "doc": {
        "url": "https://diabetestalk.net/blood-sugar/conversion-of-carbohydrates-to-glucose",
        "title": "Conversion Of Carbohydrates To Glucose | DiabetesTalk.Net",
        "headings": "...",
        "body": "..."
      }
    },
    {
      "docid": "msmarco_v2.1_doc_15_416962410",
      "score": 14.2271,
      "doc": {
        "url": "https://diabetestalk.net/insulin/how-is-starch-converted-to-glucose-in-the-body",
        "title": "How Is Starch Converted To Glucose In The Body? | DiabetesTalk.Net",
        "headings": "...",
        "body": "..."
      }
    },
    ...
  ]
}

MS MARCO V1 Passage

❗ Beware, the (automatically downloaded) indexes for running these experiments take up 200 GB in total.

Currently, Anserini provides support for the following models:

  • BM25
  • SPLADE++ EnsembleDistil: cached queries and ONNX query encoding
  • cosDPR-distil: cached queries and ONNX query encoding
  • bge-base-en-v1.5: cached queries and ONNX query encoding
  • cohere-embed-english-v3.0: cached queries and ONNX query encoding

The following snippet will generate the complete set of results for MS MARCO V1 Passage:

# BM25
TOPICS=(msmarco-v1-passage.dev dl19-passage dl20-passage); for t in "${TOPICS[@]}"
do
    java -cp $ANSERINI_JAR io.anserini.search.SearchCollection -index msmarco-v1-passage -topics ${t} -output $OUTPUT_DIR/run.${t}.bm25.txt -threads 16 -bm25
done

# SPLADE++ ED
TOPICS=(msmarco-v1-passage.dev dl19-passage dl20-passage); for t in "${TOPICS[@]}"
do
    # Using cached queries
    java -cp $ANSERINI_JAR io.anserini.search.SearchCollection -index msmarco-v1-passage.splade-pp-ed -topics ${t}.splade-pp-ed -output $OUTPUT_DIR/run.${t}.splade-pp-ed.cached_q.txt -threads 16 -impact -pretokenized
    # Using ONNX
    java -cp $ANSERINI_JAR io.anserini.search.SearchCollection -index msmarco-v1-passage.splade-pp-ed -topics ${t} -encoder SpladePlusPlusEnsembleDistil -output $OUTPUT_DIR/run.${t}.splade-pp-ed.onnx.txt -threads 16 -impact -pretokenized
done

# cosDPR-distil
TOPICS=(msmarco-v1-passage.dev dl19-passage dl20-passage); for t in "${TOPICS[@]}"
do
    # Using fp32 index, cached queries
    java -cp $ANSERINI_JAR io.anserini.search.SearchHnswDenseVectors -index msmarco-v1-passage.cos-dpr-distil -topics ${t}.cos-dpr-distil -output $OUTPUT_DIR/run.${t}.cos-dpr-distil.fp32.cached_q.txt -threads 16 -efSearch 1000
    # Using fp32 index, ONNX
    java -cp $ANSERINI_JAR io.anserini.search.SearchHnswDenseVectors -index msmarco-v1-passage.cos-dpr-distil -topics ${t} -encoder CosDprDistil -output $OUTPUT_DIR/run.${t}.cos-dpr-distil.fp32.onnx.txt -threads 16 -efSearch 1000
    # Using int8 index, cached queries
    java -cp $ANSERINI_JAR io.anserini.search.SearchHnswDenseVectors -index msmarco-v1-passage.cos-dpr-distil.quantized -topics ${t}.cos-dpr-distil -output $OUTPUT_DIR/run.${t}.cos-dpr-distil.int8.cached_q.txt -threads 16 -efSearch 1000
    # Using int8 index, ONNX
    java -cp $ANSERINI_JAR io.anserini.search.SearchHnswDenseVectors -index msmarco-v1-passage.cos-dpr-distil.quantized -topics ${t} -encoder CosDprDistil -output $OUTPUT_DIR/run.${t}.cos-dpr-distil.int8.onnx.txt -threads 16 -efSearch 1000
done

# bge-base-en-v1.5
TOPICS=(msmarco-v1-passage.dev dl19-passage dl20-passage); for t in "${TOPICS[@]}"
do
    # Using fp32 index, cached queries
    java -cp $ANSERINI_JAR io.anserini.search.SearchHnswDenseVectors -index msmarco-v1-passage.bge-base-en-v1.5 -topics ${t}.bge-base-en-v1.5 -output $OUTPUT_DIR/run.${t}.bge-base-en-v1.5.fp32.cached_q.txt -threads 16 -efSearch 1000
    # Using fp32 index, ONNX
    java -cp $ANSERINI_JAR io.anserini.search.SearchHnswDenseVectors -index msmarco-v1-passage.bge-base-en-v1.5 -topics ${t} -encoder BgeBaseEn15 -output $OUTPUT_DIR/run.${t}.bge-base-en-v1.5.fp32.onnx.txt -threads 16 -efSearch 1000
    # Using int8 index, cached queries
    java -cp $ANSERINI_JAR io.anserini.search.SearchHnswDenseVectors -index msmarco-v1-passage.bge-base-en-v1.5.quantized -topics ${t}.bge-base-en-v1.5 -output $OUTPUT_DIR/run.${t}.bge-base-en-v1.5.int8.cached_q.txt -threads 16 -efSearch 1000
    # Using int8 index, ONNX
    java -cp $ANSERINI_JAR io.anserini.search.SearchHnswDenseVectors -index msmarco-v1-passage.bge-base-en-v1.5.quantized -topics ${t} -encoder BgeBaseEn15 -output $OUTPUT_DIR/run.${t}.bge-base-en-v1.5.int8.onnx.txt -threads 16 -efSearch 1000
done

# cohere-embed-english-v3.0
TOPICS=(msmarco-v1-passage.dev dl19-passage dl20-passage); for t in "${TOPICS[@]}"
do
    # Using fp32 index, cached queries
    java -cp $ANSERINI_JAR io.anserini.search.SearchHnswDenseVectors -index msmarco-v1-passage.cohere-embed-english-v3.0 -topics ${t}.cohere-embed-english-v3.0 -output $OUTPUT_DIR/run.${t}.cohere-embed-english-v3.0.fp32.cached_q.txt -threads 16 -efSearch 1000
    # Using int8 index, cached queries
    java -cp $ANSERINI_JAR io.anserini.search.SearchHnswDenseVectors -index msmarco-v1-passage.cohere-embed-english-v3.0.quantized -topics ${t}.cohere-embed-english-v3.0 -output $OUTPUT_DIR/run.${t}.cohere-embed-english-v3.0.int8.cached_q.txt -threads 16 -efSearch 1000
done

Here are the expected scores (dev using MRR@10, DL19 and DL20 using nDCG@10):

dev DL19 DL20
BM25 0.1840 0.5058 0.4796
SPLADE++ ED (cached queries) 0.3830 0.7317 0.7198
SPLADE++ ED (ONNX) 0.3828 0.7308 0.7197
cosDPR-distil w/ HNSW fp32 (cached queries) 0.3887 0.7250 0.7025
cosDPR-distil w/ HNSW fp32 (ONNX) 0.3887 0.7250 0.7025
cosDPR-distil w/ HNSW int8 (cached queries) 0.3897 0.7240 0.7004
cosDPR-distil w/ HNSW int8 (ONNX) 0.3899 0.7247 0.6996
bge-base-en-v1.5 w/ HNSW fp32 (cached queries) 0.3574 0.7065 0.6780
bge-base-en-v1.5 w/ HNSW fp32 (ONNX) 0.3575 0.7016 0.6768
bge-base-en-v1.5 w/ HNSW int8 (cached queries) 0.3572 0.7016 0.6738
bge-base-en-v1.5 w/ HNSW int8 (ONNX) 0.3575 0.7017 0.6767
cohere-embed-english-v3.0 w/ HNSW fp32 (cached queries) 0.3647 0.6956 0.7245
cohere-embed-english-v3.0 w/ HNSW int8 (cached queries) 0.3656 0.6955 0.7262
Evaluation

And here's the snippet of code to perform the evaluation (which will yield the results above):

java -cp $ANSERINI_JAR trec_eval -c -M 10 -m recip_rank msmarco-passage.dev-subset $OUTPUT_DIR/run.msmarco-v1-passage.dev.bm25.txt
java -cp $ANSERINI_JAR trec_eval -m ndcg_cut.10 -c dl19-passage                    $OUTPUT_DIR/run.dl19-passage.bm25.txt
java -cp $ANSERINI_JAR trec_eval -m ndcg_cut.10 -c dl20-passage                    $OUTPUT_DIR/run.dl20-passage.bm25.txt
echo ''
java -cp $ANSERINI_JAR trec_eval -c -M 10 -m recip_rank msmarco-passage.dev-subset $OUTPUT_DIR/run.msmarco-v1-passage.dev.splade-pp-ed.cached_q.txt
java -cp $ANSERINI_JAR trec_eval -m ndcg_cut.10 -c dl19-passage                    $OUTPUT_DIR/run.dl19-passage.splade-pp-ed.cached_q.txt
java -cp $ANSERINI_JAR trec_eval -m ndcg_cut.10 -c dl20-passage                    $OUTPUT_DIR/run.dl20-passage.splade-pp-ed.cached_q.txt
echo ''
java -cp $ANSERINI_JAR trec_eval -c -M 10 -m recip_rank msmarco-passage.dev-subset $OUTPUT_DIR/run.msmarco-v1-passage.dev.splade-pp-ed.onnx.txt
java -cp $ANSERINI_JAR trec_eval -m ndcg_cut.10 -c dl19-passage                    $OUTPUT_DIR/run.dl19-passage.splade-pp-ed.onnx.txt
java -cp $ANSERINI_JAR trec_eval -m ndcg_cut.10 -c dl20-passage                    $OUTPUT_DIR/run.dl20-passage.splade-pp-ed.onnx.txt
echo ''
java -cp $ANSERINI_JAR trec_eval -c -M 10 -m recip_rank msmarco-passage.dev-subset $OUTPUT_DIR/run.msmarco-v1-passage.dev.cos-dpr-distil.fp32.cached_q.txt
java -cp $ANSERINI_JAR trec_eval -m ndcg_cut.10 -c dl19-passage                    $OUTPUT_DIR/run.dl19-passage.cos-dpr-distil.fp32.cached_q.txt
java -cp $ANSERINI_JAR trec_eval -m ndcg_cut.10 -c dl20-passage                    $OUTPUT_DIR/run.dl20-passage.cos-dpr-distil.fp32.cached_q.txt
echo ''
java -cp $ANSERINI_JAR trec_eval -c -M 10 -m recip_rank msmarco-passage.dev-subset $OUTPUT_DIR/run.msmarco-v1-passage.dev.cos-dpr-distil.fp32.onnx.txt
java -cp $ANSERINI_JAR trec_eval -m ndcg_cut.10 -c dl19-passage                    $OUTPUT_DIR/run.dl19-passage.cos-dpr-distil.fp32.onnx.txt
java -cp $ANSERINI_JAR trec_eval -m ndcg_cut.10 -c dl20-passage                    $OUTPUT_DIR/run.dl20-passage.cos-dpr-distil.fp32.onnx.txt
echo ''
java -cp $ANSERINI_JAR trec_eval -c -M 10 -m recip_rank msmarco-passage.dev-subset $OUTPUT_DIR/run.msmarco-v1-passage.dev.cos-dpr-distil.int8.cached_q.txt
java -cp $ANSERINI_JAR trec_eval -m ndcg_cut.10 -c dl19-passage                    $OUTPUT_DIR/run.dl19-passage.cos-dpr-distil.int8.cached_q.txt
java -cp $ANSERINI_JAR trec_eval -m ndcg_cut.10 -c dl20-passage                    $OUTPUT_DIR/run.dl20-passage.cos-dpr-distil.int8.cached_q.txt
echo ''
java -cp $ANSERINI_JAR trec_eval -c -M 10 -m recip_rank msmarco-passage.dev-subset $OUTPUT_DIR/run.msmarco-v1-passage.dev.cos-dpr-distil.int8.onnx.txt
java -cp $ANSERINI_JAR trec_eval -m ndcg_cut.10 -c dl19-passage                    $OUTPUT_DIR/run.dl19-passage.cos-dpr-distil.int8.onnx.txt
java -cp $ANSERINI_JAR trec_eval -m ndcg_cut.10 -c dl20-passage                    $OUTPUT_DIR/run.dl20-passage.cos-dpr-distil.int8.onnx.txt
echo ''
java -cp $ANSERINI_JAR trec_eval -c -M 10 -m recip_rank msmarco-passage.dev-subset $OUTPUT_DIR/run.msmarco-v1-passage.dev.bge-base-en-v1.5.fp32.cached_q.txt
java -cp $ANSERINI_JAR trec_eval -m ndcg_cut.10 -c dl19-passage                    $OUTPUT_DIR/run.dl19-passage.bge-base-en-v1.5.fp32.cached_q.txt
java -cp $ANSERINI_JAR trec_eval -m ndcg_cut.10 -c dl20-passage                    $OUTPUT_DIR/run.dl20-passage.bge-base-en-v1.5.fp32.cached_q.txt
echo ''
java -cp $ANSERINI_JAR trec_eval -c -M 10 -m recip_rank msmarco-passage.dev-subset $OUTPUT_DIR/run.msmarco-v1-passage.dev.bge-base-en-v1.5.fp32.onnx.txt
java -cp $ANSERINI_JAR trec_eval -m ndcg_cut.10 -c dl19-passage                    $OUTPUT_DIR/run.dl19-passage.bge-base-en-v1.5.fp32.onnx.txt
java -cp $ANSERINI_JAR trec_eval -m ndcg_cut.10 -c dl20-passage                    $OUTPUT_DIR/run.dl20-passage.bge-base-en-v1.5.fp32.onnx.txt
echo ''
java -cp $ANSERINI_JAR trec_eval -c -M 10 -m recip_rank msmarco-passage.dev-subset $OUTPUT_DIR/run.msmarco-v1-passage.dev.bge-base-en-v1.5.int8.cached_q.txt
java -cp $ANSERINI_JAR trec_eval -m ndcg_cut.10 -c dl19-passage                    $OUTPUT_DIR/run.dl19-passage.bge-base-en-v1.5.int8.cached_q.txt
java -cp $ANSERINI_JAR trec_eval -m ndcg_cut.10 -c dl20-passage                    $OUTPUT_DIR/run.dl20-passage.bge-base-en-v1.5.int8.cached_q.txt
echo ''
java -cp $ANSERINI_JAR trec_eval -c -M 10 -m recip_rank msmarco-passage.dev-subset $OUTPUT_DIR/run.msmarco-v1-passage.dev.bge-base-en-v1.5.int8.onnx.txt
java -cp $ANSERINI_JAR trec_eval -m ndcg_cut.10 -c dl19-passage                    $OUTPUT_DIR/run.dl19-passage.bge-base-en-v1.5.int8.onnx.txt
java -cp $ANSERINI_JAR trec_eval -m ndcg_cut.10 -c dl20-passage                    $OUTPUT_DIR/run.dl20-passage.bge-base-en-v1.5.int8.onnx.txt
echo ''
java -cp $ANSERINI_JAR trec_eval -c -M 10 -m recip_rank msmarco-passage.dev-subset $OUTPUT_DIR/run.msmarco-v1-passage.dev.cohere-embed-english-v3.0.fp32.cached_q.txt
java -cp $ANSERINI_JAR trec_eval -m ndcg_cut.10 -c dl19-passage                    $OUTPUT_DIR/run.dl19-passage.cohere-embed-english-v3.0.fp32.cached_q.txt
java -cp $ANSERINI_JAR trec_eval -m ndcg_cut.10 -c dl20-passage                    $OUTPUT_DIR/run.dl20-passage.cohere-embed-english-v3.0.fp32.cached_q.txt
echo ''
java -cp $ANSERINI_JAR trec_eval -c -M 10 -m recip_rank msmarco-passage.dev-subset $OUTPUT_DIR/run.msmarco-v1-passage.dev.cohere-embed-english-v3.0.int8.cached_q.txt
java -cp $ANSERINI_JAR trec_eval -m ndcg_cut.10 -c dl19-passage                    $OUTPUT_DIR/run.dl19-passage.cohere-embed-english-v3.0.int8.cached_q.txt
java -cp $ANSERINI_JAR trec_eval -m ndcg_cut.10 -c dl20-passage                    $OUTPUT_DIR/run.dl20-passage.cohere-embed-english-v3.0.int8.cached_q.txt

BEIR

❗ Beware, the (automatically downloaded) indexes for running these experiments take up 246 GB in total.

Currently, Anserini provides support for the following models:

  • Flat = BM25, "flat" bag-of-words baseline
  • MF = BM25, "multifield" bag-of-words baseline
  • S = SPLADE++ EnsembleDistil:
    • cached queries (Sp)
    • ONNX query encoding (So)
  • D = bge-base-en-v1.5
    • cached queries (Dp)
    • ONNX query encoding (Do)

The following snippet will generate the complete set of results for BEIR:

CORPORA=(trec-covid bioasq nfcorpus nq hotpotqa fiqa signal1m trec-news robust04 arguana webis-touche2020 cqadupstack-android cqadupstack-english cqadupstack-gaming cqadupstack-gis cqadupstack-mathematica cqadupstack-physics cqadupstack-programmers cqadupstack-stats cqadupstack-tex cqadupstack-unix cqadupstack-webmasters cqadupstack-wordpress quora dbpedia-entity scidocs fever climate-fever scifact); for c in "${CORPORA[@]}"
do
    # "flat" indexes
    java -cp $ANSERINI_JAR io.anserini.search.SearchCollection -index beir-v1.0.0-${c}.flat -topics beir-${c} -output $OUTPUT_DIR/run.beir.${c}.flat.txt -bm25 -removeQuery
    # "multifield" indexes
    java -cp $ANSERINI_JAR io.anserini.search.SearchCollection -index beir-v1.0.0-${c}.multifield -topics beir-${c} -output $OUTPUT_DIR/run.beir.${c}.multifield.txt -bm25 -removeQuery -fields contents=1.0 title=1.0
    # SPLADE++ ED, cached queries
    java -cp $ANSERINI_JAR io.anserini.search.SearchCollection -index beir-v1.0.0-${c}.splade-pp-ed -topics beir-${c}.splade-pp-ed -output $OUTPUT_DIR/run.beir.${c}.splade-pp-ed.cached_q.txt -impact -pretokenized -removeQuery
    # SPLADE++ ED, ONNX
    java -cp $ANSERINI_JAR io.anserini.search.SearchCollection -index beir-v1.0.0-${c}.splade-pp-ed -topics beir-${c} -encoder SpladePlusPlusEnsembleDistil -output $OUTPUT_DIR/run.beir.${c}.splade-pp-ed.onnx.txt -impact -pretokenized -removeQuery
    # BGE-base-en-v1.5, cached queries
    java -cp $ANSERINI_JAR io.anserini.search.SearchHnswDenseVectors -index beir-v1.0.0-${c}.bge-base-en-v1.5 -topics beir-${c}.bge-base-en-v1.5 -output $OUTPUT_DIR/run.beir.${c}.bge.cached_q.txt -threads 16 -efSearch 1000 -removeQuery
    # BGE-base-en-v1.5, ONNX
    java -cp $ANSERINI_JAR io.anserini.search.SearchHnswDenseVectors -index beir-v1.0.0-${c}.bge-base-en-v1.5 -topics beir-${c} -encoder BgeBaseEn15 -output $OUTPUT_DIR/run.beir.${c}.bge.onnx.txt -threads 16 -efSearch 1000 -removeQuery
done

Here are the expected nDCG@10 scores:

Corpus Flat MF Sp So Dp Do
trec-covid 0.5947 0.6559 0.7274 0.7270 0.7834 0.7835
bioasq 0.5225 0.4646 0.4980 0.4980 0.4042 0.4042
nfcorpus 0.3218 0.3254 0.3470 0.3473 0.3735 0.3738
nq 0.3055 0.3285 0.5378 0.5372 0.5413 0.5415
hotpotqa 0.6330 0.6027 0.6868 0.6868 0.7242 0.7241
fiqa 0.2361 0.2361 0.3475 0.3473 0.4065 0.4065
signal1m 0.3304 0.3304 0.3008 0.3006 0.2869 0.2869
trec-news 0.3952 0.3977 0.4152 0.4169 0.4411 0.4410
robust04 0.4070 0.4070 0.4679 0.4651 0.4467 0.4437
arguana 0.3970 0.4142 0.5203 0.5218 0.6361 0.6228
webis-touche2020 0.4422 0.3673 0.2468 0.2464 0.2570 0.2571
cqadupstack-android 0.3801 0.3709 0.3904 0.3898 0.5075 0.5076
cqadupstack-english 0.3453 0.3321 0.4079 0.4078 0.4855 0.4855
cqadupstack-gaming 0.4822 0.4418 0.4957 0.4959 0.5965 0.5967
cqadupstack-gis 0.2901 0.2904 0.3150 0.3148 0.4129 0.4133
cqadupstack-mathematica 0.2015 0.2046 0.2377 0.2379 0.3163 0.3163
cqadupstack-physics 0.3214 0.3248 0.3599 0.3597 0.4722 0.4724
cqadupstack-programmers 0.2802 0.2963 0.3401 0.3399 0.4242 0.4238
cqadupstack-stats 0.2711 0.2790 0.2990 0.2980 0.3731 0.3728
cqadupstack-tex 0.2244 0.2086 0.2530 0.2529 0.3115 0.3115
cqadupstack-unix 0.2749 0.2788 0.3167 0.3170 0.4219 0.4220
cqadupstack-webmasters 0.3059 0.3008 0.3167 0.3166 0.4065 0.4072
cqadupstack-wordpress 0.2483 0.2562 0.2733 0.2718 0.3547 0.3547
quora 0.7886 0.7886 0.8343 0.8344 0.8890 0.8876
dbpedia-entity 0.3180 0.3128 0.4366 0.4374 0.4077 0.4076
scidocs 0.1490 0.1581 0.1591 0.1588 0.2170 0.2172
fever 0.6513 0.7530 0.7882 0.7879 0.8620 0.8620
climate-fever 0.1651 0.2129 0.2297 0.2298 0.3119 0.3117
scifact 0.6789 0.6647 0.7041 0.7036 0.7408 0.7408
Evaluation

And here's the snippet of code to perform the evaluation (which will yield the results above):

CORPORA=(trec-covid bioasq nfcorpus nq hotpotqa fiqa signal1m trec-news robust04 arguana webis-touche2020 cqadupstack-android cqadupstack-english cqadupstack-gaming cqadupstack-gis cqadupstack-mathematica cqadupstack-physics cqadupstack-programmers cqadupstack-stats cqadupstack-tex cqadupstack-unix cqadupstack-webmasters cqadupstack-wordpress quora dbpedia-entity scidocs fever climate-fever scifact); for c in "${CORPORA[@]}"
do
    echo $c
    java -cp $ANSERINI_JAR trec_eval -c -m ndcg_cut.10 qrels.beir-v1.0.0-${c}.test.txt $OUTPUT_DIR/run.beir.${c}.flat.txt
    java -cp $ANSERINI_JAR trec_eval -c -m ndcg_cut.10 qrels.beir-v1.0.0-${c}.test.txt $OUTPUT_DIR/run.beir.${c}.multifield.txt
    java -cp $ANSERINI_JAR trec_eval -c -m ndcg_cut.10 qrels.beir-v1.0.0-${c}.test.txt $OUTPUT_DIR/run.beir.${c}.splade-pp-ed.cached_q.txt
    java -cp $ANSERINI_JAR trec_eval -c -m ndcg_cut.10 qrels.beir-v1.0.0-${c}.test.txt $OUTPUT_DIR/run.beir.${c}.splade-pp-ed.onnx.txt
    java -cp $ANSERINI_JAR trec_eval -c -m ndcg_cut.10 qrels.beir-v1.0.0-${c}.test.txt $OUTPUT_DIR/run.beir.${c}.bge.cached_q.txt
    java -cp $ANSERINI_JAR trec_eval -c -m ndcg_cut.10 qrels.beir-v1.0.0-${c}.test.txt $OUTPUT_DIR/run.beir.${c}.bge.onnx.txt
done