Codes for the EMNLP2021 paper: Benchmarking Commonsense Knowledge Base Population with an Effective Evaluation Dataset
We have an expert-annotated and adversarially constructed evaluation set CKBP v2.
For data preprocessing and the codes released upon EMNLP 2021, checkout the branch emnlp2021
(https://github.com/HKUST-KnowComp/CSKB-Population/tree/emnlp2021).
CKBP v1: training data, evaluation set.
CKBP v2: training data (updated), evaluation data
We tested our codes on python 3.8.5
. Here are some packages their corresponding versions.
networkx 2.5
numpy 1.19.2
pandas 1.2.2
scikit-learn 0.24.1
scipy 1.6.3
sklearn 0.0
spacy 3.2.0
stanfordnlp 0.2.0
torch 1.7.1
torch-geometric 1.7.2
torch-scatter 2.0.7
torch-sparse 0.6.9
torchsummary 1.5.1
torchtext 0.8.1
tqdm 4.56.0
transformers 3.4.0
See the branch emnlp2021
kaggle datasets download -d tianqingfang/ckbp-emnlp2021
Or download at https://www.kaggle.com/datasets/tianqingfang/ckbp-emnlp2021.
Put the train.csv
under data/ckbp_csv/emnlp2021/
. The dev.csv
and test.csv
are automatically constructed dev and test sets, sampled from ASER and CSKBs (CSKB ground truth triples are labeled 1, negative sampled examples are labeled 0.)
The annotated evaluation set is at data/evaluation_set.csv
. Please use this data set to test.
KG-BERT baseline is provided here.
CUDA_VISIBLE_DEVICES=0 python models/train_kgbert_baseline.py \
--ptlm bert-base-uncased \
--lr 1e-5 \
--epochs 1 \
--output_dir results \
--train_csv_path data/ckbp_csv/emnlp2021/train.csv \
--relation_as_special_token \
--save_best_model \
--seed 100 --batch_size 64 --test_batch_size 128 --save_best_model --experiment_name ""
Check out Model Training
section on the branch emnlp2021
.
(Note that the evaluation metric is a bit different from that in the paper. In the paper we use a grouped AUC where AUC scores within different groups are calculated and averaged. Here we report the overall AUC across all relations.)
Model | All | Ori Test Set | CSKB head | ASER edge |
---|---|---|---|---|
KG-BERT (BERT-base) | 62.5 | 74.2 | 51.9 | 54.7 |
KG-BERT (RoBERTa-large) | 70.9 | 78.0 | 63.4 | 64.6 |
Checkout the DISCOS-reproduce
folder under the branch emnlp2021
.