Official scripts for "Compensatory Debiasing for Gender Imbalances in Language Models."
This repository is available in Ubuntu 18.04.5 LTS, and it is not tested in other OS.
git clone https://github.com/squiduu/guidebias.git
cd guidebias
conda create -n guidebias python=3.7.10
conda activate guidebias
pip install -r requirements.txt
Fine-tune a pre-trained BERT to debias.
cd guidebias
mkdir ./out/
sh run_finetune.sh
Then, a debiased BERT model will be saved in ./out/
.
Download StereoSet test set from here.
mkdir ../stereoset/data/
cd ../stereoset/data/
Put the test.json
in ../stereoset/data/
.
All of the evaluation scripts are followed Bias Bench. However, some minor modifications were made to suit our experimental environment.
SEAT
cd ../seat/
mkdir ./out/
sh run_seat_original.sh
StereoSet
cd ../stereoset/
mkdir ./out/
sh run_stereoset_original.sh
sh evaluate_original.sh
CrowS-Pairs
cd ../crows_pairs/
mkdir ./out/
sh run_crows_pairs_original.sh
GLUE
cd ../glue/
mkdir ./out/
sh run_glue_original.sh
SEAT
cd ../seat/
sh run_seat_debiased.sh
StereoSet
cd ../stereoset/
rm -rf ./out/results/
sh run_stereoset_debiased.sh
sh evaluate_debiased.sh
CrowS-Pairs
cd ../crows_pairs/
sh run_crows_pairs_debiased.sh
GLUE
cd ../glue/
sh run_glue_debiased.sh
SEAT
Model | SEAT-6 | SEAT-6b | SEAT-7 | SEAT-7b | SEAT-8 | SEAT-8b | Avg. |
---|---|---|---|---|---|---|---|
BERT | 0.931 | 0.090 | -0.124 | 0.937 | 0.783 | 0.858 | 0.620 |
CDA | 0.846 | 0.186 | -0.278 | 1.342 | 0.831 | 0.849 | 0.722 |
Dropout | 1.136 | 0.317 | 0.138 | 1.179 | 0.879 | 0.939 | 0.765 |
Sent-Debias | 0.350 | -0.298 | -0.626 | 0.458 | 0.413 | 0.462 | 0.434 |
INLP | 0.317 | -0.354 | -0.258 | 0.105 | 0.187 | -0.004 | 0.204 |
Context-Debias | 0.409 | 0.159 | -0.222 | 0.848 | 0.537 | 0.176 | 0.392 |
Auto-Debias | 0.344 | 0.016 | 0.173 | 1.123 | 0.734 | 0.783 | 0.529 |
Ours | -0.023 | -0.249 | -0.405 | 0.144 | -0.353 | -0.001 | 0.196 |
StereoSet
Model | LMS | SS | ICAT |
---|---|---|---|
BERT | 84.17 | 60.28 | 66.86 |
CDA | 83.08 | 59.61 | 67.11 |
Dropout | 83.04 | 60.66 | 65.34 |
Sent-Debias | 84.20 | 59.37 | 68.42 |
INLP | 80.63 | 57.25 | 68.94 |
Context-Debias | 85.34 | 59.21 | 69.62 |
Auto-Debias | 74.09 | 53.11 | 69.48 |
Ours | 83.83 | 55.36 | 74.84 |
GLUE
Model | CoLA | MNLI | MRPC | QNLI | QQP | RTE | SST2 | STSB | WNLI | Avg. |
---|---|---|---|---|---|---|---|---|---|---|
BERT | 55.64 | 84.12 | 82.19 | 91.31 | 89.23 | 61.73 | 92.32 | 87.75 | 36.15 | 75.60 |
CDA | 55.31 | 84.56 | 82.76 | 91.16 | 90.18 | 65.46 | 92.54 | 88.03 | 32.86 | 75.87 |
Dropout | 50.90 | 84.37 | 80.64 | 91.20 | 89.94 | 63.18 | 92.58 | 87.42 | 39.91 | 75.57 |
Sent-Debias | 48.55 | 84.26 | 81.86 | 91.43 | 90.78 | 61.37 | 92.35 | 87.74 | 34.74 | 74.79 |
INLP | 55.91 | 84.09 | 84.10 | 91.17 | 89.15 | 62.22 | 92.39 | 87.83 | 34.74 | 75.73 |
Context-Debias | 53.91 | 84.28 | 82.98 | 91.43 | 89.18 | 61.48 | 92.24 | 87.00 | 36.15 | 75.41 |
Auto-Debias | 55.89 | 84.25 | 84.20 | 91.57 | 89.21 | 62.58 | 92.51 | 87.68 | 39.44 | 76.37 |
Ours | 56.15 | 84.16 | 86.17 | 91.26 | 89.19 | 62.34 | 92.39 | 87.78 | 39.44 | 76.54 |
CrowS-Pairs
Model | SS | AntiSS | Avg. |
---|---|---|---|
BERT | 57.86 | 56.31 | 7.09 |
CDA | 54.09 | 60.19 | 7.14 |
Dropout | 57.23 | 55.34 | 6.29 |
Sent-Debias | 37.74 | 74.76 | 18.51 |
INLP | 42.77 | 63.11 | 10.17 |
Context-Debias | 61.01 | 51.46 | 6.24 |
Auto-Debias | 48.43 | 59.22 | 5.40 |
Ours | 55.35 | 54.37 | 4.86 |
This work was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government(MSIT) (No. 2019-0-00079, Artificial Intelligence Graduate School Program(Korea University) and No. 2022-0-00984, Development of Artificial Intelligence Technology for Personalized Plug-and-Play Explanation and Verification of Explanation).