Sources:
Task 1: finetuning BioBERT with diff-vector on BC2GM
- Precision: 85.56
- Recall: 87.02
- F1: 86.28
Parameter | Value |
---|---|
epochs | 80 |
w learning rate | 1e-5 |
alpha learning rate | 1e-5 |
weight decay | 1e-2 |
batch size | 32 |
Task 2: diff pruning BioBERT with diff-vector on BC2GM
Parameter | Value |
---|---|
epochs | 80 |
w learning rate | 1e-5 |
alpha learning rate | 1e-1 |
sparsity penalty | 1.25e-7 |
weight decay | 1e-2 |
batch size | 32 |
- Nonzero parameters: 1.1%
- Precision: 84.73
- Recall: 83.32
- F1: 86.19
Magnitude Pruning and Fixmask Finetuning:
Parameter | Value |
---|---|
epochs | 80 |
w learning rate | 1e-5 |
alpha learning rate | - |
sparsity penalty | - |
weight decay | 1e-2 |
batch size | 32 |
- Nonzero parameters:0.5%
- Precision: 86.43
- Recall: 84.33
- F1: 85.37