This release collects some additional resources relevant to the paper.
This .pth file contains the weights of a model trained on the training set only. The performance is slightly better than the mean results reported in the paper.
python eval-acc.py logs/pretrained-on-train.pth returns
number (single) : 49.59% +- nan
number (pair) : 23.37% +- nan
count (single) : 57.29% +- nan
count (pair) : 26.96% +- nan
all (single) : 65.42% +- nan
all (pair) : 37.25% +- nan
Use the --resume <path> command-line option for train.py to load it.
This .pth file contains the weights of a model trained on both training and validation set, ready for evaluation on the VQA evaluation server. The performance of this is basically the same on test-dev (I didn't check test-standard) as the results in the paper.
Using python train.py --test --resume logs/pretrained-on-trainval.pth to generate a results.json file and uploading this to the evaluation server test-dev split, this gives us:
| test-dev | |
|---|---|
| yes/no | 83.22 |
| number | 51.51 |
| other | 58.87 |
| overall | 68.07 |
-
Poster
Presented at ICLR 2018 -
results.json
This is the submission file that was used in the paper containing answers to the VQA test questions. When uploaded to the test server, this gives the following results:
| test-dev | |
|---|---|
| yes/no | 83.14 |
| number | 51.62 |
| other | 58.97 |
| overall | 68.09 |
| test-standard | |
|---|---|
| yes/no | 83.56 |
| number | 51.39 |
| other | 59.11 |
| overall | 68.41 |