Skip to content

Repository for AAAI 2018 paper "Using Syntax for Referring Expression Recognition"

License

Notifications You must be signed in to change notification settings

volkancirik/groundnet

Repository files navigation

GroundNet

Repository for AAAI 2018 paper "Using Syntax for Referring Expression Recognition".

@article{cirik2018using,
title={Using Syntax to Ground Referring Expressions in Natural Images},
author={Cirik, Volkan and Berg-Kirkpatrick, Taylor and Morency, Louis-Philippe},
journal = {AAAI},
year = {2018}
}

Preparing Environment:

The code is written in an old version of Pytorch. So, first create a virtual environment for your pytorch and other packages like h5py:

conda create -n pytorch0.1.12 python=2.7
source activate pytorch0.1.12
conda install pytorch=0.1.12 h5py tqdm scipy pillow matplotlib scikit-image nltk
conda install -c conda-forge ipython jupyter ipywidgets

python -m ipykernel install --user --name=pytorch0.1.12

Compile Google_Refex_toolbox

cd Google_Refexp_toolbox; python setup.py

To be able to run the parser, you also need Java 8. Note that the first time you run a new installation of pytorch it takes some time to compile the cuda kernels, just be patient.

Now you need to download data I used for the experiments by running following commands. Under data/:

bash download_coco.sh
bash download_imdb.sh
bash download_refcocog.sh

Under parser/:

bash download_parser.sh

Under notebooks/:

bash download_sup_ann.sh

Under downloaded_models/:

bash download_models.sh

Training & Testing Model

In exp-refcocog/ run the following command to list the options for training models

source activate pytorch0.1.12
PYTHONPATH=.. python train.py --help

To test a trained model run as follows:

source activate pytorch0.1.12
PYTHONPATH=.. python test.py --resume ../downloaded_models/snapshot.groundnet.model --out-file groundnet.test.json

To evaluate a model for supporting object annotations:

PYTHONPATH=.. python analyze.py --category sup_acc --prediction groundnet.test.json

Visualizing Errors

Run your jupyter notebook under the root folder of the repo and don't forget to switch pytorch0.1.12 kernel. Then simply run the notebooks/interactiveModel.ipynb. You should be able to enter a referring expression to test a model. To play with CMN model, just uncomment the args.resume and args.dump line.

Acknowledgements

I thank Licheng Yu, Ronghang Hu, Varun Nagaraja for their help.

About

Repository for AAAI 2018 paper "Using Syntax for Referring Expression Recognition"

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published