Skip to content

kimiyoung/review_net

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

89 Commits
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Review Network for Caption Generation

Image Captioning on MSCOCO

You can use the code in this repo to genearte a MSCOCO evaluation server submission with CIDEr=0.96+ with just a few hours.

No fine-tuning required. No fancy tricks. Just train three end-to-end review networks and do an ensemble.

  • Feature extraction: 2 hours in parallel
  • Single model training: 6 hours
  • Ensemble model training: 30 mins
  • Beam search for caption generation: 3 hours in parallel

Below is a comparison with other state-of-the-art systems (with according published papers) on the MSCOCO evaluation server:

Model BLEU-4 METEOR ROUGE-L CIDEr Fine-tuned Task specific features
Attention 0.537 0.322 0.654 0.893 No No
MS Research 0.567 0.331 0.662 0.925 No Yes
Google NIC 0.587 0.346 0.682 0.946 Yes No
Semantic Attention 0.599 0.335 0.682 0.958 No Yes
Review Net 0.597 0.347 0.686 0.969 No No

In the diretcory image_caption_online, you can use the code therein to reproduce our evaluation server results.

In the directory image_caption_offline, you can rerun experiments in our paper using offline evaluation.

Code Captioning

Predicting comments for a piece of source code is another interesting task. In the repo we also release a dataset with train/dev/test splits, along with the code of a review network.

Check out the directory code_caption.

Below is a comparison with baselines on the code captioning dataset:

Model LLH CS-1 CS-2 CS-3 CS-4 CS-5
LSTM Language Model -5.34 0.2340 0.2763 0.3000 0.3153 0.3290
Encoder-Decoder -5.25 0.2535 0.2976 0.3201 0.3367 0.3507
Encoder-Decoder (Bidir) -5.19 0.2632 0.3068 0.3290 0.3442 0.3570
Attentive Encoder-Decoder (Bidir) -5.14 0.2716 0.3152 0.3364 0.3523 0.3651
Review Net -5.06 0.2889 0.3361 0.3579 0.3731 0.3840

References

This repo contains the code and data used in the following paper:

Review Networks for Caption Generation

Zhilin Yang, Ye Yuan, Yuexin Wu, Ruslan Salakhutdinov, William W. Cohen

NIPS 2016

About

Review Network for Caption Generation

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published