Skip to content

RSVG: Exploring Data and Model for Visual Grounding on Remote Sensing Data, 2022

Notifications You must be signed in to change notification settings

ZhanYang-nwpu/RSVG-pytorch

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

98 Commits
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

RSVG: Exploring Data and Model for Visual Grounding on Remote Sensing Data

Author: Yang Zhan, Zhitong Xiong, Yuan Yuan

This is the offical dataset for paper "RSVG: Exploring Data and Model for Visual Grounding on Remote Sensing Data", Paper.

School of Artificial Intelligence, OPtics, and ElectroNics (iOPEN), Northwestern Polytechnical University

Please share a STAR ⭐ if this project does help

📢 News

Release the DIOR_RSVG dataset.

[2022/10/22]: Publish the manuscript on arXiv.

💬 Introduction

This is Multi-Granularity Visual Language Fusion (MGVLF) Network, the PyTorch source code of the paper "RSVG: Exploring Data and Model for Visual Grounding on Remote Sensing Data". It is built on top of the TransVG in PyTorch. Our method is a transformer-based method for visual grounding for remote sensing data (RSVG). It has achieved the SOTA performance in the RSVG task on our constructed RSVG dataset.

📦DIOR-RSVG Dataset

📦Statistics of the Visual Grounding Dataset

Dataset train val test Overall
Flickr30k [Paper] [Code] [Website] 29783 (94%) 1000 (3%) 1000 (3%) 31783
ReferItGame [Paper] [Website] 54127 (45%) 5842 (5%) 60103 (50%) 120072
RefCOCO [Paper][Code] 120624 (85%) 10834 (7%) 5657 (3%) 142210
RefCOCO+ [Paper][Code] 120191 (85%) 10758 (7%) 5726 (4%) 141564
GuessWhat [Paper] [Code] [Website] 70% 15% 15% 100%
Cops-Ref [Paper] [Code] 119603 (80.5%) 16524 (11%) 12586 (8.5%) 148713
KB-Ref [Paper] [Code] 31284 (72%) 4000 (10%) 8000 (18%) 43284
Ref-Reasoning [Paper] [Code] [Website] 721164 (91%) 36183 (4.6%) 34609 (4.4%) 791956
RSVG [Paper] [Website] 5505 (70%) 1201 (15%) 1227 (15%) 7933
DIOR-RSVG [Paper] [Dataset] 26991 (70%) 3829 (10%) 7500 (20%) 38320

🚀Network Architecture

👁️Requirements and Installation

We recommended the following dependencies.

  • Python 3.6.13
  • PyTorch 1.9.0
  • NumPy 1.19.2
  • cuda 11.1
  • opencv 4.5.5
  • torchvision

🔍Download Dataset

Download our constructed RSVG dataset files. We build the first large-scale dataset for RSVG, termed DIOR-RSVG, which can be downloaded from our Google Drive. The download link is available below:

https://drive.google.com/drive/folders/1hTqtYsC6B-m4ED2ewx5oKuYZV13EoJp_?usp=sharing

We expect the directory and file structure to be the following:

./                      # current (project) directory
├── data_loader.py      # Load data
├── main.py             # Main code for training, validation, and test
├── README.md
└── DIOR_RSVG/          # DIOR-RSVG dataset
    ├── Annotations/    # Query expressions and bounding boxes
    │   ├── 00001.xml/
    │   └── ..some xml files..
    ├── JPEGImages/     # Remote sensing images
    │   ├── 00001.jpg/
    │   └── ..some jpg files..
    ├── train.txt       # ID of training set    (26991)
    ├── val.txt         # ID of validation set  (3829)
    └── test.txt        # ID of test set        (7500)

📜Reference

If you found this code useful, please cite the paper. Welcome 👍Fork and Star👍, then I will let you know when we update.

@ARTICLE{10056343,
  author={Zhan, Yang and Xiong, Zhitong and Yuan, Yuan},
  journal={IEEE Transactions on Geoscience and Remote Sensing}, 
  title={RSVG: Exploring Data and Models for Visual Grounding on Remote Sensing Data}, 
  year={2023},
  volume={61},
  number={},
  pages={1-13},
  doi={10.1109/TGRS.2023.3250471}
  }

🙏Acknowledgments

Our DIOR-RSVG is constructed based on the DIOR remote sensing image dataset. We thank to the authors for releasing the dataset. Part of our code is borrowed from TransVG. We thank to the authors for releasing codes. I would like to thank Xiong zhitong and Yuan yuan for helping the manuscript. I also thank the School of Artificial Intelligence, OPtics, and ElectroNics (iOPEN), Northwestern Polytechnical University for supporting this work.

About

RSVG: Exploring Data and Model for Visual Grounding on Remote Sensing Data, 2022

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages