One model is enough: Toward multiclass weakly supervised remote sensing image semantic segmentation

Zhenshi Li, Xueliang Zhang, and Pengfeng Xiao

Introduction | Usage| Visual results| Acknowledgement | Statement

Introduction

This is the official repository for the paper “One model is enough: Toward multiclass weakly supervised remote sensing image semantic segmentation”, based on SEAM and mmsegmentation.

Abstract: Semantic segmentation of remote sensing images is effective for large-scale land cover mapping, which heavily relies on a large amount of training data with laborious pixellevel labeling. Weakly supervised semantic segmentation (WSSS) based on image-level labels has attracted intensive attention due to its easy availability. However, existing image-level WSSS methods for remote sensing images mainly focus on binary segmentation, which are difficult to apply to multiclass scenarios. This study proposes a comprehensive framework for image-level multiclass WSSS of remote sensing images, consisting of appropriate image-level label generation, high-quality pixel-level pseudo mask generation, and segmentation network iterative training. Specifically, a training sample filtering method, as well as a dataset cooccurrence evaluation metric, is proposed to demonstrate proper image-level training samples. Leveraging multiclass class activation maps, an uncertainty-driven pixel-level weighted mask is proposed to relieve the overfitting of labeling noise in pseudo masks when training the segmentation network. Extensive experiments demonstrate that the proposed framework can achieve high-quality multiclass WSSS performance with image-level labels, which can attain 94.23% and 90.77% of the IoUs from pixel-level labels for the ISPRS Potsdam and Vaihingen datasets, respectively. Beyond that, for the DeepGlobe dataset with more complex landscapes, the WSSS framework can achieve an accuracy close to 99% of the fully supervised case. Additionally, we further demonstrate that compared to adopting multiple binary WSSS models, directly training a multiclass WSSS model can achieve better results, which can provide new thoughts to achieve WSSS of remote sensing images for multiclass application scenarios.

Usage

Classification network

Several classification networks are adopted in our OME inplement. We show our SEAM-based network here.

1. co-occurrence matrix generation: Myutils/calculate_cooccur_matrix.py
2. SEAM training: train_SEAM.py
3. SEAM inference: infer_SEAM.py
4. uncertainty generation from CAMs: Myutils/generate_uncertainty_from_cam.py

The ImageNet pretrained model weights can be downloaded from SEAM.

Segmentation network

Please refer the segmentation phase to URN based on mmsegmentation, with a improved reweighting operation as in our paper.

Visual results

Results on the ISPRS Potsdam dataset

Results on the ISPRS Vaihingen dataset

Results on the Deepglobe dataset

Acknowledgement

• Many thanks to the following repos: SEAM, URN, and mmsegmentation.

Statement

• Please cite our paper if our work is useful to your research.

    @article{li2023one,
      title={One model is enough: Toward multiclass weakly supervised remote sensing image semantic segmentation},
      author={Li, Zhenshi and Zhang, Xueliang and Xiao, Pengfeng},
      journal={IEEE Transactions on Geoscience and Remote Sensing},
      year={2023},
      publisher={IEEE}
    }
  

• Any questions please contact LZhenShi (Lzhenshi@outlook.com).