Skip to content

Exploring Structured Semantic Prior for Multi Label Recognition with Incomplete Labels [CVPR 2023]

Notifications You must be signed in to change notification settings

jameslahm/SCPNet

Repository files navigation

Official PyTorch Implementation of SCPNet, from the following paper:

Exploring Structured Semantic Prior for Multi Label Recognition with Incomplete Labels. CVPR 2023.

Zixuan Ding*, Ao Wang*, Hui Chen†, Qiang Zhang, Pengzhang Liu, Yongjun Bao, Weipeng Yan, Jungong Han,
Xidian University, Tsinghua University, JD.com

Abstract

Multi-label recognition (MLR) with incomplete labels is very challenging. Recent works strive to explore the imageto-label correspondence in the vision-language model, i.e., CLIP, to compensate for insufficient annotations. In spite of promising performance, they generally overlook the valuable prior about the label-to-label correspondence. In this paper, we advocate remedying the deficiency of label supervision for the MLR with incomplete labels by deriving a structured semantic prior about the label-to-label correspondence via a semantic prior prompter. We then present a novel Semantic Correspondence Prompt Network (SCPNet), which can thoroughly explore the structured semantic prior. A Prior-Enhanced Self-Supervised Learning method is further introduced to enhance the use of the prior. Comprehensive experiments and analyses on several widely used benchmark datasets show that our method significantly outperforms existing methods on all datasets, well demonstrating the effectiveness and the superiority of our method.

Credit to previous work

This repository is built upon the code base of ASL and SPLC, thanks very much!

Performance

Dataset mAP Ckpt Log
COCO 76.4 scpnet+coco.ckpt scpnet+coco.txt
VOC 91.2 scpnet+voc.ckpt scpnet+voc.txt
NUSWIDE 62.0 scpnet+nuswide.ckpt scpnet+nuswide.txt
CUB 25.7 scpnet+cub.ckpt scpnet+cub.txt

Training

COCO

python train.py -c configs/scpnet+coco.yaml

VOC

python train.py -c configs/scpnet+voc.yaml

NUSWIDE

python train.py -c configs/scpnet+nuswide.yaml

CUB

python train.py -c configs/scpnet+cub.yaml

Inference

Note: Please place the pretrained checkpoint to checkpoints/scpnet+coco/round1/model-highest.ckpt

COCO

python train.py -c configs/scpnet+coco.yaml -t -r 1

VOC

python train.py -c configs/scpnet+voc.yaml -t -r 1

NUSWIDE

python train.py -c configs/scpnet+nuswide.yaml -t -r 1

CUB

python train.py -c configs/scpnet+cub.yaml -t -r 1

Citation

@inproceedings{ding2023exploring,
  title={Exploring Structured Semantic Prior for Multi Label Recognition with Incomplete Labels},
  author={Ding, Zixuan and Wang, Ao and Chen, Hui and Zhang, Qiang and Liu, Pengzhang and Bao, Yongjun and Yan, Weipeng and Han, Jungong},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={3398--3407},
  year={2023}
}

About

Exploring Structured Semantic Prior for Multi Label Recognition with Incomplete Labels [CVPR 2023]

Resources

Stars

Watchers

Forks

Packages

No packages published

Languages