Skip to content

Obj2Seq: Formatting Objects as Sequences with Class Prompt for Visual Tasks (NeurIPS2022)

License

Notifications You must be signed in to change notification settings

CASIA-IVA-Lab/Obj2Seq

Repository files navigation

Obj2Seq: Formatting Objects as Sequences with Class Prompt for Visual Tasks

Introduction

This repository is an official implementation of the Obj2Seq. Obj2Seq takes objects as basic units, and regards most object-level visual tasks as sequence generation problems of objects. It first recognizes objects of given categories, and then generates a sequence to describe each of these objects. Obj2Seq is able to flexibly determine input categories and the definition of output sequences to satisfy customized requirements, and be easily extended to different visual tasks.

Obj2Seq: Arxiv | Github | Gitee

Obj2Seq

Main Results

All results are trained with a ResNet-50 backbone.

Object Detection

Epochs Params(M) $AP$ Model
DeformableDETR$^\dagger$ 50 40 44.6 model
Obj2Seq 50 40 45.7 model
+ iterative box refine 50 42 46.7 model

$^\dagger$ We convert official DeformableDETR checkpoint with this script.

Human Pose Estimation

Epochs Params(M) $AP_{box}$ $AP_{kps}$ Config/Model
Baseline 50 40 55.4 57.9 model
Obj2Seq 50 40 55.4 61.2 model
Obj2Seq 150 40 58.1 65.1 model

You may also download these models from BaiduNetdisk.

Instructions

See GET_STARTED.md.

Citation

If you find this project useful for your research, please consider citing this paper.

@inproceedings{
chen2022objseq,
title={Obj2Seq: Formatting Objects as Sequences with Class Prompt for Visual Tasks},
author={Zhiyang Chen and Yousong Zhu and Zhaowen Li and Fan Yang and Wei Li and Haixin Wang and Chaoyang Zhao and Liwei Wu and Rui Zhao and Jinqiao Wang and Ming Tang},
booktitle={Advances in Neural Information Processing Systems},
editor={Alice H. Oh and Alekh Agarwal and Danielle Belgrave and Kyunghyun Cho},
year={2022},
url={https://openreview.net/forum?id=cRNl08YWRKq}
}

Acknowledgement

Our repository is mainly built upon DETR, Deformable-DETR and Anchor-DETR. We also refer

About

Obj2Seq: Formatting Objects as Sequences with Class Prompt for Visual Tasks (NeurIPS2022)

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published