Skip to content

[NeurIPS2023] 3D-OWIS is capable of detecting unknown instances in inference, and progressively learning novel classes in the process of training.

License

Notifications You must be signed in to change notification settings

aminebdj/3D-OWIS

Repository files navigation

3D Indoor Instance Segmentation in an Open-World (NeurIPS 2023)

Mohamed El Amine Boudjoghra1, Salwa K. Al Khatib1, Jean Lahoud1, Hisham Cholakkal1, Rao Muhammad Anwer1,2, Salman Khan1,3, Fahad Khan1,4

1Mohamed Bin Zayed University of Artificial Intelligence (MBZUAI) 2Aalto University 3Australian National University 4Linköping University

paper paper

teaser.mp4

News

  • 25 September 2023: 3D-OWIS paper released on arXiv. 📝
  • 25 September 2023: Code and splits released. 💻
  • 22 September 2023: Paper accepted into NeurIPS2023. 🥳

Abstract

We formulate the problem of 3D point cloud instance segmentation in the open world, where we relax the assumption of a closed setting of classes during the training. We propose an auto-labeling scheme to produce pseudo-labels during training and induce separation to separate known and unknown category labels. We further improve the pseudo-labels quality at inference by adjusting the unknown class probability based on the objectness score distribution. We also introduce carefully curated open-world splits leveraging realistic scenarios based on inherent object distribution, region-based indoor scene exploration and randomness aspect of open-world classes. Extensive experiments reveal the efficacy of the proposed contributions leading to promising open-world 3D instance segmentation performance.

architecture

Figure I: Proposed open-world 3D instance segmentation pipeline.

Qualitative results



teaser

Figure II: The performance of 3D-OWIS in masking unknown instances is shown in the above figure. The known classes are assigned the green color, while the unknown and background objects are assigned blue and gray respectively.

teaser

Figure III: 3D-OWIS is also capable of progressively learning novel classes, as shown in the following figure, once thier labels are introduced in the training process.

Navigate in the scenes

Kindly click on the icon to navigate in the scenes

Scene 1 Scene 2 Scene 3
3D-OWIS
Ground truth
Input scene

Comparative visualization

we show the progression of learning novel classes throughout the three tasks, and a comparison to the ground truth for our model with and without PC(Probability Correction) and CT(Confidence Threshold).

Installation guide

Kindly check Installation guide on how to setup the Conda environment and how to preprocess ScanNet200 dataset.

Use our splits with another 3D instance segmentation model

You can import the labels of the PREVIOUSLY_KNOWN, CURRENTLY_KNOWN, and UNKNOWN classes from ./datasets/scannet200/owis_splits.py for usage with another codebase.

Example: the following code will load the previously known, currently known, and unknown classes for split A task1

from datasets.scannet200.owis_splits import PREV_KNOWN_CLASSES_LABELS, KNOWN_CLASSES_LABELS, UNKNOWN_CLASSES_LABELS

OW_task = 'task1'
split = 'A'

PREVIOUSLY_KNOWN = PREV_KNOWN_CLASSES_LABELS[split][OW_task]
CURRENTLY_KNOWN = KNOWN_CLASSES_LABELS[split][OW_task]
UNKNOWN = UNKNOWN_CLASSES_LABELS[split][OW_task]

Training and testing 🚆

For training a model <MODEL> with a split <SPLIT> please run the following command:

NB: <SPLIT> takes the values 'A', 'B', or 'C'.

We provide the scripts for reproducing the results for the models 3d_owis, and oracle.

sh scripts/train_<MODEL>/<SPLIT>/train.sh

For testing a task <TASK> for a specific split <SPLIT> please run the following command:

NB: <TASK> takes the values 'task1', 'task2', or 'task3'

Important for inference: please make sure to not forget to specify the <EXPERIMENT_NAME>, <SPLIT>, and <TASK> when running the inference, as the prototypes are loaded from the path saved/<EXPERIMENT_NAME>/<SPLIT>/<TASK>/ to perform the probability correction.

python main_instance_segmentation.py \
general.checkpoint='saved/<EXPERIMENT_NAME>/<SPLIT>/<TASK>/CHECKPOINT_NAME.ckpt' \
general.experiment_name=<EXPERIMENT_NAME> \
general.split=<SPLIT> \
general.OW_task=<TASK> \
general.margin=<CLUSTERING_MARGIN_USED_DURING_TRAINING> \
general.correct_unknown_cls_prob=True \
general.train_mode=false

Trained checkpoints and output predictions

Checkpoints Clustering Margin
3D-OWIS
A :1.0, B :3.0, C :1.0
3D-OWIS - PC - CT
A :1.0, B :3.0, C :1.0
Oracle
A :1.0, B :3.0, C :1.0

BibTeX 🙏

@inproceedings{boudjoghra20233d,
 author = {Mohamed El Amine Boudjoghra and Salwa K. Al Khatib and Jean Lahoud and Hisham Cholakkal and Rao Muhammad Anwer and Salman Khan and Fahad Khan},
 booktitle = {Advances in Neural Information Processing Systems},
 title = {3D Indoor Instance Segmentation in an Open-World},
 year = {2023}
}