Skip to content

Teachers in concordance for pseudo-labeling of 3D sequential data

License

Notifications You must be signed in to change notification settings

ctu-vras/T-Concord3D

Repository files navigation

T-Concord3D Teachers in concordance for pseudo-labeling of 3D sequential data

T-Concord3D The source code of our work Teachers in concordance for pseudo-labeling of 3D sequential data img│center Proposed Concordance of teachers for pseudo- labeling of sequences. A set Dℓ of sequences with central frame labeled, and a larger set Du of unannotated ones, are available for training; 1⃝ Multiple offline teachers are trained with full supervision on Dℓ, each with a different temporal range towards future and past frames; 2⃝ The teachers are run on Du to produce pseudo-labels (PLs) for central frames; 3⃝ Sequences with the most confident PLs according to Concordance of teachers are selected, forming the pseudo- labeled set Dp. The white box depicts the discarded PLs; 4⃝ The student is trained on Dℓ ∪ Dp, to work online with past and current frames only.

Installation

Requirements

Data Preparation

SemanticKITTI

The dataset consists of 22 sequences with a split of sequences from 00 to 10 for training (08 is reserved for validation) and from 11 to 21 for testing. The dataset has two challenges, namely single-scan with only 19 class categories, and multi-scan with 25 class categories, including 19 from single-scan and 6 moving-object categories. For this experiment, we cut each training sequence (00 - 10, except 08) into two parts, the first 20% for human-labeled dataset Dℓ and the latter 80% for unlabeled dataset Du as shown bellow.

./	 
├── ...
└── path_to_data_shown_in_config/
    ├──Labeled-sequences (20% training sequence (00 - 10) for training sufexted with '20')
    │    ├── 0020/    #training
    │    │   ├── velodyne/	
    │    │   │	├── 000000.bin
    │    │   │	├── 000001.bin
    │    │   │	└── ...
    │    │   ├── labels/ 
    │    │   │   ├── 000000.label
    │    │   │   ├── 000001.label
    │    │   │   └── ...
    │    │   ├── calib.txt
    │    │   ├── poses.txt
    │    │   └── times.txt
    │    ├── 0120 ...
    │    │   └── ...
    │    ├── 1020/
    │    │   └── ...
    │    │
    │    └── 08/ # for validation
    │        ├── velodyne/	
    │        │	├── 000000.bin
    │        │	├── 000001.bin
    │        │	└── ...
    │        ├── labels/ 
    │        │   ├── 000000.label
    │        │   ├── 000001.label
    │        │   └── ...
    │        ├── calib.txt
    │        ├── poses.txt
    │        └── times.txt
    └──Unlabeled-sequences (80% training sequence (00 - 10) for pseudo-labeling sufexted with '80')
        ├── 0080/   # 0080 - 1080 for pseudo-labeling
        │   ├── velodyne/	
        │   │	├── 000000.bin
        │   │	├── 000001.bin
        │   │	└── ...
        │   ├── calib.txt
        │   ├── poses.txt
        │   └── times.txt
        ├── 0180 ...
        │   └── ...
        └── 1080/
            └── ...

WOD

  • Coming soon
./
├── 
├── ...
└── path_to_data_shown_in_config/
		├──Training
		│    └──sequences (...)
		├──Validation
		│    └──sequences (...)
		└──Testing
		     └──sequences (...)

Training

Training Teacher models with access to future frame (Privileged Information)

  1. modify the config/semantickitti/semantickitti_T3_3_s20.yaml with your custom settings. We provide a sample yaml for SemanticKITTI multi-frame (both past and future) aggregation
  2. train the network by running
    sh script/sematickitti/run_train_T3_3_s20.sh
    

Generate Concordance of Teachers using a set of teachers with access to different future frame (Privileged Information) temporal window

  • e.g. \mathcal{T}^{1, ... ,3 } = {T-1,1, T-2,2, T-3,3},
  1. generate pseudo labels for the 80/% unlabeled data using the trained Teacher models (e.g., T-1_1, T-2_2, T-3_3)
    • run
    sh script/sematickitti/run_infer_T3_3_s20.sh
    
  2. generate concordance of teachers
    • run
    sh script/sematickitti/generate_T_concord.sh
    

Training Student models with distilled knowledge form Concordance of teachers

  1. modify the config/semantickitti/semantickitti_S0_0_T11_33_ssl_s20_p80.yaml with your custom settings. We provide a sample yaml for SemanticKITTI multi-frame (both past and future) aggregation
  2. train the network by running
    sh script/sematickitti/run_train_f0_0_T11_33_ssl_s20_p80.sh
    

Testing

  1. modify the config/semantickitti/semantickitti_S0_0_T11_33_ssl_s20_p80.yaml with your custom settings. We provide a sample yaml for SemanticKITTI multi-frame (both past and future) aggregation
  2. train the network by running
    sh script/sematickitti/run_test_S0_0_T11_33_ssl_s20_p80.sh
    

Pretrained Models

-- Pretrained model for SemanticKITTI (soon)

-- For Waymo Open Dataset (WOD), please refer to WOD-GUIDE

TODO List

  • Provided Inference/test code for submission to leaderboard SemanticKITTI.
  • Support Future-frame supervision semantic segmentation.
  • Support Concordance of Teachers with Privilege Information.
  • Support Knowledge Distillation on single-frame and multi-frame semantic segmentation .
  • Release pretrained model for semanticKITTI.
  • Release data preparation code.
  • Integrate Teachers in Concordance for LiDAR 3D Object Detection into the codebase.
  • Release pretrained model for WOD.

Reference

If you find our work useful in your research, please consider citing our paper:

@article{gebrehiwot2022teachers,
  title={Teachers in concordance for pseudo-labeling of 3D sequential data},
  author={Gebrehiwot, Awet Haileslassie and Vacek, Patrik and Hurych, David and Zimmermann, Karel and Perez, Patrick and Svoboda, Tom{\'a}{\v{s}}},
  journal={arXiv preprint arXiv:2207.06079},
  year={2022}
}

Acknowledgments

  • This work was supported in part by OP VVV MEYS funded project CZ.02.1.01/0.0/0.0/16 019/0000765 “Research Center for Informatics”, and by Grant Agency of the CTU Prague under Project SGS22/111/OHK3/2T/13. Authors want to thank Valeo company for a support.
  • We thank for the opensource codebase, Cylinder3D and spconv V2.0

About

Teachers in concordance for pseudo-labeling of 3D sequential data

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published