LiDARSeg3D

A repository for LiDAR 3D semantic segmentation in autonomous driving scenarios.

Also the official implementations of our ECCV 2022 paper (Self-Distillation for Robust LiDAR Semantic Segmentation in Autonomous Driving) and CVPR 2023 paper (MSeg3D: Multi-modal 3D Semantic Segmentation for Autonomous Driving).

News

• [2023-03-28] Our LiDAR-Camera based method MSeg3D (MSeg3D: Multi-modal 3D Semantic Segmentation for Autonomous Driving) is accepted at CVPR 2023. The paper, supplementary material, and code are available.

• [2022-07-19] Our method (MSeg3D) ranks 2nd on the official nuScenes lidar-seg leaderboard and achieves 81.1 mIoU with slight differences among the highly competitive Top3 submissions.

• [2022-07-04] Our LiDAR-only method SDSeg3D (Self-Distillation for Robust LiDAR Semantic Segmentation in Autonomous Driving) is accepted as a poster paper at ECCV 2022. Paper.

• [2022-06-20] Our multi-modality solution for 3D semantic segmentation won the 2nd place in the 3D semantic segmentation track of CVPR 2022 Waymo Open Dataset Challenges.

Highlights

• Simple: Modules and pipelines can be instantiated via cfg files like mmsegmentation, but more easily applicable to LiDAR 3D point clouds for voxelization, sparse convolution, devoxelization, etc.

• Extensible: Simple replacement and integration for any network components in your novel algorithms. Smooth compatibility for 3D object detector such as CenterPoint, since we try our best to preserve the features inherited from CenterPoint.

• Fast and Accurate: Accelerated by 3D sparse convolution with top performance achived on SemantiKITTI, nuScenes, and Waymo benchmarks.

• Support for multiple datasets and multimodality: SemanticKITTI，nuScenes and Waymo datasets. LiDAR point cloud and multi-camera images.

Methods

MSeg3D

MSeg3D: Multi-modal 3D Semantic Segmentation for Autonomous Driving
Jiale Li, Hang Dai, Hao Han, and Yong Ding

SDSeg3D

Self-Distillation for Robust LiDAR Semantic Segmentation in Autonomous Driving
Jiale Li, Hang Dai, and Yong Ding

Use LiDARSeg3D

Installation

Please follow INSTALL to set up libraries needed for distributed training and sparse convolution.

Benchmark Evaluation and Training

Please refer to GETTING_START to prepare the data in advance. Then follow the instruction there to play with the segmentation configurations included in configs.

Acknowlegement

This project is mainly constructed on CenterPoint as well as multiple great opensourced codebases. We list some notable examples below.

• CenterPoint
• PointAugmenting
• mmsegmentation
• mmdetection
• OpenPCDet
• Cylinder3D
• det3d
• second.pytorch
• mmcv

Citation

@inproceedings{mseg3d_cvpr2023,
author    = {Jiale Li and
            Hang Dai and
            Hao Han and
            Yong Ding},
title     = {MSeg3D: Multi-modal 3D Semantic Segmentation for Autonomous Driving},
booktitle = {CVPR},
pages     = {21694--21704},
year      = {2023},
}

@inproceedings{sdseg3d_eccv2022,
author    = {Jiale Li and
            Hang Dai and
            Yong Ding},
title     = {Self-Distillation for Robust {LiDAR} Semantic Segmentation in Autonomous Driving},
booktitle = {ECCV},
pages     = {659--676},
year      = {2022},
}