This project implements a realtime YOLO sensor into CARLA and adds distance estimation to it with IPM and two stereoscopy methods. This project is based on CARLA code examples. Developed on Ubuntu 20.04 running on WSL2 + WSLg on Windows 11.
- Python 3.8.10 (other versions might work, but not tested)
- CUDA
- OpenCV 4.5.5 compiled with CUDA/DNN support
- You may use OpenCV without CUDA, but performance will be very poor
- Alternative method
- CARLA 0.9.12
- YOLO models:
- From Darknet repository:
- yolov3.cfg
- cocos.names
- From Pytorch-YoloV4
- yolov4-weigths
- yolov4.cfg
- Copy yolov3.weights, yolov4.weights, yolov3.cfg, yolov4.cfg and cocos.names to the
modelsfolder. - Execute
pip install -r requirements.txt
-
On Windows with WSL:
- Run CARLA server on Windows Host
- Execute
make run. It should automatically get the Host IP address and start the GUI application
-
On Linux or to connect to a localhost CARLA server (not tested):
- Start CARLA server
- Execute
make run-linux
Keyboard shortcuts:
- 1: Normal camera view
- 2: YoloV3 Sensor view
- 3: YoloV4 Sensor view
- 4: YoloV5 Sensor view
- 5: IPM Sensor view
- 6: YoloV3 Stereo Sensor view
- 7: YoloV4 Stereo Sensor view
- 8: YoloV5 Stereo Sensor view
- TAB: Change camera
- P: Start autopilot (if supported on map)
- u: Start dataset generation
- Other controls: check the terminal output for default CARLA controls
After starting the application you should see a window showing a car and the loaded map. You can manually control this vehicle or press P to use the autopilot. For the intended usage of this project it is recommended to use the autopilot, the Yolo Sensor View (press 1) and the dashcam view.
This video is on an older version, a new video comming soon
- Sometimes the YoloV3 and YoloV4 sensors will hang with a CUDA error
- Distance estimation only shows correct data on the initial camera position
- Changing Yolo versions during runtime degrade performance
- Dataset generation is unreliable, especially on sync mode. To generate a dataset use
run-async