Skip to content

Wild Visual Navigation: A system for fast traversability learning via pre-trained models and online self-supervision

License

Notifications You must be signed in to change notification settings

leggedrobotics/wild_visual_navigation

Repository files navigation


Wild Visual Navigation

Features • Citing • Quick Start • Setup • Demos • Development

MIT License formatting

Overview

This package implements the Wild Visual Navigation (WVN) system presented in Frey & Mattamala et al. "Fast Traversability Estimation for Wild Visual Navigation" (2023) and later extended in Mattamala & Frey et al. "Wild Visual Navigation: Fast Traversability Learning via Pre-Trained Models and Online Self-Supervision" (2024). It implements a visual, self-supervised traversability estimation system for mobile robots, trained online after a few minutes of human demonstrations in the field.

Dino

Features

  • Implementation of the full WVN system in pure Python
  • Quick start demos for online training in simulation, as well as scripts for inference using pre-trained models
  • Robot integration packages for ANYmal and Jackal robots using ROS 1
  • Integration into elevation_mapping_cupy

Dino

Citing this work

@INPROCEEDINGS{frey23fast, 
  AUTHOR    = {Jonas Frey AND Matias Mattamala AND Nived Chebrolu AND Cesar Cadena AND Maurice Fallon AND Marco Hutter}, 
  TITLE     = {{Fast Traversability Estimation for Wild Visual Navigation}}, 
  BOOKTITLE = {Proceedings of Robotics: Science and Systems}, 
  YEAR      = {2023}, 
  ADDRESS   = {Daegu, Republic of Korea}, 
  MONTH     = {July}, 
  DOI       = {10.15607/RSS.2023.XIX.054} 
} 

If you are also building up on the STEGO integration or using the pre-trained models for comparison, please cite:

@INPROCEEDINGS{mattamala24wild, 
  AUTHOR    = {Jonas Frey AND Matias Mattamala AND Libera Piotr AND Nived Chebrolu AND Cesar Cadena AND Georg Martius AND Marco Hutter AND Maurice Fallon}, 
  TITLE     = {{Wild Visual Navigation: Fast Traversability Learning via Pre-Trained Models and Online Self-Supervision}}, 
  BOOKTITLE = {under review for Autonomous Robots}, 
  YEAR      = {2024}
} 

If you are using the elevation_mapping_cupy integration:

@INPROCEEDINGS{erni23mem,
  AUTHOR={Erni, Gian and Frey, Jonas and Miki, Takahiro and Mattamala, Matias and Hutter, Marco},
  TITLE={{MEM: Multi-Modal Elevation Mapping for Robotics and Learning}}, 
  BOOKTITLE={2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)}, 
  YEAR={2023},
  PAGES={11011-11018},
  DOI={10.1109/IROS55552.2023.10342108}
}

Dino

Quick start

We prepared a quick-start demo using a simulated Jackal robot. The demo runs on Docker, so no system dependencies are required. Please check the full instructions here

Overview

Dino

Setup

We recommend following the aforementioned Docker instructions as well as inspecting the Dockerfile for a clean, system-independent setup.

Otherwise, the next steps provide specific instructions to setup WVN on different systems.

Requirements

The next steps assume you have the following hardware & software setup.

  • ROS 1 Noetic
  • CUDA-enabled GPU
  • CUDA drivers (we use 12.0)

Minimal setup

These are the minimum requirements to use the WVN scripts (no robot integration).

Installation

First clone the WVN and our STEGO reimplementation.

mkdir ~/git && cd ~/git 
git clone git@github.com:leggedrobotics/wild_visual_navigation.git
git clone git@github.com:leggedrobotics/self_supervised_segmentation.git
./self_supervised_segmentation/models/download_pretrained.sh

(Recommended) Create new virtual environment.

mkdir ~/.venv
python -m venv ~/venv/wvn
source ~/venv/wvn/bin/activate

Install the wild_visual_navigation package.

cd ~/git
pip3 install -e ./wild_visual_navigation
pip3 install -e ./self_supervised_segmentation

Execution

Please refer to the Demos section below.

ROS setup

The following steps are required for a full installation, including the deployment tools. This enables the appropriate use of the ANYmal rosbags, enabling the robot model visualization and other deployment tools.

Installation

# Create new catkin workspace
source /opt/ros/noetic/setup.bash
mkdir -r ~/catkin_ws/src && cd ~/catkin_ws/src
catkin init
catkin config --extend /opt/ros/noetic
catkin config --cmake-args -DCMAKE_BUILD_TYPE=RelWithDebInfo

# Clone repos
git clone git@github.com:ANYbotics/anymal_d_simple_description.git
git clone git@github.com:ori-drs/procman_ros.git

# Symlink WVN-repository
ln -s ~/git/wild_visual_navigation ~/catkin_ws/src

# Dependencies
rosdep install -ryi --from-paths . --ignore-src

# Build
cd ~/catkin_ws
catkin build anymal_d_simple_description
catkin build wild_visual_navigation_ros

# Source
source /opt/ros/noetic/setup.bash
source ~/catkin_ws/devel/setup.bash

Execution

After successfully building the ROS workspace, you can run the entire pipeline by either using the launch file or by running the nodes individually. Open multiple terminals and run the following commands:

  • Run wild_visual_navigation
roslaunch wild_visual_navigation_ros wild_visual_navigation.launch
  • (ANYmal replay only) Load ANYmal description for RViZ
roslaunch anymal_d_simple_description load.launch
  • (ANYmal replay only) Replay Rosbag:
robag play --clock path_to_mission/*.bag
  • RViz:
roslaunch wild_visual_navigation_ros view.launch
  • Debugging (sometimes it is desirable to run the two nodes separately):
python wild_visual_navigation_ros/scripts/wvn_feature_extractor_node.py
python wild_visual_navigation_ros/scripts/wvn_learning_node.py

Dino

Demos

Inference of pre-trained model

We provide the python3 quick_start.py script to inference traversability from images within the input folder (assets/demo_data/*.png), given a pre-trained model checkpoint (assets/checkpoints/model_name.pt, you can obtain them from Google Drive). The script stores the result in the provided output folder (results/demo_data/*.png).

python3 quick_start.py

# python3 quick_start.py --help for more CLI information
# usage: quick_start.py  [-h] [--model_name MODEL_NAME] [--input_image_folder INPUT_IMAGE_FOLDER]
#        [--output_folder_name OUTPUT_FOLDER_NAME] [--network_input_image_height NETWORK_INPUT_IMAGE_HEIGHT] 
#        [--network_input_image_width NETWORK_INPUT_IMAGE_WIDTH] [--segmentation_type {slic,grid,random,stego}]
#        [--feature_type {dino,dinov2,stego}] [--dino_patch_size {8,16}] [--dino_backbone {vit_small}]
#        [--slic_num_components SLIC_NUM_COMPONENTS] [--compute_confidence] [--no-compute_confidence]
#        [--prediction_per_pixel] [--no-prediction_per_pixel]

Online adaptation from rosbags

To quickly test out the online training and adaption we provide some example rosbags (GDrive), collected with our ANYmal D robot. These can be tested using the ROS instructions

Here we provide some examples for the different sequences:

MPI Outdoor MPI Indoor Bahnhofstrasse Bike Trail
MPI Outdoor MPI Indoor Bahnhofstrasse Mountain Bike
MPI Outdoor MPI Indoor Bahnhofstrasse Mountain Bike

Dino

Development

Lastly, we provide some general guidelines for development. These could be useful to test WVN with your own robot platform.

Repository structure

The WVN repo is structured in different folders, which we explain in the following figure:

📦wild_visual_navigation  
 ┣ 📂assets
     ┣ 📂demo_data                            # Example images
        ┣ 🖼 example_images.png
        â”— ....
     ┗ 📂checkpoints                          # Pre-trained model checkpoints (must be downloaded from Google Drive)
        ┣ 📜 mountain_bike_trail_v2.pt
        â”— ....
 ┣ 📂docker                                   # Quick start docker container
 ┣ 📂results   
 ┣ 📂test   
 ┣ 📂wild_visual_navigation                   # Core, ROS-independent implementation of WVN
 ┣ 📂wild_visual_navigation_anymal            # ROS1 ANYmal helper package 
 ┣ 📂wild_visual_navigation_jackal            # ROS1 Jackal simulation example
 ┣ 📂wild_visual_navigation_msgs              # ROS1 message definitions
 ┣ 📂wild_visual_navigation_ros               # ROS1 nodes for running WVN 
    ┗ 📂scripts                               
       ┗ 📜 wvn_feature_extractor_node.py     # Main process for feature extraction and inference
       ┗ 📜 wvn_learning_node.py              # Main process that generates supervision signals and the online training loop
 ┗ 📜 quick_start.py                          # Inference demo from pre-trained checkpoints

Adapting WVN for your own robot

We recommend making a new ROS package to implement the overlays to run WVN with your own robot platform. We suggest inspecting the wild_visual_navigation_jackal as a reference.

In a nutshell, you need to configure:

Further notes

Here we provide additional details if you want to contribute.

Install pre-commit

pip3 install pre-commit
cd wild_visual_navigation && python3 -m pre_commit install
cd wild_visual_navigation && python3 -m pre_commit run

Code formatting

# for formatting
pip install black
black --line-length 120 .
# for checking lints
pip install flake8
flake8 .

Code format is checked on push.

Testing

Introduction to pytest.

pytest

Open-sourcing

Generating headers

pip3 install addheader

# If your are using zsh otherwise remove \
addheader wild_visual_navigation -t header.txt -p \*.py --sep-len 79 --comment='#' --sep=' '
addheader wild_visual_navigation_ros -t header.txt -p \*.py -sep-len 79 --comment='#' --sep=' '
addheader wild_visual_navigation_anymal -t header.txt -p \*.py --sep-len 79 --comment='#' --sep=' '

addheader wild_visual_navigation_ros -t header.txt -p \*CMakeLists.txt --sep-len 79 --comment='#' --sep=' '
addheader wild_visual_navigation_anymal -t header.txt -p \*.py -p \*CMakeLists.txt --sep-len 79 --comment='#' --sep=' '

Releasing ANYmal data

rosrun procman_ros sheriff -l ~/git/wild_visual_navigation/wild_visual_navigation_anymal/config/procman/record_rosbags.pmd --start-roscore 
rosbag_play --tf --sem --flp --wvn  mission/*.bag