Exercise: getting familiar with the pipeline
exercise.mp4
In order to create your own gaussian splat for the web, we have to go through some steps. That require setup on your local machine and the cluster from the university.
Issues regarding SSH Conection
Use tmux to run a task on the server without relying on the SSH connection. After connecting to the server run the following commant to start a tmux session:
tmux new -s mysession
After starting the tmux session you can run the training command or installation in the tmux session. You can detach from the tmux session by pressing ctrl + b
and then d
. Or simply close the terminal. Your tmux session will still run in the background.
After disconnecting from the server you can reconnect to the server and reattach to the tmux session. First connect to the server and then run the following command to list all tmux sessions:
tmux attach-session -t mysession
You can also have multiple sessions, just use different names for the sessions.
First, we need to setup our local and server environments. We will setup our local environment to run the web server and display the gaussian splat in our web browser. For the server setup we will use the cluster of computer graphics at the University of Tübingen. We will optimize the gaussian splats in the cluster and download the optimized splats to display them in our web browser.
![Pipeline](https://private-user-images.githubusercontent.com/9963865/325548769-4cae9848-95b7-4f88-a42e-09e14cb501ba.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQ0NDg4NzQsIm5iZiI6MTcyNDQ0ODU3NCwicGF0aCI6Ii85OTYzODY1LzMyNTU0ODc2OS00Y2FlOTg0OC05NWI3LTRmODgtYTQyZS0wOWUxNGNiNTAxYmEucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDgyMyUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDA4MjNUMjEyOTM0WiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9ZTEwNTZkMTRiMGVmNjBiMTVkNTNhMjk3NjViZDM2ZmQ3ZjQ3YmQxNmQ2ZmE1ZjZkMzQ3ZDQ0YjMwYWQ2NWExZCZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.y9NXtBRbXKF8inVd_gc3jGqOqnr4nPNw-BjJofTzpjM)
This installtion process might be lengthy and require some time. However, it is important that every one of you once understands the full pipeline of the project. Help is always available in the discord channel. Help each other first and if no one can help, ask me.
We need to prepare our local environment to run the web server and display the gaussian splat in our web browser. We will use React in combination with React Three Fiber to do so. I will provide the full code for that, so you only have to setup your environment to allow the code to run.
This tutorial is optimized for macOS and Linux. If you are using Windows, you might have to adjust some commands, I highly recommend using WSL to run a Linux distribution on your Windows machine.
First we need to install Node.js. You can download the installer from the website or use a package manager like brew on macOS. You can also install node over the node version manager nvm to manage multiple node versions on your machine (recommended).
Check if you have installed node and npm by running the following commands in your terminal:
node -v
npm -v
Next, we need to install yarn as our package manager of choice (replacing nmp). You can install yarn with npm by running the following command:
npm install -g yarn
After that you can check if yarn is installed by running:
yarn -v
Next, we need to clone the repository to our local machine. You can do so by running the following command in your terminal:
git clone git@github.com:cgtuebingen/hyperrealistic_indoor_streetview.git
After cloning the repository we need to install the dependencies for the web application. You can do so by running the following command in the root directory of the repository:
cd ./hyperrealistic_indoor_streetview
yarn install
If everything went well you can start the web server by running the following command in the root directory of the repository:
yarn dev
Goto http://localhost:5173/
(might be different for you) in your web browser to see the web application running. You should see a first gaussian splatting scene I created for you.
We will use the cluster of computer graphics at the University of Tübingen to train our gaussian splatting scene. On the server we will first setup our python environment and then install nerfstudio to train our gaussian splatting scene.
We have a full tutorial and cheatsheet on how to connect to the cluster. As I cannot share it publicly, you will find the ssh_and_vnc.pdf
in the web page linked in the Discord channel or email you received. Complete steps 1-4 (including 4) before continuing with the next steps.
Important: Complete the steps 1-4 (including 4) before continuing with the next steps.
Next we will install miniconda to manage our python environment. Be sure that you connected to the cluster and are on one of the pool-machines (e.g. cgpool1801,...,cgpool1803
or cgpool1900,...,cgpool1915
or cgpoolsand1900,...,cgpoolsand1907
).
These five commands quickly and quietly install the latest 64-bit version of the installer and then clean up after themselves (enter them one after the other):
cd ~
mkdir ~/miniconda
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda3/miniconda.sh
bash ~/miniconda3/miniconda.sh -b -u -p ~/miniconda3
rm -rf ~/miniconda3/miniconda.sh
After installing, initialize your newly-installed Miniconda. The following commands initialize for bash and zsh shells:
~/miniconda3/bin/conda init bash
You can also add the path to your .bashrc
file to automatically activate the conda environment when you open a new terminal.
cd ~
touch ~/.bashrc
echo "source ~/miniconda3/etc/profile.d/conda.sh" >> ~/.bashrc
source ~/.bashrc
Next we will install nerfstudio to train our gaussian splatting scene.
conda create -n nerfstudio python=3.8
conda activate nerfstudio
python -m pip install --upgrade pip
We need to install some dependencies to run nerfstudio, like torch
, torchvision
, functorch
and tinycudann
. We will install torch
and torchvision
from the official pytorch website and functorch
and tinycudann
from the github repositories.
pip install torch==2.1.2+cu118 torchvision==0.16.2+cu118 --extra-index-url https://download.pytorch.org/whl/cu118
conda install -c "nvidia/label/cuda-11.8.0" cuda-toolkit
pip install ninja git+https://github.com/NVlabs/tiny-cuda-nn/#subdirectory=bindings/torch
Finally we can install nerfstudio and set it up with the following commands:
pip install nerfstudio
ns-install-cli
We will use COLMAP to preprocess the images for the training. We install COLMAP with the following commands:
conda install -c conda-forge colmap
The first step is to get the data. In our case of inverse rendering we need images from a static scene. So as a first step scan your room with your mobile phone. Meaning you have to take a lot of pictures from different angles and positions.
- Check that the room is well lit and there are no moving objects.
- Take rouhgly 80-140 pictures of your room from different angles and positions.
In the next step we will preprocess the images to retreive camera poses and intrinsics for the images.We will use COLMAP to do so. Neatly it is already installed on the cluster and can be interfaced with Nerfstudio.
First, we need to upload the images to the cluster. You can use scp
to upload the images to the cluster:
scp -r /path/to/your/images/ username@servername:/path/to/save/your/images/
Replace /path/to/your/images/
with the path to your images on your local machine and /path/to/save/your/images/
with the path to save your images on the cluster. Furthermore replace username
with your username and servername
with a viable servername (e.g. cgcontact
or cgpool1801,...,cgpool1803
or cgpool1900,...,cgpool1915
or cgpoolsand1900,...,cgpoolsand1907
).
Alternatively if you are using visual studio code you can use the
Remote - SSH
extension to connect to the cluster and upload the images directly from the editor.
Run the following command to preprocess the images with COLMAP:
ns-process-data images --data /path/to/save/your/images/ --output-dir /new/path/to/processed/images/
Replace /path/to/save/your/images/
with the path to your images on the cluster and /new/path/to/processed/images/
with the path to save your processed images on the cluster.
Before starting the training be sure to be connected to the cluster and on one of the pool-machines (e.g. cgpool1801,...,cgpool1803
or cgpool1900,...,cgpool1915
or cgpoolsand1900,...,cgpoolsand1907
). Before starting a training always check the available resources on the cluster with the following command:
pool-smi
If you are sure that no one is using the cluster you can start the training with the following command (use atleast a RTX 2080Ti):
ns-train splatfacto --data /new/path/to/processed/images/ --output-dir ./outputs
You can see the progress of the training in the terminal. If you want to stop the training press
ctrl + c
.
![Nerfstudio Viewer](https://private-user-images.githubusercontent.com/9963865/325549716-39ad1ef7-49c9-43b2-8fb1-159b3037ee73.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQ0NDg4NzQsIm5iZiI6MTcyNDQ0ODU3NCwicGF0aCI6Ii85OTYzODY1LzMyNTU0OTcxNi0zOWFkMWVmNy00OWM5LTQzYjItOGZiMS0xNTliMzAzN2VlNzMucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDgyMyUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDA4MjNUMjEyOTM0WiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9ZjNhZDY4ZGU2MDliMWZhODFlNDQzMmJjYmJiNTU1MGM2Y2Q4YWVmYTY3YTZlNjMwZjA4YmUzYzQwYjc0ZDVjOSZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.ziSGom_59WlFDHamczhjpYhZRLR5cyjOWRAMNzS_6cw)
(Optional) You can also visualize the training progress by using the nerfstudio viewer. Create a port forwarding for the cluster and open the viewer in your web browser. Run this command on your local machine (not on the cluster) or use the VS Code port forwarding:
ssh -L 7007:127.0.0.1:7007 servernameThe viewer is available at
http://localhost:7007/
in your web browser. You may have to change the port to the one displayed in the terminal, when you started the training.
After the training is finished you can export the optimized scene with the following command:
ns-export gaussian-splat --load-config outputs/...[experiment_path].../config.yml --output-dir ./exports/splat
Replace [experiment_path]
with the path to the experiment folder in the outputs
directory. You can find the path in the terminal output of the training.
After you trained the model successfully you can download the optimized scene to display them in your web browser.
First, we need to download the optimized scene to our local machine. You can do so by running the following command in your terminal on your local machine:
scp -r username@servername:./exports/splat/splat.ply /.../hyperrealistic_indoor_streetview/
Again if you are using visual studio code you can use the
Remote - SSH
extension to connect to the cluster and download the optimized scene directly from the editor.
After downloading the optimized scene you can display the scene in your web browser. But before that, we need to convert our splat.ply
file to a scene.splat
file. Be sure to have the splat.ply
file in the root directory of the repository. You can convert the splat.ply
file to a scene.splat
file by running the following command in the root directory of the repository:
node ./convert_ply_to_splat.js
Then the splat.splat
file should be within the public
folder of the web application. You can start the web server by running the following command in the root directory of the repository:
yarn dev
![Bildschirmfoto 2024-04-25 um 11 23 47](https://private-user-images.githubusercontent.com/9963865/325550525-18208e63-a278-4b22-b92b-1b3b0b01bbfe.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQ0NDg4NzQsIm5iZiI6MTcyNDQ0ODU3NCwicGF0aCI6Ii85OTYzODY1LzMyNTU1MDUyNS0xODIwOGU2My1hMjc4LTRiMjItYjkyYi0xYjNiMGIwMWJiZmUucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDgyMyUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDA4MjNUMjEyOTM0WiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9OTU2ZDJjODc3MTUxYTFmYzk4MDMxNDQ2MmExNjY0YTlkZThjNjk0ZmY1OTk4OTllMDk2N2VmN2NmZGYxZGFjYiZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.LsQEtqXv_OkBXpbbw8ZufnATtKcrqn5-haBc42NKu8s)
Go to http://localhost:5173/
(might be different for you) in your web browser to see the web application running. You should see the optimized gaussian splatting scene you trained on the cluster. Navigate with your mouse and zoom, see here for full controls.