This repository is the official implementation of NeRAF: 3D Scene Infused Neural Radiance and Acoustic Field. NeRAF is a novel method that learns neural radiance and acoustic field.
The code is written in Python and uses the NerfStudio framework.
├── NeRAF
│ ├── __init__.py
│ ├── NeRAF_config.py
│ ├── NeRAF_pipeline.py
│ ├── NeRAF_model.py
│ ├── NeRAF_field.py
│ ├── NeRAF_datamanger.py
│ ├── NeRAF_resnet3d.py
│ ├── NeRAF_evaluator.py
├── pyproject.toml
Ensure that nerfstudio has been installed according to the instructions. Clone or fork this repository and run the commands:
conda activate nerfstudio
cd NeRAF/
pip install -e .
ns-install-cli
Additional audio related packages must be added to the conda environment.
- librosa
- pyroomacoustics
- torchaudio
- scipy
This code creates a new Nerfstudio method named "NeRAF-method". Change the configuration file NeRAF_config.py to set the parameters for the method. The file already contains the default parameters used in the paper.
To train the models, run the command:
ns-train NeRAF --data [PATH]
To evaluate my model on eval set, run the following command:
ns-eval --load-config [CONFIG_PATH] --output-path [OUTPUT_PATH] --render-output-path [RENDER_OUTPUT_PATH]
More informations are provided in Nerfstudio documentation. ns-eval will save rendered images and audio in the output path.
To resume training, run the following command:
ns-train NeRAF --data [PATH] --load-dir [MODEL_PATH]
More informations are provided in Nerfstudio documentation.
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.