Internal inpainting of materials with diffusion

This is the implementation of the method presented in the paper Diffusion-based image inpainting with internal learning". (https://eurasip.org/Proceedings/Eusipco/Eusipco2024/pdfs/0000446.pdf)

This repository handles the problem of materials / SVBRDFs inpainting using the following maps:

• diffuse color
• normal map
• roughness
• specular

Training

A model can be trained on a single masked image, the code makes a lot of assumptions about the structure of the folders and the name of the files. Instead of pointing to a single file, you should provide the path of a folder folderIn which contains the following files:

• diffuse.png
• normal.png
• roughness.png
• specular.png
• mask.png

Training is launched with:

python train.py --folder ${folderIn} --steps 15000

Logs and model checkpoints are saved in the folder runs/${folderIn}.

Testing

Once trained, the model can be used to generate 5 different results with:

python test.py --folder ${folderIn} --checkpoint runs/${folderIn}/model_last.pth --n 5

Results

The results from the paper have been achieved using the 100 material examples from test_set.txt, taken from the dataset of Deschaintre et al. (https://repo-sam.inria.fr/fungraph/deep-materials/)

The masks have been generated by us, and will be soon released.

Cite

@inproceedings{cherelImfusionEUSIPCO,
	Address = {Lyon},
	author = {Cherel, Nicolas and Almansa, Andr{\'e}s and Gousseau, Yann and Newson, Alasdair},
	Booktitle = {(EUSIPCO 2024) 32nd European Signal Processing Conference},
	Language = {English},
	Publisher = {IEEE},
	Shorttitle = {EUSIPCO},
	Title = {{Diffusion-based image inpainting with internal learning}},
	Year = {2024},
	url={https://arxiv.org/abs/2406.04206}
}