![](https://private-user-images.githubusercontent.com/31780571/316260806-4df6c785-8163-43e5-9c32-4b3ac7809ca8.gif?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjU5MzYyNzUsIm5iZiI6MTcyNTkzNTk3NSwicGF0aCI6Ii8zMTc4MDU3MS8zMTYyNjA4MDYtNGRmNmM3ODUtODE2My00M2U1LTljMzItNGIzYWM3ODA5Y2E4LmdpZj9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA5MTAlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwOTEwVDAyMzkzNVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTliMzg5MTY0NjQ1YWExZGMwY2VmN2E2MzQ0ODkwNThiZTFjOGE2MWE4ZThmYzkxZTNiMjMyYzUyYjViYmQxYjcmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.RrsDA44OMX7dpdMRUiwWX4jH8uNlkopaM5mhq2WL4VI)
Seamlessly generate high-quality 3D meshes from 2D images and retexture them using text-based texture generation directly in Unity.
This project integrates TripoSR, a "a state-of-the-art open-source model for fast feedforward 3D reconstruction from a single image" by StabilityAI and TripoAI, and triposr-texture-gen directly into Unity Editor.
- Transformation of 2D images into textured 3D meshes, available both in Editor and in Playmode.
- Text-based texture generation, applied directly to the generated 3D meshes.
The generated 3D meshes are imported using a modified vertex color importer (based on Andrew Raphael Lukasik's importer) and auto-assigned to a base Material with custom shader to utilize and display the vertex colors correctly (without surface normals, though).
Tested with Unity 2022.3.22f1 running on Windows 11 and Ubuntu 22.04.
The first demo clip shows the creation of a 3D mesh based on a 2D Texture within the Editor.
The generated meshes are colored (vertex colors), react to light, and (optionally) automatically use MeshCollider
and Rigidbody
for physics interaction.
tripo_sr_unity_demo.mp4
This clip shows the text-based retexturing of a TripoSR-generated 3D mesh:
tripo_sr_unity_retexture.mp4
This demo clip shows 3D mesh generation in Playmode:
tripo_sr_unity_playmode.mp4
- Download the latest release
.unitypackage
and import it into your project (Assets > Import Package
). cd
into theAssets
folder of your Unity project (using Command Prompt, Terminal, ...) and clone the latest repo of TripoSR:git clone https://github.com/VAST-AI-Research/TripoSR.git
.- After cloning TripoSR,
cd
into the TripoSR folder that you just created by cloning the repo. - Run
pip install --upgrade setuptools
,pip install torch
(in case you don't have PyTorch installed) andpip install -r requirements.txt
. - Add the
TripoSR
Prefab (found inAssets > Prefabs
) to your scene. - Configure the path to your python executable in the
TripoSR
GameObject within your scene: For Windows, runwhere python
within Command Prompt. For Unix, runwhich python
within Terminal. - Configure the other public variables in the Inspector as needed.
- Clone this repo.
- Ensure you have Python installed on your system.
- Run
pip install --upgrade setuptools
,pip install torch
(in case you don't have PyTorch installed) andpip install -r requirements.txt
from within this project'sAssets/TripoSR
folder. - In Unity, add the
TripoSR
Prefab to the scene (or simply open up mySampleScene
). - Configure the path to your python executable in the
TripoSR
GameObject within your scene: For Windows, runwhere python
within Command Prompt. For Unix, runwhich python
within Terminal. - Configure the other public variables in the Inspector as needed.
I've adapted the text2texture.py
script from triposr-texture-gen, so it doesn't open up the 3D viewer and works in-situ. When you clone the repo on your own, this won't be included.
- Follow the general setup for TripoSR from above
cd
into theAssets
folder and clone the latest triposr-texture-gen.- Run
pip install -r -requirements.txt
from within theAssets/triposr-texture-gen
directory. - Run
pip install huggingface-hub --upgrade
to fix some compatibility problems. - Add the
TextureGenerator
Prefab to your scene. - Configure the path to your python executable.
- Configure the public vars to your liking.
- Follow the general setup from above
- Run
pip install -r -requirements.txt
from within theAssets/triposr-texture-gen
directory. - Run
pip install huggingface-hub --upgrade
to fix some compatibility problems. - Add the
TextureGenerator
Prefab to your scene (or use mySampleScene
). - Configure the path to your python executable.
- Configure the public vars to your liking.
Once you have set up your scene with the TripoSRForUnity
component and configured the parameters, you can run the process by clicking the Run TripoSR
button in the inspector.
When you run TripoSR for the first time, the model weights will be downloaded and cached - this only occurs once; subsequent runs use the cached model.
autoAddMesh
: When enabled, the generated mesh is automatically added to the Unity scene as GameObject.autoAddPhysicsComponents
: Whether or not to automatically add physics components (i.e., convexMeshCollider
andRigidbody
) to the generated mesh.autoFixRotation
: If enabled, will automatically correct the wrong object rotation after adding the mesh to the scene.moveAndRename
: Moves and renames the output.obj
file based on the input image's filename if enabled.moveAndRenamePath
: The directory to which the.obj
file will be moved. Must start withAssets/
.showDebugLogs
: Enables the display of debug outputs from therun.py
script in the Unity console.
All TripoSR parameters are exposed by my script. Feel free to change them as you see fit.
I've made some of them ReadOnly
within the Inspector, since you shouldn't really change those vars (e.g. model name, device to use). You can still change them within the script of course.
Likewise, you can run the texture generator by clicking on Generate Textured Object
within the TextureGenerator
-component in the inspector.
When you run the texture generator for the first time, model weights will also be downloaded and cached, which also only occurs the first time.
Set the model file in the inspector to your .obj
from Assets > Models
, describe the design you would like as model description, and start the generator.
TripoSR .obj
s only consist of v
s and f
s, surface normals are not calculated. When Unity calculates the normals upon import, they are not smoothed correctly (even when using high smoothing angles).
For now, I've disabled normal calculation - feel free to create a PR if you know how to correctly handle this issue.
This project is licensed under the MIT License. See LICENSE
for more information.
Special thanks to StabilityAI, TripoAI, and contributors of the TripoSR project.
Please cite their work adequately in case you use it in your own publications:
@article{TripoSR2024,
title={TripoSR: Fast 3D Object Reconstruction from a Single Image},
author={Tochilkin, Dmitry and Pankratz, David and Liu, Zexiang and Huang, Zixuan and and Letts, Adam and Li, Yangguang and Liang, Ding and Laforte, Christian and Jampani, Varun and Cao, Yan-Pei},
journal={arXiv preprint arXiv:2403.02151},
year={2024}
}
Also special thanks to @ejones for triposr-texture-gen.