Installation on RTX 4090

[haoyuhsu]

Hi,

I have tried to install ChatSim environment on NVIDIA RTX 4090, and I have encountered some problems during installation.

OS: Ubuntu 22.04.4 LTS
gcc: 11.4.0
cmake: 3.22.1
cuda: 11.8

I have modified several steps to make it compatible to 4090. (since it is compute_89)

conda create -n chatsim python=3.9 git-lfs
conda activate chatsim

##### Install different version of PyTorch & CUDA #####
conda install pytorch==2.0.0 torchvision==0.15.0 torchaudio==2.0.0 pytorch-cuda=11.8 -c pytorch -c nvidia
conda install -c "nvidia/label/cuda-11.8.0" cuda-toolkit

pip install -r requirements.txt
imageio_download_bin freeimage

cd chatsim/background/mcnerf/

# mcnerf use the same data directory. 
ln -s ../../../data .

sudo apt install zlib1g-dev

cd chatsim/background/mcnerf
cd External

##### Download different version of LibTorch #####
wget https://download.pytorch.org/libtorch/cu118/libtorch-cxx11-abi-shared-with-deps-2.0.0%2Bcu118.zip
unzip ./libtorch-cxx11-abi-shared-with-deps-2.0.0+cu118.zip
rm ./libtorch-cxx11-abi-shared-with-deps-2.0.0+cu118.zip

As I tried to run cmake . -B build, it comes with a warning messages.

-- Obtained CUDA architectures automatically from installed GPUs
-- Targeting CUDA architectures: 89
-- Module support is disabled.
-- Version: 9.1.1
-- Build type: Release
-- Caffe2: CUDA detected: 11.8
-- Caffe2: CUDA nvcc is: /home/haoyuyh3/miniconda3/envs/chatsim/bin/nvcc
-- Caffe2: CUDA toolkit directory: /home/haoyuyh3/miniconda3/envs/chatsim
-- Caffe2: Header version is: 11.8
-- /usr/lib/x86_64-linux-gnu/libnvrtc.so shorthash is 65f2c18b
-- USE_CUDNN is set to 0. Compiling without cuDNN support
-- Autodetected CUDA architecture(s):  8.9
-- Added CUDA NVCC flags for: -gencode;arch=compute_89,code=sm_89
-- Configuring done
CMake Warning at CMakeLists.txt:74 (add_executable):
  Cannot generate a safe runtime search path for target main because files in
  some directories may conflict with libraries in implicit directories:

    runtime library [libnvrtc.so.11.2] in /usr/lib/x86_64-linux-gnu may be hidden by files in:
      /home/haoyuyh3/miniconda3/envs/chatsim/lib
    runtime library [libnvToolsExt.so.1] in /usr/lib/x86_64-linux-gnu may be hidden by files in:
      /home/haoyuyh3/miniconda3/envs/chatsim/lib
    runtime library [libz.so.1] in /usr/lib/x86_64-linux-gnu may be hidden by files in:
      /home/haoyuyh3/miniconda3/envs/chatsim/lib

  Some of these libraries may not be found correctly.


-- Generating done
-- Build files have been written to: /home/haoyuyh3/Documents/maxhsu/editing-related-works/ChatSim/chatsim/background/mcnerf/build

Afterwards, I ran cmake --build build --target main --config RelWithDebInfo -j and came across this error.

(chats) root@server:~/Documents/maxhsu/editing-related-works/ChatSim/chatsim/background/mcnerf$ cmake --build build --target main --config RelWithDebInfo -j
Consolidate compiler generated dependencies of target fmt
Consolidate compiler generated dependencies of target yaml-cpp
[  3%] Built target fmt
Consolidate compiler generated dependencies of target tiny-cuda-nn
[ 45%] Built target yaml-cpp
[ 61%] Built target tiny-cuda-nn
Consolidate compiler generated dependencies of target main
[ 62%] Linking CXX executable main
/usr/bin/ld: ../External/libtorch/lib/libkineto.a(CuptiActivityApi.cpp.o): in function `libkineto::CuptiActivityApi::pushCorrelationID(int, libkineto::CuptiActivityApi::CorrelationFlowType)':
CuptiActivityApi.cpp:(.text+0x95b): undefined reference to `cuptiActivityPushExternalCorrelationId'
/usr/bin/ld: CuptiActivityApi.cpp:(.text+0x976): undefined reference to `cuptiGetResultString'
/usr/bin/ld: CuptiActivityApi.cpp:(.text+0xa09): undefined reference to `cuptiActivityPushExternalCorrelationId'
/usr/bin/ld: CuptiActivityApi.cpp:(.text+0xa28): undefined reference to `cuptiGetResultString'
/usr/bin/ld: ../External/libtorch/lib/libkineto.a(CuptiActivityApi.cpp.o): in function `libkineto::CuptiActivityApi::popCorrelationID(libkineto::CuptiActivityApi::CorrelationFlowType)::{lambda()#1}::operator()() const [clone .isra.216]':
CuptiActivityApi.cpp:(.text+0xbe6): undefined reference to `cuptiActivityPopExternalCorrelationId'
/usr/bin/ld: CuptiActivityApi.cpp:(.text+0xc29): undefined reference to `cuptiGetResultString'
/usr/bin/ld: ../External/libtorch/lib/libkineto.a(CuptiActivityApi.cpp.o): in function `libkineto::CuptiActivityApi::popCorrelationID(libkineto::CuptiActivityApi::CorrelationFlowType)::{lambda()#2}::operator()() const [clone .isra.217]':
CuptiActivityApi.cpp:(.text+0xd36): undefined reference to `cuptiActivityPopExternalCorrelationId'
/usr/bin/ld: CuptiActivityApi.cpp:(.text+0xd79): undefined reference to `cuptiGetResultString'
/usr/bin/ld: ../External/libtorch/lib/libkineto.a(CuptiActivityApi.cpp.o): in function `libkineto::CuptiActivityApi::setDeviceBufferSize(unsigned long)::{lambda()#1}::operator()() const':
CuptiActivityApi.cpp:(.text+0xeb8): undefined reference to `cuptiActivitySetAttribute'
/usr/bin/ld: CuptiActivityApi.cpp:(.text+0xf01): undefined reference to `cuptiGetResultString'
/usr/bin/ld: ../External/libtorch/lib/libkineto.a(CuptiActivityApi.cpp.o): in function `libkineto::CuptiActivityApi::setDeviceBufferPoolLimit(unsigned long)::{lambda()#1}::operator()() const':
CuptiActivityApi.cpp:(.text+0x107b): undefined reference to `cuptiActivitySetAttribute'
/usr/bin/ld: CuptiActivityApi.cpp:(.text+0x10c1): undefined reference to `cuptiGetResultString'
/usr/bin/ld: ../External/libtorch/lib/libkineto.a(CuptiActivityApi.cpp.o): in function `libkineto::CuptiActivityApi::forceLoadCupti()::{lambda()#1}::operator()() const [clone .isra.218]':
CuptiActivityApi.cpp:(.text+0x1234): undefined reference to `cuptiActivityEnable'
/usr/bin/ld: CuptiActivityApi.cpp:(.text+0x1279): undefined reference to `cuptiGetResultString'
/usr/bin/ld: ../External/libtorch/lib/libkineto.a(CuptiActivityApi.cpp.o): in function `libkineto::CuptiActivityApi::activityBuffers()':
CuptiActivityApi.cpp:(.text+0x1402): undefined reference to `cuptiActivityFlushAll'
/usr/bin/ld: CuptiActivityApi.cpp:(.text+0x1571): undefined reference to `cuptiGetResultString'
/usr/bin/ld: ../External/libtorch/lib/libkineto.a(CuptiActivityApi.cpp.o): in function `libkineto::nextActivityRecord(unsigned char*, unsigned long, CUpti_Activity*&)::{lambda()#1}::operator()() const':
CuptiActivityApi.cpp:(.text+0x1751): undefined reference to `cuptiGetResultString'
/usr/bin/ld: ../External/libtorch/lib/libkineto.a(CuptiActivityApi.cpp.o): in function `libkineto::CuptiActivityApi::processActivitiesForBuffer(unsigned char*, unsigned long, std::function<void (CUpti_Activity const*)>)':
CuptiActivityApi.cpp:(.text+0x188a): undefined reference to `cuptiActivityGetNextRecord'
/usr/bin/ld: ../External/libtorch/lib/libkineto.a(CuptiActivityApi.cpp.o): in function `libkineto::CuptiActivityApi::clearActivities()::{lambda()#1}::operator()() const [clone .isra.221]':
CuptiActivityApi.cpp:(.text+0x1a71): undefined reference to `cuptiActivityFlushAll'
/usr/bin/ld: CuptiActivityApi.cpp:(.text+0x1ab1): undefined reference to `cuptiGetResultString'
/usr/bin/ld: ../External/libtorch/lib/libkineto.a(CuptiActivityApi.cpp.o): in function `libkineto::CuptiActivityApi::bufferCompleted(CUctx_st*, unsigned int, unsigned char*, unsigned long, unsigned long)':
CuptiActivityApi.cpp:(.text+0x1fec): undefined reference to `cuptiActivityGetNumDroppedRecords'
/usr/bin/ld: CuptiActivityApi.cpp:(.text+0x21a6): undefined reference to `cuptiGetResultString'
/usr/bin/ld: ../External/libtorch/lib/libkineto.a(CuptiActivityApi.cpp.o): in function `libkineto::CuptiActivityApi::enableCuptiActivities(std::set<libkineto::ActivityType, std::less<libkineto::ActivityType>, std::allocator<libkineto::ActivityType> > const&)::{lambda()#1}::operator()() const [clone .isra.222]':
CuptiActivityApi.cpp:(.text+0x231d): undefined reference to `cuptiActivityRegisterCallbacks'
/usr/bin/ld: CuptiActivityApi.cpp:(.text+0x2361): undefined reference to `cuptiGetResultString'
/usr/bin/ld: ../External/libtorch/lib/libkineto.a(CuptiActivityApi.cpp.o): in function `libkineto::CuptiActivityApi::enableCuptiActivities(std::set<libkineto::ActivityType, std::less<libkineto::ActivityType>, std::allocator<libkineto::ActivityType> > const&)::{lambda()#2}::operator()() const [clone .isra.223]':
CuptiActivityApi.cpp:(.text+0x2464): undefined reference to `cuptiActivityEnable'
/usr/bin/ld: CuptiActivityApi.cpp:(.text+0x24a9): undefined reference to `cuptiGetResultString'
/usr/bin/ld: ../External/libtorch/lib/libkineto.a(CuptiActivityApi.cpp.o): in function `libkineto::CuptiActivityApi::enableCuptiActivities(std::set<libkineto::ActivityType, std::less<libkineto::ActivityType>, std::allocator<libkineto::ActivityType> > const&)::{lambda()#3}::operator()() const [clone .isra.224]':
CuptiActivityApi.cpp:(.text+0x25b4): undefined reference to `cuptiActivityEnable'
/usr/bin/ld: CuptiActivityApi.cpp:(.text+0x25f9): undefined reference to `cuptiGetResultString'
/usr/bin/ld: ../External/libtorch/lib/libkineto.a(CuptiActivityApi.cpp.o): in function `libkineto::CuptiActivityApi::enableCuptiActivities(std::set<libkineto::ActivityType, std::less<libkineto::ActivityType>, std::allocator<libkineto::ActivityType> > const&)::{lambda()#4}::operator()() const [clone .isra.225]':
CuptiActivityApi.cpp:(.text+0x2704): undefined reference to `cuptiActivityEnable'
/usr/bin/ld: CuptiActivityApi.cpp:(.text+0x2749): undefined reference to `cuptiGetResultString'
/usr/bin/ld: ../External/libtorch/lib/libkineto.a(CuptiActivityApi.cpp.o): in function `libkineto::CuptiActivityApi::enableCuptiActivities(std::set<libkineto::ActivityType, std::less<libkineto::ActivityType>, std::allocator<libkineto::ActivityType> > const&)::{lambda()#5}::operator()() const [clone .isra.226]':
CuptiActivityApi.cpp:(.text+0x2854): undefined reference to `cuptiActivityEnable'
/usr/bin/ld: CuptiActivityApi.cpp:(.text+0x2899): undefined reference to `cuptiGetResultString'
/usr/bin/ld: ../External/libtorch/lib/libkineto.a(CuptiActivityApi.cpp.o): in function `libkineto::CuptiActivityApi::enableCuptiActivities(std::set<libkineto::ActivityType, std::less<libkineto::ActivityType>, std::allocator<libkineto::ActivityType> > const&)::{lambda()#6}::operator()() const [clone .isra.227]':
CuptiActivityApi.cpp:(.text+0x29a4): undefined reference to `cuptiActivityEnable'
/usr/bin/ld: CuptiActivityApi.cpp:(.text+0x29e9): undefined reference to `cuptiGetResultString'
/usr/bin/ld: ../External/libtorch/lib/libkineto.a(CuptiActivityApi.cpp.o): in function `libkineto::CuptiActivityApi::enableCuptiActivities(std::set<libkineto::ActivityType, std::less<libkineto::ActivityType>, std::allocator<libkineto::ActivityType> > const&)::{lambda()#7}::operator()() const [clone .isra.228]':
CuptiActivityApi.cpp:(.text+0x2af4): undefined reference to `cuptiActivityEnable'
/usr/bin/ld: CuptiActivityApi.cpp:(.text+0x2b39): undefined reference to `cuptiGetResultString'
/usr/bin/ld: ../External/libtorch/lib/libkineto.a(CuptiActivityApi.cpp.o): in function `libkineto::CuptiActivityApi::disableCuptiActivities(std::set<libkineto::ActivityType, std::less<libkineto::ActivityType>, std::allocator<libkineto::ActivityType> > const&)::{lambda()#1}::operator()() const [clone .isra.229]':
CuptiActivityApi.cpp:(.text+0x2d24): undefined reference to `cuptiActivityDisable'
/usr/bin/ld: CuptiActivityApi.cpp:(.text+0x2d69): undefined reference to `cuptiGetResultString'
/usr/bin/ld: ../External/libtorch/lib/libkineto.a(CuptiActivityApi.cpp.o): in function `libkineto::CuptiActivityApi::disableCuptiActivities(std::set<libkineto::ActivityType, std::less<libkineto::ActivityType>, std::allocator<libkineto::ActivityType> > const&)::{lambda()#2}::operator()() const [clone .isra.230]':
CuptiActivityApi.cpp:(.text+0x2e74): undefined reference to `cuptiActivityDisable'
/usr/bin/ld: CuptiActivityApi.cpp:(.text+0x2eb9): undefined reference to `cuptiGetResultString'
/usr/bin/ld: ../External/libtorch/lib/libkineto.a(CuptiActivityApi.cpp.o): in function `libkineto::CuptiActivityApi::disableCuptiActivities(std::set<libkineto::ActivityType, std::less<libkineto::ActivityType>, std::allocator<libkineto::ActivityType> > const&)::{lambda()#3}::operator()() const [clone .isra.231]':
CuptiActivityApi.cpp:(.text+0x2fc4): undefined reference to `cuptiActivityDisable'
/usr/bin/ld: CuptiActivityApi.cpp:(.text+0x3009): undefined reference to `cuptiGetResultString'
/usr/bin/ld: ../External/libtorch/lib/libkineto.a(CuptiActivityApi.cpp.o): in function `libkineto::CuptiActivityApi::disableCuptiActivities(std::set<libkineto::ActivityType, std::less<libkineto::ActivityType>, std::allocator<libkineto::ActivityType> > const&)::{lambda()#4}::operator()() const [clone .isra.232]':
CuptiActivityApi.cpp:(.text+0x3114): undefined reference to `cuptiActivityDisable'
/usr/bin/ld: CuptiActivityApi.cpp:(.text+0x3159): undefined reference to `cuptiGetResultString'
/usr/bin/ld: ../External/libtorch/lib/libkineto.a(CuptiActivityApi.cpp.o): in function `libkineto::CuptiActivityApi::disableCuptiActivities(std::set<libkineto::ActivityType, std::less<libkineto::ActivityType>, std::allocator<libkineto::ActivityType> > const&)::{lambda()#5}::operator()() const [clone .isra.233]':
CuptiActivityApi.cpp:(.text+0x3264): undefined reference to `cuptiActivityDisable'
/usr/bin/ld: CuptiActivityApi.cpp:(.text+0x32a9): undefined reference to `cuptiGetResultString'
/usr/bin/ld: ../External/libtorch/lib/libkineto.a(CuptiActivityApi.cpp.o): in function `libkineto::CuptiActivityApi::disableCuptiActivities(std::set<libkineto::ActivityType, std::less<libkineto::ActivityType>, std::allocator<libkineto::ActivityType> > const&)::{lambda()#6}::operator()() const [clone .isra.234]':
CuptiActivityApi.cpp:(.text+0x33b4): undefined reference to `cuptiActivityDisable'
/usr/bin/ld: CuptiActivityApi.cpp:(.text+0x33f9): undefined reference to `cuptiGetResultString'
/usr/bin/ld: ../External/libtorch/lib/libkineto.a(CuptiActivityApi.cpp.o): in function `libkineto::CuptiActivityApi::teardownContext()::{lambda()#1}::operator()() const':
CuptiActivityApi.cpp:(.text+0x3766): undefined reference to `cuptiActivityFlushAll'
/usr/bin/ld: CuptiActivityApi.cpp:(.text+0x3951): undefined reference to `cuptiGetResultString'
/usr/bin/ld: ../External/libtorch/lib/libkineto.a(CuptiCallbackApi.cpp.o): in function `libkineto::CuptiCallbackApi::initCallbackApi()':
CuptiCallbackApi.cpp:(.text+0x1f): undefined reference to `cuptiSubscribe'
/usr/bin/ld: ../External/libtorch/lib/libkineto.a(CuptiCallbackApi.cpp.o): in function `libkineto::CuptiCallbackApi::enableCallback(CUpti_CallbackDomain, unsigned int)':
CuptiCallbackApi.cpp:(.text+0x64b): undefined reference to `cuptiEnableCallback'
/usr/bin/ld: ../External/libtorch/lib/libkineto.a(CuptiCallbackApi.cpp.o): in function `libkineto::CuptiCallbackApi::enableCallbackDomain(CUpti_CallbackDomain)':
CuptiCallbackApi.cpp:(.text+0x7ca): undefined reference to `cuptiEnableDomain'
/usr/bin/ld: ../External/libtorch/lib/libkineto.a(CuptiCallbackApi.cpp.o): in function `libkineto::CuptiCallbackApi::reenableCallbacks()':
CuptiCallbackApi.cpp:(.text+0x8f8): undefined reference to `cuptiEnableCallback'
/usr/bin/ld: CuptiCallbackApi.cpp:(.text+0x931): undefined reference to `cuptiEnableDomain'
/usr/bin/ld: ../External/libtorch/lib/libkineto.a(CuptiCallbackApi.cpp.o): in function `libkineto::CuptiCallbackApi::disableCallback(CUpti_CallbackDomain, unsigned int)':
CuptiCallbackApi.cpp:(.text+0xbbf): undefined reference to `cuptiEnableCallback'
/usr/bin/ld: ../External/libtorch/lib/libkineto.a(CuptiCallbackApi.cpp.o): in function `libkineto::CuptiCallbackApi::disableCallbackDomain(CUpti_CallbackDomain)':
CuptiCallbackApi.cpp:(.text+0xccd): undefined reference to `cuptiEnableDomain'
/usr/bin/ld: ../External/libtorch/lib/libkineto.a(CuptiCallbackApi.cpp.o): in function `libkineto::CuptiCallbackApi::__callback_switchboard(CUpti_CallbackDomain, unsigned int, CUpti_CallbackData const*)':
CuptiCallbackApi.cpp:(.text+0x1041): undefined reference to `cuptiFinalize'
/usr/bin/ld: CuptiCallbackApi.cpp:(.text+0x1111): undefined reference to `cuptiGetResultString'
collect2: error: ld returned 1 exit status
gmake[3]: *** [CMakeFiles/main.dir/build.make:571: main] Error 1
gmake[2]: *** [CMakeFiles/Makefile2:186: CMakeFiles/main.dir/all] Error 2
gmake[1]: *** [CMakeFiles/Makefile2:193: CMakeFiles/main.dir/rule] Error 2
gmake: *** [Makefile:125: main] Error 2

I would like to know which part did I get it wrong and what is the proper way to fix it? Thanks!

[yifanlu0227]

Could you try conda install nvidia/label/cuda-11.8.0::cuda-cupti ?

[yifanlu0227]

It seems that you install cuda via conda, in which case some extra CUDA library would not be installed automatically.

The /usr/bin/ld can not find the cupti. Installing them manually may solve the problem.

[yifanlu0227]

In a system level CUDA (not in the conda environment), they can be found in the `extras` folder

[haoyuhsu]

I tried conda install nvidia/label/cuda-11.8.0::cuda-cupti and get the messages # All requested packages already installed. So I think the only way might be installing locally.

[yifanlu0227]

Maybe this helps. https://discuss.pytorch.org/t/build-errors-undefined-ref-to-cuptixxx-solved/183318

Yeah a sure-fire way is to install a local CUDA. it's not a hassle and doesn't require sudo anyway.

[haoyuhsu]

Unfortunately, I installed CUDA locally and still got the same error.

(chatsim) root@server:~/Documents/maxhsu/editing-related-works/ChatSim/chatsim/background/mcnerf$ which nvcc
/home/haoyuyh3/Documents/maxhsu/cuda/cuda-11.8/bin/nvcc
(chatsim) root@server:~/Documents/maxhsu/editing-related-works/ChatSim/chatsim/background/mcnerf$ nvcc -V
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Wed_Sep_21_10:33:58_PDT_2022
Cuda compilation tools, release 11.8, V11.8.89
Build cuda_11.8.r11.8/compiler.31833905_0

I've also tried the solution above and added those 4 lines into CMakeLists.txt. Still not working.

I observe the only difference would be one of the cmake warnings disappeared.

CMake Warning at CMakeLists.txt:74 (add_executable):
  Cannot generate a safe runtime search path for target main because files in
  some directories may conflict with libraries in implicit directories:

    runtime library [libnvrtc.so.11.2] in /usr/lib/x86_64-linux-gnu may be hidden by files in:
      /home/haoyuyh3/Documents/maxhsu/cuda/cuda-11.8/lib64
    runtime library [libnvToolsExt.so.1] in /usr/lib/x86_64-linux-gnu may be hidden by files in:
      /home/haoyuyh3/Documents/maxhsu/cuda/cuda-11.8/lib64

  Some of these libraries may not be found correctly.

[yifanlu0227]

You may try to specify those two libraries in the CMakeLists.txt, so that the cmake can find them explicitly.

[haoyuhsu]

I've tried that. Unfortunately, it also failed...Orz

[yifanlu0227]

sad to hear that, 3dgs is really a good alternative. The installation is much easier

[yifanlu0227]

export CMAKE_PREFIX_PATH=$CONDA_PREFIX

It is worth a final try :)

[haoyuhsu]

yeah I think I'll choose to move onto 3dgs.
still really appreciate your kind and detailed response!

[Jumponthemoon]

yeah I think I'll choose to move onto 3dgs. still really appreciate your kind and detailed response!

Hi, i'm wondering if you can successfully run on single RTX 4090?

[yifanlu0227]

I tested with my 4090, it is okay

[Jumponthemoon]

I tested with my 4090, it is okay

Thank you so much!