CUDA 10.2 incompatible with GCC 9.3 and Clang 9.0 #3654

jngrad · 2020-04-10T12:50:06Z

According to the CUDA 10.2 System Requirements, Table 1, the nvcc compiler in CUDA 10 doesn't support the latest versions of GCC and Clang. At the time of writing, Ubuntu 20.04 doesn't appear in that list, but I can confirm nvcc supports the older GCC 8.4 and Clang 8.0 but not the default GCC 9.3 and Clang 9.0.

On Ubuntu 20.04, nvidia-cuda-toolkit (CUDA 10.1) is incompatible with the packaged GCC version 9.3 (default from build-essential). MWE:

$> touch mwe.cu
$> /usr/bin/nvcc -ccbin /usr/bin/cc -m64 -Xcompiler=-fPIC -Xptxas=-O3 -Xcompiler=-O3,-g \
     -gencode=arch=compute_52,code=sm_52 -gencode=arch=compute_52,code=compute_52 \
     -std=c++14 -O3 -g -DNVCC -I/usr/include mwe.cu
In file included from /usr/include/cuda_runtime.h:83,
                 from <command-line>:
/usr/include/crt/host_config.h:138:2: error: #error -- unsupported GNU version! gcc versions later than 8 are not supported!
  138 | #error -- unsupported GNU version! gcc versions later than 8 are not supported!
      |  ^~~~~

Fortunately, nvidia-cuda-toolkit automatically installs GCC 8.4 as a dependency if Clang 8.0 isn't already present, so this compiler error is not an obstacle to our work in #3611.

Solution for the ubuntu-20.04 docker image:

apt -y install gcc-8 g++-8
update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-8 8
update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-8 8

Solution for end users:

cmake -DCMAKE_C_COMPILER=$(which gcc-8) -DCMAKE_CXX_COMPILER=$(which g++-8) \
      -DWITH_CUDA=ON ..

The text was updated successfully, but these errors were encountered:

mkuron · 2020-04-10T13:00:14Z

You can try patching out that check. We did that on our desktop machines with Ubuntu 16.04 and 18.04 to get the default system compiler to work with CUDA and it was always fine. We will most likely have to do that again once we upgrade them to Ubuntu 20.04. This is what we use to make CUDA 9.1 work with GCC 7 on our desktop machines:

--- /usr/include/crt/host_config.h      2017-12-02 00:36:56.000000000 +0100
+++ /usr/include/crt/host_config.h      2018-08-31 14:43:16.744783062 +0200
@@ -116,11 +116,11 @@
 
 #if defined(__GNUC__)
 
-#if __GNUC__ > 6
+#if __GNUC__ > 7
 
-#error -- unsupported GNU version! gcc versions later than 6 are not supported!
+#error -- unsupported GNU version! gcc versions later than 7 are not supported!
 
-#endif /* __GNUC__ > 6 */
+#endif /* __GNUC__ > 7 */
 
 #if defined(__APPLE__) && defined(__MACH__) && !defined(__clang__)
 #error -- clang and clang++ are the only supported host compilers on Mac OS X!

jngrad · 2020-04-10T13:10:25Z

I've just applied that patch to enable GCC 9.3, but all LB GPU files give the same compiler error:

/usr/bin/nvcc \
  /home/espresso/espresso/src/core/virtual_sites/lb_inertialess_tracers_cuda.cu -c -o \
  /home/espresso/espresso/build/src/core/CMakeFiles/EspressoCuda.dir/virtual_sites/./EspressoCuda_generated_lb_inertialess_tracers_cuda.cu.o \
  -ccbin /usr/bin/gcc-9 -m64 -DEspressoCuda_EXPORTS -Xcompiler ,\"-fPIC\" \
  -gencode=arch=compute_30,code=sm_30 -gencode=arch=compute_52,code=sm_52 \
  -gencode=arch=compute_52,code=compute_52 -std=c++14 -O3 -g -Xptxas=-O3 \
  -Xcompiler=-O3,-g -DNVCC -I/usr/include -I/home/espresso/espresso/src/core \
  -I/home/espresso/espresso/src/config -I/home/espresso/espresso/build/src/config \
  -I/home/espresso/espresso/src/shapes/include \
  -I/home/espresso/espresso/src/utils/include
/usr/include/c++/9/bits/stl_function.h(437): error: identifier "__builtin_is_constant_evaluated" is undefined

1 error detected in the compilation of "/tmp/tmpxft_00003bc2_00000000-7_lb_inertialess_tracers_cuda.compute_52.cpp1.ii".
-- Removing /home/espresso/espresso/build/src/core/CMakeFiles/EspressoCuda.dir/virtual_sites/./EspressoCuda_generated_lb_inertialess_tracers_cuda.cu.o
/usr/bin/cmake -E remove /home/espresso/espresso/build/src/core/CMakeFiles/EspressoCuda.dir/virtual_sites/./EspressoCuda_generated_lb_inertialess_tracers_cuda.cu.o
CMake Error at EspressoCuda_generated_lb_inertialess_tracers_cuda.cu.o.RelWithAssert.cmake:277 (message):
  Error generating file
  /home/espresso/espresso/build/src/core/CMakeFiles/EspressoCuda.dir/virtual_sites/./EspressoCuda_generated_lb_inertialess_tracers_cuda.cu.o

mkuron · 2020-04-10T14:48:24Z

That's annoying. You might be able to replace __builtin_is_constant_evaluated with std::is_constant_evaluated. That's what https://forums.developer.nvidia.com/t/cuda-10-installation-on-fedora-30-with-gcc-9-1-1-20190503/74989/11 suggests.

jngrad · 2020-04-10T15:13:38Z

The patch so far:

--- /usr/include/crt/host_config.h	2020-04-10 14:53:37.947817611 +0000
+++ /usr/include/crt/host_config.h	2020-04-10 14:54:08.204086102 +0000
@@ -133,11 +133,11 @@
 
 #if defined(__GNUC__)
 
-#if __GNUC__ > 8
+#if __GNUC__ > 9
 
-#error -- unsupported GNU version! gcc versions later than 8 are not supported!
+#error -- unsupported GNU version! gcc versions later than 9 are not supported!
 
-#endif /* __GNUC__ > 8 */
+#endif /* __GNUC__ > 9 */
 
 
 #if defined(__clang__) && !defined(__ibmxl_vrm__) && !defined(__ICC) && !defined(__HORIZON__) && !defined(__APPLE__)
--- /usr/include/c++/9/bits/stl_function.h	2020-04-10 14:54:36.920340490 +0000
+++ /usr/include/c++/9/bits/stl_function.h	2020-04-10 14:59:58.739168200 +0000
@@ -434,7 +434,12 @@
       {
 #if __cplusplus >= 201402L
 #ifdef _GLIBCXX_HAVE_BUILTIN_IS_CONSTANT_EVALUATED
-	if (__builtin_is_constant_evaluated())
+#ifdef __NVCC__
+#include  <type_traits>
+	if (std::is_constant_evaluated())
+#else
+ 	if (__builtin_is_constant_evaluated())
+#endif
 #else
 	if (__builtin_constant_p(__x < __y))
 #endif

CMake fails:

-- Found MPI_C: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so (found suitable version "3.1", minimum required is "3.0") 
-- Could NOT find MPI_CXX (missing: MPI_CXX_WORKS) (Required is at least version "3.0")
CMake Error at /usr/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:146 (message):
  Could NOT find MPI (missing: MPI_CXX_FOUND) (found suitable version "3.1",
  minimum required is "3.0")
Call Stack (most recent call first):
  /usr/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:393 (_FPHSA_FAILURE_MESSAGE)
  /usr/share/cmake-3.16/Modules/FindMPI.cmake:1688 (find_package_handle_standard_args)
  CMakeLists.txt:254 (find_package)

Compiler error:

$> /usr/bin/nvcc /home/espresso/espresso/src/core/virtual_sites/lb_inertialess_tracers_cuda.cu -c -o \
     /home/espresso/espresso/build/src/core/CMakeFiles/EspressoCuda.dir/virtual_sites/./EspressoCuda_generated_lb_inertialess_tracers_cuda.cu.o \
     -ccbin /usr/bin/gcc-9 -m64 -DEspressoCuda_EXPORTS -Xcompiler ,\"-fPIC\" \
     -gencode=arch=compute_30,code=sm_30 -gencode=arch=compute_52,code=sm_52 \
     -gencode=arch=compute_52,code=compute_52  -O3 -g -Xptxas=-O3 -Xcompiler=-O3,-g \
     -DNVCC -I/usr/include -I/home/espresso/espresso/src/core \
     -I/home/espresso/espresso/src/config -I/home/espresso/espresso/build/src/config \
     -I/home/espresso/espresso/src/shapes/include -I/home/espresso/espresso/src/utils/include
/usr/include/c++/9/bits/stl_function.h(439): error: namespace "std" has no member "is_constant_evaluated"

The issue is that std::is_constant_evaluated() is a C++20 feature. Removing -std=c++14 doesn't help and both -std=c++20 and -std=c++2a aren't recognized by nvcc.

mkuron · 2020-04-10T18:13:22Z

Okay, the error comes from https://github.com/gcc-mirror/gcc/blob/releases/gcc-9.3.0/libstdc++-v3/include/bits/stl_function.h#L437. This means you should be able to conditionally undef _GLIBCXX_HAVE_BUILTIN_IS_CONSTANT_EVALUATED in https://github.com/gcc-mirror/gcc/blob/releases/gcc-9.3.0/libstdc++-v3/include/bits/c++config#L649-L651 (if __GNUC__ >= 9 && !defined(__NVCC__)).

Let me know if that works. Otherwise, I'll need to have a look before we upgrade the desktop computers.

RudolfWeeber · 2020-04-10T18:29:38Z

Let's discuss this in the Espresso meeting as well.

jngrad · 2020-04-10T18:45:31Z

@mkuron The file in question isn't shipped with the header library. It's not even included in another file (grep -rF '_GLIBCXX_HAVE_BUILTIN_IS_CONSTANT_EVALUATED 1' /usr/include/c++/9/ returns nothing). Manually fixing the ifdefs solved the compiler error and all GPU tests passed. Full patch:

--- /usr/include/crt/host_config.h	2020-04-10 14:53:37.947817611 +0000
+++ /usr/include/crt/host_config.h	2020-04-10 14:54:08.204086102 +0000
@@ -133,11 +133,11 @@
 
 #if defined(__GNUC__)
 
-#if __GNUC__ > 8
+#if __GNUC__ > 9
 
-#error -- unsupported GNU version! gcc versions later than 8 are not supported!
+#error -- unsupported GNU version! gcc versions later than 9 are not supported!
 
-#endif /* __GNUC__ > 8 */
+#endif /* __GNUC__ > 9 */
 
 
 #if defined(__clang__) && !defined(__ibmxl_vrm__) && !defined(__ICC) && !defined(__HORIZON__) && !defined(__APPLE__)
--- /usr/include/c++/9/bits/stl_function.h	2020-04-10 18:39:20.692976417 +0000
+++ /usr/include/c++/9/bits/stl_function.h	2020-04-10 18:37:56.744398062 +0000
@@ -414,7 +414,7 @@
       operator()(_Tp* __x, _Tp* __y) const _GLIBCXX_NOTHROW
       {
 #if __cplusplus >= 201402L
-#ifdef _GLIBCXX_HAVE_BUILTIN_IS_CONSTANT_EVALUATED
+#if defined(_GLIBCXX_HAVE_BUILTIN_IS_CONSTANT_EVALUATED) && !defined(__CUDACC__)
 	if (__builtin_is_constant_evaluated())
 #else
 	if (__builtin_constant_p(__x > __y))
@@ -433,7 +433,7 @@
       operator()(_Tp* __x, _Tp* __y) const _GLIBCXX_NOTHROW
       {
 #if __cplusplus >= 201402L
-#ifdef _GLIBCXX_HAVE_BUILTIN_IS_CONSTANT_EVALUATED
+#if defined(_GLIBCXX_HAVE_BUILTIN_IS_CONSTANT_EVALUATED) && !defined(__CUDACC__)
 	if (__builtin_is_constant_evaluated())
 #else
 	if (__builtin_constant_p(__x < __y))
@@ -452,7 +452,7 @@
       operator()(_Tp* __x, _Tp* __y) const _GLIBCXX_NOTHROW
       {
 #if __cplusplus >= 201402L
-#ifdef _GLIBCXX_HAVE_BUILTIN_IS_CONSTANT_EVALUATED
+#if defined(_GLIBCXX_HAVE_BUILTIN_IS_CONSTANT_EVALUATED) && !defined(__CUDACC__)
 	if (__builtin_is_constant_evaluated())
 #else
 	if (__builtin_constant_p(__x >= __y))
@@ -471,7 +471,7 @@
       operator()(_Tp* __x, _Tp* __y) const _GLIBCXX_NOTHROW
       {
 #if __cplusplus >= 201402L
-#ifdef _GLIBCXX_HAVE_BUILTIN_IS_CONSTANT_EVALUATED
+#if defined(_GLIBCXX_HAVE_BUILTIN_IS_CONSTANT_EVALUATED) && !defined(__CUDACC__)
 	if (__builtin_is_constant_evaluated())
 #else
 	if (__builtin_constant_p(__x <= __y))

mkuron · 2020-04-11T17:40:52Z

Glad that worked! Using GCC 8 would have been messier because we would probably have needed to recompile Boost. Their C++ ABI is not downwards-compatible (or compatible at all).

KaiSzuttor · 2020-04-14T07:16:47Z

Why do they exclude those compiler versions by default if it actually works?

mkuron · 2020-04-14T07:38:53Z

Nvidia just excludes all compilers that were released after the CUDA version. That is, they are not officially supported but work fine.

Description of changes: - reduce number of CI images for CUDA jobs (partial fix for #3611) - test CUDA 9.1 and 10.1 using compatible compilers without patches (fixes #3654) - drop support for Ubuntu 16.04 - bump minimal Boost version to 1.65 (partial fix for #3093) - bump Python packages to the versions available in Ubuntu 18.04 (partial fix for #3421) - add missing lxml package (fixes #3686) - fix issues in docs revealed by the new Doxygen and Sphinx versions

NilsRethmeier · 2020-11-06T15:27:52Z

Anyone reading this may be interested in a simple version that worked:

install gcc-8
sudo apt -y install gcc-8
add gcc (version) alternatives
sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-8 8
(may be optional, only do if 3. complains) I had to create this for the next command to work (gcc-9 was installed by build-essentials on Ubuntu 20.04)
sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-9 9
select gcc-8 as compiler (version) -- press the number that points to gcc-8
sudo update-alternatives --config gcc
now cuda versions that require old gcc-versions work
sudo sh cuda_10.2.89_440.33.01_linux.run
(optional) -- select gcc-9 as compiler (version) -- press the number that points to gcc-8

Still testing this setup, but so far it seems to work

joeloliver · 2020-12-25T06:17:09Z

Anyone reading this may be interested in a simple version that worked:

install gcc-8
sudo apt -y install gcc-8

add gcc (version) alternatives
sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-8 8

(may be optional, only do if 3. complains) I had to create this for the next command to work (gcc-9 was installed by build-essentials on Ubuntu 20.04)
sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-9 9

select gcc-8 as compiler (version) -- press the number that points to gcc-8
sudo update-alternatives --config gcc

now cuda versions that require old gcc-versions work
sudo sh cuda_10.2.89_440.33.01_linux.run

(optional) -- select gcc-9 as compiler (version) -- press the number that points to gcc-8

Still testing this setup, but so far it seems to work

Thanks @NilsRethmeier, worked just perfect

NoWhereMan1979 · 2021-08-01T05:07:52Z

You can simply use --override to override this check.

jngrad changed the title ~~CUDA 10 incompatible with recent GCC and Clang~~ CUDA 10.2 incompatible with GCC 9.3 and Clang 9.0 Apr 18, 2020

jngrad mentioned this issue Apr 26, 2020

Simplify CI and bump version requirements #3687

Merged

jngrad closed this as completed in #3687 Apr 27, 2020

adamrankin mentioned this issue Jan 20, 2021

Question about building PlusServer with OpenCV on Ubuntu 20.04.1 PlusToolkit/PlusLib#767

Closed

Demokritus mentioned this issue Mar 24, 2021

gmake[3]: *** No rule to make target benchmark/lib/libbenchmark.a', needed by bench/sg_benchmark'. Stop. rapidsai/cuml#3654

Closed

olivierschreiber mentioned this issue Aug 20, 2021

Fatal Python error: Py_Initialize: can't initialize sys standard streams espresso 4.0.2 #3149

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CUDA 10.2 incompatible with GCC 9.3 and Clang 9.0 #3654

CUDA 10.2 incompatible with GCC 9.3 and Clang 9.0 #3654

jngrad commented Apr 10, 2020

mkuron commented Apr 10, 2020

jngrad commented Apr 10, 2020

mkuron commented Apr 10, 2020

jngrad commented Apr 10, 2020

mkuron commented Apr 10, 2020

RudolfWeeber commented Apr 10, 2020

jngrad commented Apr 10, 2020

mkuron commented Apr 11, 2020 •

edited

Loading

KaiSzuttor commented Apr 14, 2020

mkuron commented Apr 14, 2020

NilsRethmeier commented Nov 6, 2020

joeloliver commented Dec 25, 2020

NoWhereMan1979 commented Aug 1, 2021

CUDA 10.2 incompatible with GCC 9.3 and Clang 9.0 #3654

CUDA 10.2 incompatible with GCC 9.3 and Clang 9.0 #3654

Comments

jngrad commented Apr 10, 2020

mkuron commented Apr 10, 2020

jngrad commented Apr 10, 2020

mkuron commented Apr 10, 2020

jngrad commented Apr 10, 2020

mkuron commented Apr 10, 2020

RudolfWeeber commented Apr 10, 2020

jngrad commented Apr 10, 2020

mkuron commented Apr 11, 2020 • edited Loading

KaiSzuttor commented Apr 14, 2020

mkuron commented Apr 14, 2020

NilsRethmeier commented Nov 6, 2020

joeloliver commented Dec 25, 2020

NoWhereMan1979 commented Aug 1, 2021

mkuron commented Apr 11, 2020 •

edited

Loading