windows; CUDA Compiler not found

[JoeRu]

No description provided.

[JoeRu]

Hi,
i'm not used to windows - but in need to get darknet compiled to W10.

it compiles - but without cuda support.

So; This is a fresh W10 Client. i did setup VS2019 community edition; Downloaded Cuda, opencv and all with vcpk; Added a ton of environment variables; So far it compiles to a running darknet.exe - but without cuda support. It does find cuda - but not the cuda compiler.

i did find the tipp to uninstall VS and cuda - and did so - but it does not have an impact in the not finding of cmake of the nvcc file. which is in $path... so im stuck and with no idea left and would appriciate any help many times.

Mode                LastWriteTime         Length Name
----                -------------         ------ ----
d-----       18.04.2019     23:38                build_win_release
-- Selecting Windows SDK version 10.0.17763.0 to target Windows 10.0.17134.
-- The C compiler identification is MSVC 19.20.27508.1
-- The CXX compiler identification is MSVC 19.20.27508.1
-- Check for working C compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Community/VC/Tools/MSVC/14.20.27508/bin/Hostx64/x64/cl.exe
-- Check for working C compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Community/VC/Tools/MSVC/14.20.27508/bin/Hostx64/x64/cl.exe -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Community/VC/Tools/MSVC/14.20.27508/bin/Hostx64/x64/cl.exe
-- Check for working CXX compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Community/VC/Tools/MSVC/14.20.27508/bin/Hostx64/x64/cl.exe -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Looking for a CUDA compiler
-- Looking for a CUDA compiler - **NOTFOUND**
-- Found CUDA: C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v10.1 (found suitable exact version "10.1")
-- Found OpenCV: C:/Users/Shadow/source/repos/vcpkg/installed/x64-windows (found version "3.4.3")
-- Found Stb: C:/Users/Shadow/Documents/alex/darknet/3rdparty/stb/include
-- ZED SDK not enabled, since it requires CUDA
-- Looking for pthread.h
-- Looking for pthread.h - not found
-- Found Threads: TRUE
-- Found PThreads_windows: optimized;C:/Users/Shadow/source/repos/vcpkg/installed/x64-windows/lib/pthreadVC2.lib;debug;C:/Users/Shadow/source/repos/vcpkg/installed/x64-windows/debug/lib/pthreadVC2d.lib

[AlexeyAB]

@JoeRu Hi,

• Did you firstly install MSVS 2019 and only after that did you install CUDA 10.0?
• Can you show fullscreen screenshot of MSVS 2019 with opened Darknet?

[JoeRu]

Hi @AlexeyAB

@JoeRu Hi,

• Did you firstly install MSVS 2019 and only after that did you install CUDA 10.0?
  Yes; But did install CUDA 10.1; But that shouldn't be the issue - or?

• Can you show fullscreen screenshot of MSVS 2019 with opened Darknet?
  yes - of course; Shall i get on more details?
  

...
I wonder if it were more easy to create a docker-container for building/running/training. Than the whole dependency thing blows away.... For example this one is provided by nvidia
https://hub.docker.com/r/nvidia/cuda/

[JoeRu]

[JoeRu]

hmm. thats from cmake documentation; Even this warning is wrong

Added missing env variable CUDA_TOOLKIT_ROOT_DIR
https://stackoverflow.com/questions/19980412/how-to-let-cmake-find-cuda

FindCUDA
--------

.. deprecated:: 3.10

  Superseded by first-class support for the CUDA language in CMake.


Replacement
^^^^^^^^^^^

It is no longer necessary to use this module or call ``find_package(CUDA)``.
Instead, list ``CUDA`` among the languages named in the top-level
call to the :command:`project` command, or call the
:command:`enable_language` command with ``CUDA``.
Then one can add CUDA (``.cu``) sources to programs directly
in calls to :command:`add_library` and :command:`add_executable`.

[AlexeyAB]

CUDA nsight plugin isn't installed in MSVS.
You have installed MSVS after CUDA installation, but you must do it in a different order.

Just re-install CUDA.

[cenit]

There was a bug inside vcpkg for CUDA. I fixed it not long ago, please do a git pull and then ./bootstrap-vcpkg.ps1 again to rebuild vcpkg with the bugfix. This is in case you are building darknet through vcpkg

[cenit]

A docker container and distribution, for Linux CPU, Linux GPU and Windows (only CPU unfortunately, gpu is unsupported in native windows containers as of now) will come soon, I hope. It depends on how much busy I will be (probably a lot), if I have to do it alone

[cenit]

@JoeRu Sorry for the chat, I am on phone. Please let me know if rebuilding vcpkg solved your problems (it should 😉)
I can explain better what was going on later if you are interested (why not found but the line after it was found)

[JoeRu]

@AlexeyAB I did a reinstallation; The plugin is there - it just not open to see.
@cenit you got me confused; Which repo should i pull; The VCPKG? I did use the one from the explanation which is from Microsoft. Where do i find the ./bootstrap-vcpkg.ps1? another branch? puzzled

[cenit]

@JoeRu up until one week ago, it was impossible to build darknet with Cuda inside vcpkg, due to a bug in vcpkg. Now it is fixed and you have to update your vcpkg installation (pulling and rebuilding it) to fix it. Building darknet inside vcpkg means doing just

./vcpkg install darknet[opencv-cuda]

It downloads darknet from this repository, setup dependencies automatically (stb, pthreads, OpenCV and CUDA), and builds everything, in a single step.
It was not working with CUDA, only CPU, because vcpkg filters the environment and some necessary environment variables were missing. The result was exactly what you are experiencing:

-- Looking for a CUDA compiler
-- Looking for a CUDA compiler - **NOTFOUND**
-- Found CUDA: C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v10.1 (found suitable exact version "10.1")
-- Found OpenCV: C:/Users/Shadow/source/repos/vcpkg/installed/x64-windows (found version "3.4.3")

Let's understand what's going on: the first two lines come from darknet, which uses the modern enable_language(CUDA) inside the CMakeLists.txt. On your pc, it cannot find CUDA. To look for it, it searches for the PATH and the CUDACXX environment variable, in case of Ninja generator (broken for our case, but that's another topic, that's why we don't use it), or it searches for CUDA_PATH and then delegates everything to the Visual Studio CUDA plug-in, in case of the Visual Studio generator (which is our choice).
Then the same configurations says Found CUDA..., which is surprising to say the least, but it does not come from darknet, it comes from OpenCV (which is found right after), which looks for CUDA using the old find_package(CUDA) if it is built CUDA-enabled. As you also found while looking on internet about help, this method of finding CUDA is now deprecated by CMake.
Unfortunately, find_package(CUDA), while being deprecated, is still more powerful finding newer versions than enable_language(CUDA), so it is working better also in difficult environments (it was working with a broken vcpkg, it is working also for you). We are not switching to it anyway, because its future is doomed and it would be an unnecessary temporary effort.

At the beginning I was thinking, because of the symptoms, that you were building darknet inside vcpkg, but now I realised that you're not, since you are describing also the output of build.ps1, you're just building Darknet's dependencies using vcpkg (and so stb, pthreads, opencv, ... whatever you want).

Unfortunately, I have to say that Visual Studio 2019 + CUDA 10.1 is still unsupported. I had the opportunity to test and verify Visual Studio 2019, but without CUDA.
I think that from the CMake point of view this combination it is too new (they are a little bit slow with these things, unfortunately, but it's getting better). Is it really impossible for you to install VS 2017 for now? Which version of CMake are you using? Please try of course the very latest one, even a nightly if possible.

[JoeRu]

OK; So - some experiments on my side - with the following results.

i did an uninstall of VS2019 and did an installation of VS2017;
I did reinstall CUDA 10.1

i did a pull of https://github.com/Microsoft/vcpkg with the master branch;
so darknet compiles with your suggestion without any quotable error or anything:

./vcpkg install darknet[opencv-cuda]

But only CPU support - no CUDA. So if this would compile with CUDA this would be a very nice and easy way to install darknet NICE

On the behaviour of the build1.ps1 did nothing change.
The NSight-Menue is visible;
Except this line is new; Maybe i missed to install something;
Warning: no full Visual Studio setup has been found, extending search to include also partial installations

I did lost track of all the environment variables - so there is actual the following set

[JoeRu]

so - solved. For future references and to check:

Dependency : VS 2017 - vs 2019 is not working (yet)

if CUDA doesn't work
Check directory
C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\Common7\IDE\VC\VCTargets\BuildCustomizations\CUDA 10.1.props
if props-files are existent - if not uninstall ALL NVIDIA products; Delete folder for GPU in Program Files.
Delete all content of %TEMP%. Reboot;
--- Reinstall CUDA

For more detailed checking :
Open darknet.sln and set environmet-variable TRACEDESIGNTIME=true
Configure as described in manual setup; The produced Logfile in %TEMP% does give more details of missing files.

C:\Users\Shadow\Documents\alex\darknet\build\darknet\darknet.vcxproj(55,5): error MSB4019: The imported project "C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\Common7\IDE\VC\VCTargets\BuildCustomizations\CUDA 10.1.props" was not found. Confirm that the path in the <Import> declaration is correct, and that the file exists on disk.
I do hate windows and all this not solved dependencys and not working installers...

[cenit]

good that it's working now.
As a reference, if you use the CMake toolchain and VCPKG (and not the legacy VS solutions or the makefile), you don't need to define any environment variable apart from those in the guide (so only VCPKG_ROOT and VCPKG_DEFAULT TRIPLET).
So, if you want to retry with CMake and vcpkg, please remove your definitions of OpenCV_DIR, CUDACXX, CUDA_BIN_PATH (and CUDNN if you proper install it inside the CUDA folder) as they are not necessary, even damaging

[cenit]

I still don't understand what happened in your installation. If you want to check, in continuous integration we verify for each commit building on linux, macOS, Windows, with and without CUDA, and it is properly working in all configurations. I will soon add checks for CUDA and VS2019, so to verify progress with this configuration

[JoeRu]

hi @cenit
i think the CUDA Installation went wrong on some point - maybe in combination with 2019; So - the props-file referenced were not beeing installed; Then i did searched around on nvidia site for the missing props files and there an nvidia admin quoted the tipp to uninstall everything and even check some folders and remove them by hand;
I guess - some of the installation routines are checking stuff for existence and did not execute some other stuff.. but thats just wild guessing.

i will do another fresh pull of VCPKG and try it to install the suggested way - with all environment removed. But it takes a lot of time to compile - so i will report on later point on time.

[cenit]

don't need to rebuild all
the vcpkg folder should be fine. Just remove the env variables I told you, then post the full logs and eventually the CMakeCache.txt files from the build folders.

[JoeRu]

Just to close this one and maybe to update Documentation for Windows:

Download Visual-Studio 2017 and install;
If not available -- install GIT
Download CUDA and install;
Download CUDNN and unzip it into %CUDA_PATH%\..\;
Clone VCPKG
set environment-variable VCPKG_DEFAULT_TRIPLET : x64-windows
set environment-variable VCPKG_ROOT : path-where-you-put-vcpkg
build VCPKG like described in quickstart.
Open a Terminal or Powershell enter cd %VCPKG_ROOT% and run ./vcpkg install darknet[opencv-cuda]

Be patient -- it did take about 2-3 hours to compile everything

Add to environment variable PATH the following:
%VCPKG_ROOT%\installed\%VCPKG_DEFAULT_TRIPLET%\bin;%VCPKG_ROOT%\installed\%VCPKG_DEFAULT_TRIPLET%\tools\darknet

Now you should be able to start darknet.exe from every fresh CMD or Powershell.

[cenit]

exactly what I meant and what we aim in a not-so-distant future. Maybe automatically done by a script 😄 (it's still a little bit experimental to declare it ready 😉)

[cenit]

and to update, you just need to go to %VCPKG_ROOT% folder and then do git pull and .\vcpkg.exe upgrade --no-dry-run

[DarLador]

I'm having similar issue (#3028).
I use CMake (opencv 3.4.6 from opensv.org). Does someone have an idea how to make it work with CMake?

[Monkey1GIt]

My environment: Win10 + cuda 10.1 + cuddn + Visual Studio 2019.

I think I found the reason, this is problem between the cuda and Visual Studio. Because the cuda should be installed after the Visual Studio. When cuda is installed there will be a item for finding whether you have installed Visual Studio before, if you did then cuda adapting the environment, if you didn't you can ignore. Like #2971 (comment)

You can reinstall the cuda to solve the problem. There is a way to test if cuda is satisfied to Visual Studio:

open Visual Studio 2019 > New > New item

you will find a item named cuda 10.1 Runtime, like this:

The cmake will like this: