Incompatibility with CUDA 11.3

[stephenswat]

This is a quick continuation of #113, where we find that traccc is currently not compatible with CUDA 11.3, and I would like to know why. I'll keep this as a running log of my findings.

[stephenswat]

This is the compatibility matrix of CUDA versions installed on atspot01:

CUDA toolkit version	Works
10.1.243	❌ (expected)
10.2.89	❌ (expected)
11.0.3	✔️
11.1.1	✔️
11.2.2	✔️
11.3.1	❌
11.4.3	✔️
11.5.0	✔️
11.5.1	✔️

[stephenswat]

Okay, there is some extremely odd behaviour happening inside nvcc, and I think inside cudafe++. It seems that for CUDA toolkit 11.3.1, the cudafe1.cpp translations of counting_grid_capabilities.cu and populating_grid.cu are incorrectly referencing spacepoint_t:

zone_t(scalar v, const spacepoint_t< neighbor_t, 2>  &nhood) const {
dindex_sequence zone(scalar v, const spacepoint_t< dindex, 2U>  &nhood) const {
dindex_sequence zone(scalar v, const spacepoint_t< dindex, 2U>  &nhood) const {

Here are the corresponding lines for CUDA 11.4.3:

zone_t(scalar v, const array_type< neighbor_t, 2>  &nhood) const {
dindex_sequence zone(scalar v, const array_type< dindex, 2U>  &nhood) const {
dindex_sequence zone(scalar v, const array_type< scalar, 2U>  &nhood) const {

The corresponding lines from detray/core/include/detray/grids/axis.hpp are:

zone_t(scalar v, const array_type<neighbor_t, 2> &nhood) const {
dindex_sequence zone(scalar v, const array_type<dindex, 2> &nhood) const {
dindex_sequence zone(scalar v, const array_type<scalar, 2> &nhood) const {

These files are generated from the corresponding .cpp4.ii files by cudafe++.

[stephenswat]

I can confirm that the .cpp4.ii files have identical versions of these lines.

[stephenswat]

Invoking the two versions of cudafe++ (11.3.1 and 11.4.3) on exactly the same input (the cudafe1.stub.cpp generated by cicc 11.3.1, and the .cpp4.ii by the 11.3.1 preprocessor) results in the same behaviour: the 11.3.1 version erroneously inserts spacepoint_t where it shouldn't be. Running the 11.2.2 version of cudafe++ produces the same correct output that 11.4.3 does.

[stephenswat]

Okay, I am sufficiently convinced that this is a bug in cudafe++.

[stephenswat]

Okay, I can't really debug this any further, because cudafe++ is opaque as hell, and as far as I know there aren't really any changelogs or documentation for it. However, I have boiled down the issue to the detray::axis::regular type. My guess is that cudafe++ can't cope with the complex kind (* → uint → *) → (*^n → *) → *, which I suspect is either due to the n-ary nature of the kind of the second type parameter, or because the first type parameter accepts a non-* kind.

The symptom of this is that it starts substituting (seemingly) random (incompatible) types, such as spacepoint_t, where it expects the array type or the vector type. I presume that this might be some kind of indexing error happening at template resolution time, but I don't have enough evidence to make any concrete claims.

To conclude, CUDA 11.3.1 is completely bat-shit insane. The only next steps might be to investigate CUDA 11.3.0 and CUDA 11.4.0, the directly preceding and following versions.