Disable opportunistic reuse in async mr when cuda driver < 11.5

[rongou]

With NVIDIA/spark-rapids#4710 we found some issues with the async pool that may cause memory errors with older drivers. This was confirmed with the cuda team. For driver version < 11.5, we'll disable cudaMemPoolReuseAllowOpportunistic.

@abellina

[leofang]

This is not fully bullet proof: the old cudaDevAttrMemoryPoolsSupported check needs to be kept as the support could be hardware dependent.

Also, #990 is a renovation of the async MR support, so I'd suggest to keep others in the loop.

[rongou]

Added back the device attribute check.

[jrhemstad]

There is no need to fully disable cudaMallocAsync for <11.5. We just need to disable opportunistic reuse. This is done via cudaMemPoolSetAttribute and setting cudaMemPoolReuseAllowOpportunistic to zero.

https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__MEMORY__POOLS.html#group__CUDART__MEMORY__POOLS_1g0229135f7ef724b4f479a435ca300af5

[rongou]

Do we want to disable it for all cuda driver versions?

[jrhemstad]

Do we want to disable it for all cuda driver versions?

No, just for less than 11.5.

[rongou]

There is no need to fully disable cudaMallocAsync for <11.5. We just need to disable opportunistic reuse. This is done via cudaMemPoolSetAttribute and setting cudaMemPoolReuseAllowOpportunistic to zero.

https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__MEMORY__POOLS.html#group__CUDART__MEMORY__POOLS_1g0229135f7ef724b4f479a435ca300af5

Done.

[harrism]

@rongou can you please edit the title and description to better reflect the changes?

[rongou]

@harrism done.

[harrism]

@leofang please re-review.

[rongou]

@leofang please take another look. Thanks!

[jrhemstad]

@robertmaynard @rongou fyi, this is going to need to be updated with #990. So which ever merges first, the other will need to update.

[rongou]

@gpucibot merge