Update NCCL_ASYNC_ERROR_HANDLING env variable

[priba]

What does this PR do?

The environment variable NCCL_ASYNC_ERROR_HANDLING is deprecated and it is suggested to use TORCH_NCCL_ASYNC_ERROR_HANDLING instead.

What issue(s) does this change relate to?

Applies #3266

Before submitting

• Have you read the contributor guidelines?
• Is this change a documentation change or typo fix? If so, skip the rest of this checklist.
• Was this change discussed/approved in a GitHub issue first? It is much more likely to be merged if so.
• Did you update any related docs and document your change?
• Did you update any related tests and add any new tests related to your change? (see testing)
• Did you run the tests locally to make sure they pass?
• Did you run pre-commit on your change? (see the pre-commit section of prerequisites)

[mvpatel2000]

It looks like this is introduced in torch 2.2 and doesn't exist in torch 2.1. Since we try to support the last 3 minor versions of torch, we unfortunately can't switch until torch 2.4 comes out. I can merge this PR then! Thanks for opening it.

[priba]

Shall we make it conditional? Something like:

from packaging import version

if version.parse(torch.__version__) < version.parse('2.2.0'):
     to_patch = {"NCCL_ASYNC_ERROR_HANDLING": '1'}
else:
     to_patch = {"TORCH_NCCL_ASYNC_ERROR_HANDLING"='1'}

...

with _patch_env(**to_patch ):

Not sure if it is worth though.

[mvpatel2000]

Shall we make it conditional? Something like:

@priba that would also work! happy to accept that as a PR :)

[mvpatel2000]

LGTM, thanks for the PR!