[CUDA] Attention kernel provider option

[tianleiwu]

Description

• Add a cuda provider option sdpa_kernel to choose which attention kernel to run for testing purpose.
• Allow dump which attention kernel is used per node.
• Reserve a flag for cudnn flash attention which will be added soon.

CUDA provider option sdpa_kernel

Instead of setting environment variable, we also support setting it in provider option. Note that the setting is global per session. That could help performance testing of each kernel.

Attention Kernel Debug Info

Set an environment variable ORT_ENABLE_ATTENTION_KERNEL_DEBUG_INFO=1, and ORT will print sdpa kernel used in each node:

For example

ORT_ENABLE_ATTENTION_KERNEL_DEBUG_INFO=1 ./onnxruntime_test_all --gtest_filter=MultiHeadAttentionTest*

It will show debug information of kernel used in testing:

[ RUN      ] MultiHeadAttentionTest.SelfAttention_Batch2_HeadSize32_NoBias_NoMask_PackedQKV
AttentionKernelOptions: FLASH_ATTENTION=0 EFFICIENT_ATTENTION=0 TRT_FUSED_ATTENTION=1 CUDNN_FLASH_ATTENTION=0 TRT_FLASH_ATTENTION=1 TRT_CROSS_ATTENTION=0 TRT_CAUSAL_ATTENTION=0 MATH=1
Operator=MultiHeadAttention Node=node1 DataType=fp16 TRT_FUSED_ATTENTION=1
AttentionKernelOptions: FLASH_ATTENTION=0 EFFICIENT_ATTENTION=1 TRT_FUSED_ATTENTION=0 CUDNN_FLASH_ATTENTION=0 TRT_FLASH_ATTENTION=0 TRT_CROSS_ATTENTION=0 TRT_CAUSAL_ATTENTION=0 MATH=1
Operator=MultiHeadAttention Node=node1 DataType=fp16 EFFICIENT_ATTENTION=1

In this test case, the debug info shows that one session uses trt fused attention and another session use efficient attention.

Motivation and Context