fix float16 and float64 kernels

[fsx950223]

Description

Brief Description of the PR:
Fix edt float16 and float64 kernels
Fixes # (issue)

Type of change

• Bug fix
• New Tutorial
• Updated or additional documentation
• Additional Testing
• New Activation and the changes conform to the activation contribution guidelines
• New Callback and the changes conform to the callback contribution guidelines
• New Image addition and the changes conform to the image op contribution guidelines
• New Layer and the changes conform to the layer contribution guidelines
• New Loss and the changes conform to the loss contribution guidelines
• New Metric and the changes conform to the metric contribution guidelines
• New Optimizer and the changes conform to the optimizer contribution guidelines
• New RNN Cell and the changes conform to the rnn contribution guidelines
• New Seq2seq addition and the changes conform to the seq2seq contribution guidelines
• New Text addition and the changes conform to the text op contribution guidelines

Checklist:

• I've properly formatted my code according to the guidelines
  • By running Black + Flake8
  • By running pre-commit hooks
• This PR addresses an already submitted issue for TensorFlow Addons
• I have made corresponding changes to the documentation
• I have added tests that prove my fix is effective or that my feature works
• This PR contains modifications to C++ custom-ops

How Has This Been Tested?

If you're adding a bugfix or new feature please describe the tests that you ran to verify your changes:
*

[WindQAQ]

Can you explain why this can solve the issue?

[fsx950223]

Can you explain why this can solve the issue?

The error:
ERROR: too many resources requested for launch.
I don't know why float16 and float64 kernels crashed with the same setting as tf.float32, but reduce block_dim solves the bug.

[WindQAQ]

I guess this will generate wrong results when the original threads per block is larger than 256?

[fsx950223]

I guess this will generate wrong results when the original threads per block is larger than 256?

No, it's a resource exceed bug.

[WindQAQ]

I guess this will generate wrong results when the original threads per block is larger than 256?

No, it's a resource exceed bug.

Hmm, if I understand correctly, the implementation is one thread do one plane (h, w) EDT. So if block_count == 1024 and thread_per_block == 512 (which means batch \times channel == 2^19), this change will launch 2^18 threads instead, and some image planes are not going to be updated, right?

[fsx950223]

I guess this will generate wrong results when the original threads per block is larger than 256?

No, it's a resource exceed bug.

Hmm, if I understand correctly, the implementation is one thread do one plane (h, w) EDT. So if block_count == 1024 and thread_per_block == 512 (which means batch \times channel == 2^19), this change will launch 2^18 threads instead, and some image planes are not going to be updated, right?

The results are same as before, but inference may a little slower.
Just change the max value of thread_per_block from 1024 to 256.

[WindQAQ]

Can you add a test of batch x channels == 2048 or larger?

[fsx950223]

I guess this will generate wrong results when the original threads per block is larger than 256?

No, it's a resource exceed bug.

Hmm, if I understand correctly, the implementation is one thread do one plane (h, w) EDT. So if block_count == 1024 and thread_per_block == 512 (which means batch \times channel == 2^19), this change will launch 2^18 threads instead, and some image planes are not going to be updated, right?

The results are same as before, but inference may a little slower.

Can you add a test of batch x channels == 2048 or larger?

Good point, I try it.

[fsx950223]

Can you add a test of batch x channels == 2048 or larger?

I found I should set block_count*4 and set thread_per_block/4

[WindQAQ]

Can you try this?

https://github.com/tensorflow/tensorflow/blob/100c0d6538986797e1b29877b9cbbc687ba54853/tensorflow/core/util/gpu_launch_config.h#L156-L159

[fsx950223]

Can you try this?

https://github.com/tensorflow/tensorflow/blob/100c0d6538986797e1b29877b9cbbc687ba54853/tensorflow/core/util/gpu_launch_config.h#L156-L159

Good, thanks.