feat: make stream wait timeout a first class citizen

[mutianf]

Port change googleapis/gax-java#1409 because gax is going to have a major version bump.
The original PR was closed because remapping rpcTimeout is too risky. But now gax is having a major version bump soon we're hoping to fix this behavior.

From the original comment:

For a server streaming api, there are 4 conceptual timeouts:

1. overall operation timeout - the maximum amount time that passes from the user invoking a method until that method exits
2. attempt/rpc timeout - if retries are enabled the maximum amount of time that passes for each attempt in an operation
3. message wait timeout - the maximum amount of time to wait for the next message from the server
4. idle timeout - how long to wait before considering the stream orphaned by the user and closing it

Each has a usecase:

1. operation timeout is useful for users to fulfill their own slo guarantees
2. attempt timeout are useful when a client developer knows the absolute limit of an rpc but that limit happens to be shorter than the required slo for a customer. For example a point read of a bigtable key shouldnt ever take more than 100ms. If it does, then we can assume something is wrong (GFE died without sending a FIN packet) and abort the request and retry.
3. message wait timeouts have a similar use as attempt timeouts with tighter guarantees
4. idle timeouts are useful to reduce buffer bloat on the server
   Currently all of them are implemented in gax but the delineation was muddied by me a while back. This PR tries to fix the situation. In the current world, operation timeout is defined by RetrySettings#totalTimeout, idle timeout is defined by ServerStreamingCallSettings#idleTimeout. However RetrySettings#rpcTimeout is mapped to message wait timeout and attempt timeout is only configureable per call using ApiCallContext#withTimeout.

This PR cleans up the situation by mapping RetrySettings#rpcTimeout to attempt timeouts and exposes a new setting for wait timeout on ServerStreamingCallSettings.

[sonarcloud]

SonarCloud Quality Gate failed.   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

62.5% Coverage
0.0% Duplication

[suztomo]

@mutianf We planned a major release but it turned out that we can implement many of the ideas without introducing breaking changes. So GAX is not going to have a major release.

However, I think your idea here is worth discussing. Would you add it into the wishlist as a suggested edit? https://docs.google.com/document/d/1oEzsIEr7DFoyG6-QXXZyIyjIZkbe8i2obLOG90RnVKk/edit?resourcekey=0-T1uuNtaLJBuOtIlcRfe2iQ (internal doc)

[mutianf]

Thanks @suztomo, I added to the bottom of the list. Feel free to add me to meetings that discuss these issues!

[lqiu96]

I think the changes make sense. Just left a few questions (trying to get more familiar with the streaming retry logic).

[blakeli0]

Thanks Mattie for opening this PR! It would be great if we can add a showcase integration test to visualize how customers would use it, and maybe test the behavior of streamWaitTimeout as well. We already have a basic showcase integration tests for server side streaming.
See the README of showcase module for more details. Ley me know if you have any additional questions regarding showcase integration tests, @burkedavison /@lqiu96 / @mpeddada1 can provide more info as well.

[mutianf]

Thanks for reviewing! I'll add showcase integration test and address other comments later.

[sonarcloud]

[gapic-generator-java-root] Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
1 Code Smell

87.5% Coverage
0.0% Duplication

[sonarcloud]

[java_showcase_integration_tests] SonarCloud Quality Gate failed.   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
1 Code Smell

68.8% Coverage
0.0% Duplication

[igorbernstein2]

Awesome! thanks for fixing this! and I apologize for the original misfeature

[sonarcloud]

[java_showcase_unit_tests] Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
1 Code Smell

100.0% Coverage
0.0% Duplication