[cloud storage] support for reqwest_middleware

[nicolas-vivot]

original "issue"

The storage client do not offer the ability to specify some retry policy.

How to address it

While we can simply do it on our end (wrapping the cloud storage calls in some retry mechanism), one simple way is to address it directly in the cloud storage library by changing the reqwest client being used.

The client currently used for the storage part (and i believe in other modules but i have only looked at the storage part) is reqwest::Client.
Switching to the ClientWithMiddleware from reqwest_middleware allow us to pass a fully customized client by using middlewares.

Benefits doing it

Middle-wares can then be used to extend how requests get processed, without altering the cloud storage crate itself, making it quite flexible.
A couple of examples of middleware that can be used:

• authentication layer
• custom retry policy
• custom retry strategy (when it comes to specific needs or non standard APIs)
• adding custom headers (for tracing for example)

Changes proposal

I made a change proposal in this PR if you want to have a look at it.

Since the original reqwest crate do not expose trait, the reqwest_middleware client is a wrapper. For this reason, i could not refactor the cloud storage to offer support for both, and had to replace the client types.

In term of support / migration for people who are using the cloud storage library with a custom reqwest::Client, there should not be any issues since there is a From to create a ClientWithMiddleware from an existing reqwest::Client.

notes:

• the cloud storage crate could also benefit it by moving the custom headers (including the authentication token) out as middlewares.
• i did not do anything for the default ClientWithMiddleware being built, but i do believe that having one with a default retry strategy would greatly improve the default behavior of the component. I am happy to add it in a separate PR if this one get accepted.

[nicolas-vivot]

I am closing this since the PR has been accepted, and it seems like @yoshidan have changed the other modules to also use it.