internal/civisibility/utils/net: a new http client for rapid endpoints

[tonyredondo]

What does this PR do?

This PR adds a new http client for specific use on rapid endpoints used by ci visibility with json support, exponential backoff, rate-limiting status code, gzip compression and multipart bodies (attachments).

Motivation

Some CI Visibility features require getting, posting and sending multipart form data to the backend. Features like:

• Early Flake detection
• Git upload
• Intelligent test runner
• Settings api
• Retry failed tests

Code Coverage

Reviewer's Checklist

• Changed code has unit tests for its functionality at or near 100% coverage.
• System-Tests covering this feature have been added and enabled with the va.b.c-dev version tag.
• There is a benchmark for any new code, or changes to existing code.
• If this interacts with the agent in a new way, a system test has been added.
• Add an appropriate team label so this PR gets put in the right place for the release notes.
• Non-trivial go.mod changes, e.g. adding new modules, are reviewed by @DataDog/dd-trace-go-guild.

Unsure? Have a question? Request a review!

[pr-commenter]

Benchmarks

Benchmark execution time: 2024-09-19 11:56:47

Comparing candidate commit d2119cb in PR branch tony/civisibility-rapid-http-client with baseline commit b94c2c3 in branch main.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 59 metrics, 0 unstable metrics.

[romainkomorndatadog]

Maybe I just don't know Go but I find the amount of omitempty in JSON struct definitions a bit concerning.

Doesn't that mean we could unmarshal unexpectedly formatted JSON responses, and then crash later when we try to use them? Or does the compiler make sure we never try to access something that might be empty?

[tonyredondo]

t mean we could unmarshal unexpectedly formatted JSON responses, and then crash later when we try to use them? Or does the compiler make sure we never try to access something that might be empty?

omitempty is only used on serialization, so if the value of that omitempty field is nil or empty(default value) then is omitted on serialization. it doesn't affect unmarshalling.

[romainkomorndatadog]

it doesn't affect unmarshalling.

Oh, carry on then. :)

Edit: although if it doesn't affect unmarshalling, why use omitempty on things like the JSON response from the API endpoints? We never marshal those anyway, right?

[tonyredondo]

it doesn't affect unmarshalling.

Oh, carry on then. :)

Edit: although if it doesn't affect unmarshalling, why use omitempty on things like the JSON response from the API endpoints? We never marshal those anyway, right?

Yes I removed a lot of omitempty here: 7c60549