feat: Add S3 Bucket fetch speed improvement

[spangenberg]

No description provided.

[disq]

lgtm, but why not use a waitgroup and range on the (potentially unbuffered) channel? range would stop iterating when the channel is closed, and no more len(response) business

[spangenberg]

lgtm, but why not use a waitgroup and range on the (potentially unbuffered) channel? range would stop iterating when the channel is closed, and no more len(response) business

But the response channel is still needed for the diagnostics, don't see where the WaitGroup would help in that case.

[disq]

But the response channel is still needed for the diagnostics, don't see where the WaitGroup would help in that case.

Create the waitgroup/goroutines, send stuff to the channel, close the channel, run a range on the errs channel (will need buffered channel or another, single goroutine to read err results into diags...) and to make sure all workers exited, finally do wg.Wait()? Better flow and readability/maintainability imo.

[spangenberg]

But the response channel is still needed for the diagnostics, don't see where the WaitGroup would help in that case.

Create the waitgroup/goroutines, send stuff to the channel, close the channel, run a range on the errs channel (will need buffered channel or another, single goroutine to read err results into diags...) and to make sure all workers exited, finally do wg.Wait()? Better flow and readability/maintainability imo.

That makes sense, thanks for clarifying, think this should do it?

[roneli]

LGTM