Perf: Limit concurrency for parallel requests

[hopeyen]

A file may contain a large number of chunks.

To prevent overwhelming system resources, utilize tokio::sync::Semaphores to limit concurrency. A semaphore maintains a set of permits, and a task must acquire a permit from the semaphore before proceeding.

// Declare number of permits
let semaphore = Arc::new(Semaphore::new(max_concurrent_tasks));
// Before task starts
let permit = semaphore.clone().acquire_owned().await.expect("Failed to acquire semaphore permit");
// Release the permit when task finishes
drop(permit); 

[chriswessels]

Noice!