Chunk download

[horheynm]

Desctiption

Chunk download if endpoint supports chunked download.

Problem

Downloading large files tend to fail

Solution

Add chunk download, redownload the chunk if it fails less than N times.

Design

Use threads to speed up downloads. Download speed is limited by bandwidth and write speed to the disk. Write to disk speed usually differ by usage. so constant write time.

Any download is a job. If chunk download, we need to combine and delete the chunk files so they are a job also.
We have two queues.

1. Job queue - run concurrently -- doesnt matter what file gets downloaded
2. Job queues - run only when the previous job_queue finishes -- used for combining chunks

Ex.
job_queue1 = Queue()
download_job1 = Job(...)
download_job2 = Job(...)
job_queue1.put(download_job1).put(download_job2)

job_queue2 = Queue()
combine_job1 = Job(...)
job_queue2.put(combine_job1)

job_queues = Queue()
job_queues.put(job_queue1)
job_queues.put(job_queue2) # run this queue if the previous job_queue is done to guarantee downloads.

Usage:

url = "https://download-some-file.com/"
dest_path = "your/local/folder"
max_retries= 3
downloader = Downloader(
    url=url_path, download_path=dest_path, max_retries=num_retries
)
downloader.download()

Code

stub = "zoo:llama2-7b-ultrachat200k_llama2_pretrain-pruned80"
print(stub)

model = Model(stub)
p = model.onnx_model.path

Output:

(.venv) george@gpuserver6:~/sparsezoo$ python3 -m scratch.down
zoo:llama2-7b-ultrachat200k_llama2_pretrain-pruned80
Downloading Chunks: 100%|█████████████████████████████████████████████████████████| 5.27G/5.27G [00:47<00:00, 112MB/s]
Combining Chunks: 100%|██████████████████████████████████████████████████████████| 5.27G/5.27G [00:05<00:00, 1.01GB/s]
(.venv) george@gpuserver6:~/sparsezoo$ 

Testing

Basic url mock and donwload count. actual download is covered in e2e

[dbogunowicz]

So @horheynm, just so I understand. The logic now computes the amounts of chunks that are to be downloaded, uses concurrency to download all the chunks asynchronously, and the last job is there to "concatenate" all the chunks into a single file right?

[bfineran]

looks great @horheynm, very easy to follow. Would be great to add some simple unit tests to confirm the chunking is happening - e2e functionality is definitely thoroughly covered since every other pathway uses download

[horheynm]

So @horheynm, just so I understand. The logic now computes the amounts of chunks that are to be downloaded, uses concurrency to download all the chunks asynchronously, and the last job is there to "concatenate" all the chunks into a single file right?

yes