-
Notifications
You must be signed in to change notification settings - Fork 5
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Implement rate limits on writes to blob storage #55
Comments
Are there other options here. For example, could we parallelise transactions with multiple accounts, or increase the GCS limit somehow? Not suggesting that these are feasible or better solutions, just want to understand what other options we might have? |
I read the docs and it seems GCS adjusts rate limits automatically based on usage https://cloud.google.com/storage/docs/request-rate#auto-scaling. I think we can get away with retries with exponential backoffs instead of a hard rate limit |
The best way forward here is to let the google cloud Python sdk handle retries with an exponential backoff and jitter (see https://cloud.google.com/python/docs/reference/storage/latest/retry_timeout). We would replace our for loop ( servicelayer/servicelayer/archive/gs.py Line 97 in ceb20c3
|
Helpful context from @brrttwrks: We had similar issues in the past when uploading archive/package files (ZIP archives, Outlook PST files, …). During ingestion, these files are unpacked and uploaded to the storage backend individually. |
While processing large PST files and other archives on Aleph. we hit the GCS rate limit some times while writing files to our storage bucket.
We should enforce a configurable rate limit when writing files to the archive.
The text was updated successfully, but these errors were encountered: