Implement rate limits on writes to blob storage

[sunu]

While processing large PST files and other archives on Aleph. we hit the GCS rate limit some times while writing files to our storage bucket.

We should enforce a configurable rate limit when writing files to the archive.

[Rosencrantz]

Are there other options here. For example, could we parallelise transactions with multiple accounts, or increase the GCS limit somehow? Not suggesting that these are feasible or better solutions, just want to understand what other options we might have?

[sunu]

I read the docs and it seems GCS adjusts rate limits automatically based on usage https://cloud.google.com/storage/docs/request-rate#auto-scaling. I think we can get away with retries with exponential backoffs instead of a hard rate limit

[stchris]

The best way forward here is to let the google cloud Python sdk handle retries with an exponential backoff and jitter (see https://cloud.google.com/python/docs/reference/storage/latest/retry_timeout). We would replace our for loop (

servicelayer/servicelayer/archive/gs.py

Line 97 in ceb20c3

for attempt in service_retries():

) and linear backoff (https://github.com/alephdata/servicelayer/blob/ceb20c34ce141796c46585247cb88607299f3d1c/servicelayer/archive/gs.py#L109C3-L109C3) with that.

See also https://occrp.sentry.io/issues/4162555422/?project=4504916166967297&query=is%3Aunresolved&referrer=issue-stream&stream_index=0

[tillprochaska]

Helpful context from @brrttwrks: We had similar issues in the past when uploading archive/package files (ZIP archives, Outlook PST files, …). During ingestion, these files are unpacked and uploaded to the storage backend individually.

• https://github.com/alephdata/ingest-file/blob/main/ingestors/directory.py
• https://github.com/alephdata/ingest-file/blob/main/ingestors/email/outlookpst.py#L24