You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The AzureBlobStorage::delete_objects function loops over the list of objects to be deleted, without any retry behaviour. The chance of at least one failing increases exponentially with the length of the list. As we might specify long lists to the function, this is a risk.
Ideally, there is support for proper bulk deletion in the SDK (issue), but until then, we should at least do retries inside the call.
This adds retries to the bulk deletion, because if there is a certain
chance n that a request fails, the chance that at least one of the
requests in a chain of requests fails increases exponentially.
We've had similar issues with the S3 DR tests, which in the end yielded
in adding retries at the remote_storage level. Retries at the top level
are not sufficient when one remote_storage "operation" is multiple
network requests in a trench coat, especially when there is no notion of
saving the progress: even if prior deletions had been successful, we'd
still need to get a 404 in order to continue the loop and get to the
point where we failed in the last iteration. Maybe we'll fail again but
before we've even reached it.
Retries at the bottom level avoid this issue because they have the
notion of progress and also when one network operation fails, only that
operation is retried.
First part of #7931.
The
AzureBlobStorage::delete_objects
function loops over the list of objects to be deleted, without any retry behaviour. The chance of at least one failing increases exponentially with the length of the list. As we might specify long lists to the function, this is a risk.Ideally, there is support for proper bulk deletion in the SDK (issue), but until then, we should at least do retries inside the call.
cc #5567
The text was updated successfully, but these errors were encountered: