Skip to content

Commit

Permalink
gvfs-helper: V2 robust retry and throttling
Browse files Browse the repository at this point in the history
Add robust-retry mechanism to automatically retry a request after network
errors.  This includes retry after:
   [] transient network problems reported by CURL.
   [] http 429 throttling (with associated Retry-After)
   [] http 503 server unavailable (with associated Retry-After)

Add voluntary throttling using Azure X-RateLimit-* hints to avoid being
soft-throttled (tarpitted) or hard-throttled (429) on later requests.

Add global (outside of a single request) azure-throttle data to track the
rate limit hints from the cache-server and main Git server independently.

Add exponential retry backoff.  This is used for transient network problems
when we don't have a Retry-After hint.

Move the call to index-pack earlier in the response/error handling sequence
so that if we receive a 200 but yet the packfile is truncated/corrupted, we
can use the regular retry logic to get it again.

Refactor the way we create tempfiles for packfiles to use
<odb>/pack/tempPacks/ rather than working directly in the <odb>/pack/
directory.

Move the code to create a new tempfile to the start of a single request
attempt (initial and retry attempts), rather than at the overall start
of a request.  This gives us a fresh tempfile for each network request
attempt.  This simplifies the retry mechanism and isolates us from the file
ownership issues hidden within the tempfile class.  And avoids the need to
truncate previous incomplete results.  This was necessary because index-pack
was pulled into the retry loop.

Minor: Add support for logging X-VSS-E2EID to telemetry on network errors.

Minor: rename variable:
    params.b_no_cache_server --> params.b_permit_cache_server_if_defined.
This variable is used to indicate whether we should try to use the
cache-server when it is defined.  Got rid of double-negative logic.

Minor: rename variable:
    params.label --> params.tr2_label
Clarify that this variable is only used with trace2 logging.

Minor: Move the code to automatically map cache-server 400 responses
to normal 401 response earlier in the response/error handling sequence
to simplify later retry logic.

Minor: Decorate trace2 messages with "(cs)" or "(main)" to identify the
server in log messages.  Add params->server_type to simplify this.

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
  • Loading branch information
jeffhostetler authored and dscho committed Aug 11, 2023
1 parent 3b02e63 commit 0e03263
Showing 1 changed file with 1,101 additions and 302 deletions.
Loading

0 comments on commit 0e03263

Please sign in to comment.