Unhandled Exception on Error 429 - Too Many Requests

[jkupferer]

Long story short

When a watch receives a 429 error (Too Many Requests) the watch fails rather than handling with backoff.

Kopf version

1.36.0

Kubernetes version

1.23.12.12

Python version

No response

Code

# Unable to reliably reproduce.

Logs

{"message": "Watcher for oauthaccesstokens.v1.oauth.openshift.io@none has failed: (None, None)", "exc_info": "... see below ...", "timestamp": "2023-02-27T15:47:27.234080+00:00", "severity": "error"}

# exec_info reformatted with line breaks for readability:

Traceback (most recent call last):
  File \"/opt/app-root/lib64/python3.9/site-packages/kopf/_cogs/clients/errors.py\", line 148, in check_response
    response.raise_for_status() 
  File \"/opt/app-root/lib64/python3.9/site-packages/aiohttp/client_reqrep.py\", line 1005, in raise_for_status
    raise ClientResponseError(
aiohttp.client_exceptions.ClientResponseError: 429, message='Too Many Requests', url=URL('https://172.30.0.1:443/apis/oauth.openshift.io/v1/oauthaccesstokens?watch=true&resourceVersion=799875280')

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File \"/opt/app-root/lib64/python3.9/site-packages/kopf/_cogs/aiokits/aiotasks.py\", line 108, in guard
    await coro
  File \"/opt/app-root/lib64/python3.9/site-packages/kopf/_core/reactor/queueing.py\", line 175, in watcher
    async for raw_event in stream:
  File \"/opt/app-root/lib64/python3.9/site-packages/kopf/_cogs/clients/watching.py\", line 82, in infinite_watch
    async for raw_event in stream:
  File \"/opt/app-root/lib64/python3.9/site-packages/kopf/_cogs/clients/watching.py\", line 186, in continuous_watch
    async for raw_input in stream:
  File \"/opt/app-root/lib64/python3.9/site-packages/kopf/_cogs/clients/watching.py\", line 251, in watch_objs
    async for raw_input in api.stream(
  File \"/opt/app-root/lib64/python3.9/site-packages/kopf/_cogs/clients/api.py\", line 200, in stream
    response = await request(
  File \"/opt/app-root/lib64/python3.9/site-packages/kopf/_cogs/clients/auth.py\", line 45, in wrapper
    return await fn(*args, **kwargs, context=context)
  File \"/opt/app-root/lib64/python3.9/site-packages/kopf/_cogs/clients/api.py\", line 85, in request
    await errors.check_response(response)  # but do not parse it!
  File \"/opt/app-root/lib64/python3.9/site-packages/kopf/_cogs/clients/errors.py\", line 150, in check_response
    raise cls(payload, status=response.status) from e
kopf._cogs.clients.errors.APIClientError: (None, None)

Additional information

I believe a 429 Too Many Requests response should be handled with error backoff similar to an APIServerError?

[kriscode1]

I believe a 429 Too Many Requests response should be handled with error backoff similar to an APIServerError?

We have a similar issue that this solution would also assist with. We need our operator to manage a lot of pre-existing resources in the cluster. Since there's no limit to the number of requests made, the first run of the operator causes 429s because it tries to annotate many objects in a very short time.

[psontag]

This seems to be a duplicate of #962 which should be resolved with #963. The changes are not yet released though.

[a-m-w]

@nolar Would it be possible to consider cutting a new release? Thanks in advance!

[dheeg]

Was somebody able to fix this error? Our operator (kopf 1.36.2) has to deal with several thousand resources as well on startup. From time to time the following lethal error occurs.

After it happened the finalizers are stuck and the operator basically can't manage the resource type anymore.

Watcher for grafanaalerts.v1.example.com@none has failed: (None, None)

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/kopf/_cogs/clients/errors.py", line 148, in check_response
    response.raise_for_status()
  File "/usr/local/lib/python3.11/site-packages/aiohttp/client_reqrep.py", line 1005, in raise_for_status
    raise ClientResponseError(
aiohttp.client_exceptions.ClientResponseError: 429, message='Too Many Requests', url=URL('https://10.32.0.1:443/apis/example.com/v1/grafanaalerts?watch=true&resourceVersion=1771761318')

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/kopf/_cogs/aiokits/aiotasks.py", line 108, in guard
    await coro
  File "/usr/local/lib/python3.11/site-packages/kopf/_core/reactor/queueing.py", line 175, in watcher
    async for raw_event in stream:
  File "/usr/local/lib/python3.11/site-packages/kopf/_cogs/clients/watching.py", line 86, in infinite_watch
    async for raw_event in stream:
  File "/usr/local/lib/python3.11/site-packages/kopf/_cogs/clients/watching.py", line 201, in continuous_watch
    async for raw_input in stream:
  File "/usr/local/lib/python3.11/site-packages/kopf/_cogs/clients/watching.py", line 266, in watch_objs
    async for raw_input in api.stream(
  File "/usr/local/lib/python3.11/site-packages/kopf/_cogs/clients/api.py", line 200, in stream
    response = await request(
               ^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/kopf/_cogs/clients/auth.py", line 45, in wrapper
    return await fn(*args, **kwargs, context=context)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/kopf/_cogs/clients/api.py", line 85, in request
    await errors.check_response(response)  # but do not parse it!
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/kopf/_cogs/clients/errors.py", line 150, in check_response
    raise cls(payload, status=response.status) from e
kopf._cogs.clients.errors.APIClientError: (None, None)

[dheeg]

@nolar Do you have an idea about this by chance? It looks like there are types of requests which will not be retried. Maybe this is also the reason behind #1019