Unhandled UnicodeDecodeError exception if response with status 400 and request contains binary payload (for FastHttpUser)

[magupov]

Bug

I get an unhandled UnicodeDecodeError exception if the response has an invalid status code and the request contains binary payload (for FastHttpUser)

Expected behavior

UnicodeDecodeError shouldn't be raised in case of non-utf-8 characters in the payload

Actual behavior

That is a stack trace:

[2020-06-23 09:14:15,612] 7ab43366d074/ERROR/locust.user.task: 'utf-8' codec can't decode byte 0x95 in position 41: invalid start byte
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/geventhttpclient/useragent.py", line 360, in urlopen
    self._verify_status(resp.status_code, url=req.url)
  File "/usr/local/lib/python3.8/site-packages/geventhttpclient/useragent.py", line 302, in _verify_status
    raise BadStatusCode(url, code=status_code)
geventhttpclient.useragent.BadStatusCode: URL http://url/endpoint: code=400

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/locust/user/task.py", line 284, in run
    self.execute_next_task()
  File "/usr/local/lib/python3.8/site-packages/locust/user/task.py", line 309, in execute_next_task
    self.execute_task(self._task_queue.pop(0))
  File "/usr/local/lib/python3.8/site-packages/locust/user/task.py", line 416, in execute_task
    task(self.user)
  File "/app/stress_tests/locustfile.py", line 95, in test_endpoint
    result = self.client.make_request()
  File "/app/tests/base_client.py", line 77, in make_request
    return self.client.request(**kwargs)
  File "/usr/local/lib/python3.8/site-packages/locust/contrib/fasthttp.py", line 227, in request
    response = self._send_request_safe_mode(method, url, payload=data, headers=headers, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/locust/contrib/fasthttp.py", line 159, in _send_request_safe_mode
    return self.client.urlopen(url, method=method, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/geventhttpclient/useragent.py", line 366, in urlopen
    e.http_log = self._conversation_str(req.url, resp, payload=req.payload)
  File "/usr/local/lib/python3.8/site-packages/geventhttpclient/useragent.py", line 424, in _conversation_str
    ret += payload.decode('utf-8') + '\n\n'
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x95 in position 41: invalid start byte

Steps to reproduce

Basically that's a known issue: an attempt to decode the fully binary string fails with UnicodeDecodeError. It might be reproduced even without locust, just try to decode any binary string:

In [1]: import secrets                                                                                                                                                                                        
In [2]: binary_str = secrets.token_bytes(32)                                                                                                                                                                  
In [3]: binary_str.decode('utf-8')                                                                                                                                                                            
---------------------------------------------------------------------------
UnicodeDecodeError                        Traceback (most recent call last)
<ipython-input-3-11a2204405d4> in <module>
----> 1 binary_str.decode('utf-8')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb7 in position 2: invalid start byte

Environment

• OS: Ubuntu 20.04
• Python version: 3.8
• Locust version: 1.0.2
• Locust command line that you ran: locust

[cyberw]

I think this is a bug in geventhttpclient. It seems to assume anything that is of binary type can be decoded as utf-8 here:

https://github.com/gwik/geventhttpclient/blob/5e95a79f3045e0c074c24075fe31d00f1a89a1a0/src/geventhttpclient/useragent.py#L424

If you want, file a bug there (or better yet, a PR - I have admin there as well so I can merge it)

[cyberw]

Or, one could argue, geventhttpclient shouldnt even try to decode the response if it was a failure (line 366). After all, decoding doesnt normally happen until the user asks for the response text, so it is a bit weird to start doing it in error cases.

[magupov]

Agree, especially because request and response are passed to the exception and can be handled directly if necessary
I’ll make a PR soon

[magupov]

@cyberw could you please take a look, I made a PR https://github.com/locustio/geventhttpclient/pull/2

[cyberw]

Nice! Maybe the second change makes the first change unnecessary (I know I said I thought it was a bad idea to try and decode a broken response, but maybe there are cases where this is needed for debugging, so I kind of changed my mind :)

Maybe instead of just silently ignoring the error (pass) do something like:
ret += "UnicodeDecodeError"

[magupov]

Thank you. Done

[cyberw]

Thanks! I dont have time to build a release right now, but will do it in a few days...

[cyberw]

Oh wait, you should submit your PR to the main gwik/geventhttpclient repo, not the fork under locustio/geventhttpclient

[magupov]

oh, done
geventhttpclient/geventhttpclient#128