Grequests is creating Recursion errors due to invalid monkey patch

[mwitteveen]

I am having issues using the grequests package in my project.

The issue takes place when i am running pytest on my whole project. Separately all my tests will pass but combining all the tests makes about half of them crash. And this forms an issues when using pytest-cov and pytest-xdist.

After a couple of hours searching what could be wrong i ended up blaming you guys xD.

So the related issue is: gevent/gevent#1016

A warning is given while i run my tests:

=============================== warnings summary ===============================
.venv/lib/python3.8/site-packages/grequests.py:22
/.venv/lib/python3.8/site-packages/grequests.py:22: MonkeyPatchWarning: Monkey-patching ssl after ssl has already been imported may lead to errors, including RecursionError on Python 3.6. It may also silently lead to incorrect behaviour on Python 3.7. Please monkey-patch earlier. See gevent/gevent#1016. Modules that had direct imports (NOT patched): ['urllib3.util.ssl_ (.venv/lib/python3.8/site-packages/urllib3/util/ssl_.py)', 'urllib3.util (.venv/lib/python3.8/site-packages/urllib3/util/init.py)'].
curious_george.patch_all(thread=False, select=False)

The error that occurs in my pytests has the following form:

.venv/lib/python3.8/site-packages/requests/api.py:104: in head
return request('head', url, **kwargs)
.venv/lib/python3.8/site-packages/requests/api.py:61: in request
return session.request(method=method, url=url, **kwargs)
.venv/lib/python3.8/site-packages/requests/sessions.py:542: in request
resp = self.send(prep, **send_kwargs)
.venv/lib/python3.8/site-packages/requests/sessions.py:655: in send
r = adapter.send(request, **kwargs)
.venv/lib/python3.8/site-packages/requests/adapters.py:439: in send
resp = conn.urlopen(
.venv/lib/python3.8/site-packages/urllib3/connectionpool.py:699: in urlopen
httplib_response = self._make_request(
.venv/lib/python3.8/site-packages/urllib3/connectionpool.py:382: in _make_request
self._validate_conn(conn)
.venv/lib/python3.8/site-packages/urllib3/connectionpool.py:1010: in validate_conn
conn.connect()
.venv/lib/python3.8/site-packages/urllib3/connection.py:392: in connect
self.ssl_context = create_urllib3_context(
.venv/lib/python3.8/site-packages/urllib3/util/ssl.py:303: in create_urllib3_context
context.options |= options
/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/ssl.py:602: in options
super(SSLContext, SSLContext).options.set(self, value)
/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/ssl.py:602: in options
super(SSLContext, SSLContext).options.set(self, value)
E RecursionError: maximum recursion depth exceeded
!!! Recursion detected (same locals & position)
------------------------------ Captured log call -------------------------------
DEBUG urllib3.connectionpool:connectionpool.py:971 Starting new HTTPS connection (1): ec.europa.eu:443

The moment i ignore the files that use grequests the problem is solved.

[spyoungtech]

Sometimes this can be fixed by making sure grequests is the first imported module, ensuring that no other modules import modules that need to be patched by gevent.

grequests relies on gevent's monkeypatching to work. If you import a module that gevent patches before it is patched, it will result in the module being unpatched and thus incompatible.

# bad
import requests
import grequests

# good
import grequests
import requests

In other cases, there are sometimes conflicts between what grequests patches and what other libraries patch (or dont patch). This seems particularly true of other libraries that either (1) use modules patched by gevent (like ssl) or (2) also use gevent.

Arguably, grequests does a less-than-ideal (incorrect) thing by performing the monkey patch on import, whereas it may be ideal if grequests left it in the user's control when monkeypatch occurs. Unfortunately, it's probably not something that can be changed as a default this far down the road without breaking everybody using this package.

You can search for other issues in this repo with the gevent label to find previous discussion and other explanations/workarounds for this and similar issues.

[ariisan123]

Sometimes this can be fixed by making sure grequests is the first imported module, ensuring that no other modules import modules that need to be patched by gevent.

grequests relies on gevent's monkeypatching to work. If you import a module that gevent patches before it is patched, it will result in the module being unpatched and thus incompatible.

# bad
import requests
import grequests

# good
import grequests
import requests

In other cases, there are sometimes conflicts between what grequests patches and what other libraries patch (or dont patch). This seems particularly true of other libraries that either (1) use modules patched by gevent (like ssl) or (2) also use gevent.

Arguably, grequests does a less-than-ideal (incorrect) thing by performing the monkey patch on import, whereas it may be ideal if grequests left it in the user's control when monkeypatch occurs. Unfortunately, it's probably not something that can be changed as a default this far down the road without breaking everybody using this package.

You can search for other issues in this repo with the gevent label to find previous discussion and other explanations/workarounds for this and similar issues.

Sos crack man, gracias

[spyoungtech]

Glad I could help