SSLContext infinite recursion in Python 3.6

[janbuchar]

On Python 3.6.2 and gevent 1.2.2, I encounter the problem described in #903 . I do not have the rights to reopen the issue, which is why I'm making a duplicate.

[jamadden]

Can you please post a failing example and its output?

[janbuchar]

The code:

import gevent
from gevent import monkey
import requests

monkey.patch_all()
from requests.packages.urllib3.util.ssl_ import create_urllib3_context
create_urllib3_context()

The output:

Traceback (most recent call last):                                                        
  File "gevent_requests.py", line 7, in <module>                                          
    create_urllib3_context()
  File "/usr/lib/python3.6/site-packages/urllib3/util/ssl_.py", line 269, in create_urllib
3_context             
    context.options |= options
  File "/usr/lib/python3.6/ssl.py", line 459, in options
    super(SSLContext, SSLContext).options.__set__(self, value)                            
  File "/usr/lib/python3.6/ssl.py", line 459, in options
    super(SSLContext, SSLContext).options.__set__(self, value)
  File "/usr/lib/python3.6/ssl.py", line 459, in options
    super(SSLContext, SSLContext).options.__set__(self, value)
  [Previous line repeated 329 more times]    
RecursionError: maximum recursion depth exceeded while calling a Python object

[jamadden]

Thanks. The problem is that you're importing requests before you monkey patch. You must monkey patch before importing anything else. I can reproduce this if I patch in the incorrect order, but if I patch in the correct order it works as expected:

Python 3.6.2 (default, Jul 18 2017, 14:08:57)
[GCC 4.2.1 Compatible Apple LLVM 8.1.0 (clang-802.0.42)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import gevent.monkey
>>> gevent.monkey.patch_all()
>>> from requests.packages.urllib3.util.ssl_ import create_urllib3_context
>>> create_urllib3_context()
<gevent._ssl3.SSLContext object at 0x1013c2228>

[janbuchar]

Yes, that seems to be the problem - after fixing the order, everything is fine. I'm sorry for bothering you with this, I just googled the exception, found the issue I referenced and thought this must be the problem.

[jamadden]

No worries. I will add a warning to gevent to make this more clear.

See also #941

[wumpus]

Thank you for the warning! Excellent way to point out coding errors.

[garyo]

Is there a recommended way to initialize gevent with gunicorn to avoid this warning? I'm using Flask with gunicorn as server and gevent with websockets. I tried doing monkey.patch_all() in the gunicorn.conf.py in post_fork, but still get the warning.

[jamadden]

I feel this has been said other places, but I personally monkey-patch before using gunicorn, in the initial boot script.

[garyo]

So, you suggest starting gunicorn not on the command line but from a python module? And do the monkey-patching there? (Sorry to be obtuse, I'm just learning gunicorn -- only need to use it because I'm doing websockets in flask.)

[jamadden]

Well, no. We use a setuptools entry point that wraps th gunicorn entry point. But it’s application specific.

[theladyjaye]

Is there anywhere else I could look that might cause this?

This here are the first few lines of my main.py:

1 print("Main Loading!")
2 import gevent
3 from gevent import monkey
4 monkey.patch_all()
5 print("Gevent Patched!")
6
7 from typing import Iterable, Optional, List

Here's what I see in my logs:

/var/task/main.py:4: MonkeyPatchWarning: Monkey-patching ssl after ssl has already been imported may lead to errors, including RecursionError on Python 3.6. Please monkey-patch earlier. See https://github.com/gevent/gevent/issues/1016

I am running this on the AWS Lambda python3.6 runtime

I cannot for the life of me understand how I can possibly import and patch any earlier than lines 2-4 (effectively line 1-3 as I dropped those print's in to see where it was happening)

I have some other Lambda's on the same python3.6 runtime that use gevent and patch in the same way and I do not receive this error. So I'm clearly missing something.

12 days later... (2018-06-13)

Ok so here is my Lambda deployment package and example code. It uses the Lambda handler main.aws_lambda

The only source present:

import sys
from pprint import pprint

# https://github.com/gevent/gevent/blob/master/src/gevent/monkey.py#L700
pprint(sys.modules)

import gevent.monkey
gevent.monkey.patch_all()


def aws_lambda(payload, _) -> None:
    print("Trial")

And from the logged output, the only line of note:

'ssl': <module 'ssl' from '/var/lang/lib/python3.6/ssl.py'>,

So AWS Lambda seems to already have SSL imported when it calls the entry point for the function. I don't know if this matters or falls under the python2 exception as documented here:

https://github.com/gevent/gevent/blob/master/src/gevent/monkey.py#L701-L704

If it does not matter, as in the python 2 notes, it looks like there is a runtime module available in the Python 3.6 AWS Lambda Runtime that could be used to detect this and ignore it:

'runtime': <module 'runtime' from '/var/runtime/awslambda/runtime.cpython-36m-x86_64-linux-gnu.so'>,

Code
Archive.zip

Log Output

{'__future__': <module '__future__' from '/var/lang/lib/python3.6/__future__.py'>,
'__main__': <module '__main__' from '/var/runtime/awslambda/bootstrap.py'>,
'_bisect': <module '_bisect' from '/var/lang/lib/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so'>,
'_blake2': <module '_blake2' from '/var/lang/lib/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so'>,
'_bootlocale': <module '_bootlocale' from '/var/lang/lib/python3.6/_bootlocale.py'>,
'_bz2': <module '_bz2' from '/var/lang/lib/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so'>,
'_codecs': <module '_codecs' (built-in)>,
'_collections': <module '_collections' (built-in)>,
'_collections_abc': <module '_collections_abc' from '/var/lang/lib/python3.6/_collections_abc.py'>,
'_compression': <module '_compression' from '/var/lang/lib/python3.6/_compression.py'>,
'_datetime': <module '_datetime' from '/var/lang/lib/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so'>,
'_decimal': <module '_decimal' from '/var/lang/lib/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so'>,
'_frozen_importlib': <module 'importlib._bootstrap' (frozen)>,
'_frozen_importlib_external': <module 'importlib._bootstrap_external' (frozen)>,
'_functools': <module '_functools' (built-in)>,
'_hashlib': <module '_hashlib' from '/var/lang/lib/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so'>,
'_heapq': <module '_heapq' from '/var/lang/lib/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so'>,
'_imp': <module '_imp' (built-in)>,
'_io': <module 'io' (built-in)>,
'_json': <module '_json' from '/var/lang/lib/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so'>,
'_locale': <module '_locale' (built-in)>,
'_lzma': <module '_lzma' from '/var/lang/lib/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so'>,
'_operator': <module '_operator' (built-in)>,
'_posixsubprocess': <module '_posixsubprocess' from '/var/lang/lib/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so'>,
'_random': <module '_random' from '/var/lang/lib/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so'>,
'_sha3': <module '_sha3' from '/var/lang/lib/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so'>,
'_signal': <module '_signal' (built-in)>,
'_sitebuiltins': <module '_sitebuiltins' from '/var/lang/lib/python3.6/_sitebuiltins.py'>,
'_socket': <module '_socket' from '/var/lang/lib/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so'>,
'_sre': <module '_sre' (built-in)>,
'_ssl': <module '_ssl' from '/var/lang/lib/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so'>,
'_stat': <module '_stat' (built-in)>,
'_string': <module '_string' (built-in)>,
'_struct': <module '_struct' from '/var/lang/lib/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so'>,
'_sysconfigdata_m_linux_x86_64-linux-gnu': <module '_sysconfigdata_m_linux_x86_64-linux-gnu' from '/var/lang/lib/python3.6/_sysconfigdata_m_linux_x86_64-linux-gnu.py'>,
'_thread': <module '_thread' (built-in)>,
'_warnings': <module '_warnings' (built-in)>,
'_weakref': <module '_weakref' (built-in)>,
'_weakrefset': <module '_weakrefset' from '/var/lang/lib/python3.6/_weakrefset.py'>,
'abc': <module 'abc' from '/var/lang/lib/python3.6/abc.py'>,
'argparse': <module 'argparse' from '/var/lang/lib/python3.6/argparse.py'>,
'atexit': <module 'atexit' (built-in)>,
'base64': <module 'base64' from '/var/lang/lib/python3.6/base64.py'>,
'binascii': <module 'binascii' from '/var/lang/lib/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so'>,
'bisect': <module 'bisect' from '/var/lang/lib/python3.6/bisect.py'>,
'builtins': <module 'builtins' (built-in)>,
'bz2': <module 'bz2' from '/var/lang/lib/python3.6/bz2.py'>,
'calendar': <module 'calendar' from '/var/lang/lib/python3.6/calendar.py'>,
'codecs': <module 'codecs' from '/var/lang/lib/python3.6/codecs.py'>,
'collections': <module 'collections' from '/var/lang/lib/python3.6/collections/__init__.py'>,
'collections.abc': <module 'collections.abc' from '/var/lang/lib/python3.6/collections/abc.py'>,
'contextlib': <module 'contextlib' from '/var/lang/lib/python3.6/contextlib.py'>,
'copy': <module 'copy' from '/var/lang/lib/python3.6/copy.py'>,
'copyreg': <module 'copyreg' from '/var/lang/lib/python3.6/copyreg.py'>,
'datetime': <module 'datetime' from '/var/lang/lib/python3.6/datetime.py'>,
'decimal': <module 'decimal' from '/var/lang/lib/python3.6/decimal.py'>,
'email': <module 'email' from '/var/lang/lib/python3.6/email/__init__.py'>,
'email._encoded_words': <module 'email._encoded_words' from '/var/lang/lib/python3.6/email/_encoded_words.py'>,
'email._parseaddr': <module 'email._parseaddr' from '/var/lang/lib/python3.6/email/_parseaddr.py'>,
'email._policybase': <module 'email._policybase' from '/var/lang/lib/python3.6/email/_policybase.py'>,
'email.base64mime': <module 'email.base64mime' from '/var/lang/lib/python3.6/email/base64mime.py'>,
'email.charset': <module 'email.charset' from '/var/lang/lib/python3.6/email/charset.py'>,
'email.encoders': <module 'email.encoders' from '/var/lang/lib/python3.6/email/encoders.py'>,
'email.errors': <module 'email.errors' from '/var/lang/lib/python3.6/email/errors.py'>,
'email.feedparser': <module 'email.feedparser' from '/var/lang/lib/python3.6/email/feedparser.py'>,
'email.header': <module 'email.header' from '/var/lang/lib/python3.6/email/header.py'>,
'email.iterators': <module 'email.iterators' from '/var/lang/lib/python3.6/email/iterators.py'>,
'email.message': <module 'email.message' from '/var/lang/lib/python3.6/email/message.py'>,
'email.parser': <module 'email.parser' from '/var/lang/lib/python3.6/email/parser.py'>,
'email.quoprimime': <module 'email.quoprimime' from '/var/lang/lib/python3.6/email/quoprimime.py'>,
'email.utils': <module 'email.utils' from '/var/lang/lib/python3.6/email/utils.py'>,
'encodings': <module 'encodings' from '/var/lang/lib/python3.6/encodings/__init__.py'>,
'encodings.aliases': <module 'encodings.aliases' from '/var/lang/lib/python3.6/encodings/aliases.py'>,
'encodings.latin_1': <module 'encodings.latin_1' from '/var/lang/lib/python3.6/encodings/latin_1.py'>,
'encodings.utf_8': <module 'encodings.utf_8' from '/var/lang/lib/python3.6/encodings/utf_8.py'>,
'enum': <module 'enum' from '/var/lang/lib/python3.6/enum.py'>,
'errno': <module 'errno' (built-in)>,
'fnmatch': <module 'fnmatch' from '/var/lang/lib/python3.6/fnmatch.py'>,
'functools': <module 'functools' from '/var/lang/lib/python3.6/functools.py'>,
'genericpath': <module 'genericpath' from '/var/lang/lib/python3.6/genericpath.py'>,
'gettext': <module 'gettext' from '/var/lang/lib/python3.6/gettext.py'>,
'grp': <module 'grp' from '/var/lang/lib/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so'>,
'hashlib': <module 'hashlib' from '/var/lang/lib/python3.6/hashlib.py'>,
'heapq': <module 'heapq' from '/var/lang/lib/python3.6/heapq.py'>,
'html': <module 'html' from '/var/lang/lib/python3.6/html/__init__.py'>,
'html.entities': <module 'html.entities' from '/var/lang/lib/python3.6/html/entities.py'>,
'http': <module 'http' from '/var/lang/lib/python3.6/http/__init__.py'>,
'http.client': <module 'http.client' from '/var/lang/lib/python3.6/http/client.py'>,
'http.server': <module 'http.server' from '/var/lang/lib/python3.6/http/server.py'>,
'imp': <module 'imp' from '/var/lang/lib/python3.6/imp.py'>,
'importlib': <module 'importlib' from '/var/lang/lib/python3.6/importlib/__init__.py'>,
'importlib._bootstrap': <module 'importlib._bootstrap' (frozen)>,
'importlib._bootstrap_external': <module 'importlib._bootstrap_external' (frozen)>,
'importlib.abc': <module 'importlib.abc' from '/var/lang/lib/python3.6/importlib/abc.py'>,
'importlib.machinery': <module 'importlib.machinery' from '/var/lang/lib/python3.6/importlib/machinery.py'>,
'importlib.util': <module 'importlib.util' from '/var/lang/lib/python3.6/importlib/util.py'>,
'io': <module 'io' from '/var/lang/lib/python3.6/io.py'>,
'ipaddress': <module 'ipaddress' from '/var/lang/lib/python3.6/ipaddress.py'>,
'itertools': <module 'itertools' (built-in)>,
'json': <module 'json' from '/var/lang/lib/python3.6/json/__init__.py'>,
'json.decoder': <module 'json.decoder' from '/var/lang/lib/python3.6/json/decoder.py'>,
'json.encoder': <module 'json.encoder' from '/var/lang/lib/python3.6/json/encoder.py'>,
'json.scanner': <module 'json.scanner' from '/var/lang/lib/python3.6/json/scanner.py'>,
'keyword': <module 'keyword' from '/var/lang/lib/python3.6/keyword.py'>,
'linecache': <module 'linecache' from '/var/lang/lib/python3.6/linecache.py'>,
'locale': <module 'locale' from '/var/lang/lib/python3.6/locale.py'>,
'logging': <module 'logging' from '/var/lang/lib/python3.6/logging/__init__.py'>,
'lzma': <module 'lzma' from '/var/lang/lib/python3.6/lzma.py'>,
'main': <module 'main' from '/var/task/main.py'>,
'marshal': <module 'marshal' (built-in)>,
'math': <module 'math' from '/var/lang/lib/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so'>,
'mimetypes': <module 'mimetypes' from '/var/lang/lib/python3.6/mimetypes.py'>,
'numbers': <module 'numbers' from '/var/lang/lib/python3.6/numbers.py'>,
'operator': <module 'operator' from '/var/lang/lib/python3.6/operator.py'>,
'os': <module 'os' from '/var/lang/lib/python3.6/os.py'>,
'os.path': <module 'posixpath' from '/var/lang/lib/python3.6/posixpath.py'>,
'platform': <module 'platform' from '/var/lang/lib/python3.6/platform.py'>,
'posix': <module 'posix' (built-in)>,
'posixpath': <module 'posixpath' from '/var/lang/lib/python3.6/posixpath.py'>,
'pprint': <module 'pprint' from '/var/lang/lib/python3.6/pprint.py'>,
'pwd': <module 'pwd' (built-in)>,
'quopri': <module 'quopri' from '/var/lang/lib/python3.6/quopri.py'>,
'random': <module 'random' from '/var/lang/lib/python3.6/random.py'>,
're': <module 're' from '/var/lang/lib/python3.6/re.py'>,
'reprlib': <module 'reprlib' from '/var/lang/lib/python3.6/reprlib.py'>,
'runtime': <module 'runtime' from '/var/runtime/awslambda/runtime.cpython-36m-x86_64-linux-gnu.so'>,
'select': <module 'select' from '/var/lang/lib/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so'>,
'selectors': <module 'selectors' from '/var/lang/lib/python3.6/selectors.py'>,
'shutil': <module 'shutil' from '/var/lang/lib/python3.6/shutil.py'>,
'signal': <module 'signal' from '/var/lang/lib/python3.6/signal.py'>,
'site': <module 'site' from '/var/lang/lib/python3.6/site.py'>,
'socket': <module 'socket' from '/var/lang/lib/python3.6/socket.py'>,
'socketserver': <module 'socketserver' from '/var/lang/lib/python3.6/socketserver.py'>,
'sre_compile': <module 'sre_compile' from '/var/lang/lib/python3.6/sre_compile.py'>,
'sre_constants': <module 'sre_constants' from '/var/lang/lib/python3.6/sre_constants.py'>,
'sre_parse': <module 'sre_parse' from '/var/lang/lib/python3.6/sre_parse.py'>,
'ssl': <module 'ssl' from '/var/lang/lib/python3.6/ssl.py'>,
'stat': <module 'stat' from '/var/lang/lib/python3.6/stat.py'>,
'string': <module 'string' from '/var/lang/lib/python3.6/string.py'>,
'struct': <module 'struct' from '/var/lang/lib/python3.6/struct.py'>,
'subprocess': <module 'subprocess' from '/var/lang/lib/python3.6/subprocess.py'>,
'sys': <module 'sys' (built-in)>,
'sysconfig': <module 'sysconfig' from '/var/lang/lib/python3.6/sysconfig.py'>,
'tempfile': <module 'tempfile' from '/var/lang/lib/python3.6/tempfile.py'>,
'textwrap': <module 'textwrap' from '/var/lang/lib/python3.6/textwrap.py'>,
'threading': <module 'threading' from '/var/lang/lib/python3.6/threading.py'>,
'time': <module 'time' (built-in)>,
'token': <module 'token' from '/var/lang/lib/python3.6/token.py'>,
'tokenize': <module 'tokenize' from '/var/lang/lib/python3.6/tokenize.py'>,
'traceback': <module 'traceback' from '/var/lang/lib/python3.6/traceback.py'>,
'types': <module 'types' from '/var/lang/lib/python3.6/types.py'>,
'urllib': <module 'urllib' from '/var/lang/lib/python3.6/urllib/__init__.py'>,
'urllib.error': <module 'urllib.error' from '/var/lang/lib/python3.6/urllib/error.py'>,
'urllib.parse': <module 'urllib.parse' from '/var/lang/lib/python3.6/urllib/parse.py'>,
'urllib.request': <module 'urllib.request' from '/var/lang/lib/python3.6/urllib/request.py'>,
'urllib.response': <module 'urllib.response' from '/var/lang/lib/python3.6/urllib/response.py'>,
'uu': <module 'uu' from '/var/lang/lib/python3.6/uu.py'>,
'warnings': <module 'warnings' from '/var/lang/lib/python3.6/warnings.py'>,
'weakref': <module 'weakref' from '/var/lang/lib/python3.6/weakref.py'>,
'wsgi': <module 'wsgi' from '/var/runtime/awslambda/wsgi.py'>,
'wsgiref': <module 'wsgiref' from '/var/lang/lib/python3.6/wsgiref/__init__.py'>,
'wsgiref.handlers': <module 'wsgiref.handlers' from '/var/lang/lib/python3.6/wsgiref/handlers.py'>,
'wsgiref.headers': <module 'wsgiref.headers' from '/var/lang/lib/python3.6/wsgiref/headers.py'>,
'wsgiref.simple_server': <module 'wsgiref.simple_server' from '/var/lang/lib/python3.6/wsgiref/simple_server.py'>,
'wsgiref.util': <module 'wsgiref.util' from '/var/lang/lib/python3.6/wsgiref/util.py'>,
'zipimport': <module 'zipimport' (built-in)>,
'zlib': <module 'zlib' from '/var/lang/lib/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so'>}
/var/task/main.py:8: MonkeyPatchWarning: Monkey-patching ssl after ssl has already been imported may lead to errors, including RecursionError on Python 3.6. Please monkey-patch earlier. See https://github.com/gevent/gevent/issues/1016
gevent.monkey.patch_all()

[dvf]

Any chance we can re-open this?