Support secure proxies by implementing HTTPS-in-HTTPS.

[spumer]

I can't estabilish connection to target via https proxy.

:me: <-[ssh-tunnel]-> :protected-host: <-> :squid-proxy: <-> :target:

Here is my proxy settings for AsyncClient:

PROXIES = {
    'https': 'https://user:password@127.0.0.1:8443',
}

And i try to connect to 'https://target-hostname.com:8080' via that proxies.

When i use uvloop all works fine.
When i use asyncio connection can not be established and failed with traceback:

Traceback

Traceback (most recent call last):
  File "/Users/spumer/example/.venv/lib/python3.8/site-packages/httpcore/_async/http_proxy.py", line 239, in _tunnel_request
    await proxy_connection.start_tls(host, timeout)
  File "/Users/spumer/example/.venv/lib/python3.8/site-packages/httpcore/_async/connection.py", line 177, in start_tls
    self.socket = await self.connection.start_tls(hostname, timeout)
  File "/Users/spumer/example/.venv/lib/python3.8/site-packages/httpcore/_async/http11.py", line 87, in start_tls
    self.socket = await self.socket.start_tls(hostname, self.ssl_context, timeout)
  File "/Users/spumer/example/.venv/lib/python3.8/site-packages/httpcore/_backends/asyncio.py", line 113, in start_tls
    transport = await asyncio.wait_for(
  File "/Users/spumer/.pyenv/versions/3.8.6/lib/python3.8/asyncio/tasks.py", line 455, in wait_for
    return await fut
  File "/Users/spumer/.pyenv/versions/3.8.6/lib/python3.8/asyncio/base_events.py", line 1181, in start_tls
    raise TypeError(
TypeError: transport <asyncio.sslproto._SSLProtocolTransport object at 0x108d3bbe0> is not supported by start_tls()

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/spumer/example/.venv/lib/python3.8/site-packages/httpx/_exceptions.py", line 326, in map_exceptions
    yield
  File "/Users/spumer/example/.venv/lib/python3.8/site-packages/httpx/_client.py", line 1502, in _send_single_request
    (status_code, headers, stream, ext,) = await transport.arequest(
  File "/Users/spumer/example/.venv/lib/python3.8/site-packages/httpcore/_async/http_proxy.py", line 124, in arequest
    return await self._tunnel_request(
  File "/Users/spumer/example/.venv/lib/python3.8/site-packages/httpcore/_async/http_proxy.py", line 242, in _tunnel_request
    raise ProxyError(exc)
httpcore.ProxyError: transport <asyncio.sslproto._SSLProtocolTransport object at 0x108d3bbe0> is not supported by start_tls()

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/spumer/example/probe.py", line 23, in <module>
    asyncio.run(main())
  File "/Users/spumer/.pyenv/versions/3.8.6/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/Users/spumer/.pyenv/versions/3.8.6/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/Users/spumer/example/probe.py", line 15, in main
    fns = await client_factory.create_client()
  File "/Users/spumer/example/src/npd/fns_npd/client.py", line 166, in create_client
    message = await self._send_auth_request()
  File "/Users/spumer/example/src/npd/fns_npd/client.py", line 180, in _send_auth_request
    message = await self._auth_client.service.GetMessage(Message=value)
  File "/Users/spumer/example/.venv/lib/python3.8/site-packages/zeep/proxy.py", line 64, in __call__
    return await self._proxy._binding.send_async(
  File "/Users/spumer/example/.venv/lib/python3.8/site-packages/zeep/wsdl/bindings/soap.py", line 156, in send_async
    response = await client.transport.post_xml(
  File "/Users/spumer/example/.venv/lib/python3.8/site-packages/zeep/transports.py", line 230, in post_xml
    response = await self.post(address, message, headers)
  File "/Users/spumer/example/.venv/lib/python3.8/site-packages/zeep/transports.py", line 215, in post
    response = await self.client.post(
  File "/Users/spumer/example/.venv/lib/python3.8/site-packages/httpx/_client.py", line 1633, in post
    return await self.request(
  File "/Users/spumer/example/.venv/lib/python3.8/site-packages/httpx/_client.py", line 1371, in request
    response = await self.send(
  File "/Users/spumer/example/.venv/lib/python3.8/site-packages/httpx/_client.py", line 1406, in send
    response = await self._send_handling_auth(
  File "/Users/spumer/example/.venv/lib/python3.8/site-packages/httpx/_client.py", line 1444, in _send_handling_auth
    response = await self._send_handling_redirects(
  File "/Users/spumer/example/.venv/lib/python3.8/site-packages/httpx/_client.py", line 1476, in _send_handling_redirects
    response = await self._send_single_request(request, timeout)
  File "/Users/spumer/example/.venv/lib/python3.8/site-packages/httpx/_client.py", line 1502, in _send_single_request
    (status_code, headers, stream, ext,) = await transport.arequest(
  File "/Users/spumer/.pyenv/versions/3.8.6/lib/python3.8/contextlib.py", line 131, in __exit__
    self.gen.throw(type, value, traceback)
  File "/Users/spumer/example/.venv/lib/python3.8/site-packages/httpx/_exceptions.py", line 343, in map_exceptions
    raise mapped_exc(message, **kwargs) from exc  # type: ignore
httpx.ProxyError: transport <asyncio.sslproto._SSLProtocolTransport object at 0x108d3bbe0> is not supported by start_tls()

[florimondmanca]

Probably related? encode/httpx#1424 (That issue is for the sync client.)

Though the distinction b/w uvloop and plain asyncio is quite interesting here.

:me: <-[ssh-tunnel]-> :protected-host: <-> :squid-proxy: <-> :target:

The :protected-host: is the one running the program that runs HTTPX, right?

[spumer]

No, i'm running program on :me: host.
And use tunneling to :squid-proxy: (trough :protected-host:) via ssh:
ssh -L 8443:squid-proxy.host:8443 user@protected-host.host -N

Sorry for my incorrect schema. I connect to :target: through :squid-proxy:, but connection to :squid-proxy: tunneled via ssh (may be that not significant)

[spumer]

This is why proxy host looks like 127.0.0.1, but actually connection go to :squid-proxy: host :)

Probably related? encode/httpx#1424 (That issue is for the sync client.)

I tried requests lib to sync connection and all works fine (actually zeep lib use it for sync)
(May be that can helps)

[florimondmanca]

I tried requests lib to sync connection and all works fine (actually zeep lib use it for sync)

And I assume using httpx.Client (our sync implementation) won't work, right? Does it fail with the same traceback than encode/httpx#1424 or is that something else? That other issue is one we could use some more reproduction cases, so why I'm asking. :)

As for this asyncio case — honestly, my bet is that we just don't really seem to support HTTPS/HTTPS (HTTPS requests through an HTTPS proxy) just yet. I don't know why it works on uvloop, though — could be by chance that uvloop's start_tls() does something asyncio doesn't do by default and we'd have to compensate for?

Just for the sake of narrowing down the problem, were you able to determine if the SSH tunnel could be part of the problem? I'd assume not, since that should be transparent to whatever application-level transmission gets performed over that connection — but who knows.

[spumer]

Ok, i tested httpx.Client and got httpx.ProxyError: [SSL: INVALID_ALERT] invalid alert (_ssl.c:1124)

Same exception i got when try to fix this issue manually for httpx.AsyncClient:

httpcore/httpcore/_backends/asyncio.py

Line 115 in cf45ea7

transport,

Here we pass transport object to event loop. Then it checks

        if not getattr(transport, '_start_tls_compatible', False):
            raise TypeError(
                f'transport {transport!r} is not supported by start_tls()')

And object which we pass does not have it. But transport._ssl_context._transport has :)
And when i try to pass it instead transport i got same exception as mentioned above.

Then i was think what about uvloop? It is re-implement sockets, what about TLS? and uvloop does not check _start_tls_compatible attribute and httpx.AsyncClient works fine! :D

[florimondmanca]

Hmm, interesting. :-) So asyncio is doing a check that's actually… not worth checking? Or I assume it just really can't do .start_tls() on an _SSLProtocolTransport (not implemented yet).

In urllib3/urllib3#1923 we can see that the urllib3 team had to provide a custom SSLTransport implementation (not for asyncio, but for sync) to get HTTPS proxies to work… Looks like this HTTPS proxy thing is bound to give us some low-level headaches if out-of-the-box ecosystem support (http.client, asyncio) is so low.

So, hmm, unfortunately this will take some time to look into and resolve!

@spumer Out of curiosity, would you be able to try connecting to your HTTPS proxy in a trio and curio async environments?

import trio
import curio

async def main():
    async with httpx.AsyncClient(proxies="https://...") as client:
        response = await client.get("https://...")
        print(response)

trio.run(main)
curio.run(main())

I don't think those shouldn't be problematic, because both provide clean "wrap/unwrap SSL transport" APIs that asyncio doesn't have, eg:

httpcore/httpcore/_backends/trio.py

Lines 37 to 54 in cf45ea7

	async def start_tls(
	self, hostname: bytes, ssl_context: SSLContext, timeout: TimeoutDict
	) -> "SocketStream":
	connect_timeout = none_as_inf(timeout.get("connect"))
	exc_map = {
	trio.TooSlowError: ConnectTimeout,
	trio.BrokenResourceError: ConnectError,
	}
	ssl_stream = trio.SSLStream(
	self.stream,
	ssl_context=ssl_context,
	server_hostname=hostname.decode("ascii"),
	)
	with map_exceptions(exc_map):
	with trio.fail_after(connect_timeout):
	await ssl_stream.do_handshake()
	return SocketStream(ssl_stream)

But it's definitely worth checking!

[spumer]

With trio works fine
With curio failed: httpcore.ConnectError: [SSL: INVALID_ALERT] invalid alert (_ssl.c:1124)

[jborean93]

I just came across this and found out that the start_tls() method in asyncio has that _start_tls_compatible method causing the failure. I'm not sure why the _SSLProtocolTransport doesn't have that attribute set as when I manually fallback to the backport_start_tls method it works without any immediate issues.

Without understanding the full risks involved it seems like 2 things need to happen

• A bug is reported for CPython asking for _SSLProtocolTransport to have _start_tls_compatible = True set
• httpcore continues to use the backport_start_tls method even on newer Python versions until the above is in place

httpcore could also manually add that attribute to the transport before it calls the asyncio start_tls method as a workaround instead of relying on the backport method.

[jborean93]

Looks like a bug has already been reported https://bugs.python.org/issue37179 and is meant to be fixed in 3.10. When I'm looking through the CPython codebase it doesn't seem to have been fixed and that PR doesn't seem to address that particular problem so I'll add a comment there.

Edit: Testing against 3.10.0a7 still has the problem.

[jborean93]

Looks like the PR was updated to support this and merged python/cpython#17975 but subsequently it was reverted python/cpython#25848 so 3.10 is still affected by this.

[jborean93]

I've opened a PR just for this problem, I still need to go through the CLA process but once that is done I'm hoping it will get accepted python/cpython#26454.

[jborean93]

Looks like it got lost in the pile, trying one more time python/cpython#28073

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[tomchristie]

Hello @stalebot - thanks for chiming in.

I'm curious why you added a wontfix label - I didn't configure you to do that. I suppose that must be your default behaviour.

Good prompt to review if this issue is stale or not, tho. Thank you.

I think we do still want this - right folks? But we probably ought to re-title it, because the issue more generally is just "we don't support HTTPS proxies right now" rather than specifically anything to do with asyncio.

So - questions...

• Should we be co-ordinating these issues against httpcore, or try to keep everything in one place on the httpx repo? Specifically - should we move this issue across?
• Do other folks agree that I've got the right assessment in the - "we don't support HTTPS proxies right now" rather than anything specifically to do with asyncio?
• We should probably take a first pass at this by making sure our proxy docs are up to scratch on our limitations there.
• Next step is then actually resolving that. I think this one is probably fairly high up on the priority list.
• We'll also want some new configuration here - proxy_ssl_context - so that we can configure the proxy SSL separately to the URL target SSL.

[tomchristie]

Our issue on httpx for this is encode/httpx#1434

[jborean93]

I think we do still want this - right folks

I think so, I don't use proxies very often but supporting such a thing sounds important, especially considering the prevalent of TLS today.

Should we be co-ordinating these issues against httpcore, or try to keep everything in one place on the httpx repo? Specifically - should we move this issue across?

I don't have really anything to say except that the code for this sits in this repository right now AFAIK.

Do other folks agree that I've got the right assessment in the - "we don't support HTTPS proxies right now" rather than anything specifically to do with asyncio?

I haven't tried any of the other backends but I know for sure the asyncio limitation is a problem in cpython.

Next step is then actually resolving that. I think this one is probably fairly high up on the priority list.

At least for asyncio I tried to fix it up with python/cpython#28073 but unfortunately there's some weird case where the tests are failing on Windows. I was never able to figure out why but there was trouble with how the various protocol/transports were closed and how it was all set up. I never had time to investigate it further unfortunately :(

We'll also want some new configuration here - proxy_ssl_context - so that we can configure the proxy SSL separately to the URL target SSL.

That sounds like a good idea

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[tomchristie]

Didn't expect to see you back on this ticket @Stale.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.