Ability to provide authorization when tunneling to another proxy

[saadtazi]

Is your feature request related to a problem? Please describe.

I am using master branch.

I am trying to follow/modify the ProxyPoolPlugin in order to connect to some proxies that requires authentication. I am not sure if it is possible. One of the proxies I try to use is luminati. I am able to use it successfully using the following curl command or using requests python library.

http client <---> proxy.py proxy <---> proxy with authentication
                            î
                            +-------------> another proxy with authentication

Here is what I tried:

In before_upstream_connection(), I am returning the server name and port in the following format:

self.conn = new_socket_connection(
            ("zproxy.lum-superproxy.io", 22225))

Then, I modified handle_client_request() to add the Proxy-Authorization header:

    def handle_client_request(
            self, request: HttpParser) -> Optional[HttpParser]:
        request.path = self.rebuild_original_path(request)
        request.headers = request.headers or {}
        session_id = random.random()
        user_pwd_str = "username:password"
        userAndPass = "Basic " + b64encode(user_pwd_str.encode('ascii'))
        request.headers[b"proxy-authorization"] = (b"Proxy-Authorization", userAndPass)
        self.tunnel(request)
        # Returning None indicates core to gracefully
        # flush client buffer and teardown the connection
        return None

I then started proxy.py:

$ pipenv run proxy  --plugins my_plugins.custom_luminati_proxy.ProxyPoolPlugin
2020-03-02 19:47:07,221 - pid:94482 [I] load_plugins:525 - Loaded plugin proxy.http.proxy.HttpProxyPlugin
2020-03-02 19:47:07,222 - pid:94482 [I] load_plugins:525 - Loaded plugin my_plugins.custom_luminati_proxy.ProxyPoolPlugin
2020-03-02 19:47:07,223 - pid:94482 [I] listen:63 - Listening on 0.0.0.0:8899
2020-03-02 19:47:07,236 - pid:94482 [I] start_workers:84 - Started 8 workers

Then I used curl: curl -v -x localhost:8899 http://httpbin.org

I got the following curl output:

$ curl -v  -x localhost:8899 http://httpbin.org 
*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 8899 (#0)
> GET http://httpbin.org/ HTTP/1.1
> Host: httpbin.org
> User-Agent: curl/7.64.1
> Accept: */*
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 400 Bad Request
< Connection: close
<

And the following logs in proxy.py:

2020-03-02 19:49:46,071 - pid:94656 [I] access_log:343 - ::1:60491 - GET None:Nonehttp://httpbin.org:/ - None None - 0 bytes - 10126.25 ms

I am obviously doing something wrong: notice the None:None.

Note: issuing the following curl command works (after replacing with valid username/password):
curl -v --proxy zproxy.lum-superproxy.io:22225 --proxy-user valid-user:valid-password "http://lumtest.com/myip.json"

---

Note: I also tried to add username and password like this: new_socket_connection("username:password@zproxy.lum-superproxy.io", 22225)) but I get the following error:

  File "/Users/saadtazi/.pyenv/versions/3.7.4/lib/python3.7/socket.py", line 748, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno 8] nodename nor servname provided, or not known

Describe the solution you'd like
I am not sure how to provide proxy authentication for the final proxy.

Any help would be appreciated.

Thank you

[saadtazi]

ok, made some progress: after realizing that the original proxy_pool plugin was not working (same None:None), I slightly changed rebuild_original_path() plugin method so it returns the following:

return (
            request.url.scheme +
            COLON + SLASH + SLASH +
            request.host +
            COLON +
            bytes(str(request.port).encode("ascii")) +  # <-- was bytes(request.port)
            request.path
        )

I know start getting some responses! but I don't get all the html: the curl command get stuck (times out) and ends with the following (partial response):

...aria-label="View source on Github">
* transfer closed with 8631 bytes remaining to read
* Closing connection 0
curl: (18) transfer closed with 8631 bytes remaining to read
        <svg width="80" height="80" viewBox="0 0 250 250" style="fill:#1

Increasing the number of bytes read (probably not recommended...) "fixes" it: response = self.conn.recv(DEFAULT_BUFFER_SIZE * 1000)

Note that I get the same behaviour (incomplete response) with the original proxy_pool.py plugin (after starting pipenv run proxy --port 9001 and pipenv run proxy --plugins proxy.plugin.ProxyPoolPlugin)

I tried master and develop: same behaviour (stuck + partial response...)

---

Update: I just found this issue #292. I am pretty sure it is related.

[abhinavsingh]

@saadtazi Yes, you are right. Incomplete response can happen for large response bodies. I wrote proxy pool example mostly for demonstration purposes. And it (proxy pool plugin) needs more work for production usage.

#292 discusses a few approaches. You can use the short-term approach while @atsakiridis work this out.

Let me know how it goes. Thank you!!!!

PS: I am currently traveling till end of this month. If time permits I'll jump back into it. Unfortunately, currently I am not carrying my development machine.

[saadtazi]

Thank you @abhinavsingh , and thanks for the "responsiveness".

If by "short term" approach you mean "increase the number of bytes read", then it works great!

I guess we can close this issue now that I was able to authenticate. And I can definitively wait for #292 to be fixed (no rush on my end...)

One last question though: I am still wondering if the change I made to rebuild_original_path() makes sense (seems to fix the original plugin ProxyPoolPlugin... for me at least...):
from:

return (
            request.url.scheme +
            COLON + SLASH + SLASH +
            request.host +
            COLON +
            bytes(request.port) +
            request.path
        )

to:

return (
            request.url.scheme +
            COLON + SLASH + SLASH +
            request.host +
            COLON +
            bytes(str(request.port).encode("ascii")) + # <-- changed line
            request.path
        )

[abhinavsingh]

Yes it does make sense. Someone already reported about it and I think someone sent a PR (which was merged), but I never got a chance to verify it's correctness. Sounds good to close this issue. As far as authorization is concerned, I believe proxy auth should work seamlessly even with chain of proxies. Just make sure proxy headers are not stripped or just add your own based upon upstream proxy requirements. Thank you!!!!

…

On Wed, Mar 4, 2020, 12:15 AM Saad Tazi ***@***.***> wrote: Thank you @abhinavsingh <https://github.com/abhinavsingh> , and thanks for the "responsiveness". If by "short term" approach you mean "increase the number of bytes read", then it works great! I guess we can close this issue now that I was able to authenticate. And I can definitively wait for #292 <#292> to be fixed (no rush on my end...) One last question though: I am still wondering if the change I made to rebuild_original_path() makes sense (seems to fix the original plugin ProxyPoolPlugin... for me at least...): from: return ( request.url.scheme + COLON + SLASH + SLASH + request.host + COLON + bytes(request.port) + request.path ) to: return ( request.url.scheme + COLON + SLASH + SLASH + request.host + COLON + bytes(str(request.port).encode("ascii")) + # <-- changed line request.path ) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#300?email_source=notifications&email_token=AAA6Y4N4EFIJESW3D4PTJM3RFVF5FA5CNFSM4K773QCKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOENUVMWQ#issuecomment-594105946>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAA6Y4JYEJDVQY2JDXIJNKDRFVF5FANCNFSM4K773QCA> .

[saadtazi]

Thank you!