sundog, pluto: run settings generators with network proxy environment vars

[etungsten]

Issue number:
Fixes #2197

Description of changes:

    sundog: run settings generators with network proxy environment vars
    
    We make sure settings generators are kicked-off with user-specified
    network proxy environment variables.

Author: Erikson Tung <etung@amazon.com>
Date:   Tue Jun 21 18:16:38 2022 -0700

    pluto: respect `https_proxy` and `no_proxy` when making EKS API reqs
    
    Both `rusoto` and `aws-sdk-rust` do not respect `https_proxy` and
    `no_proxy` env vars. We have to handle it ourselves by creating a custom
    hyper http 'connector' that's configured with the right proxying
    configuration.

Testing done:
Builds and runs fine.
Launched instance with aws-k8s-1.23 AMI with user-data that includes the following:

[settings.network]
https-proxy = "my-proxy:9898"

Checked proxy server logs and found all traffic being tunneled including the EKS API calls for DescribeCluster

CONNECT   Jun 22 08:00:00 [3794]: Connect (file descriptor 7): ec2-35-88-231-112.us-west-2.compute.amazonaws.com [35.88.231.112]   
CONNECT   Jun 22 08:00:00 [3794]: Request (file descriptor 7): CONNECT eks.us-west-2.amazonaws.com:443 HTTP/1.1                    
INFO      Jun 22 08:00:00 [3794]: No upstream proxy for eks.us-west-2.amazonaws.com                                                
INFO      Jun 22 08:00:00 [3794]: opensock: opening connection to eks.us-west-2.amazonaws.com:443                                  
INFO      Jun 22 08:00:00 [3794]: opensock: getaddrinfo returned for eks.us-west-2.amazonaws.com:443                               
CONNECT   Jun 22 08:00:00 [3794]: Established connection to host "eks.us-west-2.amazonaws.com" using file descriptor 8.            
INFO      Jun 22 08:00:00 [3794]: Not sending client headers to remote machine                                                     
INFO      Jun 22 08:00:00 [3794]: Closed connection between local client (fd:7) and remote client (fd:8) 

Checking CloudTrail, I can see that the DescribeCluster API call source IP is my proxy server

...
    "eventTime": "2022-06-22T08:00:00Z",
    "eventSource": "eks.amazonaws.com",
    "eventName": "DescribeCluster",
    "awsRegion": "us-west-2",
    "sourceIPAddress": "my-proxy-IP",
    "userAgent": "rusoto/0.48.0 rust/1.61.0 linux",
...

If I set no-proxy for the EKS API endpoint for a new instance launch like so:

[settings.network]
https-proxy = "my-proxy:9898"
no-proxy = ["eks.us-west-2.amazonaws.com"]

Then I can see that EKS API calls do not get proxied and the source IP is my actual instance:

...
    "eventTime": "2022-06-22T08:10:04Z",
    "eventSource": "eks.amazonaws.com",
    "eventName": "DescribeCluster",
    "awsRegion": "us-west-2",
    "sourceIPAddress": "34.211.227.47",
    "userAgent": "rusoto/0.48.0 rust/1.61.0 linux",
...

Terms of contribution:

By submitting this pull request, I agree that this contribution is dual-licensed under the terms of both the Apache License, version 2.0, and the MIT license.

[etungsten]

So turns out there's a bit of complication with rusoto not respecting proxy environment variables automatically. The gist of it is that I'll have to create a custom rusoto_core::HttpClient with the right proxy config in pluto for the EksClient. See rusoto/rusoto#1172

Once we've switch over to aws-rust-sdk wholesale, we can just remove the custom http client from pluto.
See #1968

[etungsten]

Apparently this is still an issue for aws-rust-sdk as well, so switching over won't help pluto either. See awslabs/aws-sdk-rust#169

[webern]

Wow, I'm disappointed that the AWS Rust SDK does not support proxy env vars. Rusoto I could understand because it may be using an old version of hyper, but I would have expected the AWS Rust SDK to be using the latest hyper under-the-hood.

[zmrow]

🚢

Nice work!

[jpculp]

It looks like we're getting multiple versions of a couple crates due to things depending of hyper-rustls, but since aws-rust-sdk also uses 0.22.1, it will get sorted out when we transition.

[etungsten]

Push above addresses @bcressey 's comments.

Tested changes they still work as described in the PR description