Support S3 HTTP redirects to non-Amazon URI's by using the location header

[GeorgeJahad]

Is your feature request related to a problem? Please describe.
I'm a developer contributing to Apache Ozone, an S3 compatible object store.

I would like to use HTTP redirects to balance the load amongst Ozone-based s3 servers, but aws-cli/botocore doesn't currently support redirects to non-amazon URI's.

Typically, http clients implement redirects using the 'location' header in the response as the destination. aws-cli/botocore doesn't. Instead, it uses the region . From that, it constructs an amazonaws.com based URI to redirect to. This means that aws-cli only ever redirects s3 requests to amazon URI's even if it made the original request to a non-Amazon server.

Describe the solution you'd like
Allow aws-cli to support s3 redirection to non-Amazon URI's using the location header if available. This functionality would be controlled by a config variable, and off by default.

It could look something like this.

[tim-finnigan]

Hi @GeorgeJahad, thanks for the feature request. I think what you’re asking for is essentially the ability to specify an endpoint url. Can you confirm if that correct? If so then does using the --endpoint-url parameter accomplish what you're trying to do?

[GeorgeJahad]

Hey Tim:

I appreciate the quick response, but no, it isn't quite what I'm asking for.

The --endpoint-url option allows the user to specify a private, (non-amazon,) s3-compatible service, (and I do use it for that.)

But it is sometimes useful for that service to redirect the aws-cli client to a different url, using standard http redirects: https://developer.mozilla.org/en-US/docs/Web/HTTP/Redirections
https://docs.aws.amazon.com/AmazonS3/latest/userguide/RESTRedirect.html

Amazon s3 does that, when it finds that your bucket is in a different region, and it redirects the aws-cli client to the region appropriate for that bucket. The aws-cli code that receives/handles that redirect is here: https://github.com/aws/aws-cli/blob/v2/awscli/botocore/utils.py#L1292-L1309

But if you look at that code, it only allows redirects to amazon servers, not s3 compatible servers.

I'm suggesting we allow redirects to be generalized so they also work with non-amazon servers.

The changes seem simple: GeorgeJahad/aws-cli@b97cdd9...GeorgeJahad:georgeFirst

Do you think you would consider an enhancement like that? I would be happy to code it. I just want to see if that is something you'd be willing to accept.

What do you think?

Thanks much!

[tim-finnigan]

Hi @GeorgeJahad, thanks for clarifying that. I brought this up for discussion with the team and the consensus was that this is not something they plan to implement.

One reason is because allowing redirects to non-AWS URIs could be a security vulnerability. Another reason is because they expressed concern that altering the current redirect logic could lead to breaking changes.

You could potentially create a plugin to support this behavior, with the caveat of knowing that how plugins are supported may evolve over time.

[GeorgeJahad]

Thanks for checking @tim-finnigan

With respect to plugins, I'm not seeing any documentation on how to implement one. Do you have any pointers?

[michelsumbul]

@tim-finnigan do you think it will be possible to imagine to have by default redirection only to aws URI (what we have now) but if a specific option is mentioned, like --allow-redirect-non-aws-uri, then the client will accept redirection to non aws.
In that way we seriously mitigate the security concern plus by default we don't change the redirect logic except if the user explicitly use that option.
What do you think?

I think having the option to be redirect to non aws uri will be really great!

[tim-finnigan]

@GeorgeJahad we have another issue open requesting support for a stable plugin API: #2350. But as mentioned in this comment: #1261 (comment), there is no official plugin documentation as of now because it is subject to change.

@michelsumbul the proposal to add a configuration option for disabling region redirection was floated here a while back: boto/botocore#1708 (comment) but I can’t guarantee that it would be considered. You could try creating a new feature request issue in the botocore repository and fleshing out your proposed use case there.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.