-
Notifications
You must be signed in to change notification settings - Fork 409
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
remote_storage: AWS_PROFILE with endpoint overrides in ~/.aws/config (updates AWS SDKs) #7664
Conversation
… (updates AWS SDKs) Before this PR, using the AWS SDK profile feature for running against minio didn't work, * either because our SDK versions were too old and didn't include awslabs/aws-sdk-rust#1060 * or because we didn't massage the s3 client config builder correctly. This PR * udpates all the AWS SDKs we use to, respectively, the latest version I could find on crates.io (Is there a better process?) * changes the way remote_storage constructs the S3 client. Regarding the latter, if one reads the SDK docs, it is clear that the recommended way is to use `aws_config::from_env` because that sources all the default, then customize where needed. What we were doing instead is to use the `aws_sdk_s3` builder directly.
It fails with ``` 2024-05-08T18:41:09.230477Z INFO attach{tenant_id=ccd34c1399035a50e09bbc8b05de940a shard_id=0000 gen=00000001}:preload: listing remote timelines 2024-05-08T18:41:12.337353Z INFO attach{tenant_id=ccd34c1399035a50e09bbc8b05de940a shard_id=0000 gen=00000001}:preload:lazy_load_identity: constructed abstract provider from config file chain=ProfileChain { base: AccessKey(Credentials { provider_name: "ProfileFile", access_key_id: "foobar123", secret_access_key: "** redacted **", expires_after: "never" }), chain: [] } 2024-05-08T18:41:12.337430Z INFO attach{tenant_id=ccd34c1399035a50e09bbc8b05de940a shard_id=0000 gen=00000001}:preload:lazy_load_identity: loaded base credentials creds=Credentials { provider_name: "ProfileFile", access_key_id: "foobar123", secret_access_key: "** redacted **", expires_after: "never" } 2024-05-08T18:41:12.342052Z INFO attach{tenant_id=ccd34c1399035a50e09bbc8b05de940a shard_id=0000 gen=00000001}:preload: list identifiers in prefix tenants/ccd34c1399035a50e09bbc8b05de940a/timelines/ failed, will retry (attempt 1): Failed to download a remote file: Failed to list S3 prefixes Caused by: 0: dispatch failure 1: io error 2: error trying to connect: dns error: failed to lookup address information: Name or service not known 3: dns error: failed to lookup address information: Name or service not known 4: failed to lookup address information: Name or service not known ```
This reverts commit db2011d.
it fails with ``` 2024-05-08T18:46:08.713500Z INFO attach{tenant_id=ff8ceb3c7e9b0963650bee7c29cb74e4 shard_id=0000 gen=00000001}:preload: listing remote timelines 2024-05-08T18:46:11.778847Z WARN attach{tenant_id=ff8ceb3c7e9b0963650bee7c29cb74e4 shard_id=0000 gen=00000001}:preload:lazy_load_identity: profile `services local-minio-services` ignored because `services local-minio-services` was not a valid identifier 2024-05-08T18:46:11.779070Z INFO attach{tenant_id=ff8ceb3c7e9b0963650bee7c29cb74e4 shard_id=0000 gen=00000001}:preload:lazy_load_identity: constructed abstract provider from config file chain=ProfileChain { base: AccessKey(Credentials { provider_name: "ProfileFile", access_key_id: "foobar123", secret_access_key: "** redacted **", expires_after: "never" }), chain: [] } 2024-05-08T18:46:11.779102Z INFO attach{tenant_id=ff8ceb3c7e9b0963650bee7c29cb74e4 shard_id=0000 gen=00000001}:preload:lazy_load_identity: first credentials will be loaded from AccessKey(Credentials { provider_name: "ProfileFile", access_key_id: "foobar123", secret_access_key: "** redacted **", expires_after: "never" }) base=AccessKey(Credentials { provider_name: "ProfileFile", access_key_id: "foobar123", secret_access_key: "** redacted **", expires_after: "never" }) 2024-05-08T18:46:11.779129Z INFO attach{tenant_id=ff8ceb3c7e9b0963650bee7c29cb74e4 shard_id=0000 gen=00000001}:preload:lazy_load_identity: loaded base credentials creds=Credentials { provider_name: "ProfileFile", access_key_id: "foobar123", secret_access_key: "** redacted **", expires_after: "never" } 2024-05-08T18:46:11.779168Z INFO attach{tenant_id=ff8ceb3c7e9b0963650bee7c29cb74e4 shard_id=0000 gen=00000001}:preload:lazy_load_identity: identity cache miss occurred; added new identity (took 692.001<C2><B5>s) new_expiration=2024-05-08T19:01:11.778442Z valid_for=899.999275768s partition=IdentityCachePartition(6) 2024-05-08T18:46:11.782016Z INFO attach{tenant_id=ff8ceb3c7e9b0963650bee7c29cb74e4 shard_id=0000 gen=00000001}:preload: list identifiers in prefix tenants/ff8ceb3c7e9b0963650bee7c29cb74e4/timelines/ failed, will retry (attempt 1): Failed to download a remote file: Failed to list S3 prefixes Caused by: 0: dispatch failure 1: io error 2: error trying to connect: dns error: failed to lookup address information: Name or service not known 3: dns error: failed to lookup address information: Name or service not known 4: failed to lookup address information: Name or service not known ```
This reverts commit 5517893.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A bit weird that we now have both http body versions but, maybe that'll go away with the follow-up work.
3024 tests run: 2892 passed, 0 failed, 132 skipped (full report)Code coverage* (full report)
* collected from Rust tests only The comment gets automatically updated with the latest test results
3e13ac4 at 2024-05-09T08:34:14.513Z :recycle: |
Can you link to what you're referring to here? I'm not following |
Have you checked whether the commands written in #6202 still work with this PR applied? |
Yep, still works. Extended the docs in this PR. Verified manually that the tests actually use the env-var-configured bucket / credentials. Side note: the |
That's probably because the bucket doesn't have versioning enabled. It needs to be enabled explicitly (also in S3 but those buckets are long lived and the configuration lives outside of this repo). For #6533's pytest changes I had to add a |
…(updates AWS SDKs) (#7664) Before this PR, using the AWS SDK profile feature for running against minio didn't work because * our SDK versions were too old and didn't include awslabs/aws-sdk-rust#1060 and * we didn't massage the s3 client config builder correctly. This PR * udpates all the AWS SDKs we use to, respectively, the latest version I could find on crates.io (Is there a better process?) * changes the way remote_storage constructs the S3 client, and * documents how to run the test suite against real S3 & local minio. Regarding the changes to `remote_storage`: if one reads the SDK docs, it is clear that the recommended way is to use `aws_config::from_env`, then customize. What we were doing instead is to use the `aws_sdk_s3` builder directly. To get the `local-minio` in the added docs working, I needed to update both the SDKs and make the changes to the `remote_storage`. See the commit history in this PR for details. Refs: * byproduct: smithy-lang/smithy-rs#3633 * follow-up on deprecation: #7665 * follow-up for scrubber S3 setup: #7667
Before this PR, using the AWS SDK profile feature for running against minio didn't work because
Service Specific Configuration Override awslabs/aws-sdk-rust#1060 and
This PR
Regarding the changes to
remote_storage
: if one reads the SDK docs, it is clear that the recommended way is to useaws_config::from_env
, then customize.What we were doing instead is to use the
aws_sdk_s3
builder directly.To get the
local-minio
in the added docs working, I needed to update both the SDKs and make the changes to theremote_storage
. See the commit history in this PR for details.Refs: