Mock S3 "Custom" endpoint/region for local testing purposes?

[brainstorm]

Hi aws-sdk-rust team,

From what I see on the defined S3 endpoints, there does not seem to be any way to define a non-production S3 endpoint for local (mock) testing?

I'm using the excellent s3-server crate from @datenlord on my project's S3-related testing operations but I can't seem to be able to point aws-sdk-rust's S3 client to http://localhost:8041 as I need to mock test some of my functionality?

Rusoto does have such a facility via "custom Regions", see: rusoto/rusoto#759.

EDIT: while I was writing this and checking out this S3 signing test, I noticed smithy-client, which looks like a lower-level way to test things out and perhaps not requiring s3-server crate at all... I'm not sure it supports head_object or similar but I'll check that, but please let me know if there's a better way to test those ;)

[rcoh]

Hello! using s3-server is a totally reasonable testing strategy, you can accomplish by setting a custom endpoint:
(note that you'll need to a dependency on http directly and construct your client a little differently):

use aws_sdk_s3::Client;
use aws_sdk_s3::Endpoint;
use http::Uri;

#[tokio::main]
async fn main() {
    let conf = aws_config::load_from_env().await;
    let ep = Endpoint::immutable(Uri::from_static("http://localhost:8543"));
    let s3_conf = aws_sdk_s3::config::Builder::from(&conf)
        .endpoint_resolver(ep)
        .build();
    let s3 = Client::from_conf(s3_conf);
    let buckets = s3.list_buckets().send().await;
    println!("got buckets: {:#?}", buckets);
}

You can also use the hyper service as a connector directly (which I think is pretty cool, personally!):

use aws_sdk_s3::ByteStream;
use s3_server::storages::fs::FileSystem;
use s3_server::S3Service;
use smithy_http::result::ConnectorError;
use std::error::Error;
use std::future::Future;
use std::pin::Pin;

use aws_sdk_s3::Client;
use aws_sdk_s3::Endpoint;
use http::Uri;
use s3_server::SharedS3Service;
use smithy_http::body::SdkBody;

#[derive(Clone)]
struct AwsS3Server(SharedS3Service);

impl tower::Service<http::Request<SdkBody>> for AwsS3Server {
    type Response = http::Response<SdkBody>;
    type Error = ConnectorError;

    type Future = Pin<Box<dyn Future<Output = Result<Self::Response, Self::Error>> + Send>>;

    fn poll_ready(
        &mut self,
        cx: &mut std::task::Context<'_>,
    ) -> std::task::Poll<Result<(), Self::Error>> {
        self.0.poll_ready(cx).map_err(ConnectorError::io)
    }

    fn call(&mut self, req: http::Request<SdkBody>) -> Self::Future {
        let req = req.map(|sdkbody| hyper::Body::wrap_stream(ByteStream::new(sdkbody)));
        let inner = self.0.call(req);
        let fut = async move {
            let resp = inner.await;
            resp.map(|ok| ok.map(SdkBody::from))
                .map_err(ConnectorError::io)
        };
        Box::pin(fut)
    }
}

#[tokio::main]
async fn main() -> Result<(), Box<dyn Error>> {
    let conf = aws_config::load_from_env().await;
    let ep = Endpoint::immutable(Uri::from_static("http://localhost:8543"));
    let s3_conf = aws_sdk_s3::config::Builder::from(&conf).build();
    let fs = FileSystem::new("./s3")?;
    // setup the service
    let server = AwsS3Server(S3Service::new(fs).into_shared());
    let s3 = Client::from_conf_conn(s3_conf, server);
    let buckets = s3.list_buckets().send().await;
    println!("got buckets: {:#?}", buckets);
    Ok(())
}

[brainstorm]

Excellent! Thanks so much for the detailed explanation! ;)

[brainstorm]

Hello again @rcoh, fast forward to 2023, we'd like to remove the s3-server dependency since it has OpenSSL and Rusoto dependencies we'd really like to remove from our codebase... and ideally not requiring a localhost server running to respond to requests.

Unfortunately it's not entirely clear to me how to mock a few methods with the official AWS trait mock docs.

This is a followup so folks can land on this issue and improve docs and support for Client mocking, so replies can be followed up there instead?