Is it possible to inject failures into DynamoDB request handling for chaos testing?

[damc-dev]

Hi,
The AWS SDK for Java allows you to specify request handlers with the DynamoDB Java client which can be used to inject failures and latency as described in this blog post: https://aws.amazon.com/blogs/developer/injecting-failures-and-latency-using-the-aws-sdk-for-java/.

Is there something similar within the aws-sdk-rust that could be used to injecting faults into DynamoDB requests?

[jdisanti]

While there isn't any built-in support for failure/latency injection, there are two approaches you can take for this:

1. Override the default hyper/rustls HTTPS connector with your own delegating connector that injects the failures, or
2. Set up a proxy to inject failures. There are a couple different threads that explore this approach, with the second one being a real world example by another SDK user.

I recently built a custom connector to diagnose a throttling failure for STS, so I've adapted that to DynamoDB below as an example of approach 1.

Cargo.toml dependencies

[dependencies]
aws-config = "0.12.0"
aws-sdk-dynamodb = "0.12.0"
aws-smithy-client = "0.42.0"
aws-smithy-http = "0.42.0"
bytes = "1"
http = "0.2.7"
rand = "0.8.5"
tokio = { version = "1", features = ["full"] }
tower = "0.4"

Custom failure injecting connector

use aws_sdk_dynamodb as dynamo;
use aws_smithy_client::erase::DynConnector;
use aws_smithy_client::hyper_ext;
use aws_smithy_http::body::SdkBody;
use aws_smithy_http::result::ConnectorError;
use bytes::Bytes;
use rand::{thread_rng, Rng};
use std::future::Future;
use std::pin::Pin;
use std::task::Poll;
use std::time::Duration;

/// Custom connector that injects failures
#[derive(Clone, Debug)]
struct FakeFailureConnector {
    inner: DynConnector,
}

impl FakeFailureConnector {
    fn new() -> FakeFailureConnector {
        FakeFailureConnector {
            // Construct a default HTTPS connector to delegate to
            inner: DynConnector::new(
                hyper_ext::Adapter::builder().build(aws_smithy_client::conns::https()),
            ),
        }
    }
}

// The custom connector must implement Service<Request<SdkBody>> in order to work with the SDK
impl tower::Service<http::Request<SdkBody>> for FakeFailureConnector {
    type Response = http::Response<SdkBody>;
    type Error = ConnectorError;
    type Future =
        Pin<Box<dyn Future<Output = Result<Self::Response, Self::Error>> + Send + 'static>>;

    fn poll_ready(&mut self, cx: &mut std::task::Context<'_>) -> Poll<Result<(), Self::Error>> {
        self.inner.poll_ready(cx)
    }

    fn call(&mut self, req: http::Request<SdkBody>) -> Self::Future {
        // Inject a failure 30% of the time
        let failure_chance = 0.3;
        let inject_failure = thread_rng().gen_bool(failure_chance);

        // Note: The SDK will retry these failures, so if this is working, there should be more
        // of these outputs than calls made.
        println!(
            "FakeFailureConnector called; decision: {:?}",
            inject_failure
        );

        // Either inject a failure, or delegate to the underlying HTTPS connector to make a real call
        let result: Self::Future = if inject_failure {
            Box::pin(async {
                Ok(http::Response::builder()
                    .status(500)
                    .body(SdkBody::from(Bytes::from_static(b"")))
                    .unwrap())
            })
        } else {
            let inner_call = self.inner.call(req);
            Box::pin(async { Ok(inner_call.await?) })
        };
        result
    }
}

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let config = aws_config::load_from_env().await;

    // When constructing the client, use `from_conf_conn` to override the connector with our `FakeFailureConnector`
    let client = dynamo::Client::from_conf_conn((&config).into(), FakeFailureConnector::new());

    // To illustrate it working, call DynamoDB a few times
    for _ in 1..=5 {
        match client.list_tables().send().await {
            Ok(_) => println!("call succeeded!"),
            Err(err) => println!("failed: {:#?}", err),
        }
        tokio::time::sleep(Duration::from_millis(50)).await;
    }

    Ok(())
}

[damc-dev]

Thank you @jdisanti, thats exactly what I was looking for.

I appreciate the bar raising answer along with example code!

[ymwjbxxq]

You could setup a lower timeout on the client.
DynamoDB usually has latency between 10ms and 20ms, but a lot of time is going over and so if you would add this:

let timeout_config = timeout::Config::new()
        .with_api_timeouts(
            timeout::Api::new()
                // This timeout acts at the "Request to a service" level. When the SDK makes a request to a
                // service, that "request" can contain several HTTP requests. This way, you can retry
                // failures that are likely spurious, or refresh credentials.
                .with_call_timeout(TriState::Set(Duration::from_millis(20)))
                // This timeout acts at the "HTTP request" level and sets a separate timeout for each
                // HTTP request made as part of a "service request."
                .with_call_attempt_timeout(TriState::Set(Duration::from_millis(20))),
        )
        .with_http_timeouts(
            timeout::Http::new()
                // A limit on the amount of time an application takes to attempt to read the first byte over
                // an established, open connection after a write request.
                // Also known as the "time to first byte" timeout.
                .with_read_timeout(TriState::Set(Duration::from_millis(10)))
                // A time limit for completing the connect-handshake. The time starts when
                // making an initial connect attempt on a socket.
                .with_connect_timeout(TriState::Set(Duration::from_millis(10))),
        );
    let config = aws_config::from_env()
        .retry_config(RetryConfig::new().with_max_attempts(10)) -> THIS DOES NOT DO ANYTHING
        .timeout_config(timeout_config)
        .load()
        .await;

    let dynamodb_client = aws_sdk_dynamodb::Client::new(&config);

Your Lambda will crash so many times :)

[github-actions[bot]]

Hello! Reopening this discussion to make it searchable.