Skip to content

angelabad/terraform-aws-msk-cluster

Repository files navigation

AWS Msk Kafka Cluster

GitHub tag (latest by date) Msk Kafka Cluster

Terraform module which creates Msk Kafka Cluster on AWS.

These types of resources are supported:

Features

This module create a fully featured Msk Kafka Cluster on Aws. You could configure monitoring, encryption, server options, etc...

Usage

module "msk-cluster" {
  source  = "angelabad/msk-cluster/aws"

  cluster_name    = "kafka"
  instance_type   = "kafka.t3.small"
  number_of_nodes = 2
  client_subnets  = ["subnet-0ab97cbe1bd1406c2", "subnet-0d6cbf60360dbac64"]
  kafka_version   = "2.6.2"

  extra_security_groups = ["sg-019fc0f7d26f6c70f"]

  enhanced_monitoring = "PER_BROKER"

  s3_logs_bucket = aws_s3_bucket.logs.id
  s3_logs_prefix = "msklogs"

  prometheus_jmx_exporter  = true
  prometheus_node_exporter = true

  server_properties = {
    "auto.create.topics.enable"  = "true"
    "default.replication.factor" = "2"
  }

  encryption_in_transit_client_broker = "TLS"

  tags = {
    Owner       = "user"
    Environment = "dev"
  }
}

Requirements

Name Version
terraform >= 0.15
aws >= 4.16
random >= 2.1

Providers

Name Version
aws >= 4.16
random >= 2.1

Modules

No modules.

Resources

Name Type
aws_msk_cluster.this resource
aws_msk_configuration.this resource
aws_msk_scram_secret_association.this resource
aws_security_group.this resource
aws_security_group_rule.jmx-exporter resource
aws_security_group_rule.msk-plain resource
aws_security_group_rule.msk-tls resource
aws_security_group_rule.node_exporter resource
aws_security_group_rule.zookeeper-plain resource
aws_security_group_rule.zookeeper-tls resource
random_id.configuration resource
aws_subnet.this data source

Inputs

Name Description Type Default Required
client_authentication_sasl_iam_enabled Enables IAM client authentication. bool false no
client_authentication_sasl_scram_secrets_arns Associates SCRAM secrets stored in the Secrets Manager. You need secret policy. list(string) [] no
client_authentication_tls_certificate_authority_arns List of ACM Certificate Authority Amazon Resource Names (ARNs). list(string) [] no
client_authentication_unauthenticated_enabled Enables unauthenticated access. bool false no
client_subnets A list of subnets to connect to in client VPC list(string) n/a yes
cloudwatch_logs_group Name of the Cloudwatch Log Group to deliver logs to. string "" no
cluster_name Name of the MSK cluster. string n/a yes
encryption_at_rest_kms_key_arn You may specify a KMS key short ID or ARN (it will always output an ARN) to use for encrypting your data at rest. If no key is specified, an AWS managed KMS ('aws/msk' managed service) key will be used for encrypting the data at rest. string "" no
encryption_in_transit_client_broker Encryption setting for data in transit between clients and brokers. Valid values: TLS, TLS_PLAINTEXT, and PLAINTEXT. Default value is TLS_PLAINTEXT. string "TLS_PLAINTEXT" no
encryption_in_transit_in_cluster Whether data communication among broker nodes is encrypted. Default value: true. bool true no
enhanced_monitoring Specify the desired enhanced MSK CloudWatch monitoring level to one of three monitoring levels: DEFAULT, PER_BROKER, PER_TOPIC_PER_BROKER or PER_TOPIC_PER_PARTITION. See Monitoring Amazon MSK with Amazon CloudWatch. string "DEFAULT" no
extra_security_groups A list of extra security groups to associate with the elastic network interfaces to control who can communicate with the cluster. list(string) [] no
firehose_logs_delivery_stream Name of the Kinesis Data Firehose delivery stream to deliver logs to. string "" no
instance_type Specify the instance type to use for the kafka brokers. e.g. kafka.m5.large. string n/a yes
kafka_version Specify the desired Kafka software version. string n/a yes
number_of_nodes The desired total number of broker nodes in the kafka cluster. It must be a multiple of the number of specified client subnets. number n/a yes
prometheus_jmx_exporter Indicates whether you want to enable or disable the JMX Exporter. bool false no
prometheus_node_exporter Indicates whether you want to enable or disable the Node Exporter. bool false no
provisioned_volume_throughput Throughput value of the EBS volumes for the data drive on each kafka broker node in MiB per second. The minimum value is 250. The maximum value varies between broker type. See [https://docs.aws.amazon.com/msk/latest/developerguide/msk-provision-throughput.html#throughput-bottlenecks](documentation on throughput bottlenecks). number null no
s3_logs_bucket Name of the S3 bucket to deliver logs to. string "" no
s3_logs_prefix Prefix to append to the folder name. string "" no
server_properties A map of the contents of the server.properties file. Supported properties are documented in the MSK Developer Guide. map(string) {} no
tags A mapping of tags to assign to the resource. map(string) {} no
volume_size The size in GiB of the EBS volume for the data drive on each broker node. number 1000 no

Outputs

Name Description
arn Amazon Resource Name (ARN) of the MSK cluster.
bootstrap_brokers A comma separated list of one or more hostname:port pairs of kafka brokers suitable to boostrap connectivity to the kafka cluster. Only contains value if client_broker encryption in transit is set o PLAINTEXT or TLS_PLAINTEXT.
bootstrap_brokers_sasl_iam A comma separated list of one or more DNS names (or IPs) and TLS port pairs kafka brokers suitable to boostrap connectivity to the kafka cluster. Only contains value if client_broker encryption in transit is set to SASL IAM.
bootstrap_brokers_sasl_scram A comma separated list of one or more DNS names (or IPs) and TLS port pairs kafka brokers suitable to boostrap connectivity to the kafka cluster. Only contains value if client_broker encryption in transit is set to SASL SCRAM.
bootstrap_brokers_tls A comma separated list of one or more DNS names (or IPs) and TLS port pairs kafka brokers suitable to boostrap connectivity to the kafka cluster. Only contains value if client_broker encryption in transit is set to TLS_PLAINTEXT or TLS.
cluster_name Msk cluster name
current_version Current version of the MSK Cluster used for updates, e.g. K13V1IB3VIYZZH
default_security_group Msk cluster default security group id.
encryption_at_rest_kms_key_arn The ARN of the KMS key used for encryption at rest of the broker data volumes.
extra_security_groups Msk cluster extra security group ids.
zookeeper_connect_string A comma separated list of one or more hostname:port pairs to use to connect to the Apache Zookeeper cluster.

Authors

Module managed by Angel Abad

License

Apache 2 Licensed. See LICENSE for full details