[EKS] [Requesting Feedback] Support for Deploying to Kubernetes from CloudFormation

[christopherhein]

We're investigating adding support for deploying to Kubernetes from AWS CloudFormation and would like to to get feedback on if this is something that would be interesting.

Tell us about your request
This would be an AWS CloudFormation Resource Type[s] for deploying Kubernetes resources, the idea would be a generic type that would allow you to deploy any number of Kubernetes resources from CFN.

Which service(s) is this request for?
EKS and Kubernetes

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?
Right now to deploy to a Kubernetes cluster you must configure your CLI or a CI/CD pipeline to use kubectl and apply manifests when something changes. This would bring AWS and Kubernetes closer allowing you to create a single CFN stack that can deploy AWS resources and Kubernetes resources at the same time. Allowing you to use CFN Ref and Fn::* to add dynamic data into your Kubernetes resources.

Additional context
We're requesting your feedback.

[luiseduardocolon]

From the CloudFormation team: we're very interested in getting ideas from customers on this - there are multiple ways for us to attack this - what resource types to add? what existing configs can we parse and reuse? what other changes should be create abstractions for? We're all ears :)

[jaksonwkr]

Apply aws-auth-cm.yml direct on cloudformation template could be a great feature

[yyolk]

Yes I would use this.

[inductor]

This sounds really good!

[pawelprazak]

The other way around makes sense. This, don't. It's just vendor lock-in.

[christopherhein]

@pawelprazak while CFN is AWS specific you could replace this with any type of pipelining tool. In essence this could be a fast way to have a remote process run sed and inject params then kubectl apply on those files.

I 100% see where you are coming from but it doesn’t really lock you in, merely makes it a little easier.

From the other side, definitely. I’m always in support of the direction I’m taking the AWS Service Operator.

[yandy-r]

being able to pull secrets from secrets manager and parameters from SSM at cluster creation/update would be great. Being able to deploy cluster specific pods a cluster creation, say istio or service discovery, cluster autoscaler, making sure those are running, then deploy application manifests.\

Also @pawelprazak "lock-in" is the most overused, overrated term in technology, and there are a lot of them. If there's no value, there's nothing to be locked into, find the value and lock-in vendor doesn't matter. Provide what the business needs, solve the problem, don't create another. Just another opinion, we all have one 😁

[inductor]

In a way, using AWS managed services along with CloudFormation is vendor lock-in already... 😅

Not saying it's bad, nor good.

[videocursoscloud]

It would be useful to deploy everything using cloudformation so we can deploy, using the same template, the cluster, the control plane and some kubernetes extras - like external-dns and alb ingress controler.

In order to achieve that the beta annotations should be supported (currently they are not on terraform) in order to deploy the services with the annotations needed by alb ingress controller and external dns :)

[jaksonwkr]

I think it can be archived creating a new object type with the name AWS::EKS::Workload or something like this.

[runningman84]

Once this feature is implemented we get this feature without any additional effort:
#51

[christopherhein]

In order to achieve that the beta annotations should be supported (currently they are not on terraform) in order to deploy the services with the annotations needed by alb ingress controller and external dns :)

@videocursoscloud If the implementation was a generic k8s manifest which had a text type which supported any YAML would this implement what you are talking about? or are you saying that you'd want the ability to add attributes to other parts of the system which might not be in the CFN?

[videocursoscloud]

In order to achieve that the beta annotations should be supported (currently they are not on terraform) in order to deploy the services with the annotations needed by alb ingress controller and external dns :)

@videocursoscloud If the implementation was a generic k8s manifest which had a text type which supported any YAML would this implement what you are talking about? or are you saying that you'd want the ability to add attributes to other parts of the system which might not be in the CFN?

Attributes would be nice if they support ANY kind of annotation, not a restricted list of them. If that´s not possible, embedding the whole k8s manifest should do the trick

[christopherhein]

@videocursoscloud great! This is really helpful to make sure the feature will support everyones use cases.

[mumoshu]

@christopherhein This is so nice as-is!

I would use it in combination with CFN wait handles, so that I can turn CloudFormation into a simple workflow engine capable of provisioning your whole infrastructure composed of EKS control-plane, nodegroups, and now any Kubernetes resources.

[ghost]

Why you are not considering to support external kubernetes deployment files as https://github.com/reactiveops/reckoner ?
specific filenexample is here : https://github.com/reactiveops/reckoner/blob/master/reckoner/example-course.yml

in addition it will be very nice if AWS will provide reckoner file for AWS related charts ( alb ingress , external dns , aws operator , node autoscalar etc)

[christopherhein]

@yanivpaz1973 thanks for sharing that, I haven't had experience with reckoner. Is this something your organization uses?

Are other SIG projects like the alb ingress, external dns, cluster autoscaler starting to use this? I know we're going in the direction of kustomize as a way of handling multiple environment configurations.

[ghost]

@christopherhein yes I used reckoner to simplify multi charts deployment. its simple and convenient and should be part of AWS EKS BOM ( ie - EKS version X was tested with ALB chart version Y , Autoscaler verision Z etc) . I dont think that its in SIGs projects scope , but more ops oriented job that need to mix and match many many helm charts on top of EKS deployment .

[whereisaaron]

1. I'd prefer something that is roughly 1:1 with something k8s native that works outside CF, so k8s resources can be 'bundled' into a CF stack rather than translated to a vendor-specific syntax. E.g. plain/raw k8s resources or the 'kustomize' overlay/patch model would suit S3 storage (or ideally CodeCommit storage). In CF resources you could be able to create kustomize edit equivalents or YAML patches to add CF variables, SSM variables, Secret Manager secrets etc. to k8s resources. But the base resources would be k8s YAML and/or kustomize compatible overlays.

2. Enable CF to reference k8s resources from CodeCommit rather than S3 for better versioning and management.

3. An anti-pattern here would be to create CF resource types for k8s resource types. CF's tracking of API changes lags tremendously, sometimes by years, even for AWS APIs (tags on EIPs anyone? :-). It would be a constant frustration for CF to be constantly out of date with current k8s versions.

[BBonifield]

This would be a fantastic addition.

[SanderKnape]

This sounds like a great addition to CloudFormation. However, given the many ways it's already possible to get resources into k8s, to me this is one of those things that would never get high enough on the priority list to actually research. There are many feature requests open for both EKS and CloudFormation that deserve more priority in my opinion. Just to give some examples:

• Fargate for EKS
• IAM permissions for pods in EKS
• Full AWS coverage for CloudFormation
• Ability to import existing resources in CloudFormation

[jaymccon]

The Amazon EKS Quick Start provides some support for managing Kubernetes resources using CloudFormation Custom Resources that map (roughly) 1:1 to k8s manifests. Also provides a tillerless helm resource type.

AWSTemplateFormatVersion: "2010-09-09"
Resources:
  KubeManifestExample:
    Type: "Custom::KubeManifest"
    Properties:
      ....
      Manifest:
        apiVersion: v1
        kind: ConfigMap
        metadata:
          namespace: kube-system
        data:
          ConfigItem: "MyValue"
  HelmExample:
    Type: "Custom::Helm"
    Properties:
      ....
      RepoUrl: https://svc-catalog-charts.storage.googleapis.com
      Chart: svc-cat/catalog
      Values:
        apiserver.healthcheck.enabled: true
        imagePullPolicy: Always

It would be great if this could be done natively without the overhead of having to define and deploy CloudFormation custom resources

[lukeplausin]

Being able to create resources using a generic kubernetes manifest would be really useful. In that case I would combine it with Fn::Sub to template any inputs with cloudformation.

Another use case could be deploying Helm charts directly from Cloudformation. In that case it might be more useful to specify the helm repo URL and the values.yaml as a json object.

Maybe one resource type for each case would fit the bill?

[tabern]

Link to an AWS blog that shows how to do this using CFN and Helm.