-
Notifications
You must be signed in to change notification settings - Fork 318
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[EKS] [request]: Manage IAM identity cluster access with EKS API #185
Comments
@ayosec have you created something to automate this as of now? I'm running into this when setting up a cluster using CloudFormation. Do you mind sharing your current approach? |
Unfortunately, no. I haven't found a reliable way to do it 100% automatic.
My current approach is to generate the ConfigMap using a template:
|
Adding this feature on CloudFormation would allow the same feature to be added on AWS CDK. This will greatly simplify the process of adding/removing new nodes, for example. |
I also thought about this. An api to manage the config map for aws-iam-authenticator is interesting, i think would be a bit clunky. I am using terraform to create an eks cluster, and this approach is alot nicer |
I'd love this |
Anybody from AWS care to comment on this feature request? |
With the release of Managed Nodes with CloudFormation support, EKS now automatically handles updating aws-auth config map for joining nodes to a cluster. Does this satisfy the initial use case here, or is there a separate ask to manage adding users to the aws-auth config map via CloudFormation? |
@mikestef9 I think that #554 can be one of the similar issues why you would like this kind of options |
My main use case is with Lambda functions. The managed nodes feature is pretty cool, and very useful for new EKS clusters, but most of our modifications to the |
@mikestef9 It would be useful to then allow people / roles to be able to then run kubectl commands. Right now, we have a CI deploy role - but we want to allow other saml based users to be able to kubectl We do a post cluster creation to kubectl
But I'd much rather have this configMAp created by me during the cluster creation |
@mikestef9 Some relevant issues related to EKS users debugging authentication problems (kubernetes-sigs/aws-iam-authenticator#174 and kubernetes-sigs/aws-iam-authenticator#275) that imo are data points in favor of API and Cloudformation management of auth mappings (and configurable admin role: #554). |
This ^^, how can we get this implemented? Can anyone from AWS tell us if they gonna support this at CF template level? Or a workaround is needed at eksctl level? |
@nemo83 AWS team tagged |
I'm also looking into automating the updates to this configmap from cloudformation. Doing so via lambda seems doable. My main concern with automation are race-conditions on the contents of the configmap when applying updates as the content has to be parsed. A strategic merge is not possible. If the configuration would be implemented in one or more (one per entry) CRD's it would be easier to apply a patch. In that case existing efforts on Kubernetes support for CloudFormation like kubernetes-resources-provider can be reused. Update: we gave up on writing a lambda to update the configmap. The code became too complex and fragile. We now template it separately. Update 2: I had a concern for automatically updating the configmap if it would become corrupt and thereby prevent API access. With the current workings of AWS (1 sept 2020) there is a way of recovering from an aws-auth configmap corruption: aws-auth configmap recovery (tested 1 sept 2020)The prerequisite is to have a pod in the cluster running with a serviceaccount that can update the aws-auth configmap. Ideally something that you can interact with, like k8s-dashboard or in our case ArgoCD. Then if the aws-auth become corrupt you can hopefully still update the configmap that way. If that is not the case because the nodes have lost their access we can use the EKS-managed Node Group to restore node access to the Kubernetes API. You can create an EKS-managed Node Group of just 1 node with the role that is also used by your cluster nodes. (Note: this is not recommended by AWS, but we abuse AWS's power to update the configmap on the managed master nodes.) AWS will now add this role to the aws-auth configmap: apiVersion: v1
kind: ConfigMap
metadata:
annotations:
name: aws-auth
namespace: kube-system
data:
mapRoles: |
...
# this entry is added by AWS automically
- rolearn: < NodeInstanceRole ARN >
username: system:node:{{EC2PrivateDNSName}}
groups:
- system:bootstrappers
- system:nodes Deletion of that Node Group will remove that entry (for which AWS warns you), so the serviceaccount access is required to ensure another method of cluster access, like via the kubectl CLI. Update to aws-auth configmap to get that method of access. Then the Node Group can be removed, which in turn removes the added aws-auth configmap entry that was automatically created earlier. Now the persistent connection (e.g. kubectl CLI) can be used to permanently fix the configmap to ensure the nodes have access. ⚠ Note: if a service is automatically but incorrectly updating the configmap it would be harder, of not impossible to recover. ⚠ |
I would go a extra mile and ask AWS to create an API to manage aws-auth, with IAM action associated in case I delete the IAM role/user associated with the cluster creation ( detail: this user/role is not visible after, you have to save this info outside the cluster, or tag the cluster with it. ) and if I dont add another admin to the cluster, I am now locked out of the cluster, for me, this is a major issue, because I use federated auth, users ( and my day-to-day account ) are efemeral... my user can be recreated without warning with another name/ID, the ideia is: can AWS add an IAM action like ESHttpGet/ESHttpPost ? ( example from ElasticSearch, because is a third party software ) |
Hey @hellupline We are actually working on exactly that right now, an EKS API to manage IAM users and their permissions to an EKS cluster. This will allow you to manage IAM users via IaC tools like CloudFormation |
@mikestef9 how it's going to be different compared to https://github.com/aws-quickstart/quickstart-amazon-eks/blob/main/templates/amazon-eks-controlplane.template.yaml#L109 |
I wonder, why this isn't possible with eks clusters (but with selfhosted k8s clusters on AWS?) https://github.com/kubernetes-sigs/aws-iam-authenticator#crd-alpha Even looking at the cdk implementation of auth mapping, it would be simple to get rid of some limitations that exist right now (stack barrier, imported clusters ...) So if something like CF Support for Auth-Mapping will be implemented (i support this) it would be good, if it won't conflict with the crd's I hope coming soon to eks. |
Any news on this issue? |
Yes! |
great job team !!! |
EKS now supports a simplified method for managing IAM authentication and authorization. |
More details on the featureEnable Access Management APIs EKS access entry APIs require you to opt in before use. This can be done during cluster creation or updated on existing clusters. You must first ensure your cluster is on a supported platform version. Supported platform versions
Existing cluster
EKS supports the following three values for authenticationMode:
In a to be determined future Kubernetes version, EKS will stop supporting the aws-auth ConfigMap as an authentication source, and you will be blocked from upgrading if you haven't switched your cluster authenticationMode to opt in to access entry APIs. Users are strongly encouraged to migrate to access entry APIs as soon as possible. Note: The default selected option at launch for new clusters in the EKS Console is Switching authentication modes on an existing cluster is a one-way operation. You can switch from
Managing/Modifying Cluster AdminControl over which identity has cluster admin permissions when creating a cluster. The following command will create a cluster without any IAM identity having Kubernetes cluster admin access to the cluster.
Important : bootstrapClusterCreatorAdminPermissions can only be set to False if the cluster authenticationMode is set to either After cluster creation (or during, as you can create access entries while a cluster is in CREATING state), you can create a standard access entry with whatever desired IAM identity as a cluster admin. For existing clusters with authenticationMode set to
GitHub issues resolved:
If you don't set the admin bootstrap parameter (or explicitly set it to True), EKS will automatically create an access entry with Kubernetes cluster administrator permissions on your behalf. Note: The value that you set for bootstrapClusterCreatorAdminPermissions on cluster creation is not returned in the response of subsequent EKS DescribeCluster API calls. This is because the value of that field post cluster creation may not be accurate. Further changes to access control post cluster creation will always be performed with access entry APIs. The ListAccessEntries API is the source of truth for cluster access post cluster creation. Recover cluster admin access Previously if you made a typo in the aws-auth ConfigMap you could lock yourself out of the cluster. Or you could lose access to cluster if you accidentally deleted cluster creator IAM identity and didn't remember the IAM identity to re-create it. Now, as long as an IAM identity has permission to create access entries, you can always recover Kubernetes API access to your cluster.
GitHub issues resolved:
View Kubernetes resources from EKS ConsoleA common challenge is EKS Console users (who didn't create the cluster and have Kubernetes cluster admin permissions by default) struggling to figure out why they can't view Kubernetes resources in the Console. Previously, a separate IAM principal with cluster access would have to add an additional aws-auth ConfigMap entry for that Console user's IAM principal. Now, a pop up will appear for IAM principals using the EKS Console that takes them to the create access entry screen with their IAM principal pre-popluated. As the long as the Console user has IAM permissions to call EKS access entry APIs, they can give themselves Kubernetes permissions from the Console itself without needing to create aws-auth ConfigMap entries anymore. Note: The Console user's IAM identity still first requires the permission GitHub issues resolved:
AuthenticationThe following examples show to authenticate IAM principals to an EKS cluster, as the preferred alternative to the aws-auth ConfigMap. If no EKS access policies are attached to the principal (as shown in authorization examples below), no Kubernetes permissions are granted, but the principal will be able to successfully authenticate to the cluster, and more fine-grained authorization can be implemented using RBAC. Username is not a required input when creating an access entry. If not specified, the username value passed to Kubernetes for authentication decisions will map to the value returned from the IAM STS get-caller-identity API. For IAM users, this is the original IAM user ARN. For IAM roles, this is the assumed role ARN with session name, of the format We recommend leaving custom username blank, as the generated username includes the session name, which will appear in CloudTrail logs as an added security benefit. If using a custom username with IAM roles, it is strongly encouraged to set the Note: IAM principals referenced in an access entry must already exist in IAM. Authenticate an IAM user to a cluster
Note: The Kubernetes username used for further authorization RBAC in this case will be the same as the input principal ARN Note: If you change your IAM username, that user will no longer have access to the cluster. You will need to recreate an access entry with the updated IAM user ARN. Note: IAM best practices recommends using roles with temporary credentials, rather than users with long-term credentials. Authenticate an IAM role to a cluster and specify Kubernetes groups
Note: The Kubernetes username generated in this case will be Authenticate an IAM role to a cluster with a custom username
Note: No groups are specified here, because the static username of my-username can be referenced in Kubernetes RBAC policies. However, this is not recommended, as the assumed role session name will be lost from Kubernetes audit logs. Authenticate an IAM role to a cluster with a custom username using macros
Note: The macros will be expanded when username reaches Kubernetes for further RBAC authorization decision. Note: As illustrated in this example, when using the Update the username and/or groups of an existing access entry
Note: Be careful performing this operation if you already have existing RBAC resources that reference previous username/groups. RBAC will also have to be updated to support the new username/groups. Note: Groups are declarative. Enter all desired groups with the update call. Creating an access entry with a role ARN containing a path Authentication decision for EKS access entries are matched against the IAM principal ID of an IAM identity, rather than a string match against input value of the IAM identity and returned assumed role response from sts get-caller-identity (as is the case with entries in the aws-auth ConfigMap). When creating an EKS access entry, enter the fully qualified IAM role ARN like below (do not strip off the path like you need to do in the aws-auth ConfigMap today)
Note: The username in returned access entry will not include the path: Authenticate an IAM principal from a separate account From account 012345678910 run the following
An identity assuming the myOtherAccountRole role in account 567890123456 is now able to authenticate to the my-cluster cluster in account 012345678910. Eventual Consistency Similar to AWS IAM behavior, EKS access entries are eventually consistent, and may take several seconds to be effective after the initial API call returns successfully. You must design your applications to account for these potential delays. We recommend that you do not include access entry create/updates in the critical, high-availability code paths of your application. Instead, make changes in a separate initialization or setup routine that you run less frequently. Also, be sure to verify that the changes have been propagated before production workflows depend on them. GitHub issues resolved:
AuthorizationThe Kubernetes API server supports an ordered list of authorization modules, allowing modules to approve, deny, or pass on a particular action. EKS clusters now include an additional authorizer separate from RBAC, and you can optionally attach EKS defined access policies to give Kubernetes authorization permissions to the IAM principal authenticated with an access entry. Authorization can be done on a namespace or cluster scope. You can specify up to 25 namespace selectors, with optional wildcard suffix matching, to restrict that principals access to specific namespaces. In EKS clusters, RBAC policies are currently evaluated prior to EKS access policies, so any Kubernetes Role Based Access Control (RBAC) ClusterRoleBinding or RoleBinding that specifies an IAM principal will be evaluated for an allow authorization decision before EKS Access Policies. This may change in the future, although in practice the order today does not matter, because neither RBAC nor EKS access policies support deny actions. EKS predefines several managed access policies that mirror the default Kubernetes user facing roles, including cluster-admin, admin, edit, and, view. To view the list of all policies you can run the following command
The examples below assume you have already created the access entry to authenticate the IAM principal to the cluster. Give read only permissions for an IAM principal to an entire cluster
Give editor permissions for an IAM principal to specific namespaces
Give admin permissions to an IAM identity to access multiple namespaces with a wildcard selector
Update list of allowed namespaces for an existing access entry
Node Authentication and AuthorizationAccess entries have an optional input type field for use with authorizing worker node roles to join a cluster. The following types are supported by the access entry API:
Note: aws-auth ConfigMap entries not associated with MNG or Fargate are not automatically migrated when switching authentication modes. These entries must be migrated manually. Note: In practice, you will never need to manually use the The examples below would be required if using self-managed node groups or Karpenter.
Note: It is required to use separate IAM roles for the Note: You cannot attach EKS access policies to access entries with type of Note: Cross account entries are not permitted for node type access entries. Note: The IAM identity creating node type access entries of either GitHub issues resolved:
Migration fully off the aws-auth ConfigMapThe aws-auth ConfigMap can still be used until the to be announced future Kubernetes version where it is no longer supported. But migration to access entry API is strongly encouraged. Migrate an entry in the aws-auth ConfigMap to access entry API For an existing cluster, change your cluster authenticationMode from AWS Service IntegrationsThe access entries feature simplifies how other AWS services can obtain Kubernetes permissions needed to perform actions against EKS clusters. In the future, AWS services with EKS integrations (including EMR, Batch, Resilience Hub) will update their Service Linked Roles to contain permissions to create access entries. When you take an action in that service, the service SLR will automatically create an access entry and associate a specific EKS access policy for that service. There is no need for users to manually create access entries and associated permissions. For example, when Resilience Hub migrates to access entry, all of the steps currently outlined in this doc page will no longer be required. You can think of these "service-linked access entries" as the Kubernetes permission equivalent of AWS service-linked-roles. Example policy a future service might add to its SLR to give itself permissions to create a specific access entry with only specific associated access policies.
You can use this example as a reference to write your own IAM policies that scope down what type of access entries IAM principals can create. IAM policy control for access entries EKS supports the following condition keys associated with this feature
DeletionDeleting an access entry will remove any associated access policies. There is no need to first disassociate access policies.
Deleting a cluster will also automatically delete any access entries currently associated with that cluster. SSO/AWS IAM Identity CenterThere is no direct change to improve this user experience. You still need to add each auto generated role as a separate access entry. But automating that process on your own should be become easier since everything is now an AWS API. We plan to improve this UX in the future by allowing users to directly associate IAM Identity Center permission sets to EKS clusters. Follow this GitHub issue for details: EKS authentication rolearn wildcard support #474. |
Hello, thanks for your feedback ! I attempted to reproduce the issue in the following way but I wasn't able to reproduce Could you describe more on the CFN template before/after the stack update so I can try reproducing ? |
Hello, I'll try to reproduce over here again (this is my cluster definition for now):
.. also I looked through Cloudtrail logs and there are
|
Thanks for your reply! It turns out there is a code bug in CFN update, we will be working on the fix as soon as possible. |
Hello, the fix to above issue was shipped, thanks again for the feedback. |
@mikestef9 Do both the EKS User Guide and EKS Best Practices Guide need updating now that the Cluster Access Manager API has been added and is the preferred way to manage access of AWS IAM principals to Amazon EKS clusters? |
@mikestef9 Thank you very much for the detailed documentation on this set of features.
However, I cannot find a way to retrieve the list of Kubernetes groups associated with a principal via the CLI. I would expect this to be either Update🤦 Thank you @bryantbiggs for telling me the command I want is there, I just didn't see it:
|
|
Tell us about your request
CloudFormation resources to register IAM roles in the
aws-auth
ConfigMap.Which service(s) is this request for?
EKS
Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?
A Kubernetes cluster managed by EKS is able to authenticate users with IAM roles. This is very useful to grant access to Lambda functions. However, as described in the documentation, every IAM role has to be registered manually in a ConfigMap with the name
aws-auth
.For every IAM role we add to the CloudFormation stack, we have to add an entry like this:
This process is a bit tedious, and it is hard to automate.
It will be much better if those IAM roles can be registered directly in the CloudFormation template. For example, with something like this:
Thus, CloudFormation will add and remove entries in the ConfigMap as necessary, with no extra manual steps.
Another
AWS::EKS::MapUsers::Entry
can be used to register IAM users inmapUsers
.With this addition, we can automate the extra step to register the IAM role of the worker nodes when a new EKS instance is created:
The text was updated successfully, but these errors were encountered: