feat(iam-role-for-service-accounts-eks): Add support for EKS pod identity service

[pdrastil]

Description

As in title

Motivation and Context

Allow to use EKS IAM roles for add-ons with new pod identity service that allows to run workloads without service account annotations.

Breaking Changes

N/A

How Has This Been Tested?

• I have updated at least one of the examples/* to demonstrate and validate my change(s)
• I have tested and validated these changes using one or more of the provided examples/* projects
• I have executed pre-commit run -a on my pull

[bryantbiggs]

We can add the service trust, but this module will be the new home for pod identity (it will be move into this org once it's ready)

https://github.com/clowdhaus/terraform-aws-eks-pod-identity

[bryantbiggs]

Lets hold off on this for now - the addons need to support Pod Identity before the roles generated here can be used

[pdrastil]

Hi thanks for the info and I'll wait for a new module. I agree that add-ons will require some time to support pod identity and IRSA will be still needed and this was attempt to unblock migration of already working pieces.

[github-actions]

This PR has been automatically marked as stale because it has been open 30 days
with no activity. Remove stale label or comment or this PR will be closed in 10 days

[danielloader]

Lets hold off on this for now - the addons need to support Pod Identity before the roles generated here can be used

Could you explain what you mean by this? I am looking forward to the feature so as to deprecate and remove all my IRSA roles.

Edit: Had a thought that it might be the addons aren't using an SDK library new enough to use the new authentication method.

[pdrastil]

@danielloader - I think you are right about AWS SDK. At the time this PR was opened most of the commonly used AWS add-ons weren't working properly with pod identity agent (ALB, EFS, EBS, VPC, etc.).

[github-actions]

This PR has been automatically marked as stale because it has been open 30 days
with no activity. Remove stale label or comment or this PR will be closed in 10 days

[truongnht]

@pdrastil any idea on the current status of Pod Identity Agent and AWS addons and where do you track this info?

[pdrastil]

@truongnht Hi - no idea about this PR / support in TF module. That's probably question for @bryantbiggs who wanted to create new module for this. For plugins you can check in release notes. For example latest AWS LB controller added support for pod identities and it's documented in release notes for 2.7.0 (https://github.com/kubernetes-sigs/aws-load-balancer-controller/releases)

[truongnht]

@bryantbiggs can you share some updates on your module, e.g when is it expected to be available to consume 🙏🏼?

[github-actions]

This PR has been automatically marked as stale because it has been open 30 days
with no activity. Remove stale label or comment or this PR will be closed in 10 days

[pdrastil]

@bryantbiggs Hi any update on IAM support as the EKS 20.x module supports identity providers as input.

[truongnht]

@pdrastil, not sure why we have no response from @bryantbiggs. In our team, we have decided to fork the repo to consume internally

[bryantbiggs]

Been a bit busy, we'll get that released this week

[bryantbiggs]

closing now that this is available https://github.com/terraform-aws-modules/terraform-aws-eks-pod-identity

[ivan-sukhomlyn]

@bryantbiggs thanks 🚀
could you please clarify if the terraform-aws-eks-pod-identity module will include aws_eks_pod_identity_association Terraform resource management for IAM role association with appropriate k8s SA in the future? Or will it be managed outside the module, on the user side?

[bryantbiggs]

terraform-aws-modules/terraform-aws-eks-pod-identity#4

I still don't know where the best place to put this association is - I suspect it will end up in multiple places based on the different ways folks may want to manage this, and I guess thats ok

[ivan-sukhomlyn]

Thank you! 🥇
In my opinion, this module is the best place for association management, considering the module name eks-pod-identity (not a terraform-aws-eks-pod-identity-iam, etc.), which self-speaks about complete management, not only an IAM role. Furthermore, users can choose the most convenient way based on their requirements.

[truongnht]

@pdrastil, not sure why we have no response from @bryantbiggs. In our team, we have decided to fork the repo to consume internally

so, when we kinda fork this repo, we also make eks-pod-identity taking care of

1. Create pod identity role with correct assume-role policy
2. Attach whatever necessary policy from calling consumer (e.g cert-manager will provide necessary documents to attach to the role).
3. Also add aws_eks_pod_identity_association

We believe this way would make the module generic enough to be consumed by whoever needs the role. We believe the responsibility of adding correct policy should not be within the module.

[bryantbiggs]

We believe the responsibility of adding correct policy should not be within the module.

Why not? How many different policy variants can one have for external-dns, cluster-autoscaler, etc.? These commonly used applications need IAM permissions, why not make that easy for users. The new Pod Identity module is flexible enough to where you can extend these permissions, or even provide your own permissions (i.e. - not use the pre-canned policies we provide)

[truongnht]

who knows what's gonna change for external-dns and cluster-autoscaler and others? When such a change happens, where do you expect to update? Imho tt's about where the change should happen and should be updated.

[pdrastil]

@bryantbiggs In my personal opinion I think that Pod Identity association should be optional / decoupled from IAM role. The main use case I see for new pod identity is to consolidate large amount of same IAM roles for core add-ons used by fleet of clusters that live in same AWS account to single IAM role. If you want to do 1:1 mapping between role and cluster it probably doesn't matter if you use IRSA or Pod Identity resource.

[danielloader]

Have to respectfully disagree. If I can use pod identity associations for once and done things like the AWS ALB controller, cert manager etc I don't need to play the chicken and egg situation of trying to hand down namespaces, service accounts with annotations into a cluster at terraform time before helm deploys they charts.

Personally I'll almost exclusively be using pod identity associations in my cluster for cluster admin level services and charts not workloads from developers.

It's deeply frustrating trying to fully gitops your system with ArgoCD if you play such games with owning your namespaces and service accounts in terraform outside the state inside kubernetes.

[pdrastil]

@danielloader I do full GitOps with Argo CD and faced the same frustration with context injection :). The best solution I've seen so far was presented on last ArgoCon that uses AppSets to bootstrap the cluster and context is injected to cluster secret as annotations that contain SA names, IAM roles, toggles, etc. Check https://github.com/gitops-bridge-dev with samples. Second option you can do is to create custom plugin that loads this metadata from somewhere (K8s resource or external system), however it might be painful to maintain this in longterm.

[danielloader]

Yeah it's totally possible and yeah I'm using that pattern, also on the same topic there's an issue I'm debating about it on the repository in question - gitops-bridge-dev/gitops-bridge#42.

I just fundamentally don't like IRSA much, the tight coupling makes things quite painful, even if it's possible (which it is since I'm running it in prod) I'd be happier without it.

That said this isn't a hill I'd die on, and worst case scenario someone can maintain a small module of well known IAM policies for common kubernetes addons, and plug this all together painlessly.

[pdrastil]

To clarify my statement about associations being optional - it was more related to TF implementation where some users might want to have 1:1 mapping similar to IRSA withaws_eks_pod_identity_association built into the TF module and some users might want to create IAM without specific EKS cluster for 1:N mapping and create association later on. I agree that using Pod Identity for GitOps environments makes integrations easier without necessary context passing.

[github-actions]

I'm going to lock this pull request because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.