Add support for graceful shutdown options to Provisioner spec

[AlexMaksimkin]

Description

What problem are you trying to solve?
Current Provisioner spec sh.karpenter.v1alpha5.Provisioner.spec.kubeletConfiguration are missing support for graceful shutdown kubelet configuration, in particular:

• shutdownGracePeriod
• shutdownGracePeriodCriticalPods
• shutdownGracePeriodByPodPriority

https://kubernetes.io/docs/reference/config-api/kubelet-config.v1beta1/#kubelet-config-k8s-io-v1beta1-ShutdownGracePeriodByPodPriority

which block any ability to use of graceful termination features with Karpenter.

How important is this feature to you?

Many workloads are require graceful shutdown as well as doing that in particular order.
BottlerocketOS recently added related support: bottlerocket-os/bottlerocket#2930, bottlerocket-os/bottlerocket#3308, but Karpenter couldn't use it without additional support in Provisioner specs.

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[jonathan-innis]

@AlexMaksimkin Can you pass the additional values down to the bootstrapped KubeletConfig by performing overrides similar to what's described in our upstream docs?

We've generally been trying to avoid adding all the support KubeletConfiguration fields into the Provisioner (since there's a lot of fields) and tried to scope the fields to just be fields that are related to scheduling behavior that Karpenter should be aware of without having to introspect the userData.

[bwagner5]

Looks like we do need to update the Bottlerocket settings configuration to support this, but we shouldn't need to add this to the kubelet configuration in the provisioner spec.

[bwagner5]

Added support for graceful shutdown, will be available in the next Karpenter release.