Release Proposal v2.20

[floryut]

Deprecation / Removal

• Drop Ansible support for v2.9 and v2.10 (Drop Ansible support for v2.9 and v2.10 #8925, @oomichi)
• Drop support for Fedora 34 (Add Fedora36 support, remove Fedora34 support and update kube-ovn #8967, @floryut)

Feature / Major changes

• Add Rocky Linux 8 support (Add Rocky Linux 8 support for vagrant and CI #8905, @oomichi)
• Add Kylin Linux support. (Add kylin OS support #9078, @ErikJiang)
• Add Fedora36 support (Add Fedora36 support, remove Fedora34 support and update kube-ovn #8967, @floryut)
• Add 'flush ip6tables' task in reset role (Add 'flush ip6tables' task in reset role #9168, @GreatLazyMan)
• Add tar in common required package (add tar in common required package #9184, @yankay)
• Add support for NTP configuration. (Add Feature for NTP management. #9027, @yankay)
• Increase ansible fact_caching_timeout (from 2 to 24 hours) (increase ansible fact_caching_timeout  #9059, @rptaylor)
• Add kubelet systemd service hardening option kubelet_systemd_hardening: [true|false] (feat: add kubelet systemd service hardening option #9194, @alegrey91)
• Support timezone setting (Add timezone support #9263, @yankay)
• Update deprecated ansible include syntax ( update deprecated syntax #9040, @boeto)
• Update etcd download url in offline.yml to use arch (update etcd download url in offline.yml #8943, @ErikJiang)
• Add Support for Rewrite Plugin to CoreDNS/NodelocalDNS (Add Support for Rewrite Plugin to CoreDNS/NodelocalDNS #9245, @eifelmicha)
• Add SeccompDefault admission plugin for kubelet (using new variable kubelet_seccomp_default) (hardening: Add SeccompDefault admission plugin for kubelet #9074, @alegrey91)
• Add an optional extra_groups parameter for k8s_nodes (e.g. to configure calico route reflector nodes on Openstack using the calico_rr group) (add optional parameter extra_groups for k8s_nodes in openstack terraform #9211, @rptaylor)
• Add arm64 Flatcar OS's pypy bootstrapping support (Add arm64 Flatcar OS's pypy bootstrapping support #8959, @kerryeon) (see Notes 1)
• Add docker support for Kylin distributions (Add docker support for kylin distributions #9144, @ErikJiang)
• Add hashes for Kubernetes 1.24.3 , v1.22.12, v1.23.9 ([kubernetes] add hashes for v1.22.12, v1.23.9 & v1.24.3 #9092, @marcofortina)
• Add ingress nginx webhook (add ingress nginx webhook #9033, @liupeng0518)
• Add manage-offline-files.sh to collect necessary files and provides http file download service for offline deployment. (add manage offline files script #8956, @ErikJiang)
• Add missing configuration for extra tolerations ([MASTER] Add missing configuration for extra tolerations #8908, @smasset)
• Add support for node & pod pid limits (in kubelet-config file) (Add support for node & pod pid limits #9038, @h9-HSFRQDH)
• Add the option to enable default Pod Security Configuration (Add the option to enable default Pod Security Configuration #9017, @Foxlik)
• Add unsafe_show_logs switch to show more log details (default to false, same as previous behavior) (Add unsafe_show_logs switch #9164, @ErikJiang)
• Add variables (delete_node_retries,delete_node_delay_seconds) to tweak remove node process (Use a variable for roles of remove-node/post-remove #9096, @ydFu)
• Added 'avoid-buggy-ips' support of MetalLB (metallb_avoid_buggy_ips for default IP address pool and avoid_buggy_ips for additional IP address pools defined in metallb_additional_address_pools) (Add 'avoid-buggy-ips' support of MetalLB #9166, @kerryeon) (see Notes 2)
• Adjust the default value of calico blockSize ipv4 to 26, and ipv6 to 122. (feat: change default blockSize for calico #9055, @cyclinder)
• Make kubernetes owner parametrized (using kube_owner/kube_cert_group/etcd_owner variables) (feat: make kubernetes owner parametrized #8952, @alegrey91)
• Move old etcd backup removal after etcd restart, to prevent removing backup if etcd fail (Move old etcd backup removal after etcd restart #9147, @emiran-orange)
• Supports reserve ephemeral-storage (feat: Support reserve ephemeral-storage #8895, @Thearas)
• [dev/docs] add support for pre-commit hook (add pre-commit hook to facilitate local testing #9158, @cristicalin)
• [etcd] Etcd role won't run on all nodes everytime. ([etcd] Sometimes, we do not need to run etcd role on all nodes. #9173, @liupeng0518)
• [etcd] add 3.5.4 and drop 3.5.1 and 3.5.2 ([etcd] add 3.5.4 and drop 3.5.1 and 3.5.2 #9021, @cristicalin)
• [infra] bump pause container to 3.6 (bump pause container to 3.6 #9024, @cristicalin)
• Update Kubernetes dashboard to 2.6.0 (k8s 1.24 support) (Update dashboard to 2.6.0 (k8s 1.24 support) #8906, @floryut)
• [kubernetes] make 1.24.x the new default ([kubernetes] support 1.24.x #8935, @cristicalin)
• [kubernetes] drop support for 1.21.x ([kubernetes] support 1.24.x #8935, @cristicalin)
• [kubernetes] drop support for deprecated dynamic_kubelet_configuration ([kubernetes] support 1.24.x #8935, @cristicalin)
• [offline] Archive offline-files and env NO_HTTP_SERVER to skip Nginx container running. ([#9067] archive offline-files and NO_HTTP_SERVER to skip nginx running #9068, @yjqg6666)
• Adds support for multiple architectures to yq (Adds support for multiple architectures to yq #9288, @ErmalKristo)
• Add variable to tweak the vsphere-csi namespace (vsphere_csi_namespace) (Add variable for the vsphere-csi namespace #9278, @MahdiAbbasi95)
• Ensure ping package is installed on the system (Ensure the ping package #9284, @yankay)
• Add more functionalty to DNS configuration (feat: allows users to have more control on DNS #9270, @eminaktas)
• Ensure ostree variable has been defined for fcos (Ensure ostree variable has been defined #9321, @electrocucaracha)
• Support removing options in resolvconf with tab separator (support removing options in resolvconf with tab separator #9304, @2k0ri)
• preinstall: Add nodelocaldns to supersede_nameserver if enabled (preinstall: Add nodelocaldns to supersede_nameserver if enabled #9282, @azuwis)

Network

• [Calico] calico rr now supports multiple groups ([calico] calico rr supports multiple groups #9134, @liupeng0518)
• [Calico] drop support for 3.19.x and 3.20.x
• [Calico] Make Calico CNI log path configurable and allow disabling this log (Allow disabling calico CNI logs with calico_cni_log_file_path #8921, @fungusakafungus)
• [Calico] The NAT (nat_outgoing) would not be disabled automatically when enabling peer_with_router. (Remove mutual exclusivity in calico: NAT and router mode #9255, @kerryeon)
• [Calico] The variable calcio_ipam_autoallocateblocks has been renamed to calico_ipam_autoallocateblocks (Fix some typo #9056, @liupeng0518)
• [Calico] calico-typha metrics port are now exposed when metrics are enabled (Expose calico-typha metrics port when metrics are enabled #8855, @vjacynycz)
• [Calico] Add Wireguard support for Rocky Linux 9 (Calico: add wireguard support for Rocky Linux 9 #9287, @krystianmlynek)
• [Calico] The parameter name calcio_rr_id Is renamed to calico_rr_id for fixing a typo ⚠️ (Fix typos in calico #9327, @kerryeon)
• [Canal] update templates to work again with both etcd and k8s datastore (Align canal templates back with official ones #9113, @floryut)
• [Cilium] Add list/watch nodes rules to cilium-operator clusterrole. (add list/watch nodes rules to cilium-operator clusterrole #9178, @Thearas)
• [Cilium] Add support for the updated (startup|liveness|readiness)Probe.Port numbers (Add support for the updated (startup|liveness|readiness)Probe.Port numbers in Cilium #9031, @tomberget)
• [Cilium] Update cilium to v1.11.7 (update cilium to v1.11.7 #9119, @dkhachyan)
• [Cilium] Make rolling-restart readiness wait delay and count configurable via cilium_rolling_restart_wait_retries_{count, delay_seconds} (9035: Make Cilium rolling-restart delay/timeout configurable #9176, @Tristan971)
• [Cilium] Upgrades cilium to 1.11.6 and add some default variables. (feat: upgrade cilium and add default variables #9065, @eminaktas) (See Notes 3)
• [Cilium] Update Cilium default to 1.12.x (Cilium 1.12 Upgrade #9225, @necatican) (See Notes 5)
• [Cilium] Dropped support for < v1.10.0 (Cilium 1.12 Upgrade #9225, @necatican)
• [Cilium] cilium_ip_masq_agent_enable variable no longer exists. Use enable-ipv4-masquerade and enable-ipv4-masquerade to enable masquerade. (Cilium 1.12 Upgrade #9225, @necatican)
• [flannel] update to v1.18.1 & make it default ([flannel] update to v1.18.1 & make it default #9104, @mzaian)
• [flannel] update to v1.19.2 & make it default ([flannel] update to v1.19.2 & make it default #9296, @mzaian)
• [Kube-vip] Fail if kube_proxy_strict_arp is set to false in arp mode (kube-vip shoud fail if kube_proxy_strict_arp is false in arp mod #9223, @yankay)
• [Multus] Support multi-architecture installation (support multus multi-architecture installation #9012, @cyclinder)

Applications

• [Openstack] Add option to use default deny firewall policy and port allowlisting on UpCloud ([upcloud] Add firewall default deny policy and port allowlisting #9058, @Ajarmar)
• [Openstack] Fix subnet order and number of master nodes (Subnet setup order fix & Number of master nodes syntax fix #9159, @robinelastisys)
• [Metallb] Renamed matallb_auto_assign variable to metallb_auto_assign (users disabling 'auto-assign' in metallb must update the variable name) (fix a typo in the "matallb_auto_assign" variable name #8949, @orange-llajeanne)
• [vSphere-csi] Add nodeAffinity to daemonset using vsphere_csi_node_affinity variable (vSphere-csi: add nodeAffinity to daemonset #9293, @dmitrytretyakov)
• [upcload-csi] Bump driver version to v0.3.3 (upcloud csi driver: bump version to v0.3.3 #9317, @robinAwallace)

Container-Managers

• [containerd] add hashes for 1.5.12, 1.5.13, 1.6.5 and 1.6.6, make 1.6.6 the new default ([containerd] upgrade to 1.6.6 and add hashes #8980, @cristicalin)
• [containerd] Add LimitMEMLOCK parameter configuration in containerd.service (using containerd_limit_[proc_num/core/open_file_num/mem_lock) (Add LimitMEMLOCK parameter configuration in containerd.service #9269, @ErikJiang)
• [containerd] Remove duplication in containerd template (Remove duplication in containerd template #9301, @fungusakafungus)
• [containerd] Allow configuring base_runtime_spec per containerd runtime and supply a default runtime spec (Allow configuring base_runtime_spec per containerd runtime and supply a default runtime spec #9302, @fungusakafungus)
• [Docker] use cri-dockerd instead of dockershim by default
• [Docker] Enable cri-dockerd service to prevent issue with reboot (🌱 Enable cri-dockerd service #9201, @mostafaghadimi)
• [cri-o] Add dpkg hold for apt installs (9052 crio add dpkg hold #9075, @SamuelBECK1)
• [cri-o] add support for 1.24.x required by kubernetes 1.24.x ([kubernetes] support 1.24.x #8935, @cristicalin)
• [runc] update versions for 1.1.x and drop 1.0.x ([runc] update versions for 1.1.x and drop 1.0.x #9022, @cristicalin)
• [runc] Variable containerd_default_runtime is now undifined by default (but default to runc) (let containerd_default_runtime be undefined by default #9026, @rptaylor)
• [crun] add 1.4.5 and drop 1.2 and 1.3 ([crun] add 1.4.5 and drop 1.2 and 1.3 #9023, @cristicalin)
• [nerdctl] upgrade to 0.20.0 ([containerd] upgrade to 1.6.6 and add hashes #8980, @cristicalin) then 0.22.2 (Upgrade nerdctl from 0.20.0 to 0.22.2 #9180, @panpan0000)

Bug or Regression

• Fix failure to look up user etcd when adding a user (Fix the issue of the miss the etcd user when etcd_deployment_type: kubeadm #9016, @yankay)
• Fixing setting up kubespray on Azure with CSI drivers. (Fix CSI drivers issues on Azure #9153, @wayfrro)
• Add --supervisor-fss-namespace=kube-system flag to vcloud-csi installation (Fix vcloud-csi bug related to #9046 #9066, @yasintahaerol)
• Add assertion for IPv4 check in verify settings (to allow IPv6 deployments) (Add assertion for IPv4 check in verify settings #8946, @Citrullin)
• Add calico-kube-controllers missing verbs (add calico-kube-controllers missing verbs #9032, @ghostloda)
• Allow "openSUSE Tumbleweed" to be run (again) (Allow "openSUSE Tumbleweed" to be run #9072, @oomichi)
• Apply calico bgp peer definition task to all nodes (Apply calico bgp peer definition task to all nodes #8974, @orange-llajeanne)
• Create snapshot namespace only when needed (Upgrade v2.19 issues #9014, @robinAwallace)
• Disable kubelet_authorization_mode_webhook by default (disable kubelet_authorization_mode_webhook by default #9238, @cristicalin)
• Disabled DNSStubListener for Flatcar Linux (Disable DNSStubListener for Flatcar Linux #9160, @kerryeon)
• Do not run etcd role in scale.yml playbook when etcd installed by kubeadm (do not run etcd role in scale.yml playbook when etcd installed by kub… #9210, @LuckySB)
• Fix Hetzner CCM cluster-cidr (wrongly set to a static value) (Fix Hetzner CCM cluster-cidr #9127, @ym)
• Fix calicoctl.sh path error when getting calico configuration (Fix abort because calicoctl.sh is not a full path #9217, @tasekida)
• Fix failing tasks when calico_datastore is set to etcd (Fixes for calico_datastore: etcd #9228, @chadswen)
• Fix missing quote in task "See if node is schedulable" (Argument jsonpath must be single-quoted in "See if node is schedulable" task #9146, @emiran-orange)
• Fix number node name can't be added. (fix number node name can't be added #9266, @cleverhu)
• Fix regex for replacing http_proxy host in RedHat Subscription Manager (Fix regex for replacing http_proxy host in RedHat Subscription Manager #8957, @dicksontung)
• Fix some docker reset task (don't remove already uninstalled packages, ignore error on remove docker config files if already removed) (Several fixes for docker reset tasks #8966, @orange-llajeanne)
• Fix the Centos/RHEL docker installation issue in ARM64 (Fix docker option in centos arm64 #9047, @yankay)
• Fix the kube-vip missed SAN issue (add kube-vip ip to kube SANS #9099, @yankay)
• Fixed concatenate str & int in auto_renew_certificates_systemd_calendar (Fixed concatenate str & int in auto_renew_certificates_systemd_calendar var #8979, @floryut)
• Fixes the issue when it cannot correctly set the namespace for vphere-csi-driver (fix(vsphere-csi): remove namespace env variable and set namespace as kube-system #9046, @eminaktas)
• Fixes vSphere CSI for vSphere CSI >= 2.4.0 on vSphere 6.7U3 ([vsphere-csi] fix support for vsphere 6.7U3 #8944, @snowball77)
• No more errors are emitted when attempting to delete worker nodes that do not exist. (Ignore deleting worker nodes that are not in cluster #9244, @kerryeon)
• Optimize the format of evictionHard in kubelet-config.yaml template (optimize the format of evictionHard in kubelet-config.yaml template #9204, @shelmingsong)
• Remove kubeowner different than root condition for user creation (fix: remove condition for user creation #9125, @alegrey91)
• Remove unneeded socat wrapper installation for Flatcar (Remove unneeded socat wrapper installation for Flatcar #8970, @kerryeon) (See Notes 4)
• Set fallback value of kubelet ip6 (Fix: set fallback value of kubelet ip6 (#8858) #8926, @kerryeon)
• Swap calico download url, as the old primary url was deprecated and artefact no longer published (use latest calicoctl repo first #8920, @sathieu)
• Upgrade the nginx-proxy and haproxy image version , and use the alpine base image (Update load balancers Versions and use the alpine base image #9100, @yankay)
• Variable kube_pid_reserved must be a string (pid reserved must be str #9124, @liupeng0518)
• [Docker] Add restart of docker.service during install (cri-dockerd: add restart of docker.service #9205, @krystianmlynek)
• [Kube-ovn] Value check for HW_OFFLOAD is now correctly handle (and will no longer always be false) (Fix kube_ovn_hw_offload value #9218, @floryut)
• [ingress-nginx] Fix ingress-nginx RBAC rules when deployed classless ([ingress-nginx] Fix ingress-nginx RBAC rules when deployed classless #9156, @cristicalin)
• Remove the 'etcd-unsupported-arch' args to fix the etcd issue in arm64 (Remove ETCD_UNSUPPORTED_ARCH since etcdmain officially supports "arm64" #9049, @yankay)
• Fix duplicate field in ingress-nginx template (fix duplicate field in ingress-nginx template #9285, @cloud-66)
• Fix CoreDNS memory leak issue by adding max_concurrent=1000 in the CoreDNS config (add max_concurrent in coredns to fix memory leak issue #9307, @yankay)
• Fix ansible user module create_home property (erroneously written as createhome) (Changed createhome to create_home #9314, @liupeng0518)
• Hotfix containerd restart not properly restarting (Hotfix containerd restart #9322, @fungusakafungus)

Other (Cleanup or Flake)

• [CI] upgrade vagrant image for opensuse leap to 15.4 ([CI] fix molecule tests on opensuse by upgrading to 15.4 #9175, @cristicalin)
• [CI] test upgrade with defaults (containerd) instead of docker ([containerd] upgrade to 1.6.6 and add hashes #8980, @cristicalin)
• [CI] Fix cloud_init files for different distros (Fix cloud_init files for different distros #9232, @floryut)
• git ignore .terraform.lock.hcl in all folders (git ignore .terraform.lock.hcl anywhere #9109, @rptaylor)

Component versions:

• Core
  • Kubernetes v1.24.6
  • Etcd v3.5.4
  • Docker v20.10
  • Containerd v1.6.8
  • CRI-O v1.24
• Network
  • CNI-plugins v1.1.1
  • Calico v3.23.3
  • Cilium v1.12.1
  • Flannel v0.19.2
  • Kube-ovn v1.9.7
  • Kube-Router v1.5.1
  • Multus v3.8
  • Weave v2.8.1
  • kube-vip v0.4.2
• App
  • Cert-manager v1.9.1
  • CoreDNS v1.8.6
  • Nginx-ingress v1.3.1
  • krew v0.4.3
  • argocd v2.4.12
  • helm v3.9.4
  • metallb v0.12.1
  • registry v2.8.1

Known issues

• Host network might broke when an interface goes down (Cilium 1.12/Ubuntu 22.04), please read Note 5.
• If bin_dir value is changed to something other than /usr/local/bin, containerd configuration might need to be tweak, please check Custom Binary directoy bin_dir bug #9243

Notes

1. Upgrading the bootstrap pypy may cause some unexpected behaviors for Flatcar use-cases)
2. As the newly added feature uses the default value of MetalLB as same, there is no side effect for users who do not change it's value
3. This PR also implements cgroup auto-mount. By default, it is enabled. You can disable it by adding cgroup_auto_mount: false. Moreover, you can enable or disable BPF with these variables cilium_enable_bpf_masquerade and cilium_enable_host_legacy_routing
4. Some old (<2020Y) 'Flatcar Container Linux by Kinvolk' may not be supported.
5. With Cilium 1.12/Ubuntu 22.04, you might run into this issue, workaround are available while the issue is resolved on cilium end.

[krystianmlynek]

Should systemd-networkd issue that was mentioned in #9225 be listed in known issues? What do you think?

[floryut]

Should systemd-networkd issue that was mentioned in #9225 be listed in known issues? What do you think?

Thank you, you're right, I'll draft something up

[sohnaeo]

Should #9243 be listed in known_issues as well? users need to be warned if bin_dir is different than /usr/local/bin then need to add below into config.toml through containerd_extra_args

[plugins."io.containerd.internal.v1.opt"]
path = "/opt/mypath"

[floryut]

@sohnaeo done, thank you 👍

[fungusakafungus]

Add variable to tweak the vsphere-csi namespace (vsphere_csi_namespace) (#9278, @MahdiAbbasi95)

is more dangerous than it sounds, new var defaults to new value, vmware-system-csi, and there's no code to remove vsphere csi driver from kube-system namespace, so if someone used it before, they'll either need to remove csi driver from kube-system or set vsphere_csi_namespace to kube-system

[oomichi]

Thank you so much for preparing a new release @floryut !
The above release-note seems good for me.

+1

[mzaian]

Hi @floryut
Would be nice If we can have this in the release #9302

[floryut]

Add variable to tweak the vsphere-csi namespace (vsphere_csi_namespace) (#9278, @MahdiAbbasi95)

is more dangerous than it sounds, new var defaults to new value, vmware-system-csi, and there's no code to remove vsphere csi driver from kube-system namespace, so if someone used it before, they'll either need to remove csi driver from kube-system or set vsphere_csi_namespace to kube-system

Created #9312 to address that

[floryut]

Hi @floryut Would be nice If we can have this in the release #9302

Approved, needs a reviewer to check this PR 👍

[yankay]

HI @floryut , I'm sorry for there is a typo:

Fix ETCD memory leak issue by adding max_concurrent=1000 in the CoreDNS config. (#9307, @yankay) =>

Fix CoreDNS memory leak issue by adding max_concurrent=1000 in the CoreDNS config. (#9307, @yankay) =>

Thanks.

[floryut]

HI @floryut , I'm sorry for there is a typo:

Fix ETCD memory leak issue by adding max_concurrent=1000 in the CoreDNS config. (#9307, @yankay) =>

Fix CoreDNS memory leak issue by adding max_concurrent=1000 in the CoreDNS config. (#9307, @yankay) =>

Thanks.

🤦 thank you

[fungusakafungus]

I could reproduce #9019 and am working on a fix. It's mind-boggling.

[rptaylor]

I could reproduce #9019 and am working on a fix. It's mind-boggling.

That looks serious, would be good to have a fix in 2.20 otherwise clusters could still have an old containerd version from 2.18.

[rptaylor]

@floryut what about including #9026 , #9109?

[floryut]

@floryut what about including #9026 , #9109?

Don't forget to fill the Does this PR introduce a user-facing change? this is what generate the release notes (I try to edit them but sometime I can miss a few PR).

Anyway, I've added the PR in the release note, thank you.

[HoKim98]

I hope this calico typo correction commit will also be included in the next release!

• Fix typos in calico #9327

[oomichi]

I hope this calico typo correction commit will also be included in the next release!

• Fix typos in calico #9327

That makes the parameter name change and might be user-facing change.
I wrote a release-note on the pull request.

[floryut]

And done 🥳

[rptaylor]

Looks like upgrading a cluster to 2.20 fails due to the calico block size change: #9055 (comment)