Skip to content

Commit

Permalink
Merge pull request #47469 from kubernetes/dev-1.31
Browse files Browse the repository at this point in the history
Official 1.31 Release Docs
  • Loading branch information
reylejano authored Aug 13, 2024
2 parents ad6351d + b1972bd commit 20d164c
Show file tree
Hide file tree
Showing 108 changed files with 1,841 additions and 519 deletions.
4 changes: 4 additions & 0 deletions content/en/docs/concepts/architecture/leases.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,10 @@ This is used by control plane components like `kube-controller-manager` and `kub
HA configurations, where only one instance of the component should be actively running while the other
instances are on stand-by.

Read [coordinated leader election](/docs/concepts/cluster-administration/coordinated-leader-election)
to learn about how Kubernetes builds on the Lease API to select which component instance
acts as leader.

## API server identity

{{< feature-state feature_gate_name="APIServerIdentity" >}}
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
---
reviewers:
- jpbetz
title: Coordinated Leader Election
content_type: concept
weight: 200
---

<!-- overview -->

{{< feature-state feature_gate_name="CoordinatedLeaderElection" >}}

Kubernetes {{< skew currentVersion >}} includes an alpha feature that allows {{<
glossary_tooltip text="control plane" term_id="control-plane" >}} components to
deterministically select a leader via _coordinated leader election_.
This is useful to satisfy Kubernetes version skew constraints during cluster upgrades.
Currently, the only builtin selection strategy is `OldestEmulationVersion`,
preferring the leader with the lowest emulation version, followed by binary
version, followed by creation timestamp.

## Enabling coordinated leader election

Ensure that `CoordinatedLeaderElection` [feature
gate](/docs/reference/command-line-tools-reference/feature-gates/) is enabled
when you start the {{< glossary_tooltip text="API Server"
term_id="kube-apiserver" >}}: and that the `coordination.k8s.io/v1alpha1` API group is
enabled.

This can be done by setting flags `--feature-gates="CoordinatedLeaderElection=true"` and
`--runtime-config="coordination.k8s.io/v1alpha1=true"`.

## Component configuration
Provided that you have enabled the `CoordinatedLeaderElection` feature gate _and_
have the `coordination.k8s.io/v1alpha1` API group enabled, compatible control plane
components automatically use the LeaseCandidate and Lease APIs to elect a leader
as needed.

For Kubernetes {{< skew currentVersion >}}, two control plane components
(kube-controller-manager and kube-scheduler) automatically use coordinated
leader election when the feature gate and API group are enabled.
Original file line number Diff line number Diff line change
Expand Up @@ -508,7 +508,7 @@ that file but the kubelet does not categorize the space as in use.
{{% /tab %}}
{{% tab name="Filesystem project quota" %}}

{{< feature-state for_k8s_version="v1.15" state="alpha" >}}
{{< feature-state feature_gate_name="LocalStorageCapacityIsolationFSQuotaMonitoring" >}}

Project quotas are an operating-system level feature for managing
storage use on filesystems. With Kubernetes, you can enable project
Expand All @@ -534,13 +534,21 @@ If a file is created and deleted, but has an open file descriptor,
it continues to consume space. Quota tracking records that space accurately
whereas directory scans overlook the storage used by deleted files.

To use quotas to track a pod's resource usage, the pod must be in
a user namespace. Within user namespaces, the kernel restricts changes
to projectIDs on the filesystem, ensuring the reliability of storage
metrics calculated by quotas.

If you want to use project quotas, you should:

* Enable the `LocalStorageCapacityIsolationFSQuotaMonitoring=true`
[feature gate](/docs/reference/command-line-tools-reference/feature-gates/)
using the `featureGates` field in the
[kubelet configuration](/docs/reference/config-api/kubelet-config.v1beta1/)
or the `--feature-gates` command line flag.
[kubelet configuration](/docs/reference/config-api/kubelet-config.v1beta1/).

* Ensure the `UserNamespacesSupport`
[feature gate](/docs/reference/command-line-tools-reference/feature-gates/)
is enabled, and that the kernel, CRI implementation and OCI runtime support user namespaces.

* Ensure that the root filesystem (or optional runtime filesystem)
has project quotas enabled. All XFS filesystems support project quotas.
Expand All @@ -556,6 +564,13 @@ If you want to use project quotas, you should:
mounted with project quotas enabled. For both XFS and ext4fs, the
mount option is named `prjquota`.


If you don't want to use project quotas, you should:

* Disable the `LocalStorageCapacityIsolationFSQuotaMonitoring`
[feature gate](/docs/reference/command-line-tools-reference/feature-gates/)
using the `featureGates` field in the
[kubelet configuration](/docs/reference/config-api/kubelet-config.v1beta1/).
{{% /tab %}}
{{< /tabs >}}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -315,13 +315,10 @@ may also be used with field selectors when included in the `spec.versions[*].sel

{{< feature-state feature_gate_name="CustomResourceFieldSelectors" >}}

You need to enable the `CustomResourceFieldSelectors`
[feature gate](/docs/reference/command-line-tools-reference/feature-gates/) to
use this behavior, which then applies to all CustomResourceDefinitions in your
cluster.

The `spec.versions[*].selectableFields` field of a {{< glossary_tooltip term_id="CustomResourceDefinition" text="CustomResourceDefinition" >}} may be used to
declare which other fields in a custom resource may be used in field selectors.
declare which other fields in a custom resource may be used in field selectors
with the feature of `CustomResourceFieldSelectors`
[feature gate](/docs/reference/command-line-tools-reference/feature-gates/) (This feature gate is enabled by default since Kubernetes v1.31).
The following example adds the `.spec.color` and `.spec.size` fields as
selectable fields.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -160,7 +160,7 @@ The general workflow of a device plugin includes the following steps:
The processing of the fully-qualified CDI device names by the Device Manager requires
that the `DevicePluginCDIDevices` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/)
is enabled for both the kubelet and the kube-apiserver. This was added as an alpha feature in Kubernetes
v1.28 and graduated to beta in v1.29.
v1.28, graduated to beta in v1.29 and to GA in v1.31.
{{< /note >}}

### Handling kubelet restarts
Expand All @@ -170,6 +170,35 @@ kubelet instance. A new kubelet instance deletes all the existing Unix sockets u
`/var/lib/kubelet/device-plugins` when it starts. A device plugin can monitor the deletion
of its Unix socket and re-register itself upon such an event.

### Device plugin and unhealthy devices

There are cases when devices fail or are shut down. The responsibility of the Device Plugin
in this case is to notify the kubelet about the situation using the `ListAndWatchResponse` API.

Once a device is marked as unhealthy, the kubelet will decrease the allocatable count
for this resource on the Node to reflect how many devices can be used for scheduling new pods.
Capacity count for the resource will not change.

Pods that were assigned to the failed devices will continue be assigned to this device.
It is typical that code relying on the device will start failing and Pod may get
into Failed phase if `restartPolicy` for the Pod was not `Always` or enter the crash loop
otherwise.

Before Kubernetes v1.31, the way to know whether or not a Pod is associated with the
failed device is to use the [PodResources API](#monitoring-device-plugin-resources).

{{< feature-state feature_gate_name="ResourceHealthStatus" >}}

By enabling the feature gate `ResourceHealthStatus`, the field `allocatedResourcesStatus`
will be added to each container status, within the `.status` for each Pod. The `allocatedResourcesStatus`
field
reports health information for each device assigned to the container.

For a failed Pod, or or where you suspect a fault, you can use this status to understand whether
the Pod behavior may be associated with device failure. For example, if an accelerator is reporting
an over-temperature event, the `allocatedResourcesStatus` field may be able to report this.


## Device plugin deployment

You can deploy a device plugin as a DaemonSet, as a package for your node's operating system,
Expand Down
12 changes: 6 additions & 6 deletions content/en/docs/concepts/scheduling-eviction/assign-pod-node.md
Original file line number Diff line number Diff line change
Expand Up @@ -363,10 +363,10 @@ null `namespaceSelector` matches the namespace of the Pod where the rule is defi
{{< feature-state feature_gate_name="MatchLabelKeysInPodAffinity" >}}

{{< note >}}
<!-- UPDATE THIS WHEN PROMOTING TO BETA -->
The `matchLabelKeys` field is an alpha-level field and is disabled by default in
<!-- UPDATE THIS WHEN PROMOTING TO STABLE -->
The `matchLabelKeys` field is a beta-level field and is enabled by default in
Kubernetes {{< skew currentVersion >}}.
When you want to use it, you have to enable it via the
When you want to disable it, you have to disable it explicitly via the
`MatchLabelKeysInPodAffinity` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/).
{{< /note >}}

Expand Down Expand Up @@ -414,10 +414,10 @@ spec:
{{< feature-state feature_gate_name="MatchLabelKeysInPodAffinity" >}}

{{< note >}}
<!-- UPDATE THIS WHEN PROMOTING TO BETA -->
The `mismatchLabelKeys` field is an alpha-level field and is disabled by default in
<!-- UPDATE THIS WHEN PROMOTING TO STABLE -->
The `mismatchLabelKeys` field is a beta-level field and is enabled by default in
Kubernetes {{< skew currentVersion >}}.
When you want to use it, you have to enable it via the
When you want to disable it, you have to disable it explicitly via the
`MatchLabelKeysInPodAffinity` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/).
{{< /note >}}

Expand Down
Loading

0 comments on commit 20d164c

Please sign in to comment.