KEP 2527: Clarify meaning of status

[thockin]

Since basically the beginning of Kubernetes, we've had this sort of "litmus
test" for status fields: "If I erased the whole status struct, would everything
in it be reconstructed (or at least be reconstructible) from observation?". The
goal was to ensure that the delineation between "what I asked for" and "what it
actually is" was clear and to encourage active reconciliation of state.

Many of our APIs pass this test (sometimes we fudge it and say yes "in
theory"), but not all of them do. This KEP proposes to clarify or remove this
guidance, especially as it pertains to state that is not derived from
observation.

One of the emergent uses of the spec/status split is access control. It is
assumed that, for most resources, users own (can write to) all of spec and
controllers own all of status, and not the reverse. This allows patterns like
Services which set spec.type: LoadBalancer, where the controller writes the
LB's IP address to status, and kube-proxy can trust that IP address (because it
came from a controller, not a user). Compare that with Services which use
spec.externalIPs. The behavior is kube-proxy is roughly the same, but
because non-trusted users can write to spec.externalIPs and that does not
require a trusted controller to ACK, that behavior was declared a CVE.

This KEP further proposes to add guidance for APIs that want to implement an
"allocation" or "binding" pattern which requires trusted ACK.

KEP: #2527

[dims]

cc @palnabarun @nikhita

[embano1]

Since I'm (currently) in the "option 1" camp I was wondering if we could use annotations to track non-observable state there instead of diluting status? Just throwing this out here as I did not see it mentioned in the KEP (but not sure if it was discussed already and I'm just missing context).

[thockin]

@embano1 In general we try avoid enshrining non-primitive APIs as annotations. It just offloads the problem of decoding onto users. In this case, I don't think it helps because annotations are not RBAC'ed individually. A user could set "sensitive" values.

[thockin]

Hi all. I made some minor updates to this, but there has not been much feedback so far. Would love to hear more thoughts.

[liggitt]

I think this makes sense... I don't think this scuttles any future plans to routinely discard status subtrees of objects or anything.

I expect some people have been writing controllers that operate as one-way "take observed state and update status" because of this guidance will now feel free to branch out into more creative/complex paths (spec → status pre-write → other object → status post-write, etc).

As we relax this, it would be good to give pretty crisp examples/guidance/patterns for writing state to status to help those folks stay re-entrant and robust in the face of conflicts/errors.

[thockin]

I fixed the small nit. I think the text you want actually goes in the impl, not the KEP? As such, PTAL at this.

[liggitt]

I'm happy with the direction, and the details can fall in the actual docs PR. I do think it's worth at least mentioning in this KEP the risks that working from status as a data source introduces, even if the detailed recommendations / best practices / examples are in the docs PR.

[thockin]

Added, stole some of your words.

[liggitt]

/lgtm
/approve

[thockin]

@derekwaynecarr @johnbelamaric @dims - you all are listed as sig-arch leads - please approve?

[dims]

/approve

thanks for doing this! +1 to tune our guidance based on what we are observing in the field.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: dims, liggitt, thockin

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• keps/sig-architecture/OWNERS [dims]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment