Use ConfigMapList's resourceVersion for watching (w/ PropertySource)

[rdesgroppes]

Rationale: avoid ERROR notifications during rolling upgrades.
Alternative PR: #299.

Using the greatest resourceVersion of all the ConfigMaps returned
within the ConfigMapList works as expected for fresh deployments.
But, when performing a rolling upgrade (and depending on the upgrade
strategy), the watcher happens to frequently stop after having received
an ERROR notification:

[ERROR] [KubernetesConfigMapWatcher] [] Kubernetes API returned an
error for a ConfigMap watch event: ConfigMapWatchEvent{type=ERROR,
object=ConfigMap{metadata=Metadata{name='null', namespace='null',
uid='null', labels={}, resourceVersion=null}, data={}}}

What's actually streamed in that case is a Status object such as:

{
  "type": "ERROR",
  "object": {
    "kind": "Status",
    "apiVersion": "v1",
    "metadata": {},
    "status": "Expired",
    "message": "too old resource version: 123 (456)",
    "reason": "Gone",
    "code": 410
  }
}

A few references:

• How to detect 410 Gone in watch response? ManageIQ/kubeclient#452
• https://www.baeldung.com/java-kubernetes-watch#1-resource-versions

It's possible to recover by adding some logic to reinstall the watcher
starting with the newly advertised resourceVersion, but this may be
avoided at all by starting the initial watch at the resourceVersion
of the ConfigMapList itself: this one won't expire.

The proposed implementation consists in storing last received
resourceVersion as an additional PropertySource (through a dedicated
name and version key) and later using it when installing the watcher.

[CLAassistant]

All committers have signed the CLA.

[pgressa]

@rdesgroppes I prefer to use this way of resourceVersion tracking b/c of using the property source as a vehicle how to deliver and store the value that is effectively a dynamic configuration property. Though it would be great if the resource version would be updated on every add/modify event so we won't hit 504 | 410 in case of restart of the watcher.

You ok with it or do you prefer #299 for some reason?

[rdesgroppes]

@pgressa

You ok with it or do you prefer #299 for some reason?

#299 is the simplest patch @grollinger and I came up with, but #300 seems more forward looking, so I'd agree with you.

Update: I took the liberty to squash commits (w/ due Co-authored-by) so that no broken commit enters the mainline.