*: add experimental flag for watch notify interval

[jingyih]

Make watch progress notify configurable via flag --experimental-watch-progress-notify-interval. It is useful in cases such as https://github.com/kubernetes/enhancements/tree/master/keps/sig-api-machinery/1904-efficient-watch-resumption

[jingyih]

cc @xiang90 @gyuho @wojtek-t @ptabor

[wojtek-t]

LGTM - thanks!

[FTR: we were doing both some scale and correctness tests of prototype based on https://github.com//pull/11463 (which this one seem to be just rebase of) and that all worked fine.]

[gyuho]

@jingyih Can we enforce a minimum duration value for this field to prevent misuse?

This seems useful for testing. What's the other use case and what's the value that Kubernetes wants to use?

Also, please update the changelog.

[wojtek-t]

This seems useful for testing. What's the other use case and what's the value that Kubernetes wants to use?

We were scale testing it in 5k-node tests. 250ms worked perfectly fine, 100ms was signifcantly visible in metrics, though it still kind-of worked.
In Kubernetes - I'm targeting to setting it to O(5s) (at least initially).

[jingyih]

Thanks @gyuho!

Can we enforce a minimum duration value for this field to prevent misuse?

I added 100ms minimum duration.

This seems useful for testing. What's the other use case and what's the value that Kubernetes wants to use?

Please refer to @wojtek-t 's previous comment. Also as mentioned in the PR description, this is useful to https://github.com/kubernetes/enhancements/tree/master/keps/sig-api-machinery/1904-efficient-watch-resumption

Also, please update the changelog.

Done.

[wojtek-t]

Cool - great to see that merged!