Add control of the configuration refresh interval

[maxlaverse]

What this PR does / why we need it:
Make the syncRateLimiter configurable, and therefore how the often the configuration might be reloaded. This value is tightly coupled with the all the upstream pods lifecycle and therefore should be configurable.

[k8s-reviewable]

This change is 

[coveralls]

Coverage increased (+0.1%) to 36.7% when pulling 43dc5b9e9acc42339bad9487295869ec99899a58 on maxlaverse:configurable_refresh_interval into 6816630 on kubernetes:master.

[aledbf]

@maxlaverse just in case this just limit the rate of the execution of the sync function not how often it runs

[maxlaverse]

Sorry @aledbf , my description is confusing. But the PRs still achieve what I'm looking for.

This is the upper limit for the frequency of syncs. With a current value of 0.3 it means dead backends are removed from 0s to 3s after there are actually declared dead. Therefore we told all our teams to add a preStop hook with at least 4 seconds, to have a high probabilty for the dead backend to have been removed from the Nginx Ingress and not receive any traffic anymore, before there are being stopped.

This value changed twice in the past iirc. We'd like to have this configurable as it's tightly coupled to all the preStop delay we have and if it was decided that this value should change one day, we wouldn't be able to upgrade the Nginx Ingress Controller without updating all the preStop hooks or running a fork.

[coveralls]

Coverage increased (+0.04%) to 36.648% when pulling 0561ea8 on maxlaverse:configurable_refresh_interval into 6816630 on kubernetes:master.

[maxlaverse]

Is there a command to re-trigger the tests ?

[coveralls]

Coverage increased (+0.009%) to 36.614% when pulling 0561ea8 on maxlaverse:configurable_refresh_interval into 6816630 on kubernetes:master.

[aledbf]

This is the upper limit for the frequency of syncs.

Correct

With a current value of 0.3 it means dead backends are removed from 0s to 3s after there are actually declared dead.

Please understand that this flag could end affecting the uptime of nginx. Changing this flag to something like 0.1 you could end reloading nginx once per second and that means you could end with lots of worker processes (in the case of pending requests)

[maxlaverse]

Yes thanks, I'm aware of that :)

[aledbf]

/lgtm

[aledbf]

@maxlaverse thanks!