Reduce test suite time

[yuandrew]

What was changed

Set cluster cache refresh rate to seconds instead of minutes. Now our test doesn't need to wait 60+ seconds for the cache to refresh

TestOperator_Cluster_Operations goes from ~118.29 seconds to ~4.4 seconds
Overall CI runtime goes from 10m32s to 7m26s (sample size 1)

Why?

Speed up tests

Checklist

1. Closes [Feature Request] Reduce test suite time #645
2. How was this tested:

Ran test locally

3. Any docs updates needed?

[CLAassistant]

All committers have signed the CLA.

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[cretz]

This is great, thanks! Ideally we have no sleeps in tests, but even this is a big win. Any flakiness concern on sleeping exactly the amount of the refresh impl? Should we sleep 2s to be sure or reduce the refresh interval (maybe only for this test)?

[yuandrew]

This is great, thanks! Ideally we have no sleeps in tests, but even this is a big win. Any flakiness concern on sleeping exactly the amount of the refresh impl? Should we sleep 2s to be sure or reduce the refresh interval (maybe only for this test)?

Changing refresh rate to milliseconds takes another second off of the test, having a buffer of 5 millisecond sleep seems to be good, no issues after running test locally 80 times

[cretz]

Can you confirm with server any harm in changing this to 100ms just for our tests? Meaning, is this max expiry before refetch or is this some kind of loop interval. If no prob, lets do that.

[yuandrew]

YX confirmed this is fine, only side effect is more DB IO

Jotting down an observation, setting refresh rate to 1 ms causes some tests to fail due to max QPS being reached, but 100 ms seems fine.

[dandavison]

Nice.