Fix timings in kubernetes module cache cleaning

[exekias]

We have an in-memory cache of some metrics from kube-state-metrics. We reuse them from the container metricset to calculate % resource usage (current usage vs limit/request).

This PR fixes an issue in that cache, it was evicting some entries too soon, so some of the containers missed the calculated metrics.

Tests have been refactored to avoid timings issues too

[exekias]

jenkins retest this please

[ruflin]

Could you share in the PR description a bit more on what the issue exactly was? Can you add a changelog entry if one is needed?

[exekias]

done!

[ruflin]

There seems to be a race:

12:10:32 command [go test -race -tags integration -cover -coverprofile /tmp/gotestcover-945305131 github.com/elastic/beats/metricbeat/module/kubernetes/util]: exit status 1
12:10:32 ==================
12:10:32 WARNING: DATA RACE
12:10:32 Read at 0x00c42011c010 by goroutine 8:
12:10:32   github.com/elastic/beats/metricbeat/module/kubernetes/util.TestTimeout.func1()
12:10:32       /go/src/github.com/elastic/beats/metricbeat/module/kubernetes/util/metrics_cache_test.go:14 +0x4d
12:10:32   github.com/elastic/beats/metricbeat/module/kubernetes/util.(*valueMap).Set()
12:10:32       github.com/elastic/beats/metricbeat/module/kubernetes/util/_test/_obj_test/metrics_cache.go:87 +0xb9
12:10:32   github.com/elastic/beats/metricbeat/module/kubernetes/util.TestValueMap()
12:10:32       /go/src/github.com/elastic/beats/metricbeat/module/kubernetes/util/metrics_cache_test.go:44 +0x121
12:10:32   testing.tRunner()
12:10:32       /usr/local/go/src/testing/testing.go:746 +0x16c
12:10:32 
12:10:32 Previous write at 0x00c42011c010 by goroutine 7:
12:10:32   github.com/elastic/beats/metricbeat/module/kubernetes/util.TestTimeout.func1()
12:10:32       /go/src/github.com/elastic/beats/metricbeat/module/kubernetes/util/metrics_cache_test.go:14 +0x66
12:10:32   github.com/elastic/beats/metricbeat/module/kubernetes/util.(*valueMap).ensureCleanupWorker.func1()
12:10:32       github.com/elastic/beats/metricbeat/module/kubernetes/util/_test/_obj_test/metrics_cache.go:102 +0x220
12:10:32 
12:10:32 Goroutine 8 (running) created at:
12:10:32   testing.(*T).Run()
12:10:32       /usr/local/go/src/testing/testing.go:789 +0x568
12:10:32   testing.runTests.func1()
12:10:32       /usr/local/go/src/testing/testing.go:1004 +0xa7
12:10:32   testing.tRunner()
12:10:32       /usr/local/go/src/testing/testing.go:746 +0x16c
12:10:32   testing.runTests()
12:10:32       /usr/local/go/src/testing/testing.go:1002 +0x521
12:10:32   testing.(*M).Run()
12:10:32       /usr/local/go/src/testing/testing.go:921 +0x206
12:10:32   main.main()
12:10:32       github.com/elastic/beats/metricbeat/module/kubernetes/util/_test/_testmain.go:98 +0x2f0
12:10:32 
12:10:32 Goroutine 7 (running) created at:
12:10:32   github.com/elastic/beats/metricbeat/module/kubernetes/util.(*valueMap).ensureCleanupWorker()
12:10:32       github.com/elastic/beats/metricbeat/module/kubernetes/util/_test/_obj_test/metrics_cache.go:96 +0xc8
12:10:32   github.com/elastic/beats/metricbeat/module/kubernetes/util.(*valueMap).Set()
12:10:32       github.com/elastic/beats/metricbeat/module/kubernetes/util/_test/_obj_test/metrics_cache.go:86 +0x9d
12:10:32   github.com/elastic/beats/metricbeat/module/kubernetes/util.TestTimeout()
12:10:32       /go/src/github.com/elastic/beats/metricbeat/module/kubernetes/util/metrics_cache_test.go:26 +0x2b8
12:10:32   testing.tRunner()
12:10:32       /usr/local/go/src/testing/testing.go:746 +0x16c
12:10:32 ==================
12:10:32 --- FAIL: TestValueMap (0.00s)
12:10:32 	testing.go:699: race detected during execution of test
12:10:32 FAIL

[exekias]

Thanks for the ping, I fixed the issue, it was in the tests code