metrics-generator: add override to disable metrics collection

[kvrhdn]

What this PR does:
Add an override metrics_generator_disable_collection which tells the registry to not collect and write samples for this tenant. This will break the link between the registry and the storage layer (which is responsible for remote writing metrics).

The metrics-generator will still generate metrics and keep track of all the counters and histograms. If you don't want to generate metrics at all, you should clear metrics_generator_processors for this tenant.

This setting is useful if you want to ingest data into the metrics-generator, observe resource usage and active series, but don't want to write real metrics. This should be especially handy if you are concerned about the generated active series overwhelming a downstream TSDB.

Which issue(s) this PR fixes:
Related to #1303

Checklist

• Tests updated
• Documentation added, will do in docs: add metrics-generator configuration #1328
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[mapno]

Great idea. Left a comment regarding the approach.

[mapno]

I was thinking, maybe we can use MetricsGeneratorCollectionInterval == 0 instead? We set a default value for the interval and not run collection if it's disabled.

Now we can have MetricsGeneratorCollectionInterval: 0 and MetricsGeneratorDisableCollection: true and not collect metrics.

[kvrhdn]

I would be concerned about confusing 0 with 'not set', though I think this should work:

MetricsGeneratorCollectionInterval *time.Duration `yaml:"metrics_generator_collection_interval,omitempty" json:"metrics_generator_collection_interval,omitempty"`

I'll try some stuff out.

[kvrhdn]

I like combining this functionality into the existing override with metrics_generator_collection_interval: 0 but it's a bit tricky/annoying to implement.
We use collection interval both for collecting metrics and for checking the overrides:

tempo/modules/generator/registry/registry.go

Line 113 in 1ecc97c

go job(instanceCtx, r.collectMetrics, r.collectionInterval)

tempo/modules/generator/registry/job.go

Lines 8 to 29 in 1ecc97c

	// job executes f every getInterval.
	func job(ctx context.Context, f func(context.Context), getInterval func() time.Duration) {
	currentInterval := getInterval()
	ticker := time.NewTicker(currentInterval)
	for {
	select {
	case <-ctx.Done():
	return
	case <-ticker.C:
	jobCtx, cancel := context.WithTimeout(ctx, currentInterval)
	f(jobCtx)
	cancel()
	}
	newInterval := getInterval()
	if currentInterval != newInterval {
	ticker = time.NewTicker(newInterval)
	currentInterval = newInterval
	}
	}
	}

So if collection interval is 0, we can't just kill this job as we have to keep checking the overrides as well. Otherwise we won't detect when collection interval changes again.

So for simplicity I suggest just adding the new override and we can later refactor this if we find a better solution.

[joe-elliott]

This PR generally looks fine to me, but I have some questions:

• Why do this in the metrics-generator and not the distributor?
• The distributor already has a per tenant "switch" to send data to the metrics generator. How is this different?

[joe-elliott]

Ok, I just read your PR description which answers my questions. :)