Simplifying stateful behavior

[VihasMakwana]

Component(s)

extension/storage, extension/storage/filestorage, receiver/filelog

Is your feature request related to a problem? Please describe.

Currently, the collector operates in a stateless mode by default, with stateful components storing offsets in memory. However, stateful components should persist their state during shutdown if a storage extension is available.

At present, enabling stateful behavior involves a somewhat lengthy process:

1. Adding a filestorage entry to the extensions stanza in the configuration.
2. Including filestorage under service::extensions.
3. Ensuring that the storage directory exists.
4. Adding storage: file_storage/xyz to individual components.

It would be beneficial to simplify this process by introducing a feature gate or a single configuration option. With this approach, users could enable stateful mode with a single setting, and the necessary steps would be handled automatically, achieving the same effect as the manual steps described.

Describe the solution you'd like

Here’s one way to tackle the enhancement:

• Introduce a new configuration provider that provides us with a default filestorage extension.
  • We might also consider including some commonly used extensions, such as healtcheck, pprof, memorylimitter, etc.
• Inject this new configuration.
• Enable stateful receivers to automatically utilize the storage extension when the feature flag is turned on.

This is a high-level overview, and I expect it will require multiple pull requests to implement.

Describe alternatives you've considered

Please share if you have any thoughts!

Additional context

My solution would require open-telemetry/opentelemetry-collector#11046 to be merged or a similar workaround to be implemented to combine lists in config.

[github-actions]

Pinging code owners:

• extension/storage: @dmitryax @atoulme @djaglowski
• extension/storage/filestorage: @djaglowski
• receiver/filelog: @djaglowski

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[VihasMakwana]

any thoughts over this @djaglowski @atoulme ?

[djaglowski]

I think this is an issue which could be considered by the core maintainers. I will note that at one point the filelog receiver would automatically use a storage extension is exactly one were present, but this caused problems, e.g. adding a second storage extension would invalidate the config for an unrelated component.

[VihasMakwana]

I think this is an issue which could be considered by the core maintainers.

I see. @open-telemetry/collector-maintainers can you take a look here and share your thoughts?

I will note that at one point the filelog receiver would automatically use a storage extension is exactly one were present, but this caused problems, e.g. adding a second storage extension would invalidate the config for an unrelated component.

Would you happen to remember when this happened and around which collector version it was? I'll look it up.