Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Memory leak using async gauge instruments with version 1.11.2/v0.34.0 #3808

Open
naveenmall11 opened this issue Feb 27, 2023 · 5 comments
Open
Labels
bug Something isn't working invalid This doesn't seem right

Comments

@naveenmall11
Copy link

Description

We have noticed a memory leak in our code that uses async gauge instrument with version 1.11.2/v0.34.0.
It looks like while registering callbacks with meter, the allocated function is never removed from the data structure in pipeline https://github.com/open-telemetry/opentelemetry-go/blob/v1.11.2/sdk/metric/pipeline.go#L78
This could be a cause of memory leak as it keeps allocating a new instance of the function for every gauge record and that keeps piling up in the data structure.
We noticed that an issue for this was reported and an unregister method was added in the next version 1.12.0/v0.35.0
Is there a work around for this in 1.11.2/v0.34.0 as it is difficult for us to migrate to a newer version.
Here is sample code that we are using:

func RecordMetricsWithAttributes(g asyncfloat64.Gauge, val float64, tags ...attribute.KeyValue) {
	_ = meter.RegisterCallback([]instrument.Asynchronous{g}, func(ctx context.Context) {
		g.Observe(ctx, val, tags...)
	})
}

Environment

  • OS: [iOS]
  • Architecture: [x86, i386]
  • Go Version: [1.18]
  • opentelemetry-go version: [1.11.2/v0.34.0]

Steps To Reproduce

Call RecordMetricsWithAttributes in a loop and create a manualReader to process the callbacks, even after the reader has processed the callbacks, the callbacks would still be present in data structure and its size would keep increasing on every call to RecordMetricsWithAttributes.

Expected behavior

After the callback is processed it should be unregistered.

@naveenmall11 naveenmall11 added the bug Something isn't working label Feb 27, 2023
@MrAlias
Copy link
Contributor

MrAlias commented Feb 27, 2023

You seem to be registering a callback within a loop, not recording metrics.

I would expect this to continually grow as it is adding a stateful callback to the metric pipeline.

If you want to record an asynchronous instrument, register the callback you want to use once and let the SDK collection cycle call that callback to record the observation.

@MrAlias MrAlias added the invalid This doesn't seem right label Feb 27, 2023
@naveenmall11
Copy link
Author

Hi @MrAlias ,
Thanks for the prompt reply.
Yes we were registering callbacks in a loops, this is because we want to observe different values and tags on different calls.
If we register the callbacks once, how would you suggest sending different values to observe() on different calls?

@MrAlias
Copy link
Contributor

MrAlias commented Feb 27, 2023

I would write a closure around the tag structure you want to update, or look into using synchronous instruments if the tags are changed based on the context they come from.

@naveenmall11
Copy link
Author

Is there any synchronous gauge? I can not see it.

@MrAlias
Copy link
Contributor

MrAlias commented Feb 27, 2023

There is not. The specification is still discussing if this addition is desired or not.

There is an UpDownCounter that could be used for non-monotonic state reporting.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working invalid This doesn't seem right
Projects
None yet
Development

No branches or pull requests

2 participants