-
Notifications
You must be signed in to change notification settings - Fork 1.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Memory leak in akka.actor.LocalActorRef
#5431
Comments
I believe akka 2.6.20 is the first release under the non-open source BSL license, not Apache v2. Therefore changes to update OpenWhisk to akka 2.6.20 cannot be accepted by the Apache OpenWhisk project. |
@dgrove-oss 2.6.20 is still Apache. >2.7.x is BSL. They actually released another patch a couple months ago 2.6.21 to fix a TLS bug. Apache Pekko has started doing official releases over the last month. Once we get on to 2.6.20 we can start discussing migrating the project to Pekko. So far the core modules, http, and kafka have been released. They’re about to do management and then the rest of the connectors. I think there should be releases for everything by September at the pace they’re going. For the topic of this memory leak, more information is needed. Is the memory leak only with 2.6.20? Can you reproduce off master? Are you using the new scheduler which uses the v2 FPCInvoker or the original invokers? |
Cool, @bdoyle0182 thanks for clarifying. I had found an old post that said 2.6.19 was the last Apache version and 2.6.20 and beyond were going to be BSL. A strategy of getting to the most recent Apache licensed version from Lightbend and then switching to Pekko sounds right to me. |
@bdoyle0182 we are migrating our project from Akka |
Apache Pekko, a fork of Akka 2.6 has been released. v1.0.1 is out - very similar to Akka 2.6.21. https://pekko.apache.org/docs/pekko/current/project/migration-guides.html |
@joni-jones Is there any chance you provide a self contained reproducer? |
If you want to raise a Pekko issue about this, someone may be able to help. |
Since the strings are all IP addresses and it is below the stream materializer, this could be incoming connections that are hanging / not cleaned up (without knowing anything about openwhisk). Hard to say without knowing anything about the setup. |
@jrudolph I'm looking at these graphs and strings with IPs having 0% in comparison to I see that these |
What you are probably looking at is the child actors of the materializer actor where one actor is spawned for every stream you run. So, it might be a bit hard to see what the actual issue is because the memory might be spread over all these actors. One way to go about it would be to see the output of a class histogram just over the elements referenced by that children tree and see what kind of data is in there. |
Thanks @jrudolph. Yes, I tried to go down through these trees and leaves are pointing to child actors and I don't know if it's related, but some time ago when OpenWhisk was upgraded from 2.5.x Akka to 2.6.12 and the actor materialized has been removed there was a The version that we are upgrading from still uses 2.5.x Akka and we don't have issues with memory there. |
It seems the issue in https://github.com/apache/openwhisk/blob/master/common/scala/src/main/scala/org/apache/openwhisk/http/PoolingRestClient.scala#L76, without In our implementation, we use a wrapper on top of I did a couple of different implementations, including:
|
@joni-jones Thanks for sharing the update. |
It looks like I was able to fix the memory leak and it was stable on our production so far. |
Summary
I'm working on upgrading OpenWhisk to Akka 2.6.20 and Scala 2.13 and experienced the issue with OpenWhisk invokers consuming all available g1-old heap size after running for a couple of days with active traffic.
Doing heap profiling I got the following suggestions from Heap Hero:
Further analysis with Eclipse Memory Analyzer shows the following:
Environment details:
Any suggestions on where I should look to find the root cause of this memory leak?
The text was updated successfully, but these errors were encountered: