Fix for not reusable http client leading to connection leaks in Jolokia module

[mirkochip]

In a host where I have three Wildfly 9 running instances, I installed metricbeat along with Jolokia module in order to keep track of JVM heap usage, etc.

I'm able to collect regularly all the needed metrics, but after a while that Metricbeat/Jolokia is running, it reaches the limit of maximum file opened in the system, and then it breaks saying: write error: failed to open new file: open /var/log/metricbeat/metricbeat: too many open files.

Checking the status of connections by metricbeat PID: netstat --all --program | grep '<PID>', I get tons of entries like the following ones:

...
...
tcp        0      0 localhost:57346         localhost:8090          ESTABLISHED 3181/metricbeat     
tcp        0      0 localhost:52678         localhost:8090          ESTABLISHED 3181/metricbeat     
tcp        0      0 localhost:38310         localhost:8090          ESTABLISHED 3181/metricbeat     
tcp        0      0 localhost:43934         localhost:8090          ESTABLISHED 3181/metricbeat     
tcp        0      0 localhost:54988         localhost:8090          ESTABLISHED 3181/metricbeat 
...
...

Metricbeat will definitely break after a while, when the number of the above entries is equal to the maximum number of open file descriptors parameter set for the user running the process (ulimit).

Thanks also to @jsoriano, we found out that the Jolokia module creates a new client for each request, instead of reusing the already available clients, leading to an unavoidable connections leak.

This PR introduces a fix for it.

[elasticmachine]

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?

[jsoriano]

Hi @mirkochip,

Thanks for opening this PR and raising awareness about this issue. It actually looks like a potential problem that could affect all http-based metricbeat modules.

There is already a timeout option available in all metricbeat modules, but unfortunately it doesn't seem to be documented. You could try to use it, just set timeout to some value like 2s in the module configuration.

So, I don't see the need of adding an specific timeout option for the Jolokia module, but we could:

• Set a sane default (something like 10s?) for all metricbeat modules in the http helper.
• Document the timeout option in the http helper documentation.

Would you like to do these changes here? If not I am happy to do them too in a new PR.

Thanks!

[jsoriano]

Something that is probably doing this problem worse in the Jolokia module than in other modules is that it creates a new client for each request, it should reuse the client.

[mirkochip]

Something that is probably doing this problem worse in the Jolokia module than in other modules is that it creates a new client for each request, it should reuse the client.

Hello @jsoriano,

many thanks for your reply and interest in my PR!

You pointed me in the right direction: with my latest push Jolokia should now take advantage of the already existing clients, instead of instantiate always a new one!

[jsoriano]

Thanks for doing these changes! Could you edit the title and description of the pull request to match current changes?

Changes requested were addressed

[jsoriano]

jenkins, test this

[jsoriano]

@mirkochip thanks for the changes, this LGTM, could you please add a changelog entry in CHANGELOG.next.asciidoc?

[mirkochip]

@mirkochip thanks for the changes, this LGTM, could you please add a changelog entry in CHANGELOG.next.asciidoc?

Sure, done!

[jsoriano]

@mirkochip btw, I started a PR to set default timeouts, find it in #11032 in case you want to give any feedback. Thanks!

[mirkochip]

@mirkochip btw, I started a PR to set default timeouts, find it in #11032 in case you want to give any feedback. Thanks!

It sounds super, I'll have a look at it! Thanks.

[jsoriano]

@mirkochip sorry, one last thing, could you move the changelog entry from the breaking changes section to the bugfixes section for metricbeat? thanks!

[mirkochip]

@mirkochip sorry, one last thing, could you move the changelog entry from the breaking changes section to the bugfixes section for metricbeat? thanks!

Done :)

[jsoriano]

jenkins, test this

[jsoriano]

jenkins, test this