Examine Ruby memory use

[david-a-wheeler]

See
https://www.joyfulbikeshedding.com/blog/2019-03-14-what-causes-ruby-memory-bloat.html#is-fragmentation-at-the-ruby-heap-pages-level-the-cause-of-memory-bloating

[david-a-wheeler]

In the short term, investigate using

MALLOC_ARENA_MAX=2

[david-a-wheeler]

The "Joyful Bikeshedding" is, as far as I can tell, the best analysis. Many people thought it was fragmentation, but with additional analysis, it appears that the real problem is slightly different.

That said, others have also noticed that setting MALLOC_ARENA_MAX=2 is helpful for Ruby:

https://www.speedshop.co/2017/12/04/malloc-doubles-ruby-memory.html
heroku/heroku-buildpack-ruby#751
https://www.mikeperham.com/2018/04/25/taming-rails-memory-bloat/

Experimentation reveals no serious problems in setting this environment variable for our situation.

[david-a-wheeler]

I continue to monitor. I see no serious problems. The memory use is somewhat less - it's not trivial, but it looks like we have a better margin.

[david-a-wheeler]

I've continued to monitor. MALLOC_ARENA_MAX=2 is a significant improvement. Our 7-day average is 53%, and the max is 96% (at over 100% we start swapping). While swapping doesn't shut us down, a single setting that makes it rare is a good thing.

So I'm closing this issue. Further improvements would be welcome, but right now we should move on to other things.