Docker image temporary files

[shaliko]

On few installations reproduce issue with generation huge amount of temporary files.

Api-umbrella container in folder /tmp have hundreds of thousands files:

-rw-rw-rw-  1 api-umbrella api-umbrella  17K Aug 30 13:51 hs_err_pid13561.log
-rw-rw-rw-  1 api-umbrella api-umbrella  17K Aug 30 13:51 hs_err_pid13576.log
-rw-rw-rw-  1 api-umbrella api-umbrella  17K Aug 30 13:51 hs_err_pid13590.log
-rw-rw-rw-  1 api-umbrella api-umbrella  17K Aug 30 13:51 hs_err_pid13604.log
-rw-rw-rw-  1 api-umbrella api-umbrella  17K Aug 30 13:51 hs_err_pid13619.log
-rw-rw-rw-  1 api-umbrella api-umbrella  17K Aug 30 13:51 hs_err_pid13633.log
-rw-rw-rw-  1 api-umbrella api-umbrella  17K Aug 30 13:51 hs_err_pid13648.log
...

Or

-rw-------  1 root         root              0 Aug 30 20:33 api-umbrella-geoip-auto-updater.zq6Q08BSnu
-rw-------  1 root         root              0 Aug 30 19:21 api-umbrella-geoip-auto-updater.zqCAjc6AxM
-rw-------  1 root         root              0 Aug 30 19:29 api-umbrella-geoip-auto-updater.zrObizn32S
-rw-------  1 root         root              0 Aug 30 19:29 api-umbrella-geoip-auto-updater.zrPfnOe8gj
-rw-------  1 root         root              0 Aug 30 20:53 api-umbrella-geoip-auto-updater.zrSBUnUBHA
-rw-------  1 root         root              0 Aug 30 19:04 api-umbrella-geoip-auto-updater.zruSRZCwPm
-rw-------  1 root         root              0 Aug 30 18:58 api-umbrella-geoip-auto-updater.zuarQgBLrg
-rw-------  1 root         root              0 Aug 30 20:31 api-umbrella-geoip-auto-updater.zvhhH9OyMf
-rw-------  1 root         root              0 Aug 30 21:04 api-umbrella-geoip-auto-updater.zvrlVEuOD5
...

for hs_err_pidxxx files issue in :

cat hs_err_pid14412.log 
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (malloc) failed to allocate 87228416 bytes for committing reserved memory.
# Possible reasons:
#   The system is out of physical RAM or swap space
#   In 32 bit mode, the process size limit was hit
# Possible solutions:
#   Reduce memory load on the system
#   Increase physical memory or swap space
#   Check if swap backing store is full
#   Use 64 bit Java on a 64 bit OS
#   Decrease Java heap size (-Xmx/-Xms)
#   Decrease number of Java threads
#   Decrease Java thread stack sizes (-Xss)
#   Set larger code cache with -XX:ReservedCodeCacheSize=
# This output file may be truncated or incomplete.
#
#  Out of Memory Error (os_linux.cpp:2827), pid=14412, tid=140016592033536
#
# JRE version:  (7.0_111-b01) (build )
# Java VM: OpenJDK 64-Bit Server VM (24.111-b01 mixed mode linux-amd64 compressed oops)
# Derivative: IcedTea 2.6.7
# Distribution: Debian GNU/Linux 8.5 (jessie), package 7u111-2.6.7-1~deb8u1
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
...

Looks like I can add limit for Java heap size in api-umbrella.yml

flume:
  embedded_server_config:
    java_opts: "-Xmx96m -Xms32m"

Not yet found how fix generation "api-umbrella-geoip-auto-updater".

My configuration:

• "2 GB Memory / 30 GB Disk / AMS2 - Ubuntu Docker 1.11.2 on 14.04" on DigitaOcean.
• Docker image "nrel/api-umbrella:0.12.0"
• docker stats container_id:

CONTAINER           CPU %               MEM USAGE / LIMIT       MEM %               NET I/O               BLOCK I/O             PIDS
cd1da9b81722        10.08%              795.7 MiB / 1.953 GiB   39.78%              16.26 MB / 357.3 kB   45.43 MB / 181.7 MB   158

# On host
$ df -i
Filesystem      Inodes   IUsed  IFree IUse% Mounted on
udev            253267     424 252843    1% /dev
tmpfs           256058     514 255544    1% /run
/dev/vda1      1835008 1835008      0  100% /

Any advice how avoid to many files (hs_err_pid.log and api-umbrella-geoip-auto-updater) in /tmp folder?

[bajiat]

@GUI Do you have any suggestions on what could be causing this? We could potentially even contribute and help with fixing, but we would appreciate your insight on what might be causing this problem.

[GUI]

@shaliko: Does this occur after API Umbrella is in use for some time or under load? Or does this happen immediately on booting the docker container? If it takes a while for this to happen, are there any steps you can perform to reliably reproduce this? I can't reproduce this on some quick testing in Docker, but I've only tried briefly.

But a couple of general thoughts:

• The hs_err_pid* files would probably be from Elasticsearch, not Flume (Flume is only used for some of the newer experimental analytics work, but is disabled by default). So if you want to adjust the memory for Elasticsearch, you'd need to adjust the elasticsearch.embedded_server_env.heap_size variable. This defaults to 512MB and this controls the underlying Xmx and Xms settings for elasticsearch.
  • The odd thing about the error is that it's saying it can't allocate 87MB of RAM. You seem to have a decent amount of memory available (1.2GB) in your docker stats command, so that's why I'm wondering if this is only happening under certain situations, like heavy load, or when an admin is performing analytics queries. Or have you captured the docker stats output right when this happens? I'm also wondering if something is briefly spiking the memory usage by a lot.
  • Do you know how much historical data you have in this elasticsearch database? Inside your container, you could try curl "http://127.0.0.1:14002/_count"
  • Is there anything in /var/log/api-umbrella/elasticsearch/current when this occurs?
• The api-umbrella-geoip-auto-updater is a pretty simple bash script that just runs once a night and checks to see if a newer file needs to be downloaded. But based on that output, it seems like something might be killing the process, and it keeps restarting, since there appear to be incomplete temp files being generated a few times an hour (whereas, it should only really attempt to create a new file once per day). Is there anything in your /var/log/api-umbrella/geoip-auto-updater/current log file?
• Do you have any swap space configured on the system? While not ideal from a performance perspective, if there are temporary memory spikes going on, swap space might allow you to survive them in a bit more graceful manner.

[shaliko]

@GUI thanks, for feedback. I tuned "embedded_server_env.heap_size variable".

Found issue with "geoip-auto-updater", that script try load GeoIP DB by "curl", but curl dose not installed in "api-umbrella" docker image "0.12.0" and "0.13.0".

So probably issue will reproduce for all users who use "api-umbrella" docker images.