apiserver: reap exited child processes

[tjkirch]

Issue number:

Fixes #1380

Description of changes:

We spawn some background processes, like thar-be-settings, when they might take
some time and we don't want to hold up API clients.  Unless we `wait` for those
children when they exit, their process ID lives on as a zombie.  This adds a
SIGCHLD handler to apiserver to call wait when any child exits.

Testing done:

Before, each apiclient set would leave a zombie. After, no zombies!

I ran them in a tight loop with i=0; while :; do apiclient set settings.motd=hi$i; let i++; done which accomplished about 120 changes per second. I separately watched the motd file and saw that the operations were successful, and saw that no zombies were stacking up. You'd see 2-3 (live) thar-be-settings processes, presumably because they're waiting on the coarse API write lock. I did once catch a [thar-be-settings] zombie before the signal/loop/wait caught it, but it was immediately cleaned up, so we know things are working.

I also tested:

• manual PATCH, /tx/commit, /tx/apply, because that follows a slightly different code path to call thar-be-settings, and it was fine.
• apiclient update still updated a host successfully.
• repeated apiclient check doesn't leave thar-be-updates zombies, either.
• systemctl status running, pod ran OK.
• 0% CPU usage from apiserver (after some sets) in its normal idle state, so no haywire loop.

Terms of contribution:

By submitting this pull request, I agree that this contribution is dual-licensed under the terms of both the Apache License, version 2.0, and the MIT license.

[srgothi92]

Interesting !

[tjkirch]

^ This push improves the comments based on @bcressey helping me understand the one-to-many nature of CHLD signals to exited children.