linkerd-tcp crashes ungracefully from 'too many open files'

[stevej]

Problem

'Too many open files' leaves linkerd-tcp in a bad state rather than gracefully exiting.

Symptoms:

In a shell with a ulimit -n 1024, if I open 500 connections (which implies 500 outgoing connections) linkerd-tcp will print an error to stdout and stop processing incoming connections or requests from existing connections. Closing some incoming connections will allow connects to succeed again but those new sockets never have their requests processed by linkerd-tcp.

$ RUST_LOG=error RUST_BACKTRACE=yes ./linkerd-tcp-1490585634 example.yml
Listening on http://127.0.0.1:9989.


thread 'main' panicked at 'could not run proxies: Error { repr: Os { code: 24, message: "Too many open files" } }', /buildslave/rust-buildbot/slave/stable-dist-rustc-linux/build/src/libcore/result.rs:868
stack backtrace:
   1:     0x5586a7e4e7ac - std::sys::imp::backtrace::tracing::imp::write::hf33ae72d0baa11ed
                        at /buildslave/rust-buildbot/slave/stable-dist-rustc-linux/build/src/libstd/sys/unix/backtrace/tracing/gcc_s.rs:42
   2:     0x5586a7e51abe - std::panicking::default_hook::{{closure}}::h59672b733cc6a455
                        at /buildslave/rust-buildbot/slave/stable-dist-rustc-linux/build/src/libstd/panicking.rs:351
   3:     0x5586a7e516c4 - std::panicking::default_hook::h1670459d2f3f8843
                        at /buildslave/rust-buildbot/slave/stable-dist-rustc-linux/build/src/libstd/panicking.rs:367
   4:     0x5586a7e51f5b - std::panicking::rust_panic_with_hook::hcf0ddb069e7beee7
                        at /buildslave/rust-buildbot/slave/stable-dist-rustc-linux/build/src/libstd/panicking.rs:555
   5:     0x5586a7e51df4 - std::panicking::begin_panic::hd6eb68e27bdf6140
                        at /buildslave/rust-buildbot/slave/stable-dist-rustc-linux/build/src/libstd/panicking.rs:517
   6:     0x5586a7e51d19 - std::panicking::begin_panic_fmt::hfea5965948b877f8
                        at /buildslave/rust-buildbot/slave/stable-dist-rustc-linux/build/src/libstd/panicking.rs:501
   7:     0x5586a7e51ca7 - rust_begin_unwind
                        at /buildslave/rust-buildbot/slave/stable-dist-rustc-linux/build/src/libstd/panicking.rs:477
   8:     0x5586a7e7e34d - core::panicking::panic_fmt::hc0f6d7b2c300cdd9
                        at /buildslave/rust-buildbot/slave/stable-dist-rustc-linux/build/src/libcore/panicking.rs:69
   9:     0x5586a7bb6642 - core::result::unwrap_failed::h52f3f53af574d319
  10:     0x5586a7bbbf41 - linkerd_tcp::main::h2f95da4c40bc36fe
  11:     0x5586a7e58f7a - __rust_maybe_catch_panic
                        at /buildslave/rust-buildbot/slave/stable-dist-rustc-linux/build/src/libpanic_unwind/lib.rs:98
  12:     0x5586a7e526c6 - std::rt::lang_start::hd7c880a37a646e81
                        at /buildslave/rust-buildbot/slave/stable-dist-rustc-linux/build/src/libstd/panicking.rs:436
                        at /buildslave/rust-buildbot/slave/stable-dist-rustc-linux/build/src/libstd/panic.rs:361
                        at /buildslave/rust-buildbot/slave/stable-dist-rustc-linux/build/src/libstd/rt.rs:57
  13:     0x7fa60f0823f0 - __libc_start_main
  14:     0x5586a7bb4f68 - <unknown>
  15:                0x0 - <unknown>```