add hab-director

[bookshelfdave]

(this text copied verbatim from src/main.rs)

The Habitat Director is a supervisor for a group of hab-sup processes. It loads packages to start from it's config.toml file. The director will automatically restart child process upon failure detection. Each
child service runs in it's own hab-sup process. The director can be run inside of a hab-sup instance as well.

Components

• Task
  • manages a hab-sup as a child process
  • tracks a single child process PID
  • generates CLI arguments for hab-sup start
  • creates a PID file for it's child process
  • starts a thread to read stdout from the child process
• Controller
  • A controller "has" and supervises many Tasks (children)
  • calculates gossip and sidecar port #'s for all children before starting.
  • runs in a tight loop to see if children are down and start/restarts them.
  • catches OS signals
• ExecContext
  • A task "execution context". The ExecContext is used to
    decouple service root directory and path to a supervisor executable.
    Decoupling these values into a struct allows us to easily test
    Tasks + Controllers.
• ExecParams
  • Config values for a Task that the Controller calculates during
    startup. ExecParams currently includes:
    • gossip_listen
    • sidecar_listen
    • Option<peer_ip_port>
• ServiceDef
  • A combination of PackageIdent, ServiceGroup, and CLI args. These
    values are loaded from the config file and are set by the user, as
    opposed to ExecContext values which are set by the Controller.
  • Examples:
    • core.redis.somegroup.someorg corresponds to the
      core/redis PackageIdent, and the
      redis.somegroup@someorg ServiceGroup.
    • core.redis.somegroup corresponds to the
      core/redis PackageIdent, and the redis.somegroup ServiceGroup
      (org-less).

┌──────────┐
│  hab-sup │
└──────────┘
      │
      │
      │    ┌───────────────┐
      │    │               │
      └───▶│   Controller  │────┐    ┌────────────┐   ┌────────┐    ┌──────────┐
           │               │    ├───▶│ ExecParams │──▶│  Task  │───▶│  hab-sup │
           └───────────────┘    │    └────────────┘   └────────┘    └──────────┘
             ┌─────────────┐    │    ┌────────────┐   ┌────────┐    ┌──────────┐
             │ ExecContext │────┼───▶│ ExecParams │──▶│  Task  │───▶│  hab-sup │
             └─────────────┘    │    └────────────┘   └────────┘    └──────────┘
                                │    ┌────────────┐   ┌────────┐    ┌──────────┐
                                └───▶│ ExecParams │──▶│  Task  │───▶│  hab-sup │
                                     └────────────┘   └────────┘    └──────────┘

Config file format

ServiceDefs are parsed from the config.toml file upon startup.

All services must be described as children of the services toml table.
Note, when toml is rendered, the values for services will be
located under cfg.services.*.

Each service definition is a . separated list of values as a TOML table name.

[services.<origin>.<name>.<group>.<organization>]

or rendered by hab-sup:

[cfg.services.<origin>.<name>.<group>.<organization>]

A service definition can additionally specify a start key/value under
the service table definition:

# Start core/redis with --group somegroup and --org someorg
# Additionally, pass in --permanent-peer to the start CLI
[cfg.services.core.redis.somegroup.someorg]
start = "--permanent-peer"

[cfg.services.core.rngd.foo.someorg]
start = "--permanent-peer --foo=bar"

Signal handling

• If started from bash: when hab-director receives SIGINT or SIGTERM,
  the director exits. Child processes will have already been sent the signal
  from bash because they are in the same session group, and will die as well.
• TODO: If NOT started from bash: when hab-director receives SIGINT or SIGTERM,
  signal behavior is undefined and signals are NOT forwarded to child tasks.
• When hab-director receives any other signal (that doesn't
  kill this process), they are re-sent to each Task in the same
  order that services are defined in config.toml.

PID files

• for each Task created, a pid file is created in the hab-director service directory. For example, for core.redis.somegroup.someorg, we'll have a /hab/svc/hab-director/core-redis-somegroup-someorg.pid file created. Creation/removal of these files is handled automatically by the director.

Ring behavior

• The first task that is created will attempt to join sys.gossip_ip:sys.gossip_port by specifying a --peerto hab-sup. This only occurs if the director is running under a supervisor.
• Each subsequent task that is created uses the previous tasks IP:port as a value for --peer.

---

• upon startup, the director tries to get the path for the latest installed version of hab-sup. If it can't find it, it falls back to the hab-sup in /src/components/sup...
• running the director via hab-sup is demoed here
• Run hab-director with RUST_LOG=habitat_director=debug to see process start params, path to hab-sup etc.
• argument parsing: cli args specified in config.toml are split on whitespace to pass to std::process::Command. This will break if we allow string args that contain whitespace, but we don't currently have any args that allow whitespace.
• sup/supervisor and director/controller behavior when not started from bash should be investigated and updated in another PR.
• there are many enhancements we could make in the director, but they'll have to be prioritized by product.
  • unified HTTP endpoint
  • use a hook to send a HUP to the director to reload service defs without killing existing services.
  • additional parameters for a service definition in config.toml
• TODO:
  • the hab-director plan needs to be built and tested. I couldn't get hab-plan-build to work for any Rust plan.
  • fix interleaved output

[thesentinels]

By analyzing the blame information on this pull request, we identified @adamhjk, @fnichol, @smith, @jtimberman and @reset to be potential reviewers

[adamhjk]

/pedant You have extra whitespace.

[bookshelfdave]

There should be error handling here.

[bookshelfdave]

fixed!

[bookshelfdave]

is this path correct?

[bookshelfdave]

@habitat-sh/habitat-core-maintainers I fixed up some things to make this work under hab-sup:

• when hab-director dies/exits, send a SIGINT, wait for 8 seconds, then send a SIGKILL to each child.
• the hab-director plan runs as root, otherwise we have director permission issues in `/hab/svc/ for each child process
• pids are written to /hab/svc/hab-director/data/
• use the latest hab-sup installed package path by default, otherwise, fall back to /src/components/sup/target/debug/hab-sup
• I'd like for the hab-director children to join the hab-sup that's running the director, but I don't see the port # in config.toml after it's rendered.

[bookshelfdave]

The first child that is created now connects to the parent hab-sup ring, IFF director is running under a hab-sup.

[bookshelfdave]

anyone have a moment to review the newest changes so I can get this merged?

[adamhjk]

@thesentinels r+

[thesentinels]

📌 Commit 6ee8961 has been approved by adamhjk

[thesentinels]

⌛ Testing commit 6ee8961 with merge f2b3756...

[thesentinels]

☀️ Test successful - travis