Production failure: Dotmesh server pods ending up in Completed state

[alaric-dotmesh]

In production, it was thusly seen that two of our server pods just... finished:

server-gke-dothub-cluster-default-pool-6cc559e3-d4j9   0/1       Completed   0          1h
server-gke-dothub-cluster-default-pool-6cc559e3-gmdw   1/1       Running     0          1h
server-gke-dothub-cluster-default-pool-6cc559e3-j1w6   0/1       Completed   0          1h

They're not supposed to do that. The operator didn't know what to do about this (see #448 about fixing that) because it shouldn't ever happen.

The logs for the dead pods don't show any useful errors, but do show a strange mixture of everyday logging and what looks like a goroutine dump, annotated with what each goroutine is blocked on.

Here's the logs from all three dots:

d4j9.log
gmdw.log
j1w6.log

[Godley]

Describe on one of the pods did also show:

Warning  Unhealthy  38m   kubelet, gke-dothub-cluster-default-pool-6cc559e3-j1w6  Readiness probe failed: Get http://10.60.2.34:32607/check: dial tcp 10.60.2.34:32607: getsockopt: connection refused
  Warning  Unhealthy  38m   kubelet, gke-dothub-cluster-default-pool-6cc559e3-j1w6  Liveness probe failed: Get http://10.60.2.34:32608/check: dial tcp 10.60.2.34:32608: getsockopt: connection refused

But...it's 38m old and there's no "last seen" there, so I'm inclined to say that's only when the pod started up