featureflagservice database migrations fail in kubernetes

[joshleecreates]

Bug Report

Which version of the demo you are using? (please provide either a specific
b6e75ee

Symptom

When using the helm charts to deploy to k8s, the featureflagservice fails to run it's migrations because it does not wait for the postgres pod to be ready.

What is the expected behavior?

I expect to see ecto database migrations run automatically as they do when using docker compose.

What is the actual behavior?

Instead, the featureflagservice starts with the following errors:

02:51:47.123 [error] Could not create schema migrations table. This error usually happens due to the following:

  * The database does not exist
  * The "schema_migrations" table, which Ecto uses for managing
    migrations, was defined by another library
  * There is a deadlock while migrating (such as using concurrent
    indexes with a migration_lock)

To fix the first issue, run "mix ecto.create".

To address the second, you can run "mix ecto.drop" followed by
"mix ecto.create". Alternatively you may configure Ecto to use
another table and/or repository for managing migrations:

    config :featureflagservice, Featureflagservice.Repo,
      migration_source: "some_other_table_for_schema_migrations",
      migration_repo: AnotherRepoForSchemaMigrations

The full error report is shown below.

** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 2971ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:

  1. Ensuring your database is available and that you can connect to it
  2. Tracking down slow queries and making sure they are running fast enough
  3. Increasing the pool_size (although this increases resource consumption)
  4. Allowing requests to wait longer by increasing :queue_target and :queue_interval

See DBConnection.start_link/2 for more information

Reproduce

Deploy the latest helm charts and check the logs for featureflagservice.

Additional context

It seems like a fix would be to modify the ffs docker image to wait for connectivity on the database port before running the release.

[puckpuck]

I wonder if we can have the entire process fail on error, and leverage container restart policies in docker/kubernetes instead. @tsloughter is this something we can have Elixir do?