Determine containerised database migrations approach

[shrink]

https://pythonspeed.com/articles/schema-migrations-server-startup/
https://engineering.instawork.com/elegant-database-migrations-on-ecs-74f3487da99f

Database migrations are the primary use case for this but it may capture other things, such as configuration changes, so it should be thought of more as dependency changes between versions than just database migrations.

• All dependency changes are backwards compatible by one version, database changes are additive -- deprecated database configuration is cleaned up in later deploys
• Application image has an alternative "upgrade" entrypoint which is triggered explicitly by passing in a flag at runtime -- by default the entrypoint is nginx and the upgrade (migrate + any other tasks) can only happen explicitly
• Container healthcheck determines if it is compatible with all dependencies
  • e.g, database: if the application version (baked into the image) is greater than the database version (stored in the database) then the healthcheck fails
  • The reason to check for greater than rather than matching is because we need the old application versions to be able to continue to start, even if the database has been upgraded, because it's possible the new application versions won't start and the old will need to continue to run
• Deployment process is responsible for triggering a single instance of upgrade and pushing a new version of the application to each running instance: healthchecks will fail for the new versions of the application until the migration is finished
  • optionally, in a strict environment (where any failing healthcheck during a deployment is an alarm) the upgrade task could be waited on before triggering the new application version deployment -- this would create a dependency between deployment command order but that shouldn't be an issue

[shrink]

A dependency has a version: an application has a list of dependencies and the minimum versions which must be met for the healthcheck to pass. For example, a Database has a Schema Version.

[shrink]

This is now supported with the addition of Conductor which provides a Schema Version check. Last outstanding step is to document the implementation (i.e: how it can be used as part of a deployment).