Consolidate to fewer templates

[BryanQuigley]

I was wondering if there was any interest in generating more of the Docker files from templates at build time? This could make it easier to update, add more archs, or support more versions of PostGIS.

(we were doing something similar here.)

Designing this does somewhat require a better understanding of how many different variations to build/support at a time. This might overlap with @ImreSamu WIP PR where a recommended versions is specified - I really like the idea of a recommended version.

As I see the overall variants right now:
OSes: Alpine (1 version at a time). Debian (maybe more than one, right now we have stretch - very unsupported {maybe drop} to support postgis 2.5 and bullseye/stable)
Postgres: Clearly says all supported versions, could see the argument for dropping the oldest supported one
PostGIS: 3 versions (2.5, 3.2 and master), but it might be helpful to be able to support 3.1/3.3 as well
Archs: amd64 (but interest in more)

We could support newer releases with more options - like only arm for 14-3.2, or 14-master.

What would the ideal matrix of variants look like?

[ImreSamu]

Hi BryanQuigley,

we don't have a clear roadmap yet ..

What would the ideal matrix of variants look like?

As I understand - the project has some constraints:

1.) The project ~ have to use the upstream (docker-library/postgres) supported postgres versions:

• https://github.com/docker-library/postgres/tree/master/11 -> ( 'alpine','bullseye','stretch' )
• https://github.com/docker-library/postgres/tree/master/15 -> ('alpine',bullseye' )

2.) The Debian postgis packaged versions is very limited ; so no postgis 3.0 , 3.1 ( 3.3 ) package for debians

• postgresql-14-postgis-3 --> now 3.2.1
• postgresql-11-postgis-2.5 --> now 2.5.5 ( the latest -> PostGIS 2.5.6 )

3.) Long time support for the current users ( ~ debian based images , and if it is easy )

• 10-2.5 : base image https://hub.docker.com/_/postgres?tab=tags&page=1&name=10-stretch ~ "Last pushed 17 days ago"
  • so no security problems.

4.) The project's current https://github.com/postgis/docker-postgis/blob/master/update.sh scripts have some limitations

• The alpine version depends on the debian version
  • If no debian Postgis apt package exists ..
    • Then We can't create an alternative alpine image ( pg3.0 , pg3,1, pg3.3)

Workarounds for 4.)

• temporary - using a Debian "placeholder" dockerfile ( ~ hack ) ..
  • like in my current PR
    • with this hack - we can add any alpine based test images.
• long term: refactoring our codebase like the upstream docker-library + using jq template
  • this is planned ..

now my recommended images based on my: #295

Debian based:

DockerHub image	OS	Postgres	PostGIS
postgis/postgis:10-2.5	debian:stretch	10	2.5.5
postgis/postgis:10-3.2	debian:bullseye	10	3.2.1
postgis/postgis:11-2.5	debian:stretch	11	2.5.5
postgis/postgis:11-3.2	debian:bullseye	11	3.2.1
postgis/postgis:12-3.2	debian:bullseye	12	3.2.1
postgis/postgis:13-3.2	debian:bullseye	13	3.2.1
postgis/postgis:14-3.2	debian:bullseye	14	3.2.1

not easy to create more variations - with supported base images.

Alpine based: ( now in 3.15 .. )

DockerHub image	OS	Postgres	PostGIS
postgis/postgis:10-2.5-alpine	alpine:3.16	10	2.5.5
postgis/postgis:10-3.2-alpine	alpine:3.16	10	3.2.1
postgis/postgis:11-2.5-alpine	alpine:3.16	11	2.5.5
postgis/postgis:11-3.2-alpine	alpine:3.16	11	3.2.1
postgis/postgis:12-3.2-alpine	alpine:3.16	12	3.2.1
postgis/postgis:13-3.2-alpine	alpine:3.16	13	3.2.1
postgis/postgis:14-3.2-alpine	alpine:3.16	14	3.2.1

Test images ( planned )

DockerHub image	OS	Postgres	PostGIS
postgis/postgis:13-master	debian:bullseye	13	development: postgis, geos, proj, gdal
postgis/postgis:14-3.3.0alpha1-alpine	alpine:3.16	14	3.3.0alpha1
postgis/postgis:14-master	debian:bullseye	14	development: postgis, geos, proj, gdal
postgis/postgis:15beta1-3.3.0alpha1-alpine	alpine:3.16	15beta1	3.3.0alpha1
postgis/postgis:15beta1-master	debian:bullseye	15beta1	development: postgis, geos, proj, gdal

My opinion:

• It is not advisable to add too many new images before refactoring to jq template .. because it is difficult to support later.
  • exceptions:
    • adding test images
    • new arm64/v8 / arm/v7 images - based on the current debian scripts.

And Later we can add multiple alpine images:

• postgis/postgis:14-3.0-alpine
• postgis/postgis:14-3.1-alpine
• postgis/postgis:14-3.2-alpine
• postgis/postgis:14-3.3-alpine

etc ..

I was wondering if there was any interest in generating more of the Docker files from templates at build time?
This could make it easier to update, add more archs, or support more versions of PostGIS.

I do not fully understand your proposal, can you give me a simple example?

honestly I prefer recommendations that are compatible with the upstream ( docker-library/postgres ) best practices ( + jq templating )
see the current version docker-library/postgres repo,
that I would like to target :

• https://github.com/docker-library/postgres/blob/master/versions.json
• https://github.com/docker-library/postgres/blob/master/Dockerfile-alpine.template ( with jq templating )

and the upstream repo is pre-generating Dockerfiles

• https://github.com/docker-library/postgres/blob/master/15/bullseye/Dockerfile
• https://github.com/docker-library/postgres/tree/master/10/bullseye
• https://github.com/docker-library/postgres/tree/master/10/stretch

arm for 14-master.

it has a very long build time ( X86-64 ~ 30-40 min ) so without a strong hardware CI support - I don't recommend in the first iteration.

[BryanQuigley]

Thanks for the detailed response!

2.) Understood. Postgis 3.1 is packaged with Postgres 13 in the stock Debian repos, but that might be to divergent to try using.
3.) I forgot Stretch has LTS - although it is ending in 15 days (https://wiki.debian.org/LTS)
4.) Missed those limitations

jq templating seems similar - but maybe better than what I was proposing.. Will investigate it more. Happy to help however I can be useful - if that's the jq bit, sure :).

[phillipross]

In the limited time I've had to think about some of these things, I thought that maybe it could be helpful to start working on a newer build framework in parallel to the one that exists now.

The limitations we've had thus far have been constraining things for too long. The constraints have been accepted/tolerated mostly to prevent breaking compatibility for the existing user base. Maybe a new framework could be established in parallel to develop/modernize things? Traditionally we modeled things after the docker-library postgresql repo in hopes of some day becoming the definitive top-level postgis image, but the postgresql repo has since evolved to use jq to parse richer build metadata for configurations.

Maybe a separate branch needs to be created and development started on an updated framework? Github actions supports multiple workflow configurations. We could start development to modernize things without breaking compatibility for existing users, pushing images to dockerhub with different tag naming schemes, and making them available for the community to test with?

[BryanQuigley]

It looks like the jq parser in postgres may be refactored once stretch is EOL (end of this month), will approach doing something then. I think it should be quite possible to refactor and maintain the existing images (once stretch is gone).