Use the version 0.0.4 of the build plugin

Drops support for non-buildkit builds. Adds the following: - multi-arch builds (amd64, arm64, etc) - custom build of confd to support arm64 - unique build caches per image (should reduce cache misses) - fixes fits (mediainfo, exifdata, file info) - update gradle version to latest (match plugin) - child images of tomcat can override tomcat properties - child images of nginx can override php/nginx properties - updated to use latest alpine 3.13 - php is no 7.4 - imagemagick only builds what is required (no longer make clibs, etc) - general refactors
Islandora-Devops · Mar 23, 2021 · f429178 · f429178
1 parent 4995819
commit f429178
Show file tree

Hide file tree

Showing 65 changed files with 1,270 additions and 373 deletions.
diff --git a/.github/workflows/pr.yml b/.github/workflows/pr.yml
@@ -7,7 +7,7 @@ jobs:
   build:
     name: Build and Test PR 
     runs-on: ubuntu-latest
-    timeout-minutes: 30
+    timeout-minutes: 60
     steps:
     - name: Checkout
       uses: actions/checkout@v2
@@ -22,11 +22,7 @@ jobs:
         key: ${{ runner.os }}-gradle-${{ hashFiles('**/*.gradle*') }}
         restore-keys: |
           ${{ runner.os }}-gradle-
-    - name: Enable buildkit
-      shell: bash
-      run: |
-        echo '{"experimental": "enabled"}' > ~/.docker/config.json
     - name: Build Docker images
       uses: eskatos/gradle-command-action@v1
       with:
-        arguments: build --info
+        arguments: build -Pdocker.driver=docker-container -Pdocker.platforms=linux/amd64,linux/arm64 --info
diff --git a/.github/workflows/push.yml b/.github/workflows/push.yml
@@ -20,20 +20,15 @@ jobs:
         key: ${{ runner.os }}-gradle-${{ hashFiles('**/*.gradle*') }}
         restore-keys: |
           ${{ runner.os }}-gradle-
-    - name: Log in to Docker Hub
-      run: docker login -u '${{ secrets.REGISTRY_USER }}' -p '${{ secrets.REGISTRY_PASS }}' '${{ secrets.REGISTRY_URL }}'
-    - name: Enable buildkit
-      shell: bash
-      run: |
-        echo '{"experimental": "enabled"}' > ~/.docker/config.json
+    - name: Login to Docker Hub
+      uses: docker/login-action@v1
+      with:
+        username: ${{ secrets.REGISTRY_USER }}
+        password: ${{ secrets.REGISTRY_PASS }}
     - name: Set TAG Environment Variable
       run: |
         echo "TAG=$GITHUB_SHA" >> $GITHUB_ENV
-    - name: Build Docker images
-      uses: eskatos/gradle-command-action@v1
-      with:
-        arguments: build '-Prepository=${{ secrets.REPOSITORY }}' --info
-    - name: Push Docker images
+    - name: Build and Push Docker images
       uses: eskatos/gradle-command-action@v1
       with:
-        arguments: push '-Ptags=${{ env.TAG }}' '-Prepository=${{ secrets.REPOSITORY }}' '-PregistryUrl=${{ secrets.REGISTRY_URL }}' '-PregistryUsername=${{ secrets.REGISTRY_USER }}' '-PregistryPassword=${{ secrets.REGISTRY_PASS }}' --info
+        arguments: push '-Pdocker.tags=${{ env.TAG }}' '-Pdocker.repository=${{ secrets.REPOSITORY }}' -Pdocker.driver=docker-container -Pdocker.cacheTo=true -Pdocker.platforms=linux/amd64,linux/arm64 --info
diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml
@@ -21,22 +21,17 @@ jobs:
         key: ${{ runner.os }}-gradle-${{ hashFiles('**/*.gradle*') }}
         restore-keys: |
           ${{ runner.os }}-gradle-
-    - name: Log in to Docker Hub
-      run: docker login -u '${{ secrets.REGISTRY_USER }}' -p '${{ secrets.REGISTRY_PASS }}' '${{ secrets.REGISTRY_URL }}'
-    - name: Enable buildkit
-      shell: bash
-      run: |
-        echo '{"experimental": "enabled"}' > ~/.docker/config.json
+    - name: Login to Docker Hub
+      uses: docker/login-action@v1
+      with:
+        username: ${{ secrets.REGISTRY_USER }}
+        password: ${{ secrets.REGISTRY_PASS }}
     - name: Set TAG Environment Variable
       run: |
         export TAG=$(echo "${GITHUB_REF}" | cut -d "/" -f3)
         echo "TAG=$TAG" >> $GITHUB_ENV
-    - name: Build Docker images
-      uses: eskatos/gradle-command-action@v1
-      with:
-        arguments: build '-Ptags=${{ env.TAG }}' '-Prepository=${{ secrets.REPOSITORY }}' --info
     - name: Push Docker images
       uses: eskatos/gradle-command-action@v1
       with:
-        arguments: push '-Ptags=${{ env.TAG }}' '-Prepository=${{ secrets.REPOSITORY }}' '-PregistryUrl=${{ secrets.REGISTRY_URL }}' '-PregistryUsername=${{ secrets.REGISTRY_USER }}' '-PregistryPassword=${{ secrets.REGISTRY_PASS }}' --info
+        arguments: push '-Pdocker.tags=${{ env.TAG }}' '-Pdocker.repository=${{ secrets.REPOSITORY }}' -Pdocker.driver=docker-container -Pdocker.cacheTo=true -Pdocker.platforms=linux/amd64,linux/arm64 --info
     # @todo add tests.
diff --git a/README.md b/README.md
@@ -7,7 +7,6 @@
 - [Requirements](#requirements)
 - [Building](#building)
   - [Build All Images](#build-all-images)
-  - [Building without Buildkit](#building-without-buildkit)
   - [Build Specific Image](#build-specific-image)
   - [Building Continuously](#building-continuously)
 - [Running](#running)
@@ -28,13 +27,13 @@ Islandora 8 site.
 
 ## Requirements
 
-To build the Docker images using the provided Gradle build scripts with [BuildKit] requires:
+To build the Docker images using the provided Gradle build scripts requires:
 
-- [Docker 18.09+](https://docs.docker.com/get-docker/)
+- [Docker 19.03+](https://docs.docker.com/get-docker/)
 - [OpenJDK or Oracle JDK 8+](https://www.java.com/en/download/)
 
-That being said the images themselves are compatible of running with older
-versions of Docker.
+That being said the images themselves are compatible with older versions of
+Docker.
 
 ## Building
 
@@ -114,15 +113,6 @@ The following will build all the images in the correct order.
 ./gradlew build
 ```
 
-### Building without Buildkit
-
-If you are having trouble building, consider building without BuildKit as it's
-supported by older versions of Docker.
-
-```bash
-./gradlew build -PuseBuildKit=false
-```
-
 ### Build Specific Image
 
 To build a specific image and it's dependencies, for example
@@ -370,13 +360,27 @@ detect which folders should be considered
 what dependencies exist between them. The only caveat is
 that the projects cannot be nested, though that use case does not really apply.
 
-The dependencies are resolved by parsing the Dockerfile and looking for ``FROM``
-statements to determine which images are required to build it.
+The dependencies are resolved by parsing the Dockerfile and looking for:
+
+- ``FROM``statements
+- ``--mount=type=bind`` statements
+- ``COPY --from`` statements
+
+As they are capable of referring to other images.
 
 This means to add a new Docker image to the project you do not need to modify
 the build scripts, simply add a new folder and place your Dockerfile inside of
-it and it will be discovered built in the correct order relative to the other
-images.
+it. It will be discovered and built in the correct order relative to the other
+images assuming you refer to the other image using the `repository` build
+argument.
+
+For example:
+
+```Dockerfile
+ARG repository=local
+ARG tag=latest
+FROM ${repository}/base:${tag}
+```
 
 ## Design Constraints
 
@@ -426,8 +430,7 @@ This allows container to start up in any order, and to be orchestrated by any to
 failed to solve with frontend dockerfile.v0: failed to solve with frontend gateway.v0: runc did not terminate successfully: context canceled
 ```
 
-**Answer:** If possible upgrade Docker to the latest version, and switch to using the
-[Overlay2](https://docs.docker.com/storage/storagedriver/overlayfs-driver/#configure-docker-with-the-overlay-or-overlay2-storage-driver)
-filesystem with Docker. If that doesn't work trying building [without BuildKit](#building-without-buildkit).
+**Answer:** If possible upgrade Docker to the latest version, and switch to
+using the [Overlay2] filesystem with Docker.
 
-[Buildkit]: https://github.com/moby/buildkit/blob/main/frontend/dockerfile/docs/experimental.md
+[Overlay2]: https://docs.docker.com/storage/storagedriver/overlayfs-driver#configure-docker-with-the-overlay-or-overlay2-storage-driver
diff --git a/abuild/Dockerfile b/abuild/Dockerfile
@@ -1,12 +1,17 @@
 # syntax=docker/dockerfile:1.2.1
-FROM alpine:3.11.6
+ARG repository=local
+ARG tag=latest
+FROM ${repository}/download:${tag} AS download
+FROM alpine:3.13.2 AS cache
+FROM alpine:3.13.2
 
-RUN --mount=type=cache,target=/var/cache/apk \ 
-    --mount=type=cache,target=/etc/cache/apk \
+RUN --mount=type=cache,id=abuild-apk,sharing=locked,from=cache,target=/var/cache/apk \
     ln -s /var/cache/apk /etc/apk/cache && \
-    apk add --update \
+    apk add \
         alpine-sdk \
-        coreutils \
+        sudo \
     && \
     adduser -G abuild -g "Alpine Package Builder" -s /bin/ash -D builder && \
     echo "builder ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers
+
+COPY --from=download /usr/local/bin/*.sh /usr/local/bin
diff --git a/abuild/README.md b/abuild/README.md
@@ -24,9 +24,8 @@ Define a docker file that:
 FROM islandora/abuild:latest
 
 # Include packages required for building the package (not necessarily the ones require for running).
-RUN --mount=type=cache,target=/var/cache/apk \
-    --mount=type=cache,target=/etc/cache/apk \
-    apk --update add \
+RUN --mount=type=cache,id=base-apk,sharing=locked,target=/var/cache/apk \
+    apk-install.sh \
         package-require-for-building-1 \
         package-require-for-building-2 \
 

diff --git a/activemq/Dockerfile b/activemq/Dockerfile
@@ -1,8 +1,9 @@
 # syntax=docker/dockerfile:1.2.1
-FROM local/java:latest
+ARG repository=local
+ARG tag=latest
+FROM --platform=$BUILDPLATFORM ${repository}/download:${tag} AS download
 
-RUN --mount=id=downloads,type=cache,target=/opt/downloads \
-    DOWNLOAD_CACHE_DIRECTORY="/opt/downloads" && \
+RUN --mount=type=cache,id=activemq-downloads,sharing=locked,target=/opt/downloads \
     ACTIVEMQ_VERSION="5.14.5" && \
     ACTIVEMQ_FILE="apache-activemq-${ACTIVEMQ_VERSION}-bin.tar.gz" && \
     ACTIVEMQ_URL="https://archive.apache.org/dist/activemq/${ACTIVEMQ_VERSION}/${ACTIVEMQ_FILE}" && \
@@ -16,7 +17,7 @@ RUN --mount=id=downloads,type=cache,target=/opt/downloads \
         --file "${DOWNLOAD_CACHE_DIRECTORY}/${ACTIVEMQ_FILE}" \
         examples webapps-demo docs
 
-WORKDIR /opt/activemq
+FROM ${repository}/java:${tag}
 
 EXPOSE 61616 5672 61613 1883 61614 8161
 
@@ -28,4 +29,11 @@ ENV \
     ACTIVEMQ_WEB_ADMIN_PASSWORD=password \
     ACTIVEMQ_WEB_ADMIN_ROLES=admin
 
+COPY --from=download /etc/group /etc/group
+COPY --from=download /etc/passwd /etc/passwd
+COPY --from=download /etc/shadow /etc/shadow
+COPY --from=download --chown=activemq:activemq /opt/activemq /opt/activemq
+
 COPY rootfs /
+
+WORKDIR /opt/activemq
diff --git a/alpaca/Dockerfile b/alpaca/Dockerfile
@@ -1,5 +1,7 @@
 # syntax=docker/dockerfile:1.2.1
-FROM local/karaf:latest
+ARG repository=local
+ARG tag=latest
+FROM --platform=$BUILDPLATFORM ${repository}/karaf:${tag} AS download
 
 # Install common features and repos
 RUN bin/start && \
@@ -9,21 +11,21 @@ RUN bin/start && \
     bin/client -r 10 -d 5  "feature:repo-add mvn:ca.islandora.alpaca/islandora-karaf/${ALPACA_VERSION}/xml/features" && \
     bin/client -r 10 -d 5  "feature:repo-add mvn:org.apache.activemq/activemq-karaf/${ACTIVEMQ_VERSION}/xml/features" && \
     bin/client -r 10 -d 5  "feature:repo-add mvn:org.apache.camel.karaf/apache-camel/${CAMEL_VERSION}/xml/features" && \
-    bin/client -r 10 -d 5  "feature:install fcrepo-service-activemq" && \
-    bin/client -r 10 -d 5  "feature:install fcrepo-service-camel" && \
-    bin/client -r 10 -d 5  "feature:install islandora-http-client" && \
+    bin/client -r 10 -d 5  "feature:install -v fcrepo-service-activemq" && \
+    bin/client -r 10 -d 5  "feature:install -v fcrepo-service-camel" && \
+    bin/client -r 10 -d 5  "feature:install -v islandora-http-client" && \
     bin/stop && \
     rm -rf instances/*
 
 # Derivative connector
 RUN bin/start && \
-    bin/client -r 10 -d 5  "feature:install islandora-connector-derivative" && \
+    bin/client -r 10 -d 5  "feature:install -v islandora-connector-derivative" && \
     bin/stop && \
     rm -rf instances/*
 
 # Fcrepo indexing
 RUN bin/start && \
-    bin/client -r 10 -d 5  "feature:install islandora-indexing-fcrepo" && \
+    bin/client -r 10 -d 5  "feature:install -v islandora-indexing-fcrepo" && \
     bin/stop && \
     rm -rf instances/*
 
@@ -33,8 +35,11 @@ RUN bin/start && \
     bin/stop && \
     rm -rf instances/*
 
+# Fix up ownership.
 RUN chown -R karaf:karaf /opt/karaf
 
+FROM ${repository}/karaf:${tag}
+
 ENV \
     ALPACA_ACTIVEMQ_PASSWORD=password \
     ALPACA_ACTIVEMQ_URL=tcp://activemq:61616 \
@@ -74,4 +79,7 @@ ENV \
     ALPACA_OCR_SERVICE=http://hypercube:8000 \
     JAVA_OPTS="-Dfile.encoding=UTF-8 -Dnet.sf.ehcache.skipUpdateCheck=true -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+UseParNewGC -XX:MaxPermSize=128m -Xms512m -Xmx8g"
 
+COPY --from=download --chown=karaf:karaf /opt/karaf /opt/karaf
+COPY --from=download /root/.m2 /root/.m2
+
 COPY rootfs /
diff --git a/base/Dockerfile b/base/Dockerfile
@@ -1,15 +1,16 @@
 # syntax=docker/dockerfile:1.2.1
-FROM alpine:3.11.6
+ARG repository=local
+ARG tag=latest
+FROM ${repository}/download:${tag} AS download
+FROM alpine:3.13.2 AS cache
+FROM alpine:3.13.2
 
-COPY build/ /usr/local/bin/
+ENV DOWNLOAD_CACHE_DIRECTORY=/opt/downloads
 
 # Install packages and tools required by all downstream images.
-RUN --mount=type=cache,target=/var/cache/apk \ 
-    --mount=type=cache,target=/etc/cache/apk \
-    --mount=id=downloads,type=cache,target=/opt/downloads \
-    DOWNLOAD_CACHE_DIRECTORY="/opt/downloads" && \
+RUN --mount=type=cache,id=base-apk,sharing=locked,from=cache,target=/var/cache/apk \
     ln -s /var/cache/apk /etc/apk/cache && \
-    apk add --update \
+    apk add \
         bash \
         curl \
         git \
@@ -22,24 +23,36 @@ RUN --mount=type=cache,target=/var/cache/apk \
         wget \
     && \
     addgroup -g 2000 jwt && \
-    S6_VERSION="1.22.1.0" && \
-    S6_FILE="s6-overlay-amd64.tar.gz" && \
+    echo '' > /root/.ash_history
+
+ARG TARGETARCH
+
+COPY --from=download /usr/local/bin/*.sh /usr/local/bin
+
+# Install s6
+RUN --mount=type=cache,id=base-downloads,sharing=locked,target=/opt/downloads \
+    if [ "${TARGETARCH}" = "arm64" ]; then \
+        S6_FILE="s6-overlay-aarch64.tar.gz"; \
+        S6_SHA256="84f585a100b610124bb80e441ef2dc2d68ac2c345fd393d75a6293e0951ccfc5"; \
+        S6_SIG_SHA256="90b241a1a8e525c431fefcbbb0bbf90c5cc57b5496d718c441ac56265ede4817"; \
+    fi; \
+    if [ "${TARGETARCH}" = "amd64" ]; then \
+        S6_FILE="s6-overlay-amd64.tar.gz"; \
+        S6_SHA256="a7076cf205b331e9f8479bbb09d9df77dbb5cd8f7d12e9b74920902e0c16dd98"; \
+        S6_SIG_SHA256="a0f2e744893361fbf9958dca4b6df0363851dda5ce5e91dfe85e4c2a253541c7"; \
+    fi; \
+    S6_VERSION="2.2.0.3" && \
     S6_URL="https://github.com/just-containers/s6-overlay/releases/download/v${S6_VERSION}/${S6_FILE}" && \
-    S6_SHA256="73f9779203310ddf9c5132546a1978e1a2b05990263b92ed2c34c1e258e2df6c" && \
-    S6_SIG_SHA256="7e9c33f45bca1f89b3a1702175a4109e99c911e47ae9de62fbec013406db2b01" && \
     download.sh --url "${S6_URL}" --sha256 "${S6_SHA256}" "${DOWNLOAD_CACHE_DIRECTORY}" && \
     download.sh --url "${S6_URL}.sig" --sha256 "${S6_SIG_SHA256}" "${DOWNLOAD_CACHE_DIRECTORY}" && \
     gpg-receive-keys.sh --key 2536CA16DF4FCDA2 && \
     gpg "${DOWNLOAD_CACHE_DIRECTORY}/${S6_FILE}.sig" && \
-    tar -xzf "${DOWNLOAD_CACHE_DIRECTORY}/${S6_FILE}" -C / && \
-    CONFD_VERSION="0.15.0" && \
-    CONFD_URL="https://github.com/kelseyhightower/confd/releases/download/v${CONFD_VERSION}/confd-${CONFD_VERSION}-linux-amd64" && \
-    CONFD_SHA256="7f3aba1d803543dd1df3944d014f055112cf8dadf0a583c76dd5f46578ebe3c2" && \
-    download.sh --url "${CONFD_URL}" --sha256 "${CONFD_SHA256}" "${DOWNLOAD_CACHE_DIRECTORY}" && \
-    cp "${DOWNLOAD_CACHE_DIRECTORY}/confd-${CONFD_VERSION}-linux-amd64" /usr/local/bin/confd && \
-    chmod a+x /usr/local/bin/confd && \
+    tar -xzf "${DOWNLOAD_CACHE_DIRECTORY}/${S6_FILE}" -C /  && \
     echo '' > /root/.ash_history
 
+# Install confd
+COPY confd/confd-0.15.0-linux-${TARGETARCH} /usr/local/bin/confd
+
 # Start s6
 ENTRYPOINT [ "/init" ]
 

diff --git a/base/README.md b/base/README.md
@@ -7,7 +7,7 @@ It's based off off [Alpine Linux], and includes [s6 overlay] and [confd].
 
 ## Dependencies
 
-Requires `alpine:3.11.6`
+Requires `alpine:3.13.2`
 
 ## Settings
 

diff --git a/base/confd/confd-0.15.0-linux-amd64 b/base/confd/confd-0.15.0-linux-amd64
diff --git a/base/confd/confd-0.15.0-linux-arm64 b/base/confd/confd-0.15.0-linux-arm64
diff --git a/base/rootfs/sbin/apk-install.sh b/base/rootfs/sbin/apk-install.sh
diff --git a/base/rootfs/sbin/apk-uninstall.sh b/base/rootfs/sbin/apk-uninstall.sh