Skip to content

Commit

Permalink
Refactor of environment variable usage
Browse files Browse the repository at this point in the history
- Removed prefixes from each image confd settings.
- Allows the use/overriding of settings defined by parent images.
- Shared database/jwt configuration.
- Conslidated common behavoir for importing/overriding variables.
- Conslidated common behavoir for database creation
- Updates to documentation
- Removed references to GEMINI
  • Loading branch information
nigelgbanks committed Mar 17, 2021
1 parent 18a6631 commit 4995819
Show file tree
Hide file tree
Showing 187 changed files with 1,651 additions and 1,244 deletions.
9 changes: 8 additions & 1 deletion .vscode/settings.json
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,14 @@
"composer": "shellscript"
},
"cSpell.words": [
"MODESHAPE",
"POSTGRESQL",
"getenv"
"SIGTERM",
"binarystorage",
"catchable",
"classpath",
"elif",
"getenv",
"nativeplatform"
]
}
77 changes: 39 additions & 38 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -161,14 +161,14 @@ The following docker images are provided:
- [alpaca](./alpaca/README.md)
- [base](./base/README.md)
- [blazegraph](./blazegraph/README.md)
- [build](./build/README.md)
- [cantaloupe](./cantaloupe/README.md)
- [crayfish](./crayfish/README.md)
- [crayfits](./crayfits/README.md)
- [demo](./demo/README.md)
- [drupal](./drupal/README.md)
- [fcrepo](./fcrepo/README.md)
- [fits](./fits/README.md)
- [gemini](./gemini/README.md)
- [handle](./handle/README.md)
- [homarus](./homarus/README.md)
- [houdini](./houdini/README.md)
- [hypercube](./hypercube/README.md)
Expand All @@ -179,8 +179,8 @@ The following docker images are provided:
- [matomo](./matomo/README.md)
- [milliner](./milliner/README.md)
- [nginx](./nginx/README.md)
- [postgresql](./postgresql/README.md)
- [recast](./recast/README.md)
- [demo](./demo/README.md)
- [solr](./solr/README.md)
- [tomcat](./tomcat/README.md)

Expand Down Expand Up @@ -214,55 +214,59 @@ folder ``rootfs/etc/confd`` that has the following layout:
./rootfs/etc/confd
├── conf.d
│ └── file.ext.toml
├── confd.toml
└── templates
└── file.ext.tmpl
```

``confd.toml`` Is the configuration of ``confd`` and will typically limit the
namespace from which ``confd`` will read key values. For example in ``activemq``:
The ``file.ext.toml`` and ``file.ext.tmpl`` work as a pair. The ``toml`` file
defines where the template will be render to and who owns it, and the ``tmpl``
file being the template in question. Ideally these files should match the same
name of the file they are generating minus the ``toml`` or ``tmpl`` suffix. This
is to make their discovery easier.

Additionally in the ``base`` image there is ``confd.toml`` which sets defaults
such a the ``log-level``:

```toml
backend = "env"
confdir = "/etc/confd"
log-level = "debug"
log-level = "error"
interval = 600
noop = false
prefix = "/activemq"
```

The prefix is set to ``/activemq`` which means only keys / value pairs under
this prefix can be used by templates. We restrict images by prefix to force them
to define their own settings, reducing dependencies between images, and to allow
for greater customization. For example you could have Gemini use PostgreSQL as a
backend and Drupal using MariaDB since they do not share the same Database
configuration.

The ``file.ext.toml`` and ``file.ext.tmpl`` work as a pair where the ``toml``
file defines where the template will be render to and who owns it, and the
``tmpl`` file being the template in question. Ideally these files should match
the same name of the file they are generating minus the ``toml`` or ``tmpl``
suffix. This is to make the discovery of them easier.

``confd`` is also the source of all truth when it comes to configuration. We've
established a order of precedence in which environment variables can be
provided.
``confd`` is also the source of all truth when it comes to configuration. We
have established a order of precedence in which environment variables at runtime
are defined.

1. Confd backend (highest)
2. Secrets kept in `/run/secrets`
2. Secrets kept in `/run/secrets` (Except when using ``Kubernetes``)
3. Environment variables passed into the container
4. Environment variables defined in Dockerfile(s)
5. Environment variables defined in the `/etc/defaults` directory (lowest only used for multiline variables, such as JWT)

If not defined in the highest level the next level applies and so forth down the
list.

`/etc/defaults` and the environment variables declared in the Dockerfile(s) used
to create the image are **required** to define all environment variables used by
scripts and Confd templates.
> N.B. `/etc/defaults` and the environment variables declared in the
> Dockerfile(s) used to create the image are **required** to define all
> environment variables used by scripts and ``confd`` templates. If not
> specified in either of those locations the environment variables will not be
> available even if its defined at a **higher** level i.e. ``confd``.
The logic which enforces these rules is performed in
[00-container-environment-00-init.sh](./base/rootfs/etc/cont-init.d/00-container-environment-00-init.sh)

> N.B Some containers derive environment variables dynamically from other
> environment variables. In these cases they are expected to provided an
> additional startup script prefixed with ``00-container-environment-01-*.sh``
> so that the variables are defined before ``confd`` is used to render
> templates.
``confd`` templates are **required** to use `getenv` function for all default
values to ensure this order of precedence is followed.
By either using the command ``with-contenv`` or starting a script with
``#!/usr/bin/with-contenv bash`` the environment defined will follow the order
of precedence above. Additionally Within ``confd`` templates it is **required**
to use `getenv` function for fetching data.

### S6 Overlay

Expand Down Expand Up @@ -334,7 +338,6 @@ are arranged in a hierarchy, that roughly follows below:
├── mariadb
└── nginx
├── crayfish
│ ├── gemini
│ ├── homarus
│ ├── houdini (consumes "imagemagick" as well during its build stage)
│ ├── hypercube
Expand Down Expand Up @@ -378,7 +381,7 @@ images.
## Design Constraints

To be able to support a wide variety of backends for ``confd``, as well as
orchestration tools, all calls to ``getv`` **must use getenv for the default
orchestration tools, all calls **must use getenv for the default
value**. With the exception of keys that do not get used unless defined like
``DRUPAL_SITE_{SITE}_NAME``. This means the whatever backend for configuration,
wether it be ``etcd``, ``consul``, or ``environment variables``, containers can
Expand All @@ -393,21 +396,19 @@ block until another container is available or a timeout has been reached. For
example:

```bash
local fcrepo_host="{{ getv "/fcrepo/host" "fcrepo.isle-dc.localhost" }}"
local fcrepo_port="{{ getv "/fcrepo/host" "80" }}"
local fcrepo_url=

# Indexing fails if port 80 is given explicitly.
if [[ "${fcrepo_port}" == "80" ]]; then
fcrepo_url="http://${fcrepo_host}/fcrepo/rest/"
if [[ "${DRUPAL_DEFAULT_FCREPO_PORT}" == "80" ]]; then
fcrepo_url="http://${DRUPAL_DEFAULT_FCREPO_HOST}/fcrepo/rest/"
else
fcrepo_url="http://${fcrepo_host}:${fcrepo_port}/fcrepo/rest/"
fcrepo_url="http://${DRUPAL_DEFAULT_FCREPO_HOST}:${DRUPAL_DEFAULT_FCREPO_PORT}/fcrepo/rest/"
fi

#...

# Need access to Solr before we can actually import the right config.
if timeout 300 wait-for-open-port.sh "${fcrepo_host}" "${fcrepo_port}" ; then
if timeout 300 wait-for-open-port.sh "${DRUPAL_DEFAULT_FCREPO_HOST}" "${DRUPAL_DEFAULT_FCREPO_PORT}" ; then
echo "Fcrepo Found"
else
echo "Could not connect to Fcrepo"
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,4 +4,4 @@ dest = "/opt/activemq/conf/credentials.properties"
uid = 100
gid = 1000
mode = "0640"
keys = [ "/user", "/password" ]
keys = [ "/" ]
2 changes: 1 addition & 1 deletion activemq/rootfs/etc/confd/conf.d/groups.properties.toml
Original file line number Diff line number Diff line change
Expand Up @@ -4,4 +4,4 @@ dest = "/opt/activemq/conf/groups.properties"
uid = 100
gid = 1000
mode = "0640"
keys = [ "/group" ]
keys = [ "/" ]
Original file line number Diff line number Diff line change
Expand Up @@ -4,4 +4,4 @@ dest = "/opt/activemq/conf/jetty-realm.properties"
uid = 100
gid = 1000
mode = "0640"
keys = [ "/web/admin", "/web/user" ]
keys = [ "/" ]
2 changes: 1 addition & 1 deletion activemq/rootfs/etc/confd/conf.d/users.properties.toml
Original file line number Diff line number Diff line change
Expand Up @@ -4,4 +4,4 @@ dest = "/opt/activemq/conf/users.properties"
uid = 100
gid = 1000
mode = "0640"
keys = [ "/user" ]
keys = [ "/" ]
6 changes: 0 additions & 6 deletions activemq/rootfs/etc/confd/confd.toml

This file was deleted.

Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
# Defines credentials that will be used by components (like web console) to access the broker
activemq.username={{ getv "/user" (getenv "ACTIVEMQ_USER") }}
activemq.password={{ getv "/password" (getenv "ACTIVEMQ_PASSWORD") }}
activemq.username={{ getenv "ACTIVEMQ_USER" }}
activemq.password={{ getenv "ACTIVEMQ_PASSWORD" }}
2 changes: 1 addition & 1 deletion activemq/rootfs/etc/confd/templates/groups.properties.tmpl
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Defines groups and the users that belong to them.
# group=user[,user ...]
{{ range $dir := lsdir "/group" }}{{ getv (printf "/group/%s/name" $dir) }}={{ getv (printf "/group/%s/members" $dir) }}
{{ range $dir := lsdir "/activemq/group" }}{{ getv (printf "/activemq/group/%s/name" $dir) }}={{ getv (printf "/activemq/group/%s/members" $dir) }}
{{ end }}
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# Defines users that can access the web (console, demo, etc.)
# username: password [,rolename ...]
{{ getv "/web/admin/name" (getenv "ACTIVEMQ_WEB_ADMIN_NAME") }}: {{ getv "/web/admin/password" (getenv "ACTIVEMQ_WEB_ADMIN_PASSWORD") }}, {{ getv "/web/admin/roles" (getenv "ACTIVEMQ_WEB_ADMIN_ROLES") }}
{{ range $dir := lsdir "/web/user" }}{{ getv (printf "/web/user/%s/name" $dir) }}: {{ getv (printf "/web/user/%s/password" $dir) }}, {{ getv (printf "/web/user/%s/roles" $dir) }}
{{ getenv "ACTIVEMQ_WEB_ADMIN_NAME" }}: {{ getenv "ACTIVEMQ_WEB_ADMIN_PASSWORD" }}, {{ getenv "ACTIVEMQ_WEB_ADMIN_ROLES" }}
{{ range $dir := lsdir "/activemq/web/user" }}{{ getv (printf "/activemq/web/user/%s/name" $dir) }}: {{ getv (printf "/activemq/web/user/%s/password" $dir) }}, {{ getv (printf "/activemq/web/user/%s/roles" $dir) }}
{{ end }}
2 changes: 1 addition & 1 deletion activemq/rootfs/etc/confd/templates/users.properties.tmpl
Original file line number Diff line number Diff line change
@@ -1,2 +1,2 @@
{{ range $dir := lsdir "/user" }}{{ getv (printf "/user/%s/name" $dir) }}={{ getv (printf "/user/%s/password" $dir) }}
{{ range $dir := lsdir "/activemq/user" }}{{ getv (printf "/activemq/user/%s/name" $dir) }}={{ getv (printf "/activemq/user/%s/password" $dir) }}
{{ end }}
Original file line number Diff line number Diff line change
Expand Up @@ -4,4 +4,4 @@ dest = "/opt/karaf/deploy/ca.islandora.alpaca.connector.fits.blueprint.xml"
uid = 100
gid = 1000
mode = "0644"
keys = [ "/fits" ]
keys = [ "/" ]
Original file line number Diff line number Diff line change
Expand Up @@ -4,4 +4,4 @@ dest = "/opt/karaf/deploy/ca.islandora.alpaca.connector.homarus.blueprint.xml"
uid = 100
gid = 1000
mode = "0644"
keys = [ "/homarus" ]
keys = [ "/" ]
Original file line number Diff line number Diff line change
Expand Up @@ -4,4 +4,4 @@ dest = "/opt/karaf/deploy/ca.islandora.alpaca.connector.houdini.blueprint.xml"
uid = 100
gid = 1000
mode = "0644"
keys = [ "/houdini" ]
keys = [ "/" ]
Original file line number Diff line number Diff line change
Expand Up @@ -4,4 +4,4 @@ dest = "/opt/karaf/deploy/ca.islandora.alpaca.connector.ocr.blueprint.xml"
uid = 100
gid = 1000
mode = "0644"
keys = [ "/ocr" ]
keys = [ "/" ]
Original file line number Diff line number Diff line change
Expand Up @@ -4,4 +4,4 @@ dest = "/opt/karaf/etc/ca.islandora.alpaca.http.client.cfg"
uid = 100
gid = 1000
mode = "0644"
keys = [ "/http" ]
keys = [ "/" ]
Original file line number Diff line number Diff line change
Expand Up @@ -4,4 +4,4 @@ dest = "/opt/karaf/etc/ca.islandora.alpaca.indexing.fcrepo.cfg"
uid = 100
gid = 1000
mode = "0644"
keys = [ "/indexing" ]
keys = [ "/" ]
Original file line number Diff line number Diff line change
Expand Up @@ -4,4 +4,4 @@ dest = "/opt/karaf/etc/ca.islandora.alpaca.indexing.triplestore.cfg"
uid = 100
gid = 1000
mode = "0644"
keys = [ "/indexing" ]
keys = [ "/" ]
Original file line number Diff line number Diff line change
Expand Up @@ -4,4 +4,4 @@ dest = "/opt/karaf/etc/org.fcrepo.camel.indexing.triplestore.cfg"
uid = 100
gid = 1000
mode = "0644"
keys = [ "/indexing" ]
keys = [ "/" ]
Original file line number Diff line number Diff line change
Expand Up @@ -4,4 +4,4 @@ dest = "/opt/karaf/etc/org.fcrepo.camel.service.activemq.cfg"
uid = 100
gid = 1000
mode = "0644"
keys = [ "/activemq" ]
keys = [ "/" ]
Original file line number Diff line number Diff line change
Expand Up @@ -4,4 +4,4 @@ dest = "/opt/karaf/etc/org.fcrepo.camel.service.cfg"
uid = 100
gid = 1000
mode = "0644"
keys = [ "/fcrepo" ]
keys = [ "/" ]
Original file line number Diff line number Diff line change
Expand Up @@ -4,4 +4,4 @@ dest = "/opt/karaf/etc/org.ops4j.pax.logging.cfg"
uid = 100
gid = 1000
mode = "0644"
keys = [ "/log" ]
keys = [ "/" ]
6 changes: 0 additions & 6 deletions alpaca/rootfs/etc/confd/confd.toml

This file was deleted.

Original file line number Diff line number Diff line change
Expand Up @@ -10,9 +10,9 @@

<cm:property-placeholder id="properties" persistent-id="ca.islandora.alpaca.connector.fits" update-strategy="reload" >
<cm:default-properties>
<cm:property name="error.maxRedeliveries" value="{{ getv "/fits/redeliveries" (getenv "ALPACA_FITS_REDELIVERIES") }}"/>
<cm:property name="in.stream" value="{{ getv "/fits/queue" (getenv "ALPACA_FITS_QUEUE") }}"/>
<cm:property name="derivative.service.url" value="{{ getv "/fits/service" (getenv "ALPACA_FITS_SERVICE") }}"/>
<cm:property name="error.maxRedeliveries" value="{{ getenv "ALPACA_FITS_REDELIVERIES" }}"/>
<cm:property name="in.stream" value="{{ getenv "ALPACA_FITS_QUEUE" }}"/>
<cm:property name="derivative.service.url" value="{{ getenv "ALPACA_FITS_SERVICE" }}"/>
</cm:default-properties>
</cm:property-placeholder>

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -10,9 +10,9 @@

<cm:property-placeholder id="properties" persistent-id="ca.islandora.alpaca.connector.homarus" update-strategy="reload" >
<cm:default-properties>
<cm:property name="error.maxRedeliveries" value="{{ getv "/homarus/redeliveries" (getenv "ALPACA_HOMARUS_REDELIVERIES") }}"/>
<cm:property name="in.stream" value="{{ getv "/homarus/queue" (getenv "ALPACA_HOMARUS_QUEUE") }}"/>
<cm:property name="derivative.service.url" value="{{ getv "/homarus/service" (getenv "ALPACA_HOMARUS_SERVICE") }}"/>
<cm:property name="error.maxRedeliveries" value="{{ getenv "ALPACA_HOMARUS_REDELIVERIES" }}"/>
<cm:property name="in.stream" value="{{ getenv "ALPACA_HOMARUS_QUEUE" }}"/>
<cm:property name="derivative.service.url" value="{{ getenv "ALPACA_HOMARUS_SERVICE" }}"/>
</cm:default-properties>
</cm:property-placeholder>

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -10,9 +10,9 @@

<cm:property-placeholder id="properties" persistent-id="ca.islandora.alpaca.connector.houdini" update-strategy="reload" >
<cm:default-properties>
<cm:property name="error.maxRedeliveries" value="{{ getv "/houdini/redeliveries" (getenv "ALPACA_HOUDINI_REDELIVERIES") }}"/>
<cm:property name="in.stream" value="{{ getv "/houdini/queue" (getenv "ALPACA_HOUDINI_QUEUE") }}"/>
<cm:property name="derivative.service.url" value="{{ getv "/houdini/service" (getenv "ALPACA_HOUDINI_SERVICE") }}"/>
<cm:property name="error.maxRedeliveries" value="{{ getenv "ALPACA_HOUDINI_REDELIVERIES" }}"/>
<cm:property name="in.stream" value="{{ getenv "ALPACA_HOUDINI_QUEUE" }}"/>
<cm:property name="derivative.service.url" value="{{ getenv "ALPACA_HOUDINI_SERVICE" }}"/>
</cm:default-properties>
</cm:property-placeholder>

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -10,9 +10,9 @@

<cm:property-placeholder id="properties" persistent-id="ca.islandora.alpaca.connector.ocr" update-strategy="reload" >
<cm:default-properties>
<cm:property name="error.maxRedeliveries" value="{{ getv "/ocr/redeliveries" (getenv "ALPACA_OCR_SERVICE") }}"/>
<cm:property name="in.stream" value="{{ getv "/ocr/queue" (getenv "ALPACA_OCR_QUEUE") }}"/>
<cm:property name="derivative.service.url" value="{{ getv "/ocr/service" (getenv "ALPACA_OCR_SERVICE") }}"/>
<cm:property name="error.maxRedeliveries" value="{{ getenv "ALPACA_OCR_SERVICE" }}"/>
<cm:property name="in.stream" value="{{ getenv "ALPACA_OCR_QUEUE" }}"/>
<cm:property name="derivative.service.url" value="{{ getenv "ALPACA_OCR_SERVICE" }}"/>
</cm:default-properties>
</cm:property-placeholder>

Expand Down
Original file line number Diff line number Diff line change
@@ -1 +1 @@
token.value={{ getv "/http/token" (getenv "ALPACA_HTTP_TOKEN") }}
token.value={{ getenv "ALPACA_HTTP_TOKEN" }}
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
error.maxRedeliveries={{ getv "/indexing/redeliveries" (getenv "ALPACA_INDEXING_REDELIVERIES") }}
file.external.stream={{ getv "/indexing/stream/file/external" (getenv "ALPACA_INDEXING_STREAM_FILE_EXTERNAL") }}
file.delete.stream={{ getv "/indexing/stream/file/delete" (getenv "ALPACA_INDEXING_STREAM_FILE_DELETE") }}
file.stream={{ getv "/indexing/stream/file/index" (getenv "ALPACA_INDEXING_STREAM_FILE_INDEX") }}
media.stream={{ getv "/indexing/stream/media/index" (getenv "ALPACA_INDEXING_STREAM_MEDIA_INDEX") }}
milliner.baseUrl={{ getv "/indexing/milliner/url" (getenv "ALPACA_INDEXING_MILLINER_URL") }}
node.delete.stream={{ getv "/indexing/stream/node/delete" (getenv "ALPACA_INDEXING_STREAM_NODE_DELETE") }}
node.stream={{ getv "/indexing/stream/node/index" (getenv "ALPACA_INDEXING_STREAM_NODE_INDEX") }}
error.maxRedeliveries={{ getenv "ALPACA_INDEXING_REDELIVERIES" }}
file.external.stream={{ getenv "ALPACA_INDEXING_STREAM_FILE_EXTERNAL" }}
file.delete.stream={{ getenv "ALPACA_INDEXING_STREAM_FILE_DELETE" }}
file.stream={{ getenv "ALPACA_INDEXING_STREAM_FILE_INDEX" }}
media.stream={{ getenv "ALPACA_INDEXING_STREAM_MEDIA_INDEX" }}
milliner.baseUrl={{ getenv "ALPACA_INDEXING_MILLINER_URL" }}
node.delete.stream={{ getenv "ALPACA_INDEXING_STREAM_NODE_DELETE" }}
node.stream={{ getenv "ALPACA_INDEXING_STREAM_NODE_INDEX" }}
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
index.stream={{ getv "/indexing/stream/triplestore/index" (getenv "ALPACA_INDEXING_STREAM_TRIPLESTORE_INDEX") }}
delete.stream={{ getv "/indexing/stream/triplestore/delete" (getenv "ALPACA_INDEXING_STREAM_TRIPLESTORE_DELETE") }}
triplestore.baseUrl={{ getv "/indexing/url" (getenv "ALPACA_INDEXING_URL") }}
error.maxRedeliveries={{ getv "/indexing/redeliveries" (getenv "ALPACA_INDEXING_REDELIVERIES") }}
index.stream={{ getenv "ALPACA_INDEXING_STREAM_TRIPLESTORE_INDEX" }}
delete.stream={{ getenv "ALPACA_INDEXING_STREAM_TRIPLESTORE_DELETE" }}
triplestore.baseUrl={{ getenv "ALPACA_INDEXING_URL" }}
error.maxRedeliveries={{ getenv "ALPACA_INDEXING_REDELIVERIES" }}
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
input.stream={{ getv "/indexing/stream/input" (getenv "ALPACA_INDEXING_STREAM_INPUT") }}
triplestore.baseUrl={{ getv "/indexing/url" (getenv "ALPACA_INDEXING_URL") }}
triplestore.reindex.stream={{ getv "/indexing/stream/triplestore/reindex" (getenv "ALPACA_INDEXING_STREAM_TRIPLESTORE_REINDEX") }}
input.stream={{ getenv "ALPACA_INDEXING_STREAM_INPUT" }}
triplestore.baseUrl={{ getenv "ALPACA_INDEXING_URL" }}
triplestore.reindex.stream={{ getenv "ALPACA_INDEXING_STREAM_TRIPLESTORE_REINDEX" }}
Loading

0 comments on commit 4995819

Please sign in to comment.