K8s #77
Merged
K8s #77
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
michaelweiser
force-pushed
the
k8s
branch
5 times, most recently
from
e2f6687
to
f1c17ba
Compare
michaelweiser
force-pushed
the
k8s
branch
5 times, most recently
from
1a0b5e5
to
922837a
Compare
|
This can now be updated to use the published images from our scvenus/peekabooav ghcr repo. Bear with me. |
Merged
michaelweiser
force-pushed
the
k8s
branch
3 times, most recently
from
e88585e
to
50592f1
Compare
This commits my current progress in re-creating the docker-compose pipeline for PeekabooAV in Kubernetes. That includes deployments, services, and hard-coded config files for each step in the pipeline, modeled after what was done in the [pipeline](/Sett17/PeekabooAV-Installer/tree/pipeline). The yamls for cortex, and the set-up job, are included, although cortex does currently not work inside of Kubernetes, due to the missing docker runner. Except for above mentioned cortex, the pipeline is fully functional. Meaning one can send an email to the postfix_tx deployment, which is then sent to the postfix_rx deployment and then processed by rspamd and Peekaboo. This was tested and developed with microk8s and a single node.
Switch to pre-build images from Docker Hub, particularly one for Cortex that does contain the kubernetes job runner from a pending PR (link not given here as to not spam it when pushing this commit multiple times). Add the required shared persistent volume claim to the specs and configs. Together with a service account that is allowed to manage jobs this allows to run analyses on k8s.
Move the cortex-setup job into its down helm chart. The skeleton was created using helm create and then turned from a deployment into a job.
Add a Helm chart for PeekabooAV. Skeleton was created using helm create. The deployment was extended to mount drop files from a custom config map and secret. These can be populated from values.yaml by placing sections and options into dicts config, analyzers and ruleset. Drop-files support form scVENUS/PeekabooAV#212 is necessary for that to work. A pre-built image on ghcr.io is configured via its shasum for now.
Add a simple helm chart to run our PeeabooAV-specific rspamd image on k8s. Skeleton was created using helm create. A pre-built image on ghcr.io is configured via its shasum for now. On the somewhat fancy side we support configuration of arbitrary rspamd settings via a recursive template in _helpers.tpl which transforms a config dict from values.yaml into the environment variables required by the rspamd image's entrypoint.
Add a simple helm chart for running PeekabooAV-specific postfix on Kubernetes. Skeleton was created using helm create. Configuration of the container is provides from a map in values.yaml whose keys are mapped onto the environment variable logic provides by the container image entrypoint. A pre-built image on ghcr.io is configured via its shasum for now.
Add a Helm chart that sets up a complete PeekabooAV pipeline including postfix, rspamd and Cortex. As a meta-chart is orchestrates various other charts to build the overall system. This supplants the individual spec files from our previous PoC at getting PeekabooAV to work on Kubernetes, so we remove them.
Add some basic notes on how to use the demo pipeline and interpret its behaviour.
Add a short README that guides the user to the pipeline chart as starting point.
Add a workflow that uses the helm chart-releaser action to package and publish our helm charts via out gh-pages branch.
Change the main compose.yaml to use our released images for a more positive out-of-the-box experience. Add a new compose.dev.yaml to retain the option to build the images locally directly through docker-compose.
Mainly to simplify Github Actions workflows for building the container images, we change all underscore-separated image, container and service names in our compose config to dashes. As a side-effect, this gives us RFC 1035-compliant DNS host names (i.e. with only internal dashes) and strictly syntactically correct Markdown in pipeline/README.md (no intra-word emphasis).
Similarly to how we're doing it in Kuberntes we can simplify the postfix configuration by exploiting the fact that the systems know each other by their names by way of the compose network name resolution. Therefore we can tell the client talking to postfix-tx to send a mail to root@postfix-rx directly. We only need to allow this unauthenticated relaying on postfix-tx. This allows us to get rid of the routing from postfix-tx's root account to postfix-rx through the virtual table.
There's a minor change in the current rspamd release 3.5 making our integration patch fail. Adjust to suit.
Github have retired their Ubuntu 18.04 runner. So we can no longer test on it. Remove from matrix build.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Based on @Sett17 's work for the pipeline and k8s this PR is intended to provide helm charts for setting up some of our individual system components via Helm and reproduce the pipeline for testing and validation in Kubernetes.
The changes are on top of #76 and we should discuss how to divide the changes between the two in the first place to ease review. The whole drop-file and image build and publish oddyssey was triggered by work on k8s and could live here in this PR instead.