[Version 1.1] Rspamd, GPG, Unbound, ARC standard and Redis

[hardware]

hardware/mailserver - version 1.1

Migration steps from 1.0 to 1.1-stable :

https://github.com/hardware/mailserver/wiki/Migrating-from-1.0-stable-to-1.1-stable

Changes

current component (1.0)	new component (1.1)
Spamassassin	Rspamd
OpenDKIM	Rspamd DKIM and DKIM signing module
OpenDMARC	Rspamd DMARC module
Postgrey	Rspamd Greylisting module
Gross	Rspamd Greylisting module
Amavisd	Rspamd proxy worker
Supervisor	s6 - skarnet's small supervision suite

Upgrades

component	current version	new version
Debian	Jessie	Stretch
Docker compose file	2	2.1
Postfix	2.11.3	3.1.4
Dovecot	2.2.13	2.2.27
ClamAV	0.98.7	0.99.2

Links

https://rspamd.com/
https://github.com/vstakhov/rspamd
http://skarnet.org/software/s6/
https://github.com/skarnet/s6

Author : @vstakhov

Automatic GPG encryption

Zeyple catches email from the postfix queue, then encrypts it if a corresponding recipient's GPG public key is found. Finally, it puts it back into the queue.

More information in the readme : https://github.com/hardware/mailserver/#automatic-gpg-encryption-of-all-your-e-mails

Links

https://infertux.com/labs/zeyple/
https://github.com/infertux/zeyple

Unbound local DNS resolver

Unbound is a validating, recursive, and caching DNS resolver inside the container, you can control it with the remote server control utility.

Some examples :

# Display server status
docker exec -ti mailserver unbound-control status

# Print server statistics
docker exec -ti mailserver unbound-control stats_noreset

# Reload the server. This flushes the cache and reads the config file.
docker exec -ti mailserver unbound-control reload

Links

https://www.unbound.net/

ARC standard

ARC support added thanks to the Rspamd ARC module.

Links

http://arc-spec.org/
https://datatracker.ietf.org/doc/draft-ietf-dmarc-arc-usage/
https://dmarc.org/presentations/ARC-Overview-2016Q2-v03.pdf
https://blog.returnpath.com/how-to-explain-authenticated-received-chain-arc-in-plain-english-2/

Redis cache

Redis added for some features :

• Greylisting
• Rate limiting
• Whitelisting of reply messages
• Bayesian statistics
• Some other rspamd modules

Links

https://redis.io/
https://github.com/antirez/redis

[AndrewSav]

https://www.mail-tester.com says that the message is not dkim signed on latest 1.1. How do I troubleshoot?

[AndrewSav]

"This is a valid DKIM key record"
I have to add that it worked before upgrade to 1.1. And I did not change the record. Can you tell me how rspamd knows where are the private keys for each of the domains?

[AndrewSav]

Thank you. So those are present and appear correct. Search in rspamd webui, History tab Errors pane for dkim gives /var/lib/rspamd/spf_dkim_whitelist.inc.local: map file is unavailable for reading but I'm not sure it's relevant. Any other ideas / logs to investigate?

[hardware]

@AndrewSav the 1.1 isn't ready yet, I'll let you know when you can use it.

You can ignore all map file is unavailable for reading warning :

rspamd/rspamd#1474
https://github.com/hardware/mailserver/blob/v1.1/rootfs/etc/rsyslog/rsyslog.conf#L8

[AndrewSav]

@hardware thank you for this. It's just you closed the issue about spam learning so I wrongly assumed that the rspamd part is ready.

[hardware]

@AndrewSav The new version is ready !

Please read the migration procedure before upgrading your mail server : https://github.com/hardware/mailserver/wiki/Migrating-from-1.0-stable-to-1.1-beta

I will merge the v1.1 branch to master and create the 1.1-stable docker tag next week after some final stress tests :D

For the next 6 months, the :latest docker tag will always point to 1.0 repository branch to not break compatibility with older versions. After January 2018, those who have not yet migrated, will receive an error at the next update and will be prompted to update the mail server or to switch to the 1.0-oldstable docker tag.

Feel free to give your feedback, improve this new version and make suggestions :)

[denji]

1.0-stable -> 1.0-oldstable (1.0-legacy)

[AndrewSav]

@hardware thank you for that, appreciated. I'm testing it now.
There are discrepancies between docker-compose.sample.yml and Migrating-from-1.0-stable-to-1.1-beta

The former has the following and the latter does not:

- "11334:11334" # HTTP                - Optional - Rspamd WebUI

Also, the former does not have nginx dependency on the mailserver and the latter does.

Update: Also this is present in the former but not in the latter for redis:

sysctls:
      - net.core.somaxconn=1024

[hardware]

@denji Yeah it's better.
@AndrewSav Thank you for pointing this out. 11334 port is not needed if the webserver is on the same server.

[AndrewSav]

@hardware everything seems to be working here. You might want to mention in the migration doco that one needs to set up certs for rspamd too. I know, it's kind of obvious, but I have a feeling that people would appreciate pointing that out.

[hardware]

You might want to mention in the migration doco that one needs to set up certs for rspamd too.

Done ;)

New dovecot version takes more time to generate DH params (6 minutes...) on my tiny VM :

2017-08-21T18:41:05.652783+00:00 mail dovecot: ssl-params: Generating SSL parameters
2017-08-21T18:47:32.774817+00:00 mail dovecot: ssl-params: SSL parameters regeneration completed

You think I should advise using haveged entropy daemon in the readme ?

[AndrewSav]

You think I should advise the use of haveged entropy daemon in the readme ?

Uhm, I did not stumble across this. When does this happen?

[hardware]

When the container start, dovecot hang completely during ssl parameters generation with a small CPU. But i think I will store the Diffie-Hellman parameters in /var/mail/dovecot/ssl-parameters.dat to avoid this.

[hardware]

But i think I will store the Diffie-Hellman parameters in /var/mail/dovecot/ssl-parameters.dat to avoid this.

Done : e3ce2c7

[AndrewSav]

Is it safe/secure to have everyone use the same DH parameters?

[hardware]

Is it safe/secure to have everyone use the same DH parameters?

No one shares the same DH parameters, each instance has a different one. /var/mail is your docker volume.

https://github.com/hardware/mailserver/tree/v1.1#filesfolders-tree

[AndrewSav]

Ah, it's symlinked, missed that bit. Gotcha.

Also spam detection is so much better. I'm curious to see if the learning really works the way it's supposed to ;)

[arckosfr]

i think it will nice to mark the rspamd port, actually it is only writen on the ngxproxy procedure.
Apart from that, the upgrade works well! 👍

[hardware]

@arckosfr added in migration procedure ;)

This new port is not needed in the docker-compose file if the webserver is on the same host because both containers are on the same local network. For security reasons, this port should not be public, even if the worker-controller is protected by a password.

If this port is made public, your firewall should filter and accept only authorized external hosts.

[arckosfr]

Correct, i don't expose the port but just for user that use another reverse than ngxproxy (especially with label support like Traefik or Xataz reverse per exemple).

[ksylvan]

Works well on my setup. Running on 3 domains. Will let you know if I run into any issues.

[hardware]

New features recently added :

• Automatic GPG encryption with Zeyple.
• Local DNS resolver with Unbound.

I added a local resolver for stability, security and of course to have dnssec validation.
Enjoy !

[ksylvan]

Now running with 1.1-stable. Also added the option to easily enable the GPG encryption when using the docker-mail-server setup.

Question: Are rspamd stats persistent across reboots?

[hardware]

Question: Are rspamd stats persistent across reboots ?

https://github.com/hardware/mailserver/wiki/Migrating-from-1.0-stable-to-1.1-stable#known-issues

Yes but there is a bug in Rspamd which causes the counters are not saved on the filesystem when rspamd terminated too quickly or unexpectedly. It seems stats.ucl is never updating during rspamd execution.

Stats are stored in dbdir : /mnt/docker/mail/rspamd/stats.ucl

I will open an issue on rspamd github repository. Only counters are affected by this problem.

[ksylvan]

Thanks. That's exactly the behavior I noticed when I restarted the stack.

[Antexa]

Thanks for this procedure 🥇

On section "Update you docker-compose file" you point it out to mention the line to modify, I think you can also add the mention on the image line.

When we upgrade the cert file we need to stop the nginx container before trying to upgrade, it can be relevant to add it to the doc

docker-compose nginx stop
docker run... 
docker-compose up -d

[hardware]

@Antexa added

[vizv]

Migrated successfully, thanks! 🎉

[hardware]

@ksylvan I opened an issue on rspamd github repository : rspamd/rspamd#1823

[sknight80]

I will update my current 1.0-legacy environment to the 1-1-stable version. Should I use the stable or the master branch for a production environment? (About 30-40 domains)

[hardware]

1.1-latest tag is the latest development build. These builds have been validated through the CI automation system but they are not meant for deployment in production.