You should make these changes on a test server and backup any ZCS & rspamd configuration files that you are going to modify before you modify them - when you're happy that it works you can try it on a live server but again, make sure you backup any modified configuration files.
These are notes for installing rspamd on ZCS 8.7.x (my current version is 8.7.11), I use CentOS and these instructions are for that distribution and the equivalent version of RHEL although installing rspamd on Ubuntu should not be that different.
NB: Rmilter from the Rspamd project and documented on their web site is deprecated and won't be available as a separate package in future, do not use it.
[EDIT] As of today (2017-09-06) I have updated my server to the latest ZCS 8.8.3GA release and these instructions work and are valid for that version as well.
I've been running various versions of Rspamd for the last eight months and have not had any ill effects, although I only run this on my home server there has been a huge reduction in CPU usage from 15-20% to less than 5% for the same levels of inbound/outbound email. I have also removed all DNS & Protocol checks from within the ZCS configuration and the Postscreen settings have been set to their default values – this leads to rspamd processing all inbound email and also further CPU usage reductions.
The current rspamd install now uses an inbuilt milter compatible process for the mail that's sent to it, the anti-spam etc. components of rspamd for processing the mail through it's various modules and pass it back to postfix via the milter process. Do note that this configuration still uses the ZCS inbuilt amavisd & clamav for anti-virus processing.
The multiple-milter function in Postfix is broken by ZCS and only allows one milter to be functional (the bugzilla reference is 97706), although that bug seems to require a trivial fix it’s unfortunately still languishing without any attention for the past two-plus years - please add your comments and vote if you feel it should be implemented.
Install the rspamd & epel repositories, details on the rspamd wiki: https://rspamd.com/downloads.html - once that's done install rspamd & redis then modify the ZCS and rspamd config files as shown below: If you are going to use redis then a more recent version than the one in the EPEL repo should be used. Further details on the install of the REMI & Rspamd repositories are in the wiki article .
We need to set some attributes in ZCS LDAP for rspamd (these changes will survive upgrades):
Code: Select all
$ su - zimbra
$ zmprov ms $(zmhostname) zimbraMtaMilterDefaultAction accept
$ zmprov ms $(zmhostname) zimbraMtaSmtpdMilters "inet:localhost:11332, inet:[::1]:11332"
$ zmprov ms $(zmhostname) zimbraMtaNonSmtpdMilters "inet:localhost:11332, inet:[::1]:11332"
Code: Select all
$ zmprov gs $(zmhostname) | grep -i milter
Code: Select all
zimbraMilterBindPort: 7026
zimbraMilterMaxConnections: 20000
zimbraMilterNumThreads: 100
zimbraMilterServerEnabled: FALSE
zimbraMtaMilterCommandTimeout: 30s
zimbraMtaMilterConnectTimeout: 30s
zimbraMtaMilterContentTimeout: 300s
zimbraMtaMilterDefaultAction: accept
zimbraMtaSmtpdMilters: smtpd_milters=inet:localhost:11332
Code: Select all
$ su - zimbra
$ postconf smtpd_milters="inet:localhost:11332, inet:[::1]:11332"
$ postconf non_smtpd_milters="inet:localhost:11332, inet:[::1]:11332"
Code: Select all
$ postconf | grep smtpd_milters
Code: Select all
$ zmprov gs $(zmhostname) | grep zimbraServiceEnabled
Code: Select all
$ zmprov ms $(zmhostname) -zimbraServiceEnabled antispam
$ zmantispamctl stop
$ zmprov gs $(zmhostname) | grep zimbraServiceEnabled
These following are the basic changes you need to get rspamd working and the only rspamd config files you should need to create and modify are the ones below (these are not set by default):
Code: Select all
vim /etc/rspamd/local.d/worker-proxy.inc
upstream "local" {
self_scan = true;
}
Code: Select all
vim /etc/rspamd/local.d/milter_headers.conf
extended_spam_headers = true;
Code: Select all
vim /etc/rspamd/local.d/logging.inc
# Included from top-level .conf file
# enable systemd & console logging
# type = "console";
# systemd = true;
type = "file";
level = "info";
filename = "$LOGDIR/rspamd.log";
log_format =<< EOD
id: <$mid>,$if_qid{ qid: <$>,}$if_ip{ ip: $,}$if_user{ user: $,}$if_smtp_from{ from: <$>,}
(default: $is_spam ($action): [$scores] [$symbols]),
len: $len, time: $time_real real,
$time_virtual virtual, dns req: $dns_req
EOD
# Show statistics for regular expressions
log_re_cache = true;
# Can be used for console logging
color = false
# Enable debug for specific modules (e.g. `debug_modules = ["dkim", "re_cache"];`)
debug_modules = []
There is also a web ui available for monitoring and configuration of rspamd, check the web site for further documentation on this feature.
Code: Select all
vim /etc/rspamd/local.d/worker-controller.inc
Code: Select all
# Included from top-level .conf file
# all we need in here is the user and admin passwords
# password for normal user
password = "apassword"; # obviously, your password of choice
# password for 'admin'
enable_password = "a-secret-password"; # create this admin password with the rspamd utilities
If you think this is a worthwhile change for Zimbra then feel free to add you comments and votes to the RFE I' ve filed for removing spamassassin and adding this as a replacement in ZCS: https://bugzilla.zimbra.com/show_bug.cgi?id=108168
I'd be interested to hear feedback on how this went for you and if anyone with a large install wants to try Rspamd it would be great hear if it's as good as I think it is
![Smile :)](./images/smilies/icon_e_smile.gif)
NB: because of the high number of RBL (and other) DNS lookups, it's advisable to have a caching namesever on you lan (either the ZCS dnscache or your own DNS server, I use PowerDNS), if you don't do this you may get your own IP blacklisted with the RBL sites.
The zmtrainsa script has been modified to remove (some of) the DSPAM & SA training, I've added a section for training the rspamd system from the same spam/ham accounts in ZCS so users continue using the 'Spam' & Not Spam' function in the ZCS Web UI. You can take a look at the processing of this script by installing the following (make a copy of your original file first) modified zmtrainsa files and running the following command:
Code: Select all
sh -x /opt/zimbra/bin/zmtrainsa_test2 &>>/opt/zimbra/log/zmtrainsa_test.log <-- check the output to see if it's working correctly.
Do not modify any of the config files in the /etc/rspamd directory, read the Rspamd documentation on the web site for details on how to add config override files and make further modifications to the rspamd config.
What I haven't mentioned is the current features in Rspamd and the fact that some of it's modules could also replace their ZCS equivalents, take a look at the web site here: https://rspamd.com/features.html
If you have any comments, suggestions, improvements for what I've posted then feel free to add to this thread. I hope you find this post of some useful.
PS To any moderators that pass by, I've made this a sticky but if you think that should be removed then feel free to do so.
![Smile :)](./images/smilies/icon_e_smile.gif)
PPS For some strange reason the forum software won't let me add an attachment to this post so I've pasted the modified zmtrainsa script below.
Code: Select all
#!/bin/bash
#
# ***** BEGIN LICENSE BLOCK *****
# Zimbra Collaboration Suite Server
# Copyright (C) 2005, 2006, 2007, 2008, 2009, 2010, 2012, 2013, 2014, 2015, 2016 Synacor, Inc.
#
# This program is free software: you can redistribute it and/or modify it under
# the terms of the GNU General Public License as published by the Free Software Foundation,
# version 2 of the License.
#
# This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY;
# without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
# See the GNU General Public License for more details.
# You should have received a copy of the GNU General Public License along with this program.
# If not, see < gnu dot org /licenses/>.
# ***** END LICENSE BLOCK *****
#
# This section trains the system ham/spam accounts
#
autoTrainSystem() {
# This is the section for extracting the email to a
# couple of temp directories for spam & ham
timestampit "Starting spam/ham extraction from system accounts."
spamdir=`mktemp -d -t spam.XXXXXXX` || exit 1
hamdir=`mktemp -d -t ham.XXXXXXX` || exit 1
/opt/zimbra/libexec/zmspamextract ${spam_account} -o ${spamdir}
/opt/zimbra/libexec/zmspamextract ${ham_account} -o ${hamdir}
timestampit "Finished extracting spam/ham from system accounts."
# This is the actual section for rspamd training
timestampit "Starting rspamd system accounts training."
# Let's do a test here to see if rspamc is doing it's thing!
# List some stats before training
# the passwords for these rspamc commands need to be changed for your server
timestampit "List rspam stats before training."
/usr/bin/rspamc -h 127.0.0.1:11334 -P apassword stat
# do the spam directory
/usr/bin/rspamc -h 127.0.0.1:11334 -P apassword learn_spam ${spamdir}
# do the ham directory
/usr/bin/rspamc -h 127.0.0.1:11334 -P apassword learn_ham ${hamdir}
# List some stats after training
timestampit "List rspam stats after training."
/usr/bin/rspamc -h 127.0.0.1:11334 -P apassword stat
timestampit "Finished rspamd training."
# End of the rspamd training section for system ham/spam accounts
/bin/rm -rf ${spamdir} ${hamdir}
}
# The following is the section that trains rspamd for the user $FOLDER (ham or spam)
#
trainAccountFolder() {
timestampit "Starting rspamd user accounts training"
tempdir=`mktmpdir ${MODE}`
if [ "x${MODE}" = "xspam" ]; then
FOLDER=${FOLDER:=junk}
elif [ "x${MODE}" = "xham" ]; then
FOLDER=${FOLDER:=inbox}
fi
# extract the user ham/spam and train rspamd
timestampit "Starting rspamd $MODE training for $USER using folder $FOLDER"
/opt/zimbra/libexec/zmspamextract -r -m $USER -o ${tempdir} -q in:${FOLDER}
if [ "x${MODE}" = "xspam" ]; then
/usr/bin/rspamc -h 127.0.0.1:11334 -P apassword learn_spam ${tempdir} || exit 1
FOLDER=${FOLDER:=junk}
elif [ "x${MODE}" = "xham" ]; then
/usr/bin/rspamc -h 127.0.0.1:11334 -P apassword learn_ham ${tempdir} || exit 1
FOLDER=${FOLDER:=inbox}
fi
timestampit "Finished rspamd $MODE training for $USER using folder $FOLDER"
/bin/rm -rf ${tempdir}
}
mktmpdir() {
mktemp -d "${zmtrainsa_tmp_directory:-${zimbra_tmp_directory}}/rspamd.$$.$1.XXXXXX" || exit 1
}
timestampit() {
SIMPLE_DATE=`date +%Y%m%d%H%M%S`
echo "$SIMPLE_DATE $1"
}
usage() {
echo "Usage: $0 <user> <spam|ham> [folder]"
exit 1
}
if [ x`whoami` != xzimbra ]; then
echo Error: must be run as zimbra user
exit 1
fi
source `dirname $0`/zmshutil || exit 1
zmsetvars
amavis_dspam_enabled=`/opt/zimbra/bin/zmprov -l gs ${zimbra_server_hostname} zimbraAmavisDSPAMEnabled | grep zimbraAmavisDSPAMEnabled: | awk '{print $2}'`
amavis_dspam_enabled=$(echo $amavis_dspam_enabled | tr A-Z a-z)
antispam_mysql_enabled=$(echo $antispam_mysql_enabled | tr A-Z a-z)
zmtrainsa_cleanup_host=$(echo $zmtrainsa_cleanup_host | tr A-Z a-z)
if [ "x${zimbra_spam_externalIsSpamAccount}" = "x" ]; then
spam_account="-s"
else
spam_account="-m ${zimbra_spam_externalIsSpamAccount}"
fi
if [ "x${zimbra_spam_externalIsNotSpamAccount}" = "x" ]; then
ham_account="-n"
else
ham_account="-m ${zimbra_spam_externalIsNotSpamAccount}"
fi
# Set db_path
if [ x"$antispam_mysql_enabled" = "xtrue" ]; then
db_path='/opt/zimbra/data/amavisd/mysql/data'
else
db_path='/opt/zimbra/data/amavisd/.spamassassin'
fi
# No argument mode uses zmspamextract for auto-training.
if [ x$1 = "x" ]; then
autoTrainSystem
exit
fi
if [ x$1 = "x--cleanup" ]; then
if [ x${zmtrainsa_cleanup_host} = "xtrue" ]; then
timestampit "Starting spam/ham cleanup"
mydir=`mktemp -d -t cleanup.XXXXXX` || exit 1
/opt/zimbra/libexec/zmspamextract ${spam_account} -o ${mydir} -d
/opt/zimbra/libexec/zmspamextract ${ham_account} -o ${mydir} -d
/bin/rm -rf ${mydir}
timestampit "Finished spam/ham cleanup"
else
timestampit "Cleanup skipped: $zimbra_server_hostname is not a spam/ham cleanup host."
fi
exit
fi
USER=$1
MODE=`echo $2 | tr A-Z a-z`
FOLDER=$3
if [ "x${MODE}" != "xspam" -a "x${MODE}" != "xham" ]; then
usage
fi
if [ "x${USER}" = "x" ]; then
usage
fi
trainAccountFolder
exit 0
After each upgrade of ZCS we need to check the following as some of our modifications get overwritten, do the following:
verify the ldap setting is OK (it should be):
Code: Select all
zmprov gs $(hostname) zimbraMtaSmtpdMilters
Code: Select all
zimbraMtaSmtpdMilters: smtpd_milters=inet:localhost:11332
Code: Select all
vim /opt/zimbra/conf/zmconfigd.cf
Code: Select all
POSTCONF smtpd_milters
if VAR zimbraMtaSmtpdMilters
POSTCONF smtpd_milters VAR zimbraMtaSmtpdMilters
fi
Code: Select all
postconf | grep smtpd_milters
non_smtpd_milters =
smtpd_milters = smtpd_milters=inet:localhost:11332
Code: Select all
vim /opt/zimbra/common/conf/main.cf
Code: Select all
cp /opt/zimbra/bin/zmtrainsa.rspamd_version /opt/zimbra/bin/zmtrainsa