Rspamd: Fast, free and open-source spam filtering system

[phoenix]

I've been looking for a suitable replacement for DSPAM, a project that was good but died and became unmaintained then totally removed from Zimbra. In recent searches I came across Rspamd which does all that DSPAM did and more, in addition, it can completely replace spamassassin - it's lighter, more configurable and more performant than spamassassin. These notes are a brief explanation of how to easily integrate rspamd into a ZCS server and disable spamassassin completely.

You should make these changes on a test server and backup any ZCS & rspamd configuration files that you are going to modify before you modify them - when you're happy that it works you can try it on a live server but again, make sure you backup any modified configuration files.

These are notes for installing rspamd on ZCS 8.7.x (my current version is 8.7.11), I use CentOS and these instructions are for that distribution and the equivalent version of RHEL although installing rspamd on Ubuntu should not be that different.

NB: Rmilter from the Rspamd project and documented on their web site is deprecated and won't be available as a separate package in future, do not use it.

[EDIT] As of today (2017-09-06) I have updated my server to the latest ZCS 8.8.3GA release and these instructions work and are valid for that version as well.

I've been running various versions of Rspamd for the last eight months and have not had any ill effects, although I only run this on my home server there has been a huge reduction in CPU usage from 15-20% to less than 5% for the same levels of inbound/outbound email. I have also removed all DNS & Protocol checks from within the ZCS configuration and the Postscreen settings have been set to their default values – this leads to rspamd processing all inbound email and also further CPU usage reductions.

The current rspamd install now uses an inbuilt milter compatible process for the mail that's sent to it, the anti-spam etc. components of rspamd for processing the mail through it's various modules and pass it back to postfix via the milter process. Do note that this configuration still uses the ZCS inbuilt amavisd & clamav for anti-virus processing.

The multiple-milter function in Postfix is broken by ZCS and only allows one milter to be functional (the bugzilla reference is 97706), although that bug seems to require a trivial fix it’s unfortunately still languishing without any attention for the past two-plus years - please add your comments and vote if you feel it should be implemented.

Install the rspamd & epel repositories, details on the rspamd wiki: https://rspamd.com/downloads.html - once that's done install rspamd & redis then modify the ZCS and rspamd config files as shown below: If you are going to use redis then a more recent version than the one in the EPEL repo should be used. Further details on the install of the REMI & Rspamd repositories are in the wiki article .

We need to set some attributes in ZCS LDAP for rspamd (these changes will survive upgrades):

Code: Select all

 $ su - zimbra
 $ zmprov ms $(zmhostname) zimbraMtaMilterDefaultAction accept
 $ zmprov ms $(zmhostname) zimbraMtaSmtpdMilters "inet:localhost:11332, inet:[::1]:11332"
 $ zmprov ms $(zmhostname) zimbraMtaNonSmtpdMilters "inet:localhost:11332, inet:[::1]:11332"

Check all the relevant settings:

Code: Select all

 $ zmprov gs $(zmhostname) | grep -i milter 

That should give the following result:

Code: Select all

 zimbraMilterBindPort: 7026
 zimbraMilterMaxConnections: 20000
 zimbraMilterNumThreads: 100
 zimbraMilterServerEnabled: FALSE
 zimbraMtaMilterCommandTimeout: 30s
 zimbraMtaMilterConnectTimeout: 30s
 zimbraMtaMilterContentTimeout: 300s
 zimbraMtaMilterDefaultAction: accept
 zimbraMtaSmtpdMilters: smtpd_milters=inet:localhost:11332

The LDAP attributes that we modified above survive an upgrade but the following one using postconf '''does not survive an upgrade and it needs to be reapplied after every upgrade'''. Run the following command as the zimbra user:

Code: Select all

 $ su - zimbra
   $ postconf smtpd_milters="inet:localhost:11332, inet:[::1]:11332"
 $ postconf non_smtpd_milters="inet:localhost:11332, inet:[::1]:11332"

You can check the current status of the smtpd_milters entry with:

Code: Select all

 $ postconf | grep smtpd_milters

The aim is to not use spamassassin so we need to disable it (still as the zimbra user), check the current status of the enabled services:

Code: Select all

 $ zmprov gs $(zmhostname) | grep zimbraServiceEnabled

The above command will list all the services that are enabled on your server. Next, modify the antispam service with the following commands to disable and stop antispam service:

Code: Select all

 $ zmprov ms $(zmhostname) -zimbraServiceEnabled antispam
 $ zmantispamctl stop
 $ zmprov gs $(zmhostname) | grep zimbraServiceEnabled

The antispam service should not now be shown in the list of enabled services.

These following are the basic changes you need to get rspamd working and the only rspamd config files you should need to create and modify are the ones below (these are not set by default):

Code: Select all

vi /etc/rspamd/local.d/worker-proxy.inc

        upstream "local" {
            self_scan = true;
        }

If you want extended headers in messages you need just one line to the file to enable them:

Code: Select all

vi /etc/rspamd/local.d/milter_headers.conf

        extended_spam_headers = true;

To send rspamd log message to a file:

Code: Select all

vi /etc/rspamd/local.d/logging.inc

# Included from top-level .conf file
# enable systemd & console logging
#   type = "console";
#   systemd = true;

    type  = "file";
    level = "info";

    filename = "$LOGDIR/rspamd.log";

log_format =<< EOD
id: <$mid>,$if_qid{ qid: <$>,}$if_ip{ ip: $,}$if_user{ user: $,}$if_smtp_from{ from: <$>,}
(default: $is_spam ($action): [$scores] [$symbols]),
len: $len, time: $time_real real,
$time_virtual virtual, dns req: $dns_req
EOD

# Show statistics for regular expressions
log_re_cache = true;

# Can be used for console logging
color = false

# Enable debug for specific modules (e.g. `debug_modules = ["dkim", "re_cache"];`)
debug_modules = []

save and exit that file.

There is also a web ui available for monitoring and configuration of rspamd, check the web site for further documentation on this feature.

Code: Select all

vi /etc/rspamd/local.d/worker-controller.inc

Code: Select all

# Included from top-level .conf file
# all we need in here is the user and admin passwords

# password for normal user
password = "apassword";    # obviously, your password of choice

# password for 'admin'
enable_password = "a-secret-password";  # create this admin password with the rspamd utilities

To summarise, you should now have a working rspamd daemon running with your ZCS server and mail passing between them both for processing. I would suggest that the modified zmtrainsa script is added as a new cron job for training ZCS otherwise you will have to replace the original ZCS supplied script after every update.

If you think this is a worthwhile change for Zimbra then feel free to add you comments and votes to the RFE I' ve filed for removing spamassassin and adding this as a replacement in ZCS: https://bugzilla.zimbra.com/show_bug.cgi?id=108168

I'd be interested to hear feedback on how this went for you and if anyone with a large install wants to try Rspamd it would be great hear if it's as good as I think it is

NB: because of the high number of RBL (and other) DNS lookups, it's advisable to have a caching namesever on you lan (either the ZCS dnscache or your own DNS server, I use PowerDNS), if you don't do this you may get your own IP blacklisted with the RBL sites.

The zmtrainsa script has been modified to remove (some of) the DSPAM & SA training, I've added a section for training the rspamd system from the same spam/ham accounts in ZCS so users continue using the 'Spam' & Not Spam' function in the ZCS Web UI. You can take a look at the processing of this script by installing the following (make a copy of your original file first) modified zmtrainsa files and running the following command:

Code: Select all

sh -x /opt/zimbra/bin/zmtrainsa_test2 &>>/opt/zimbra/log/zmtrainsa_test.log  <-- check the output to see if it's working correctly.

The attachment (see my comment at the end of this post for the script) is a modified zmtrainsa file, you can replace the ZCS version but you'll have to replace it after each upgrade or you can add it as a separate cron job and run that then you can disable the ZCS version. FWIW, I make a copy of the original file then replace zmtrainsa with my modified version, I just replace it when I update ZCS.

Do not modify any of the config files in the /etc/rspamd directory, read the Rspamd documentation on the web site for details on how to add config override files and make further modifications to the rspamd config.

What I haven't mentioned is the current features in Rspamd and the fact that some of it's modules could also replace their ZCS equivalents, take a look at the web site here: https://rspamd.com/features.html

If you have any comments, suggestions, improvements for what I've posted then feel free to add to this thread. I hope you find this post of some useful.

PS To any moderators that pass by, I've made this a sticky but if you think that should be removed then feel free to do so.

PPS For some strange reason the forum software won't let me add an attachment to this post so I've pasted the modified zmtrainsa script below.

Code: Select all

 #!/bin/bash
 #
 # ***** BEGIN LICENSE BLOCK *****
 # Zimbra Collaboration Suite Server
 # Copyright (C) 2005, 2006, 2007, 2008, 2009, 2010, 2012, 2013, 2014, 2015, 2016 Synacor, Inc.
 #
 # This program is free software: you can redistribute it and/or modify it under
 # the terms of the GNU General Public License as published by the Free Software Foundation,
 # version 2 of the License.
 #
 # This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY;
 # without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
 # See the GNU General Public License for more details.
 # You should have received a copy of the GNU General Public License along with this program.
 # If not, see < gnu dot org /licenses/>.
 # ***** END LICENSE BLOCK *****
 #
  
 # This section trains the system ham/spam accounts
 #
 autoTrainSystem() {   
 
 # This is the section for extracting the email to a
 # couple of temp directories for spam & ham
  timestampit "Starting spam/ham extraction from system accounts."
  spamdir=`mktemp -d -t spam.XXXXXXX` || exit 1
  hamdir=`mktemp -d -t ham.XXXXXXX` || exit 1
  /opt/zimbra/libexec/zmspamextract ${spam_account} -o ${spamdir}
  /opt/zimbra/libexec/zmspamextract ${ham_account} -o ${hamdir}
  timestampit "Finished extracting spam/ham from system accounts."
 
 # This is the actual section for rspamd training
  timestampit "Starting rspamd system accounts training."
 
 # Let's do a test here to see if rspamc is doing it's thing!
 # List some stats before training
 # the passwords for these rspamc commands need to be changed for your server
 
   timestampit "List rspam stats before training."
   /usr/bin/rspamc -h 127.0.0.1:11334 -P apassword stat
 
 # do the spam directory
   /usr/bin/rspamc -h 127.0.0.1:11334 -P apassword learn_spam  ${spamdir}
 
 # do the ham directory
   /usr/bin/rspamc -h 127.0.0.1:11334 -P apassword learn_ham  ${hamdir}
 
 # List some stats after training
   timestampit "List rspam stats after training."
   /usr/bin/rspamc -h 127.0.0.1:11334 -P apassword stat
   timestampit "Finished rspamd training."
 # End of the rspamd training section for system ham/spam accounts
 
  /bin/rm -rf ${spamdir} ${hamdir}
 
 }
 
 # The following is the section that trains rspamd for the user $FOLDER (ham or spam)
 #
 trainAccountFolder() {
 
 timestampit  "Starting rspamd user accounts training"
  tempdir=`mktmpdir ${MODE}`
  if [ "x${MODE}" = "xspam" ]; then
    FOLDER=${FOLDER:=junk}
  elif [ "x${MODE}" = "xham" ]; then
    FOLDER=${FOLDER:=inbox}
  fi
 
 # extract the user ham/spam and train rspamd 
 timestampit  "Starting rspamd $MODE training for $USER using folder $FOLDER"
  /opt/zimbra/libexec/zmspamextract -r -m $USER -o ${tempdir} -q in:${FOLDER}
   
  if [ "x${MODE}" = "xspam" ]; then
    /usr/bin/rspamc -h 127.0.0.1:11334 -P apassword learn_spam ${tempdir} || exit 1
    FOLDER=${FOLDER:=junk}
  elif [ "x${MODE}" = "xham" ]; then
    /usr/bin/rspamc -h 127.0.0.1:11334 -P apassword learn_ham ${tempdir} || exit 1
    FOLDER=${FOLDER:=inbox}
   fi
 timestampit  "Finished rspamd $MODE training for $USER using folder $FOLDER"
  
  /bin/rm -rf ${tempdir}
 
 }

 mktmpdir() {
  mktemp -d "${zmtrainsa_tmp_directory:-${zimbra_tmp_directory}}/rspamd.$$.$1.XXXXXX" || exit 1
 }
 
 timestampit() {
   SIMPLE_DATE=`date +%Y%m%d%H%M%S`
   echo "$SIMPLE_DATE $1"
 }
 
 usage() {
   echo "Usage: $0 <user> <spam|ham> [folder]"
   exit 1
 }
 
 if [ x`whoami` != xzimbra ]; then
     echo Error: must be run as zimbra user
   exit 1
 fi
 
 source `dirname $0`/zmshutil || exit 1 
 zmsetvars 
 
 amavis_dspam_enabled=`/opt/zimbra/bin/zmprov -l gs ${zimbra_server_hostname} zimbraAmavisDSPAMEnabled | grep zimbraAmavisDSPAMEnabled: | awk  '{print $2}'`
 amavis_dspam_enabled=$(echo $amavis_dspam_enabled | tr A-Z a-z)
 antispam_mysql_enabled=$(echo $antispam_mysql_enabled | tr A-Z a-z)
 zmtrainsa_cleanup_host=$(echo $zmtrainsa_cleanup_host | tr A-Z a-z)
 
 if [ "x${zimbra_spam_externalIsSpamAccount}" = "x" ]; then
   spam_account="-s"
 else
   spam_account="-m ${zimbra_spam_externalIsSpamAccount}"
 fi
 
 if [ "x${zimbra_spam_externalIsNotSpamAccount}" = "x" ]; then
   ham_account="-n"
 else
   ham_account="-m ${zimbra_spam_externalIsNotSpamAccount}"
 fi
 
 # Set db_path
 if [ x"$antispam_mysql_enabled" = "xtrue" ]; then
   db_path='/opt/zimbra/data/amavisd/mysql/data' 
 else
   db_path='/opt/zimbra/data/amavisd/.spamassassin'
 fi
 
 # No argument mode uses zmspamextract for auto-training.
 if [ x$1 = "x" ]; then
   autoTrainSystem
   exit
 fi
 
 if [ x$1 = "x--cleanup" ]; then
   if [ x${zmtrainsa_cleanup_host} = "xtrue" ]; then
     timestampit "Starting spam/ham cleanup"
     mydir=`mktemp -d -t cleanup.XXXXXX` || exit 1
     /opt/zimbra/libexec/zmspamextract ${spam_account} -o ${mydir} -d
     /opt/zimbra/libexec/zmspamextract ${ham_account} -o ${mydir} -d
     /bin/rm -rf ${mydir}
     timestampit "Finished spam/ham cleanup"
   else
     timestampit "Cleanup skipped: $zimbra_server_hostname is not a spam/ham cleanup host."
  fi
   exit
 fi
  
 USER=$1 
 MODE=`echo $2 | tr A-Z a-z` 
 FOLDER=$3
 
 if [ "x${MODE}" != "xspam" -a "x${MODE}" != "xham" ]; then
   usage
 fi
 
 if [ "x${USER}" = "x" ]; then
  usage
 fi
 
 trainAccountFolder
 
 exit 0

[UPDATE]I seem to have overlooked the needed changes for ZCS after an upgrade (this is also in the wiki article), they are as follows:

After each upgrade of ZCS we need to check the following as some of our modifications get overwritten, do the following:

verify the ldap setting is OK (it should be):

Code: Select all

 zmprov gs $(hostname) zimbraMtaSmtpdMilters 

You should see the following output:

Code: Select all

 zimbraMtaSmtpdMilters: smtpd_milters=inet:localhost:11332

Remove the configd settings that overwrite our changes from the following file:

Code: Select all

 vi /opt/zimbra/conf/zmconfigd.cf

Remove these four lines:

Code: Select all

 POSTCONF smtpd_milters
 if VAR zimbraMtaSmtpdMilters
     POSTCONF smtpd_milters VAR zimbraMtaSmtpdMilters
 fi

Check to see if the settings are correct:

Code: Select all

 postconf | grep smtpd_milters 
 non_smtpd_milters =
 smtpd_milters = smtpd_milters=inet:localhost:11332

If the above settings have remained unchanged then the following file should be ok, if they are wrong then this file needs changing as mentioned in the initial installation settings:

Code: Select all

 vi /opt/zimbra/common/conf/main.cf

The zmtrainsa will also need to be replaced by the modified version for rspamd, I keep a copy of the zmtrainsascript in the /opt/zimbra/bin directory:

Code: Select all

 cp /opt/zimbra/bin/zmtrainsa.rspamd_version /opt/zimbra/bin/zmtrainsa

[JDunphy]

Promising!

The hyperscan of regexp and modern PCRE lookahead for perl because our version with centos 6 does limit our ability - resulting in excessive false positives without proper scoping. We got wrapped around the axle trying to build a simple SA rule such as the body contains an img src that does not have an extension of jpg/gif/png/etc for what is best described as tracking - so out of desperation, I wrote a plugin which was simple enough but now I am on the hook for more manual updates with my zimbra updates, etc. Crazy.

I am still reading about rspamd but what I am seeing looks extremely positive.

Your howto recipe is much appreciated Bill.

I like that they have replaced the html parser and the AST optimizer sounds interesting. I don't know what a Bayes window of 5 vs Bayes window of 1 means but if more is better than I welcome it. LOL

After the amount of perl code I have been through the past few week with SA, I have serious doubts that the SA plugin code can be brought up to current html standards and the adhoc nature of those rules/plugin's is becoming a problem when scoring is basically broken from my perspective without rule/score updates. While I don't like change, I welcome something simpler that we can customize and that it's c/lua is bonus. My perl isn't what it use to be.

Jim

[phoenix]

I'm not a great expert on managing linux systems or, as you can probably guess, a documentation writer but those notes got me through installing it a few times on test machines. I've always had problems with spamassassin and writing rules just ain't my thing, this sort of product is for the more technically adept than me so I hope that it's useful to others, I'll be interested to see if what it's like in a larger system than my server - don't forget my RFE if you think it's a worth while addition to ZCS. There is also the possibility that rspamd can replace a couple of the other modules such as DKIM signing. SPF checking, DMARC, greylisting etc. As I mentioned, I've removed all the policy, DNS restrictions, RBLs and postscreen configurations from my server and let rspamd handle it all and that provided another drop in CPU usage.

[JDunphy]

There is also the possibility that rspamd can replace a couple of the other modules such as DKIM signing. SPF checking, DMARC, greylisting etc. As I mentioned, I've removed all the policy, DNS restrictions, RBLs and postscreen configurations from my server and let rspamd handle it all and that provided another drop in CPU usage.

That is a good point on the other modules... From what I can tell, It's not possible to filter on the Authentication-Results header with Zimbra SA. I wanted to augment a meta rule and the best explanation I could come up with is that it was added after SA ran. I could match it when I ran SA from the command line and with a filter but not with the same simple SA header rule for a live message. It's still on my todo list but my goto solution always seems to be add a milter to the perimeter MX to test there and pass this header to Zimbra/SA to score. I need a better way.

I often wonder how many zimbra sites go 3rd party for spam filtering given the time consuming nature involved with this or outsource to gmail, etc in frustration. It's funny what many get caught up on. They won't use cloud storage solutions like dropbox, etc but they will allow all incoming email to pass through a 3rd party MX host for spam scoring or block all incoming traffic based on a tiny blacklist. Not saying there are not mitigation strategies but running zimbra in house should be a base for licence growth that could use some enhanced/advanced spam infrastructure IMHO. I also believe letsencrypt should work out of the box but you probably knew I was going to say that.

Maybe rspamd is that start. I think we are at a crossroads myself with spam,security,html 5+, etc and hope the SA and Zimbra communities can rise with it. I view spam more from the security side these days vs a nuisance issue given how many threats exist against browsers/companies/users. As I mention in that other post viewtopic.php?f=15&t=62447#p277751, It was harder than it should be and we are concerned. For me, I am going to try this on a test server because I need a hedge and develop some rspamd operational experience with their html parser should things not get better with the current solution. Bill your timing is pretty good because I had been looking the past few weeks but nothing was a comprehensive as what this appears to be.

[phoenix]

I'm glad that was a relatively easy install for you, it's always a concern when I post a set of instructions as I know they work for me but......

I'll be interested in hearing your comments on it's ease of configuration and how it compares to your current ZCS server with SA, not being a 'professional' admin the finer points of configuring rspamd are a bit hard for me to grasp but I'll keep plugging away at it. Have a good week-end.

[JDunphy]

A few comments. Very straightforward. This is for centos 6/RHEL 6

Code: Select all

curl http://rspamd.com/rpm-stable/centos-6/rspamd.repo > /etc/yum.repos.d/rspamd.repo
rpm --import http://rspamd.com/rpm-stable/gpg.key
yum update
yum install rspamd

Then it was cut/paste from your document above and that worked perfectly. I did modify a few things for my environment.

1) I didn't use $(hostname) which is wrong for my test environment. Went with `zmhostname` in its place.
2) centos 6 rc script for rspamd assumed the following so made this change.

Code: Select all

vi /etc/init.d/rspamd
#RSPAMD_CONF_FILE="/etc/rspamd/rspamd.sysvinit.conf"
RSPAMD_CONF_FILE="/etc/rspamd/rspamd.conf"

then I started it with:

Code: Select all

/etc/init.d/rspamd start

3) modified your training script (zmtrainsa_test2) so that: 192.X.X.X was now the following:
hostname=`zmhostname`

then removed the ip address and used this line in your script.

rspamc -h $hostname:11334 -P XXYYZZ stat

And issued the command to set a password to XXYYZZ:

Code: Select all

rspamadm pw
XXYYZZ

copied the resulting password from rspamadm to the /etc/rspamd/local.d/worker-controller.inc

Ran your test case of training and it worked perfectly.

Ok... so now I just need to figure out how all this works next. Then I have some tests to run through the parser.

Thanks for the very good directions in getting this working with Zimbra. I don't think I spent more than 5 mins. I have no idea how it works at this point but it trains spam and it appears to work from your test method. Next, I will probably just send email to both zimbra instances at the same time and see what can be learned about spam detection over a few weeks in comparison. For me performance isn't as big an issue vs accurate and better detection with the html parser given ways that html can be obfuscated by current best practices in spam delivery.

[ccelis5215]

Hi Bill and J,

Think Bill is doing a great effort showing how to use rspamd, i just didn't hear about that.

J has a very good point, nowadays performance it's not the issue.

ccelis

[phoenix]

J has a very good point, nowadays performance it's not the issue.

I'd slightly disagree that performance is not the issue. Yes, we all want it to be as accurate as possible but the side effect of reducing CPU usage is to be able to process more email. Don't forget that my simple home server that gets a very small amount of legitimate of mail per day plus the usual load of spam, the reduction of CPU usage from 15-17% to 3% is quite startling and shows that SA is too cumbersome. If you look at the rspamd site at the bottom of the first page there's a list of users and one of those is the Russian search engine rambler, they claim to process 100 million emails a day - I think CPU usage and performance might be a major factor for them.

[MisterM75]

Hello
I will not add any comment on the subject, but I use RSPAMD on another server and I can tell you that, the software works perfectly.
My question is, is it possible to go back to previous configuration by having activate postscreen?
Yours sincerely
Mz

[phoenix]

I will not add any comment on the subject, but I use RSPAMD on another server and I can tell you that, the software works perfectly.

Why not? Your experience would be interesting to read?

My question is, is it possible to go back to previous configuration by having activate postscreen?

Yes, just revert any changes you made to Postscreen or add any changes that you require in your new configuration.