Rspamd: Fast, free and open-source spam filtering system

Discuss your pilot or production implementation with other Zimbra admins or our engineers.
User avatar
zimico
Outstanding Member
Outstanding Member
Posts: 225
Joined: Mon Nov 14, 2016 8:03 am
Location: Vietnam
ZCS/ZD Version: 8.8.15 P3
Contact:

Re: Rspamd: A replacement for Spamassassin & Postscreen

Post by zimico »

Hi phoenix,
I have just started installing rspamd on my small lab zimbra server following the wiki and have some issues that need your help.
My zimbra version is 8.8.12 P3 on Centos 7. I enabled cbpoliyd. I also use dnsmasq.
In /var/log/zimbra.log I see a lot of:

Code: Select all

Jun 15 16:09:51 mail postfix/smtps/smtpd[28261]: warning: invalid transport name: smtpd_milters=inet in Milter service: smtpd_milters=inet:localhost:11332
and

Code: Select all

Jun 15 04:06:00 mail postfix/dkimmilter/smtpd[5656]: timeout after END-OF-MESSAGE from localhost[127.0.0.1]
and in /var/log/rspamd/rspamd.log:

Code: Select all

2019-06-15 16:28:01 #18891(controller) <3nxzfe>; monitored; rspamd_monitored_dns_cb: DNS query blocked on multi.uribl.com (127.0.0.1 returned), possibly due to high volume
It seems that my rspamd configuration is not good. Here is the output of:

Code: Select all

[zimbra@mail ~]$ zmprov gs $(zmhostname) | grep -i milter
zimbraMilterBindPort: 7026
zimbraMilterMaxConnections: 20000
zimbraMilterNumThreads: 100
zimbraMilterServerEnabled: TRUE
zimbraMtaMilterCommandTimeout: 30s
zimbraMtaMilterConnectTimeout: 30s
zimbraMtaMilterContentTimeout: 300s
zimbraMtaMilterDefaultAction: accept
zimbraMtaSmtpdMilters: smtpd_milters=inet:localhost:11332
[zimbra@mail ~]$ postconf | grep smtpd_milters
non_smtpd_milters =
smtpd_milters = smtpd_milters=inet:localhost:11332, inet:127.0.0.1:7026
[zimbra@mail ~]$ zmprov gs $(hostname) zimbraMtaSmtpdMilters
# name mail.zimilab.com
zimbraMtaSmtpdMilters: smtpd_milters=inet:localhost:11332
Best regards,
Minh.
phoenix
Ambassador
Ambassador
Posts: 27262
Joined: Fri Sep 12, 2014 9:56 pm
Location: Liverpool, England

Re: Rspamd: A replacement for Spamassassin & Postscreen

Post by phoenix »

As far as I can see there's nothing wrong with your configuration and I also see the "invalid transport name:" and have done for years, it doesn't appear to have any effect on rspamd. Does mail currently go through rspamd, do you see it rejecting mail etc.? If you set-up the rspamd web ui you can easily check there for mail that goes through your server.

As for the "multi.uribl.com" it could just be that it's because of the mentioned 'high volume' and that should disappear after a while. Are you using the inbuilt dnsmasq in ZCS? I use PDNS-Resolver for my caching nameserver and I don't know what the lifetime is for the cache dnsmasq, perhaps you could check that and increase it if necessary.

I do all my DKIM signing in rspamd so I don't currently use the dkimmilter, was that working before you installed rspamd? I did run the dkimmilter when I first stared with rspamd and signing worked fine for the overlap period while I configured rspamd to do the signing and once it worked I disabled the ZCS dkimmilter. Whatever services overlap between rspamd and ZCS I use the rspamd ones and disable the ZCS equivalent.
Regards

Bill

Rspamd: A high performance spamassassin replacement

Per ardua ad astra
User avatar
zimico
Outstanding Member
Outstanding Member
Posts: 225
Joined: Mon Nov 14, 2016 8:03 am
Location: Vietnam
ZCS/ZD Version: 8.8.15 P3
Contact:

Re: Rspamd: A replacement for Spamassassin & Postscreen

Post by zimico »

Dear phoenix,
This is a small lab server so I am managing to receiving more spam mail to check :). Currently mail still goes into inbox and I do not see any action log in rspamd.log. In zimbra.log I only see amavis activity.
I disabled zimbra's unbound dnscache and use dnsmasq. The strange thing is if I use dig, I see the response time is 0ms

Code: Select all

[root@mail ~]# dig yahoo.com | grep Query
;; Query time: 45 msec
[root@mail ~]# dig yahoo.com | grep Query
;; Query time: 0 msec
However when using host, I see it take some time to response:

Code: Select all

[root@mail ~]# host -a yahoo.com | grep ms
Received 789 bytes from 127.0.0.1#53 in 81 ms
[root@mail ~]# host -a yahoo.com | grep ms
Received 789 bytes from 127.0.0.1#53 in 91 ms
So I am not very sure about my dnsmasq cache...
Currently I set cache size in dnsmasq:

Code: Select all

# Increase the number of host lookups cached from the default 150
cache-size=9500
I haven't known about dkimmilter. Currently I do not enable dkim signing.
Here is what I see in the rspamd.log now:

Code: Select all

2019-06-16 22:24:40 #10025(controller) <3nxzfe>; monitored; rspamd_monitored_propagate_error: invalid return on resolving multi.uribl.com, disable object
2019-06-16 22:24:40 #10025(controller) <zqd379>; cfg; rspamd_worker_monitored_on_change: broadcast monitored update for 3nxzfegumbi67tq1kjtuupxnd493zxt: dead
2019-06-16 22:24:40 #10024(rspamd_proxy) <zqd379>; cfg; rspamd_worker_monitored_handler: updated monitored status for 3nxzfegumbi67tq1kjtuupxnd493zxt: dead
2019-06-16 22:24:40 #10028(normal) <zqd379>; cfg; rspamd_worker_monitored_handler: updated monitored status for 3nxzfegumbi67tq1kjtuupxnd493zxt: dead
2019-06-16 22:24:40 #10026(normal) <zqd379>; cfg; rspamd_worker_monitored_handler: updated monitored status for 3nxzfegumbi67tq1kjtuupxnd493zxt: dead
2019-06-16 22:25:55 #10025(controller) <3nxzfe>; monitored; rspamd_monitored_dns_cb: DNS query blocked on multi.uribl.com (127.0.0.1 returned), possibly due to high volume
2019-06-16 22:28:44 #10025(controller) <9zc4wc>; map; http_map_finish: data is not modified for server maps.rspamd.com, next check at Sun, 16 Jun 2019 19:28:43 GMT
Regards,
Minh.
User avatar
zimico
Outstanding Member
Outstanding Member
Posts: 225
Joined: Mon Nov 14, 2016 8:03 am
Location: Vietnam
ZCS/ZD Version: 8.8.15 P3
Contact:

Re: Rspamd: A replacement for Spamassassin & Postscreen

Post by zimico »

Dear all,
I use testing point of uribl:

Code: Select all

[root@mail ~]# host -tTXT 2.0.0.127.multi.uribl.com
2.0.0.127.multi.uribl.com descriptive text "127.0.0.1 -> Query Refused. See ttp://uribl.com/refused.shtml for more information [Your DNS IP: 74.....]"
So I decide to change from google DNS 8.8.8.8 to Cloudflare DNS 1.1.1.1 and reboot the server and give it a try:

Code: Select all

[root@mail ~]# host -tTXT 2.0.0.127.multi.uribl.com
2.0.0.127.multi.uribl.com descriptive text "permanent testpoint"
Review the rspamd.log and every seems to be ok now.
Regards,
Minh.
siavash
Posts: 2
Joined: Mon Apr 29, 2019 4:11 am

Re: Rspamd: A replacement for Spamassassin & Postscreen

Post by siavash »

Hi all

I've spent the past couple of days working on "why rspamd is not working on my Zimbra server". And here is why (Tested on 8.8.15 FOSS single server):

1. The LDAP configuration for milter does NOT need the phrase "smtpd_milters=" at all.
2. RSPAMD must be configured to insert the correct spam tag to emails in Zimbra-required format: X-Spam-Flag=YES

So, this is my working configuration of rspamd, packed as shell script:

Code: Select all

#!/bin/bash

# here enable milters, disable spamasassin, and redirect emails to rspamd
su - zimbra -c "zmprov ms \$(zmhostname) zimbraMtaMilterDefaultAction accept;\
zmprov ms \$(zmhostname) zimbraMtaSmtpdMilters \"inet:localhost:11332\";\
zmprov ms \$(zmhostname) zimbraMtaNonSmtpdMilters \"inet:localhost:11332\";\
zmprov -l ms \$(zmhostname) -zimbraServiceEnabled antispam;\
postconf smtpd_milters=inet:localhost:11332;\
postconf non_smtpd_milters=inet:localhost:11332;"

#rspamd configurations to match zimbra spam filter system
tee /etc/rspamd/local.d/milter_headers.conf > /dev/null <<EOT
extended_spam_headers = true;

use = ["spam-header"]

routines {
        spam-header {
                header = "X-Spam-Flag";
                value = "YES";
                remove = 0;
        }
}
EOT

tee /etc/rspamd/local.d/worker-proxy.inc > /dev/null <<EOT
upstream "local" {
	self_scan = true;
	}
EOT

#restart rspamd service
systemctl restart rspamd

#restart zimbra
su - zimbra -c "zmcontrol restart"
Hope it works for you
tonyg
Advanced member
Advanced member
Posts: 51
Joined: Fri Mar 16, 2018 5:25 pm
Location: USA
ZCS/ZD Version: 8.8.12.GA.3794.UBUNTU18.64 FOSS
Contact:

Re: Rspamd: A replacement for Spamassassin & Postscreen

Post by tonyg »

I've read every word of this 19 page thread. I've read through the wiki page. I've taken a Lot of notes in preparation for what turns out to be a rather trivial initial switch. I've been eager to try out Rspamd, but I don't think I'm ready. For all of its faults, SpamAssassin is already built-in and an integral part of this Zimbra environment. For my very modest usuage, self-hosting for about 6 domains, no responsibility to clients, I don't feel compelled to quest for the rewards that might be possible with Rspamd, when it seems SA might be fine for now. The observations and conclusions by everyone here weighed significantly in my decision.

My greatest inspiration has really been due to my respect for @phoenix. I trust whatever Bill says, and have since my very first post here and his response, just over a year ago. I've learned a Lot since then and am very thankful for every bit of information that I've been able to find here. I also appreciate that Vsevolod Stakhov, @vstakhov is author of Rspamd, has contributed to this thread.

But for me, Rspamd needs to remain on the horizon. I'll move forward with SA for some number of months and then I might come back if I feel the "brand spankin new v2.0" might scratch an itch or two.

For now, I'm posting some "errata" notes about Bill's wiki page on Rspamd, which I hope are accurate and helpful.
  1. There is a duplicate section at top, starting with "Configuring Rspamd on the Zimbra Server" down to after the "yum install rspamd redis" command.
  2. Under "Modify the ZCS…" it shows '[/code]', indicating mismatched markdown.
  3. The listing under 'zmprov gs' is missing zimbraMtaNonSmtpdMilters.
  4. By text "the following 'one', run the following (singular) command…" ... There are two commands.
  5. There is a note "If you want extended headers…" But the value is already set in the previous code block, so the reference is redundant.
  6. "The only modification you should need to make after each upgrade are the ones to the zmconfigd.cf file and..." … There was no previous mention of zmconfigd.cf.
  7. Minor grammar/typo : "and also given it" … should be " and have also given it".
  8. The example for non_smtp_milters shows null but the instruction says to change it. (There is a value)
  9. "this file needs changing as mentioned in the initial installation settings: " ... There was no previous mention of main.cf.
  10. The wiki doesn't mention zimbraServiceEnabled.
  11. The forum page1 has a lot more detail on logging.inc. I'm not sure if that was new material or code that should be added. It's subject to changes in Zimbra releases so too much reproduction of core code is probably not desirable.
  12. From page 1 of forum thread: "in addition you'll also need to modify the nightly scheduled cron job that trains the anti-spam system." There's no info about this. It's discussed later in the thread.
  13. Perhaps most of the install/configuration can be implemented in the wiki as a script, as contributed by @siavash.
  14. As I went through the wiki I was wondering why the installation couldn't be repeated through each Zimbra update. For example, the wiki documents the removal of four lines from /opt/zimbra/conf/zmconfigd.cf after an update. Why don't we need to remove those as part of the installation? If the installation includes "remove these four lines if they exist", then the intallation and become almost the exact same process, which can be scripted and simply re-run whenevever desired. That includes the creation of zmtrainsa and it's copying/overwrite of the default Zimbra code after each update.
HTH

Thanks all!
MisterM_74
Posts: 3
Joined: Sun Dec 29, 2019 11:19 am

Re: Rspamd: A replacement for Spamassassin & Postscreen

Post by MisterM_74 »

Hello

I'm the twin of MisterM75, which I'm trying to contact support to find out the email address associated with this account.

Phoenix, you forgot this in your tutorial:

Code: Select all

zimbra@zimbra-sn-u14-01:/home/oper$ zmprov ms `zmhostname` -zimbraServiceEnabled amavis

zimbra@zimbra-sn-u14-01:/home/oper$ zmcontrol restart
Because your version doesn't work, because even if you stop the antispam module, when you display the active modules again, the antispam is still there.

Trust me on this.
Mz
phoenix
Ambassador
Ambassador
Posts: 27262
Joined: Fri Sep 12, 2014 9:56 pm
Location: Liverpool, England

Re: Rspamd: A replacement for Spamassassin & Postscreen

Post by phoenix »

MisterM_74 wrote:Phoenix, you forgot this in your tutorial:
No, I didn't. I don't have a separate anti-virus scanner installed and I let Zimbra clam A/V scan the email for any viruses.
MisterM_74 wrote:Because your version doesn't work,
It's been working fine on my server for the past two and a half years without problems.
MisterM_74 wrote:...because even if you stop the antispam module, when you display the active modules again, the antispam is still there.
Not on my systems it doesn't, spamassassin is nowhere to be seen.

Code: Select all

zmprov gs $(zmhostname) | grep zimbraServiceEnabled
zimbraServiceEnabled: amavis
zimbraServiceEnabled: antivirus
zimbraServiceEnabled: logger
zimbraServiceEnabled: service
zimbraServiceEnabled: zimbra
zimbraServiceEnabled: zimbraAdmin
zimbraServiceEnabled: zimlet
zimbraServiceEnabled: mailbox
zimbraServiceEnabled: memcached
zimbraServiceEnabled: mta
zimbraServiceEnabled: stats
zimbraServiceEnabled: proxy
zimbraServiceEnabled: snmp
zimbraServiceEnabled: ldap
zimbraServiceEnabled: spell
Regards

Bill

Rspamd: A high performance spamassassin replacement

Per ardua ad astra
MisterM_74
Posts: 3
Joined: Sun Dec 29, 2019 11:19 am

Re: Rspamd: A replacement for Spamassassin & Postscreen

Post by MisterM_74 »

Hello

Another problem, how to undo amavis, because in the zimbra logs, it's him (version 8.8.15) who sends the messages.

Logical?

Mz
yeeP6rai
Posts: 41
Joined: Mon Feb 12, 2018 10:16 am

Re: Rspamd: A replacement for Spamassassin & Postscreen

Post by yeeP6rai »

How to stop scan messages from my local network?
rspamd marks system messages from local servers as spam.
Does anybody have email solution with "zimbra(with rspamd) + external mail relay"? How to configure it on antispam level? Now it's worked fine for email delivery
Post Reply