Help needed - Amavis deleting healthy mail items

[Labsy]

Hi,

I have already started similar thread here viewtopic.php?f=13&t=63828
...but unfortunatelly it is kind of urgent, as mail items are disappearing randomly and many without trace.

Symptoms:
Users are complaining, that after upgrading ZCS to 8.8.7 they simply miss some of incoming mails.
Some even say, that they could swear they actually saw particular mail already in inbox, but later could not find it anymore.

Technically:
In /opt/log/zimbra.log I can see such entries for REAL SPAM:

Code: Select all

Mar 14 16:57:30 seven amavis[22112]: (22112-18) Blocked SPAM {DiscardedInbound}, [180.123.161.71]:2307 [180.123.161.71] <kxrn@ljky.com> -> <recipient@zimbraserver.com>, Queue-ID: A03961690324, Message-ID: <20180314155727.A03961690324@zimbra.server.com>, mail_id: p5OL-_SM0FhN, Hits: 18.349, size: 1901, 1668 ms

...but also HUNDREDS of those for HEALTHY MAIL messages:

Code: Select all

Mar 14 17:53:50 seven amavis[3021]: (03021-13) Blocked SPAM {DiscardedInbound}, [109.203.209.228]:55079 [109.203.209.228] <accounts@cpanel.net> -> <recipient@zimbraserver.com>, Queue-ID: E2A4B16903A5, Message-ID: <20180314235336.99CF81F466ED6B4D@cpanel.net>, mail_id: y8g_O6SgjAav, Hits: 37.48, size: 214090, 1479 ms

And the latest (healthy ones) I cannot find anywhere on the server.
Not in /opt/zimbra/data/amavis nor anywhere else.
I looked up into particular USER ID in /opt/zimbra/store and messages are NOT there.

I am lost and becoming desparate...what can I do?
If I disable Amavis, there will be a lot of real spam.
But something obviously changed with upgrade to 8.8.7, rendering Amavis and underlying algorithms to be too strict and to simply DISCARD the incoming messages, rather than marking it as SPAM.

[JDunphy]

I am lost and becoming desparate...what can I do?
If I disable Amavis, there will be a lot of real spam.

Have you thought about bumping up 15 to something higher so the mail stays in the users junk folder instead of being discarded while you focus on root cause and any local rules you have created. Check out amavisd.conf to change that 15 to something high enough for your environment... There might be a more zimbra way of changing that value but I would just use an editor.

Note: If you are setting local SA rules with big scores. ie. past a score of 15 then you need to lower them short term IMHO. I think this thread shows some of that: viewtopic.php?f=15&t=63809&p=282244#p282244 You could have scored SPF_FAIL to 5 and forced it to junk but you went to 30 which would discarded it with a single rule... even a BAYES_00 can't rescue that if it was a FP. A few 4 and 5 point rules and one gets to 15 fairly fast.

You could bring in Kevin McGrail rules (KAM) https://www.pccc.com/downloads/SpamAssa ... rib/KAM.cf who is actively involved in SA development.They are more aggressive than the default SA rules and you would add them to your existing mix of zimbra rules.

[Labsy]

Hi JDunphy,

thank you for tips, but unfortunatelly I would ask for some clarification. I am quite a noob here regarding which engine does what and how are they linked or embedded one inside other. Amavis, SpamAssasin, Bayes...I know those and other terms, but I don't know, if there is one on the top, controlling all others...or they work in parallel or in series and they need to be managed separately.
Kevin's rules seem excellent, ready for production....but with my lack of knowledge - how to implement those and even if I do, those are SA rules, what about Amavis and other mechanisms?

As you can see, I am lost due to lack of knowledge, which translates to being lost

[JDunphy]

Sure... SA is run by amavisd-new which also does a few other things like clam, etc. It's the quarterback. There are two scores of importance here... When SA with all its rules returns scores at 5 or above, it will end up in the users junk folder. If it is more than 15, it isn't delivered. You can change some of that behavior via:

Code: Select all

zmprov gs `zmhostname` zimbraAmavisFinalSpamDestiny

where Valid values are:

D_PASS -> Deliver all spam regardless of score
D_BOUNCE -> Generate a bounce message
D_REJECT -> Reject the message
D_DISCARD -> (default) Silently discard the message

Now Check out: sa_tag2_level_deflt and sa_kill_level_deflt inside amavisd.conf to see those scores default at 5 and 15 scores. SA is perl and includes various plugins and modules including the bayes statistical training that gets data when the users hit junk or not junk, etc.

If you look at each email message, you will find this header X-Spam-Status which tells you why the system thought it was junk or not junk and which rules fired. Search some of the forums posts about running '/opt/zimbra/common/bin/spamassassin --lint' to see how you can test your own rules for any email. I show some examples where I show lint so that should locate those entries.

Your rules

Code: Select all

ls -l /opt/zimbra/data/spamassassin/state/3.004001/updates_spamassassin_org

Your SA perl code

Code: Select all

ls -l /opt/zimbra/common/lib/perl5/Mail/SpamAssassin

Your local changes or overrides

Code: Select all

/opt/zimbra/data/spamassassin/localrules/sauser.cf

You could put KAM.cf in sauser.cf or you can reference it in a path that SA looks at in the rules directory.
It's a lot of fun once you get the hang of it. Creating custom rules is incredibly easy.

[Labsy]

Wow, that's clear as in school
So, Amavis-new is kinda "wrapper" for all AS/AV filters.

I added Kevin's rules now easily.
I also changed:

Code: Select all

$defang_by_ccat{CC_BADH.",3"} = 1;  # NUL or CR character in header
$defang_by_ccat{CC_BADH.",5"} = 1;  # header line longer than 998 characters
$defang_by_ccat{CC_BADH.",6"} = 1;  # header field syntax error

to

Code: Select all

$defang_by_ccat{CC_BADH.",3"} = 0;  # NUL or CR character in header
$defang_by_ccat{CC_BADH.",5"} = 1;  # header line longer than 998 characters
$defang_by_ccat{CC_BADH.",6"} = 0;  # header field syntax error

to avoid discarding malformed headers (those are messages from automated systems, like accounting systems, web forms...where developers won't change the whole system for the sake of few users, which do not get such mails.

And somehow I found that $my_networks from SERVER and GENERAL settings were not the same!
In SERVER section there were missing IPs of my spam proxy, so I added them.

I also tried to add RAMdisk for Amavis /tmp directory...
...but I found so many different instructions, and I don't want to mess with fstab - so I quit doing this.
(/dev/shm on my syystem is already there. Also tmpfs is listed many times already, where sum is over allocated RAM. And it looks too much for me...)

Code: Select all

df -h
Filesystem      Size  Used Avail Use% Mounted on
udev             16G     0   16G   0% /dev
tmpfs           3.2G   17M  3.2G   1% /run
/dev/sda1       723G  555G  132G  81% /
tmpfs            16G     0   16G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs            16G     0   16G   0% /sys/fs/cgroup
cgmfs           100K     0  100K   0% /run/cgmanager/fs
tmpfs           3.2G     0  3.2G   0% /run/user/999
tmpfs           3.2G     0  3.2G   0% /run/user/1000

Code: Select all

 ls -al /dev/shm/
total 0
drwxrwxrwt  2 root root   40 Mar 14 22:50 .
drwxr-xr-x 19 root root 3920 Mar 14 22:50 ..

[JDunphy]

I added Kevin's rules now easily.
I also changed:

Code: Select all

$defang_by_ccat{CC_BADH.",3"} = 1;  # NUL or CR character in header
$defang_by_ccat{CC_BADH.",5"} = 1;  # header line longer than 998 characters
$defang_by_ccat{CC_BADH.",6"} = 1;  # header field syntax error

to

Code: Select all

$defang_by_ccat{CC_BADH.",3"} = 0;  # NUL or CR character in header
$defang_by_ccat{CC_BADH.",5"} = 1;  # header line longer than 998 characters
$defang_by_ccat{CC_BADH.",6"} = 0;  # header field syntax error

to avoid discarding malformed headers (those are messages from automated systems, like accounting systems, web forms...where developers won't change the whole system for the sake of few users, which do not get such mails.

Great stuff ... you have passed me in changes. I haven't deployed his rules myself as they appear they could require more frequent updates given what they are testing for. He is really good about keeping them updated however from what I see in the SA mailing lists. I would verify they linted in case you are missing some other rules they depend on... ie.

Code: Select all

su - zimbra
/opt/zimbra/common/bin/spamassassin --lint

We tend to focus our rules on envelope, phishing, embedded user tracking in the body's html, and obfuscation methods that tend to be of benefit longer and reduce our effort from having to update the rules too often. So we don't look for the word "money" in the subject line. <grin>

And somehow I found that $my_networks from SERVER and GENERAL settings were not the same!
In SERVER section there were missing IPs of my spam proxy, so I added them.

Yep, it is pretty important to have your ip's listed from any incoming MX's so SA and its rules operate on the next hop and not your own servers. I tend to put them in my sauser.cf file via this directive just to be sure.
trusted_networks 127.0.0.0/8 X.X.X.X/32 Y.Y.Y.Y Z.Z.Z.Z/24

You can then verify that fact with any email you saved like:

Code: Select all

su - zimbra
spamassassin -D  < /tmp/cyrillic-nm.mail > /dev/null 2> /tmp/3.err

Then look inside /tmp/3.err to see which rules fired and if it was the correct hop, etc. Adding the -L flag doesn't do the network parts of the SA rules. ie.

Code: Select all

su - zimbra
spamassassin -L -D  < /tmp/cyrillic-nm.mail > /dev/null 2> /tmp/3.err

I also tried to add RAMdisk for Amavis /tmp directory...
...but I found so many different instructions, and I don't want to mess with fstab - so I quit doing this.

I tend to side with the kernel in deciding how to manage it's memory. Most of the time, a lot of those short lived files don't get written out to disk and reducing the pool of memory from memory hungry java apps or apps that can utilize memory with more application level caching may cause bottlenecks elsewhere. Not saying it can't help in some circumstances but without knowing if that is an actual bottleneck, I tend to be very conservative with these type of changes. Spent too many years as a system programmer and being wrong too often that its hard for me to second guess without more details.

[Labsy]

That's some more stuff I need to digest from your posts, thanx

Meanwhile, I received another bunch of complaints, users saying some INTERNAL mail got lost. Sent from user A to users BCDE inside the same domain....but there's no mail in user's B mailbox.
Checked /var/log/zimbra.log and found:
- Original message 1 from the same A user to the same BCDE users with same subject PASSED Amavis filters and was delivered OK
- Reply-to-All message 2 from the same A user to the same BCDE users with same subject (just Re:) also PASSED Amavis filters, but is not found in BCDE user's mailboxes?!

So I went deeper and examined /opt/zimbra/log/mailbox.log
And WOILA! found out what was wrong with 2nd message:

Code: Select all

lmtp - Not delivering message with duplicate Message-ID

Actually, original message and Replied-to-all message (15 minutes later) actually do have same Message-ID.

Notice 1st message at 13:17 hours, then 17 minutes later 2nd mail using Reply-to-All, same ID:

Code: Select all

zgrep -ai "AAAAAAAAAAAuAAAAAAAAAJbnhu/cNuNCrZHYGMTwC3sBAMO2jhD3dRHOtM0AqgC7tuYAAAAAAA4AABAAAADVRjSFFaABSb6VIRaEz0JgAQAAAAA=@domain.com" /opt/zimbra/log/mailbox.log.2018-03-13.gz
2018-03-13 13:17:52,798 INFO  [LmtpServer-66] [ip=10.10.11.50;] lmtp - Delivering message: size=14985 bytes, nrcpts=5, sender=sender@domain.com, msgid=<!&!AAAAAAAAAAAuAAAAAAAAAJbnhu/cNuNCrZHYGMTwC3sBAMO2jhD3dRHOtM0AqgC7tuYAAAAAAA4AABAAAADVRjSFFaABSb6VIRaEz0JgAQAAAAA=@domain.com>

2018-03-13 13:34:43,326 INFO  [LmtpServer-66] [ip=10.10.11.50;] lmtp - Delivering message: size=20403 bytes, nrcpts=5, sender=sender@domain.com, msgid=<!&!AAAAAAAAAAAuAAAAAAAAAJbnhu/cNuNCrZHYGMTwC3sBAMO2jhD3dRHOtM0AqgC7tuYAAAAAAA4AABAAAADVRjSFFaABSb6VIRaEz0JgAQAAAAA=@domain.com>
2018-03-13 13:34:43,326 INFO  [LmtpServer-66] [name=recipientA@domain.com;mid=596;ip=10.10.11.50;] lmtp - Not delivering message with duplicate Message-ID <!&!AAAAAAAAAAAuAAAAAAAAAJbnhu/cNuNCrZHYGMTwC3sBAMO2jhD3dRHOtM0AqgC7tuYAAAAAAA4AABAAAADVRjSFFaABSb6VIRaEz0JgAQAAAAA=@domain.com>
2018-03-13 13:34:43,326 INFO  [LmtpServer-66] [name=recipientB@domain.com;mid=601;ip=10.10.11.50;] lmtp - Not delivering message with duplicate Message-ID <!&!AAAAAAAAAAAuAAAAAAAAAJbnhu/cNuNCrZHYGMTwC3sBAMO2jhD3dRHOtM0AqgC7tuYAAAAAAA4AABAAAADVRjSFFaABSb6VIRaEz0JgAQAAAAA=@domain.com>
2018-03-13 13:34:43,326 INFO  [LmtpServer-66] [name=recipientC@domain.com;mid=412;ip=10.10.11.50;] lmtp - Not delivering message with duplicate Message-ID <!&!AAAAAAAAAAAuAAAAAAAAAJbnhu/cNuNCrZHYGMTwC3sBAMO2jhD3dRHOtM0AqgC7tuYAAAAAAA4AABAAAADVRjSFFaABSb6VIRaEz0JgAQAAAAA=@domain.com>

I tried to replicate the behaviour, but without success.
Is this a BUG of 8.8.7?

***EDIT***
Found old topic on the same subject: viewtopic.php?t=28155
The culprit seems to be Outlook, sending the same MessageID with the same/replied message.

[Labsy]

This is nuts! Users are getting mad, my phone will overheat of complaints.
Amavis discarding messages like crazy:

Code: Select all

amavis[32456]: (32456-19) Blocked SPAM {DiscardedInbound}, ..., Queue-ID: 7F317168EBB2, mail_id: O9Pr3MirlGCG, Hits: 31.534, size: 13480, 1367 ms
postfix/smtp[10848]: 7F317168EBB2: ..., relay=127.0.0.1[127.0.0.1]:10024, delay=2.9, delays=1.5/0/0/1.4, dsn=2.7.0, status=sent (250 2.7.0 Ok, discarded, id=32456-19 - spam)

I have no idea, how to determine, which filter bumped spam score to 30+, so I cannot adjust.
Please, desperatelly need ideas what to do.

[phoenix]

Possibly not the best suggestion in the world but have you considered installing the latest ZCS on a new server and migrating your current users & config to that with ZeXtras?

[Labsy]

Possibly not the best suggestion in the world but have you considered installing the latest ZCS on a new server and migrating your current users & config to that with ZeXtras?

Hi,
I have been trying with ZExtras before, but many items and settings were not migrated:
- some of mailboxes were empty
- calendars were also migrated poorly
- external account retreivals were not migrated either
Maybe it was because underlying OS was different at that time (source Ubuntu 10.04, target 16.04, both same ZCS versions), but all in all I don't trust ZExtras migration since.