Help needed - Amavis deleting healthy mail items

Discuss your pilot or production implementation with other Zimbra admins or our engineers.
phoenix
Ambassador
Ambassador
Posts: 27272
Joined: Fri Sep 12, 2014 9:56 pm
Location: Liverpool, England

Re: Help needed - Amavis deleting healthy mail items

Post by phoenix »

Ah yes, I remember you've mentioned that before. There should be no reason for ZeXtras to have those problems regardless of the underlying operating system, I've used it for years without problems. If you had those sorts of problems with ZeXtras it would suggest to me that you probably had problems with the your 'current' installation at the time you tried it. If ZeXtras is not your preferred solution how about just exporting accounts and importing them into a new server? I can't really comment about Ubuntu, I have used it and also done a migration of ZCS from one release to another and still didn't have problems but Ubuntu is not my favourite distribution.

If you don't have any outrageous modifications on ZCS (i.e. a fairly standard install) then it would appear to me that new new build would be the best option, preferably to CentOS7 purely from my point of view of course. :)
Regards

Bill

Rspamd: A high performance spamassassin replacement

Per ardua ad astra
User avatar
JDunphy
Outstanding Member
Outstanding Member
Posts: 889
Joined: Fri Sep 12, 2014 11:18 pm
Location: Victoria, BC
ZCS/ZD Version: 9.0.0_P39 NETWORK Edition

Re: Help needed - Amavis deleting healthy mail items

Post by JDunphy »

Labsy wrote:This is nuts! Users are getting mad, my phone will overheat of complaints.
Amavis discarding messages like crazy:

Code: Select all

amavis[32456]: (32456-19) Blocked SPAM {DiscardedInbound}, ..., Queue-ID: 7F317168EBB2, mail_id: O9Pr3MirlGCG, Hits: 31.534, size: 13480, 1367 ms
postfix/smtp[10848]: 7F317168EBB2: ..., relay=127.0.0.1[127.0.0.1]:10024, delay=2.9, delays=1.5/0/0/1.4, dsn=2.7.0, status=sent (250 2.7.0 Ok, discarded, id=32456-19 - spam)
I have no idea, how to determine, which filter bumped spam score to 30+, so I cannot adjust.
Please, desperatelly need ideas what to do.
Discarding is normal for high scoring spam but in case this wasn't that type.
Did you miss making sure that the default 15 is much higher while you investigate... say:

Code: Select all

$sa_kill_level_deflt = 150.0;
So you could investigate. Given that 31.534 points which is cumulative with all the rules is over the default 15... How sure are you that you were able to make this change?
Here is a link describing some of those important variables: https://blog.bravi.org/?p=683

Determining the rule is fairly simple once you have the email... Two methods:
1) Look at this header in the email

Code: Select all

X-Spam-Status: No, score=-104.21 required=4.8 tests=[BAYES_00=-1.9,
	DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1,
	HTTP_IN_BODY=0.1, RCVD_IN_DNSWL_MED=-2.3, T_RP_MATCHES_RCVD=-0.01,
	USER_IN_DKIM_WHITELIST=-100] autolearn=ham autolearn_force=no
And verify or adjust scores for those rules.
2) capture the email and run it through spamassassin in debug mode as I showed previously and verify the rules that way.

I guess the other thing is how did you tell amavis to restart after adjusting it parameters?

Code: Select all

zmamavisdctl restart
User avatar
JDunphy
Outstanding Member
Outstanding Member
Posts: 889
Joined: Fri Sep 12, 2014 11:18 pm
Location: Victoria, BC
ZCS/ZD Version: 9.0.0_P39 NETWORK Edition

Re: Help needed - Amavis deleting healthy mail items

Post by JDunphy »

I should note this: viewtopic.php?t=144 I had discarded D_PASS because I thought it would disable spam checking... I am beginning to think it might work the same as raising the score really high... reference: https://www.ijs.si/software/amavisd/ama ... ml#actions ... Hmm lots of ways apparently.
Labsy
Outstanding Member
Outstanding Member
Posts: 411
Joined: Sat Sep 13, 2014 12:52 am

Re: Help needed - Amavis deleting healthy mail items

Post by Labsy »

Thanx, JDunphy, I was now able to think a bit more (I am so stressed and under pressure about the issue, that I barely can think normally).
So first I did now is to D_PASS all messages through, so I will be able to see message headers and brake down the spam score.

Code: Select all

 zmprov ms `zmhostname` zimbraAmavisFinalSpamDestiny D_PASS
 zmamavisdctl restart
Now just wait and catch some e-mails.

BTW...as now all mail will pass, do you have idea how to catch only those, which othervise wouldn't?
User avatar
JDunphy
Outstanding Member
Outstanding Member
Posts: 889
Joined: Fri Sep 12, 2014 11:18 pm
Location: Victoria, BC
ZCS/ZD Version: 9.0.0_P39 NETWORK Edition

Re: Help needed - Amavis deleting healthy mail items

Post by JDunphy »

Labsy wrote:Thanx, JDunphy, I was now able to think a bit more (I am so stressed and under pressure about the issue, that I barely can think normally).
So first I did now is to D_PASS all messages through, so I will be able to see message headers and brake down the spam score.

Code: Select all

 zmprov ms `zmhostname` zimbraAmavisFinalSpamDestiny D_PASS
 zmamavisdctl restart
Now just wait and catch some e-mails.

BTW...as now all mail will pass, do you have idea how to catch only those, which othervise wouldn't?
You will be looking at the headers and the score. Initially, I thought D_PASS would disable spam scoring but now I think it only disables spam discarding with amavisd-new. If that is the case, any email including this that is sent to you by the forum software should have an X-Spam-Status header.

Code: Select all

X-Spam-Status: No, score=1.567 required=5.0 tests=[BAYES_50=0.8,
	DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1,
	HEADER_FROM_DIFFERENT_DOMAINS=0.249, HTML_IMAGE_ONLY_32=0.001,
	HTML_MESSAGE=0.001, HTTP_IN_BODY=0.1, J_IMG_NO_EXTENS=0.1,
	J_RCVD_IN_HOSTKARMA_YEL=0.003, RCVD_IN_DNSWL_NONE=-0.0001,
	SPF_HELO_PASS=-0.001, T_RP_MATCHES_RCVD=-0.01, URIBL_GREY=0.424]
	autolearn=no autolearn_force=no
If you no longer have X-Spam-Status then this isn't the method you want. So in recap... score under 5 will be in your inbox or a folder by some user defined filter and anything higher will be in your junk folder. Any score higher than 15 would be normally discarded and not delivered to the junk folder. That last case is what you are most interested in.
Last edited by JDunphy on Thu Mar 15, 2018 6:44 pm, edited 1 time in total.
User avatar
ccelis5215
Outstanding Member
Outstanding Member
Posts: 632
Joined: Sat Sep 13, 2014 2:04 am
Location: Caracas - Venezuela
ZCS/ZD Version: 8.8.15.GA.3869.UBUNTU18.64 P12

Re: Help needed - Amavis deleting healthy mail items

Post by ccelis5215 »

Labsy,

You can search zimbra log

Code: Select all

grep -i spammy /var/log/zimbra.log


Hope it help.

ccelis
User avatar
JDunphy
Outstanding Member
Outstanding Member
Posts: 889
Joined: Fri Sep 12, 2014 11:18 pm
Location: Victoria, BC
ZCS/ZD Version: 9.0.0_P39 NETWORK Edition

Re: Help needed - Amavis deleting healthy mail items

Post by JDunphy »

Labsy wrote: BTW...as now all mail will pass, do you have idea how to catch only those, which othervise wouldn't?
Ohhh great tip from caccelis5215 about 'grep spammy /var/log/zimbra.log' ... learn something new every day. :-)

In the past, I have pulled the junk folder for a user that is having a problem. I'll see if I can find that program. I want to write a zimlet that a user could click and then it would provide a bunch of details about the email including why it was flagged or not flagged as spam and an option to forward on that email or parts of it to the admin for further analysis. I was going to use the zeta alliances unsubscribe zimlet as the base.

Darn... I can't find that script. Here is the general idea from what I have done in the past.

Code: Select all

zmmailbox -z -m user@example.net -t 0 getRestURL "/?fmt=tgz&query=in:junk"| tar -xz -O --wildcards '*.eml'
That would pull the junk folder so my idea was to do something like this.

Code: Select all

% zmmailbox -z -m user@example.net -t 0 getRestURL "/?fmt=tgz&query=in:junk" | tar -xz -O --wildcards '*.eml' | grep -A 10 X-Spam-Score
X-Spam-Score: 13.268
X-Spam-Level: *************
X-Spam-Status: Yes, score=13.268 required=4.8 tests=[BAYES_99=4,
	BAYES_999=0.2, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1,
	HTML_FONT_FACE_BAD=0.981, HTML_IMAGE_ONLY_20=1.546,
	HTML_MESSAGE=0.001, HTML_SHORT_LINK_IMG_3=0.148, HTTP_IN_BODY=0.1,
	J_BELOW_FOLD=1.5, J_DNSBL_MILTER_META=0.3, J_IMG_NO_EXTENS=0.1,
	J_OBFUSCATED_URL=1, KAM_INFOUSMEBIZ=0.5, RCVD_IN_IVMSIP24=2,
	RDNS_DYNAMIC=0.982, T_REMOTE_IMAGE=0.01]
	autolearn=no autolearn_force=no
gbillat
Zimbra Alumni
Zimbra Alumni
Posts: 44
Joined: Fri Oct 18, 2013 9:08 am

Re: Help needed - Amavis deleting healthy mail items

Post by gbillat »

Labsy wrote:This is nuts! Users are getting mad, my phone will overheat of complaints.

I have no idea, how to determine, which filter bumped spam score to 30+, so I cannot adjust.
Please, desperatelly need ideas what to do.
Hi,

Thanks for bringing this to our attention. We want to get your issue resolved asap, but it sounds complex. Please open a support ticket to get help directly from the Zimbra Support Team.

Thanks,
Gayle
User avatar
JDunphy
Outstanding Member
Outstanding Member
Posts: 889
Joined: Fri Sep 12, 2014 11:18 pm
Location: Victoria, BC
ZCS/ZD Version: 9.0.0_P39 NETWORK Edition

Re: Help needed - Amavis deleting healthy mail items

Post by JDunphy »

ccelis5215 wrote:Labsy,
You can search zimbra log

Code: Select all

grep -i spammy /var/log/zimbra.log
Been playing around with this a little today. Going further with your suggestion.

Code: Select all

grep -i blocked /var/log/zimbra.log
and 
grep -i blocked /var/log/zimbra.log | awk '{print $22, $12}' | sort
to see what wasn't delivered.

Our lowest was DrOzzfatburner scored at 15.013 that didn't get delivered to my own junk folder and our highest was 68.885 to our noc from SerbianBeauties. They will be disappointed. :-)
User avatar
ccelis5215
Outstanding Member
Outstanding Member
Posts: 632
Joined: Sat Sep 13, 2014 2:04 am
Location: Caracas - Venezuela
ZCS/ZD Version: 8.8.15.GA.3869.UBUNTU18.64 P12

Re: Help needed - Amavis deleting healthy mail items

Post by ccelis5215 »

ahhh.., those spammers..

Code: Select all

6.746, <MoneyNews@mysheddss.bid>
9.91, <5GMale@lawsuitss.bid>
:D

ccelis
Post Reply