zmspamextract fails to pull emails from junk or ham account

Discuss your pilot or production implementation with other Zimbra admins or our engineers.
guilhermehnog
Posts: 12
Joined: Fri Aug 02, 2019 1:56 pm

zmspamextract fails to pull emails from junk or ham account

Post by guilhermehnog »

Hi.
i'm new to zimbra and i would be really grateful if one of you could help me.

I'm running zimbra 8.8.15 on a CentOS 7 and is sending and receiving email just fine.
The problem is that i don't think spamassassin is learning, and ou users mailbox is flooded with spam.

i followed this tutorials on improving spamassassin:

https://wiki.zimbra.com/wiki/Anti-spam_Strategies
viewtopic.php?t=65628

I get this error running zmtrainsa:

Code: Select all

[zimbra@mail localrules]$ zmtrainsa email spam
20190815180903 Starting spamassassin spam training for email using folder junk
Exception in thread "main" java.lang.NullPointerException
        at com.zimbra.common.httpclient.HttpClientUtil.executeMethod(HttpClientUtil.java:53)
        at com.zimbra.common.httpclient.HttpClientUtil.executeMethod(HttpClientUtil.java:49)
        at com.zimbra.cs.util.SpamExtract.extractMessages(SpamExtract.java:367)
        at com.zimbra.cs.util.SpamExtract.extract(SpamExtract.java:291)
        at com.zimbra.cs.util.SpamExtract.main(SpamExtract.java:220)
netset: cannot include 127.0.0.0/8 as it has already been included
netset: cannot include 0:0:0:0:0:0:0:1/128 as it has already been included
netset: cannot include IP as it has already been included
Learned tokens from 0 message(s) (0 message(s) examined)
netset: cannot include 127.0.0.0/8 as it has already been included
netset: cannot include 0:0:0:0:0:0:0:1/128 as it has already been included
netset: cannot include IP as it has already been included
bayes: synced databases from journal in 0 seconds: 730 unique entries (1255 total entries)
20190815180908 Finished spamassassin spam training for Email using folder junk
[zimbra@mail localrules]$
and this is a spam header of an email that was clearly a spam:

X-Spam-Flag: NO
X-Spam-Score: -0.757
X-Spam-Level:
X-Spam-Status: No, score=-0.757 required=2 tests=[BAYES_00=-0.5,
CK_HELO_GENERIC=0.249, DKIM_SIGNED=0.1, DKIM_VALID=-0.1,
DKIM_VALID_AU=-0.1, DMARC_PASS_REJECT=-1.2,
HTML_FONT_LOW_CONTRAST=0.001, HTML_MESSAGE=0.001, MPART_ALT_DIFF=0.79,
RCVD_IN_DNSWL_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001,
URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no

this is our MTA restricion:

Code: Select all

[zimbra@mail localrules]$ zmprov gacf | grep zimbraMtaRestriction
zimbraMtaRestriction: check_reverse_client_hostname_access pcre:/opt/zimbra/conf/fqrdns.pcre
zimbraMtaRestriction: reject_invalid_helo_hostname
zimbraMtaRestriction: reject_non_fqdn_sender
zimbraMtaRestriction: reject_unknown_reverse_client_hostname
zimbraMtaRestriction: reject_rbl_client dnsbl.spfbl.net
zimbraMtaRestriction: reject_rbl_client psbl.surriel.com
zimbraMtaRestriction: reject_rbl_client zen.spamhaus.org
zimbraMtaRestriction: reject_rbl_client bl.spamcop.net
zimbraMtaRestriction: reject_rbl_client b.barracudacentral.org
zimbraMtaRestriction: reject_rhsbl_client dbl.spamhaus.org
zimbraMtaRestriction: reject_rhsbl_client multi.uribl.com
zimbraMtaRestriction: reject_rhsbl_client multi.surbl.org
zimbraMtaRestriction: reject_rhsbl_client rhsbl.sorbs.net
zimbraMtaRestriction: reject_rhsbl_reverse_client dbl.spamhaus.org
zimbraMtaRestriction: reject_rhsbl_sender multi.uribl.com
zimbraMtaRestriction: reject_rhsbl_sender multi.surbl.org
zimbraMtaRestriction: reject_rhsbl_sender rhsbl.sorbs.net
zimbraMtaRestriction: reject_rhsbl_sender dbl.spamhaus.org
also i get this error when runnin sa-learn --dump all -D

Code: Select all

dbg: sa-learn: spamtest initialized
Aug 15 19:07:12.597 [23720] dbg: plugin: Mail::SpamAssassin::Plugin::Bayes=HASH(0x331f258) implements 'learner_dump_database', priority 0
Aug 15 19:07:12.597 [23720] dbg: bayes: no dbs present, cannot tie DB R/O: /opt/zimbra/.spamassassin/bayes_toks
Aug 15 19:07:12.597 [23720] dbg: plugin: Mail::SpamAssassin::Plugin::Bayes=HASH(0x331f258) implements 'learner_close', priority 0
this is my saucer.cf

Code: Select all

# pyzor
use_pyzor 1
pyzor_path /usr/bin/pyzor
# DNS lookups for pyzor can time out easily.  Set the following line IF you want to give pyzor up to 20 seconds to respond
# may slow down email delivery
pyzor_timeout 20

# razor
use_razor2 1


score URIBL_BLACK 3.250
score RAZOR2_CHECK 3.250
score PYZOR_CHECK 3.250
score BAYES_99 4.000
score BAYES_60 2.250
score BAYES_50 1.500
score BAYES_00 -0.500
score RP_MATCHES_RCVD -0.000
score SPF_FAIL 3.000
score SPF_HELO_FAIL 3.000
score SPAMCOP_BL 3.500
score RCVD_IN_BRBL_LASTEXT 3.500
score HEADER_FROM_DIFFERENT_DOMAINS 2.000
thanks in advance
User avatar
L. Mark Stone
Ambassador
Ambassador
Posts: 2796
Joined: Wed Oct 09, 2013 11:35 am
Location: Portland, Maine, US
ZCS/ZD Version: 10.0.6 Network Edition
Contact:

Re: SpamAssassin not learning

Post by L. Mark Stone »

Per haps this might help:

https://www.missioncriticalemail.com/20 ... ices-2019/

All the best,
Mark
___________________________________
L. Mark Stone
Mission Critical Email - Zimbra VAR/BSP/Training Partner https://www.missioncriticalemail.com/
AWS Certified Solutions Architect-Associate
guilhermehnog
Posts: 12
Joined: Fri Aug 02, 2019 1:56 pm

Re: SpamAssassin not learning

Post by guilhermehnog »

L. Mark Stone wrote:Per haps this might help:

https://www.missioncriticalemail.com/20 ... ices-2019/

All the best,
Mark
Thanks.
I believe i passed through this page as well.
i'm going to double check that.

i also noted that spamtrain.log is empty and my spam account is not being cleaned up.
the same java error appears.
User avatar
L. Mark Stone
Ambassador
Ambassador
Posts: 2796
Joined: Wed Oct 09, 2013 11:35 am
Location: Portland, Maine, US
ZCS/ZD Version: 10.0.6 Network Edition
Contact:

Re: SpamAssassin not learning

Post by L. Mark Stone »

guilhermehnog wrote:
L. Mark Stone wrote:Per haps this might help:

https://www.missioncriticalemail.com/20 ... ices-2019/

All the best,
Mark
Thanks.
I believe i passed through this page as well.
i'm going to double check that.

i also noted that spamtrain.log is empty and my spam account is not being cleaned up.
the same java error appears.
Post up the errors and maybe we can help!

All the best,
Mark
___________________________________
L. Mark Stone
Mission Critical Email - Zimbra VAR/BSP/Training Partner https://www.missioncriticalemail.com/
AWS Certified Solutions Architect-Associate
guilhermehnog
Posts: 12
Joined: Fri Aug 02, 2019 1:56 pm

Re: SpamAssassin not learning

Post by guilhermehnog »

L. Mark Stone wrote:
guilhermehnog wrote:
L. Mark Stone wrote:Per haps this might help:

https://www.missioncriticalemail.com/20 ... ices-2019/

All the best,
Mark
Thanks.
I believe i passed through this page as well.
i'm going to double check that.

i also noted that spamtrain.log is empty and my spam account is not being cleaned up.
the same java error appears.
Post up the errors and maybe we can help!

All the best,
Mark
Error when i run the zmtrainsa clean up:

Code: Select all

[zimbra@mail root]$ zmtrainsa --cleanup
20190816085142 Starting spam/ham cleanup
Exception in thread "main" java.lang.NullPointerException
        at com.zimbra.common.httpclient.HttpClientUtil.executeMethod(HttpClientUtil.java:53)
        at com.zimbra.common.httpclient.HttpClientUtil.executeMethod(HttpClientUtil.java:49)
        at com.zimbra.cs.util.SpamExtract.extractMessages(SpamExtract.java:367)
        at com.zimbra.cs.util.SpamExtract.extract(SpamExtract.java:291)
        at com.zimbra.cs.util.SpamExtract.main(SpamExtract.java:220)
Exception in thread "main" java.lang.NullPointerException
        at com.zimbra.common.httpclient.HttpClientUtil.executeMethod(HttpClientUtil.java:53)
        at com.zimbra.common.httpclient.HttpClientUtil.executeMethod(HttpClientUtil.java:49)
        at com.zimbra.cs.util.SpamExtract.extractMessages(SpamExtract.java:367)
        at com.zimbra.cs.util.SpamExtract.extract(SpamExtract.java:291)
        at com.zimbra.cs.util.SpamExtract.main(SpamExtract.java:220)
20190816085146 Finished spam/ham cleanup
Error when i run zmtrainsa manualy:

Code: Select all

[zimbra@mail root]$ zmtrainsa "my email" spam
20190816085310 Starting spamassassin spam training for "my email" using folder junk
Exception in thread "main" java.lang.NullPointerException
        at com.zimbra.common.httpclient.HttpClientUtil.executeMethod(HttpClientUtil.java:53)
        at com.zimbra.common.httpclient.HttpClientUtil.executeMethod(HttpClientUtil.java:49)
        at com.zimbra.cs.util.SpamExtract.extractMessages(SpamExtract.java:367)
        at com.zimbra.cs.util.SpamExtract.extract(SpamExtract.java:291)
        at com.zimbra.cs.util.SpamExtract.main(SpamExtract.java:220)
netset: cannot include 127.0.0.0/8 as it has already been included
netset: cannot include 0:0:0:0:0:0:0:1/128 as it has already been included
netset: cannot include "my IP" as it has already been included
ERROR: configuration specifies 'use_bayes 0', sa-learn disabled
netset: cannot include 127.0.0.0/8 as it has already been included
netset: cannot include 0:0:0:0:0:0:0:1/128 as it has already been included
netset: cannot include "my IP" as it has already been included
ERROR: configuration specifies 'use_bayes 0', sa-learn disabled
20190816085316 Finished spamassassin spam training for "my email" using folder junk
Error when i run sa-learn -- dump all:
i can eliminate this error creating a link from /opt/zimbra/data/amavisd/.spamassassin/ on /opt/zimbra/.spamassassin/

Code: Select all

dbg: sa-learn: spamtest initialized
Aug 16 08:53:58.392 [12504] dbg: plugin: Mail::SpamAssassin::Plugin::Bayes=HASH(0x3664ed8) implements 'learner_dump_database', priority 0
Aug 16 08:53:58.392 [12504] dbg: bayes: no dbs present, cannot tie DB R/O: /opt/zimbra/.spamassassin/bayes_toks
Aug 16 08:53:58.392 [12504] dbg: plugin: Mail::SpamAssassin::Plugin::Bayes=HASH(0x3664ed8) implements 'learner_close', priority 0
ERROR: Bayes dump returned an error, please re-run with -D for more information
I checked and when the user mark a message as spam, it is sended to the spam account, but then nothing more happens. the messages get stuck in the mailbox
and spam assassin don't appear to process them.

Also spamtrain.log is empty.

Thanks!
User avatar
JDunphy
Outstanding Member
Outstanding Member
Posts: 889
Joined: Fri Sep 12, 2014 11:18 pm
Location: Victoria, BC
ZCS/ZD Version: 9.0.0_P39 NETWORK Edition

Re: SpamAssassin not learning

Post by JDunphy »

That java error from zmtrainsa is probably this: /opt/zimbra/libexec/zmspamextract that is used by zmtrainsa to extract email from the ham and spam account. Can you run that java component by hand to see if that is the issue.

Code: Select all

# su - zimbra
% /opt/zimbra/libexec/zmspamextract -s -o /tmp/some_spam_directory_not_created_or_created
% /opt/zimbra/libexec/zmspamextract -n -o /tmp/some_ham_directory_not_created_or_created
Note: if you don't have any spam or ham then these directories could be empty so train something as spam first if you are not sure.

ref: viewtopic.php?f=15&t=65303
guilhermehnog
Posts: 12
Joined: Fri Aug 02, 2019 1:56 pm

Re: SpamAssassin not learning

Post by guilhermehnog »

JDunphy wrote:That java error from zmtrainsa is probably this: /opt/zimbra/libexec/zmspamextract that is used by zmtrainsa to extract email from the ham and spam account. Can you run that java component by hand to see if that is the issue.

Code: Select all

# su - zimbra
% /opt/zimbra/libexec/zmspamextract -s -o /tmp/some_spam_directory_not_created_or_created
% /opt/zimbra/libexec/zmspamextract -n -o /tmp/some_ham_directory_not_created_or_created
Note: if you don't have any spam or ham then these directories could be empty so train something as spam first if you are not sure.

ref: viewtopic.php?f=15&t=65303
Hi.
Thanks for answering.

here is the output of the command:

Code: Select all

[zimbra@mail tmp]$ /opt/zimbra/libexec/zmspamextract -s -o /tmp/spambox
Exception in thread "main" java.lang.NullPointerException
        at com.zimbra.common.httpclient.HttpClientUtil.executeMethod(HttpClientUtil.java:53)
        at com.zimbra.common.httpclient.HttpClientUtil.executeMethod(HttpClientUtil.java:49)
        at com.zimbra.cs.util.SpamExtract.extractMessages(SpamExtract.java:367)
        at com.zimbra.cs.util.SpamExtract.extract(SpamExtract.java:291)
        at com.zimbra.cs.util.SpamExtract.main(SpamExtract.java:220)

Code: Select all

[zimbra@mail tmp]$ /opt/zimbra/libexec/zmspamextract -n -o /tmp/hambox
Exception in thread "main" java.lang.NullPointerException
        at com.zimbra.common.httpclient.HttpClientUtil.executeMethod(HttpClientUtil.java:53)
        at com.zimbra.common.httpclient.HttpClientUtil.executeMethod(HttpClientUtil.java:49)
        at com.zimbra.cs.util.SpamExtract.extractMessages(SpamExtract.java:367)
        at com.zimbra.cs.util.SpamExtract.extract(SpamExtract.java:291)
        at com.zimbra.cs.util.SpamExtract.main(SpamExtract.java:220)
Note: i found somewhere in the forum that deleting the emails of spam account and starting over would fix the problem.
i did that, and when i execute the cleaning no error appears but as soon as an email is sent to the spam account the error returns. the same happens when i use the command for a single account

here is the content of zmspamextract:

Code: Select all

#!/bin/bash
# 
# ***** BEGIN LICENSE BLOCK *****
# Zimbra Collaboration Suite Server
# Copyright (C) 2006, 2007, 2009, 2010, 2013, 2014, 2016 Synacor, Inc.
#
# This program is free software: you can redistribute it and/or modify it under
# the terms of the GNU General Public License as published by the Free Software Foundation,
# version 2 of the License.
#
# This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY;
# without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
# See the GNU General Public License for more details.
# You should have received a copy of the GNU General Public License along with this program.
# If not, see <https://www.gnu.org/licenses/>.
# ***** END LICENSE BLOCK *****
# 

exec `dirname $0`/../bin/zmjava com.zimbra.cs.util.SpamExtract "$@"
User avatar
JDunphy
Outstanding Member
Outstanding Member
Posts: 889
Joined: Fri Sep 12, 2014 11:18 pm
Location: Victoria, BC
ZCS/ZD Version: 9.0.0_P39 NETWORK Edition

Re: SpamAssassin not learning

Post by JDunphy »

I don't run 8.8.15 myself but you would think they have delivered a version that can pull email from an account or how would your web interface work? Curious!

Check logs for permission or other errors as you drill down further on this. Start with /opt/zimbra/log/mailbox.log and continue from there. /var/log/zimbra.log is another location to look if you have exhausted /opt/zimbra/log/ ... perhaps something like this:

Code: Select all

# su - zimbra
% cd /opt/zimbra/log
% egrep -i '(crash|^core|error|panic|fatal)' *.log
You could also add debug and verbose flags to zmspamextract to see if any additional information is provided prior to that exception. You can see the options here:

Ref: https://github.com/jimedler/Zimbra-1/bl ... tract.java

Another thing to verify is if the ham and spam accounts match what the local configuration thinks are their names. That you can see them from the admin interface and that they are populated is a mystery to me.
I would first verify if ham extraction works... ie. trained as NOT 'spam/junk' ... if that works then focus on the spam account and if something got messed up with its creation.

Too many possibilities at this point so further information is necessary for me to help as I have never seen this error myself. I have heard of people recreating the spam/ham accounts.

Ref:https://wiki.zimbra.com/wiki/Steps_to_r ... e_accounts

It works pretty well for us here.

Jim
guilhermehnog
Posts: 12
Joined: Fri Aug 02, 2019 1:56 pm

Re: SpamAssassin not learning

Post by guilhermehnog »

JDunphy wrote:I don't run 8.8.15 myself but you would think they have delivered a version that can pull email from an account or how would your web interface work? Curious!

Check logs for permission or other errors as you drill down further on this. Start with /opt/zimbra/log/mailbox.log and continue from there. /var/log/zimbra.log is another location to look if you have exhausted /opt/zimbra/log/ ... perhaps something like this:

Code: Select all

# su - zimbra
% cd /opt/zimbra/log
% egrep -i '(crash|^core|error|panic|fatal)' *.log
You could also add debug and verbose flags to zmspamextract to see if any additional information is provided prior to that exception. You can see the options here:

Ref: https://github.com/jimedler/Zimbra-1/bl ... tract.java

Another thing to verify is if the ham and spam accounts match what the local configuration thinks are their names. That you can see them from the admin interface and that they are populated is a mystery to me.
I would first verify if ham extraction works... ie. trained as NOT 'spam/junk' ... if that works then focus on the spam account and if something got messed up with its creation.

Too many possibilities at this point so further information is necessary for me to help as I have never seen this error myself. I have heard of people recreating the spam/ham accounts.

Ref:https://wiki.zimbra.com/wiki/Steps_to_r ... e_accounts

It works pretty well for us here.

Jim

I tried to recreate spam|ham accounts, no change on the results.

this is the output the debug of zmspamextract:

Code: Select all

/opt/zimbra/libexec/zmspamextract -D -s -o /tmp/spambox

Code: Select all

[] DEBUG: http-outgoing-2 << "<soap:Envelope xmlns:soap="http://www.w3.org/2003/05/soap-envelope"><soap:Header><context xmlns="urn:zimbra"><change token="99"/></context></soap:Header><soap:Body><SearchResponse offset="0" more="0" sortBy="dateDesc" xmlns="urn:zimbraMail"><m rev="2" s="73400" sf="1566004430000" d="1566004430000" f="au" cm="1" id="257" l="2" cid="-257"><fr>Classified-By: EMAIL Classified-As: spam Action: move Source-Folder: /Inbox Destination-Folder: /Junk Destination-Mailbox:</fr><e a="MAILER-DAEMON@server.domainr" d="MAILER-DAEMON" t="f"/><su>zimbra-spam-report: EMAIL: spam</su></m></SearchResponse></soap:Body></soap:Envelope>"
[] DEBUG: http-outgoing-2 << HTTP/1.1 200 OK
[] DEBUG: http-outgoing-2 << Date: Sat, 17 Aug 2019 01:34:58 GMT
[] DEBUG: http-outgoing-2 << Content-Type: text/xml;charset=utf-8
[] DEBUG: http-outgoing-2 << Cache-Control: no-store, no-cache
[] DEBUG: http-outgoing-2 << Vary: Accept-Encoding, User-Agent
[] DEBUG: http-outgoing-2 << Content-Length: 672
[] DEBUG: Connection can be kept alive indefinitely
[] DEBUG: Cancelling request execution
[] DEBUG: http-outgoing-2: Shutdown connection
[] DEBUG: Connection discarded
[] DEBUG: Connection released: [id: 2][route: {s}->https://server:8443][total kept alive: 0; route allocated: 0 of 100; total allocated: 0 of 300]
[] DEBUG: <SearchResponse offset="0" more="0" sortBy="dateDesc" xmlns="urn:zimbraMail">
  <m rev="2" s="73400" sf="1566004430000" d="1566004430000" f="au" cm="1" id="257" l="2" cid="-257">
    <fr>Classified-By: email Classified-As: spam Action: move Source-Folder: /Inbox Destination-Folder: /Junk Destination-Mailbox:</fr>
    <e a="MAILER-DAEMON@server.domain" d="MAILER-DAEMON" t="f"/>
    <su>zimbra-spam-report: email: spam</su>
  </m>
</SearchResponse>
[] DEBUG: adding id 257
[] DEBUG: sending request for path /service/user/spam.account/?fmt=tgz&list=257
[] DEBUG: Fetching /service/user/spam.account/?fmt=tgz&list=257
Exception in thread "main" java.lang.NullPointerException
        at com.zimbra.common.httpclient.HttpClientUtil.executeMethod(HttpClientUtil.java:53)
        at com.zimbra.common.httpclient.HttpClientUtil.executeMethod(HttpClientUtil.java:49)
        at com.zimbra.cs.util.SpamExtract.extractMessages(SpamExtract.java:367)
        at com.zimbra.cs.util.SpamExtract.extract(SpamExtract.java:291)
        at com.zimbra.cs.util.SpamExtract.main(SpamExtract.java:220)
About the access to the spam and ham accoun, i just type spam or ham on the search bar and the account appear. then i click to open the email.

I'll go look for errors in the logs.
andrey.ivanov
Advanced member
Advanced member
Posts: 50
Joined: Wed Aug 08, 2018 8:44 am

Re: SpamAssassin not learning

Post by andrey.ivanov »

The exception is in the com.zimbra.common.httpclient.HttpClientUtil class. It could be a new Zimbra bug related to the major change of version of HttpCLient ("Upgraded 3rd Party HttpClient to version 4.5.5." from https://wiki.zimbra.com/wiki/Zimbra_Releases/8.8.15).

There is already a known issue with the same reason, with a workaround "WebMail Login Failure - when zimbraCsrfTokenCheckEnabled is set to FALSE Apache HttpClient Connection pool shuts down and login to webmail fails.". So maybe it's a second side effect of HttpClient upgrade in 8.8.15...
Post Reply