amavis is very slow to deliver messages

[kokok552]

Dear all,

I've been trying to solve my problem for a week without success. So I decide to get support.

When I try to enable amavis, messages are held for a long time (aroud 20 mn) before being delivered.

I have installed ZCS 8.8.15.GA.3869.UBUNTU18.64 UBUNTU18_64 edition patch 8.8.15_P13.
I have also tried to make a fresh installation of amavis but it did not solve the problem.

Thanks in advance.

[JDunphy]

A few questions:

1) Is the amavisd cpu bound?
2) Do the messages end up in the deferred queue after tim-eout or do they complete?
3) Do you have any custom rules?
4) What version of SA are you running (spamassassin --version) ... 3.4.4 is the default for 8.8.15

A few timeouts... $active_ttl is set for 15min by default. amavisd-services will eventually send a kill -s term (SIGTERM) to wake up the stuck amavisd child who is most likely processing a SA rule. You can move it along by doing this yourself if things get really bad. Sometimes, I will move the "stuck" email to the hold queue and then run spamassassin in debug mode from the command line on the email message to isolate SA rules that might be the problem. Hint: regex101.com now has speed of execution of those regexs. I do something like this to get the email message:

Code: Select all

# /opt/zimbra/common/sbin/postcat -q QUEID | extract_msg_content.pl > message2test.txt
% su - zimbra
% spamassassin -D < message2test.txt 2>&1

If there is a long delay then comment out any custom rules and try again until you have isolated the rule. Note: amavisd has loaded a copy of the rules so you can do this without affecting those processes. It will reload the rules on a restart.
Because the messages in postfix's hold queue are self-contained to a single file, they have addition information which I parse out with this to grab just the email message with this.

Code: Select all

% cat extract_msg_content.pl 
#!/usr/bin/perl
#
# Extract the message contents from email in the hold queue
#
# Input: Parses output of postcat -q QUEID
# Ouput: Returns email msg
#

use strict;
use warnings;


my $start_str = "*** MESSAGE CONTENTS";
my $end_str = "*** HEADER EXTRACTED";
my $started = 0;

while (my $line = <STDIN>) {
   if (!$started && ($line =~ /\*\*\*\s+MESSAGE\s+CONTENTS/)) {
	$started = 1;
	next;
   }

   last if ($started && ($line =~ /\*\*\*\s+HEADER\s+EXTRACTED/));

   print $line if $started; 
}

I began writing a document last month to document some of this. It's a start but there is also a custom tool check_queue.sh that I use to isolate queue problems in zimbra. BTW, The lastest amavisd-new which has been renamed to amavisd again will now support rspamd but Zimbra hasn't kept this tool up to date so it will probably be us in the community that ports this over. In the future, we could have a remote/local rspamd in addition to SA to create more effective spam engines.

Ref: https://wiki.zimbra.com/wiki/JDunphy-Amavisd-Debugging
Ref: https://gitlab.com/amavis/amavis/-/blob ... EASE_NOTES

[kokok552]

Hi JDunphy,
Thanks for your reply.

To answer to your questions :
1- amavis is cpu-bound
2- messages are deferred
3- I did not set any rules
4- spamassassin --version gives : SpamAssassin version 3.4.2 running on perl 5.26.1

Otherwise, I don't know if it may help, I suspect that amavis is listening on port 10032 whereas postfix is delivering message on 10030. The process is hanging on that port

If you parameter files are needed, I can share.

Thanks in advance.

[JDunphy]

Hi JDunphy,
To answer to your questions :
1- amavis is cpu-bound
2- messages are deferred
3- I did not set any rules
4- spamassassin --version gives : SpamAssassin version 3.4.2 running on perl 5.26.1

You need to update SA to 3.4.4 ...

Ref: https://spamassassin.apache.org/news.html

There are a few CVE's that will cause what you describe. I spent a few painful days recently tracking how/what/why which triggered that amavisd debugging wiki article. The problem is that if you get a lot of these (more than amavisd children), it will take out ALL your amavisd's and bring your system to a crawl as you wait to time out these emails (every 15 mins)... If new email arrives, they go directly to deferred until there are amavisd that can process them. Then the older email gets processed again and the cycle repeats. Sounds like you only have a few now but it can get a lot worse as more email that causes the bug arrives.

SA 3.4.4 is the default with 8.8.15P14+ so perhaps you haven't done a yum/apt to grab an updated SA since patch 13... P16 is out now.
If that isn't an option, here is how you can apply it yourself from source or you can just upgrade the SA package via yum/apt.

https://wiki.zimbra.com/wiki/JDunphy-SA-Upgrade

HTH,

Jim

[kokok552]

Hi JDunphy,

Thanks a lot for your support. I will upgrade for sure.

By the way, I have continued to investigate my problem with ports. What I did was to enforce amavis to deliver at port 10025. Magically, this solved the slowness problem. Now it takes less than 2mn for the server to deliver messages in users mailbox. This speed is good enough for us.

Anyway, it was a rewarding experience.

Thanks