8.8.9 -> 8.8.10 new bug: Impossible to send mail to large distribution lists using web interface (ZWC)

[andrey.ivanov]

It is impossible to send mail to large distribution lists (more than 600-1000 members in our production server) using web interface (ZWC). The problem is not present in 8.8.9, it appeared in 8.8.10 (cf the section "Analysis").

Code: Select all

[zimbra@zimbra ~]$ zmcontrol -v
Release 8.8.10_GA_3039.RHEL6_64_20180928094617 RHEL6_64 NETWORK edition, Patch 8.8.10_P3.

Here is how to reproduce it:
1. Create a large distribution list in Zimbra :

cdl large-test-dl@listes.mydomain.com
mdl large-test-dl@listes.mydomain.com zimbraDistributionListSubscriptionPolicy REJECT
mdl large-test-dl@listes.mydomain.com zimbraDistributionListUnsubscriptionPolicy REJECT
mdl large-test-dl@listes.mydomain.com displayName "Large distribution list for tests"
adlm large-test-dl@listes.mydomain.com user1@zimbra.mydomain.com
adlm large-test-dl@listes.mydomain.com user2@zimbra.mydomain.com
...
adlm large-test-dl@listes.mydomain.com user1000@zimbra.mydomain.com

2. Use Zimbra Web Client (usual web interface) to send a mail to this distribution list. In 8.8.9 it works fine. In 8.8.10 ZWC waits for a couple of minutes and then gives an error "message not sent".
The real world log file (anonymized) zimbra.log shows ~135 seconds (11:32:51 -> 11:35:06) wait and then abandon of web interface submission (ZWC timeout trying to send mail i imagine):

Code: Select all

Nov 22 11:32:51 zimbra postfix/smtpd[2284]: connect from zimbra.mydomain.com[1.2.3.4]
Nov 22 11:32:51 zimbra postfix/smtpd[2284]: NOQUEUE: filter: RCPT from zimbra.mydomain.com[1.2.3.4]: <some_user@zimbra.mydomain.com>: Sender address triggers FILTER smtp-amavis:[127.0.0.1]:10026; from=<some_user@zimbra.mydomain.com> to=<large-list@listes.mydomain.com> proto=ESMTP helo=<zimbra.mydomain.com>
Nov 22 11:32:51 zimbra postfix/smtpd[2284]: 341E0720212: client=zimbra.mydomain.com[1.2.3.4]
Nov 22 11:35:06 zimbra postfix/smtpd[2284]: lost connection after DATA (6 bytes) from zimbra.mydomain.com[1.2.3.4]
Nov 22 11:35:06 zimbra postfix/smtpd[2284]: disconnect from zimbra.mydomain.com[11.2.3.4] ehlo=1 mail=1 rcpt=1 data=0/1 commands=3/4
Nov 22 11:35:06 zimbra postfix/cleanup[6664]: 341E0720212: message-id=<20181122103251.341E0720212@zimbra.mydomain.com>

Analysis:
as i said these lists worked fine in 8.8.9 and the ZWC timeout in 8.8.10 happens when postfix tries to expand the large mailing list using ldap (to be precise, using the file /opt/zimbra/conf/ldap-vam.cf).
There is a new feature of 8.8.10 :

Code: Select all

* Added ChangePrimaryEmailRequest to zmprov to change the primary email address of an account while keeping the existing account active as an alias, with no account outage. (Bug 104915)

This new feature has introduced a new LDAP attribute - "zimbraOldMailAddress". This attribute now is a part of an LDAP filter used by postfix to expand distribution lists using the file
/opt/zimbra/conf/ldap-vam.cf:
in 8.8.9 :

Code: Select all

 query_filter = (&(|(zimbraMailDeliveryAddress=%s)(zimbraMailAlias=%s)(zimbraMailCatchAllAddress=%s))(zimbraMailStatus=enabled))

in 8.8.10:

Code: Select all

query_filter = (&(|(zimbraMailDeliveryAddress=%s)(zimbraMailAlias=%s)(zimbraOldMailAddress=%s)(zimbraMailCatchAllAddress=%s))(zimbraMailStatus=enabled))

The new additional part of this filter "(zimbraOldMailAddress=%s)" slows down the query by the factor of 20. Either 'zimbraOldMailAddress' attribute is not indexed for presence and equality or there is some other reason, i have made a test file checking the old and new filter query time.
The file test-perf.sh:

Code: Select all

PW=`zmlocalconfig -s ldap_postfix_password| cut -f 2 -d '='|sed -e "s/\s//g"`
TESTMAIL='some_mail@mydomain.com'

echo "New LDAP filter with (zimbraOldMailAddress=$TESTMAIL):"
time ldapsearch -x -h zimbra.mydomain.com  -D "uid=zmpostfix,cn=appaccts,cn=zimbra" -w $PW "(&(|(zimbraMailDeliveryAddress=$TESTMAIL)(zimbraMailAlias=$TESTMAIL)(zimbraOldMailAddress=$TESTMAIL)(zimbraMailCatchAllAddress=$TESTMAIL))(zimbraMailStatus=enabled))" zimbraMailDeliveryAddress zimbraMailForwardingAddress zimbraPrefMailForwardingAddress zimbraMailCatchAllForwardingAddress > /dev/null
echo
echo
echo "Old LDAP filter without (zimbraOldMailAddress=$TESTMAIL):"
time ldapsearch -x -h zimbra.mydomain.com  -D "uid=zmpostfix,cn=appaccts,cn=zimbra" -w $PW "(&(|(zimbraMailDeliveryAddress=$TESTMAIL)(zimbraMailAlias=$TESTMAIL)(zimbraMailCatchAllAddress=$TESTMAIL))(zimbraMailStatus=enabled))" zimbraMailDeliveryAddress zimbraMailForwardingAddress zimbraPrefMailForwardingAddress zimbraMailCatchAllForwardingAddress > /dev/null

Using this file we can check that the time of search with 8.8.10 filter is 20 times larger :

New LDAP filter with (zimbraOldMailAddress=andrey.ivanov@polytechnique.edu):
real 0m0.097s
user 0m0.004s
sys 0m0.000s

Old LDAP filter without (zimbraOldMailAddress=andrey.ivanov@polytechnique.edu):
real 0m0.005s
user 0m0.004s
sys 0m0.000s


Workaround:
the only urgent workaround i have found is simply using the old LDAP filter in /opt/zimbra/conf/ldap-vam.cf. This file is generated by /opt/zimbra/libexec/zmmtainit (lines 95-109). So i have changed the query_filter at line 99 of /opt/zimbra/libexec/zmmtainit to the old value:

Code: Select all

query_filter = (&(|(zimbraMailDeliveryAddress=%s)(zimbraMailAlias=%s)(zimbraMailCatchAllAddress=%s))(zimbraMailStatus=enabled))

Then i restarted the mta (zmmtactl restart). The file /opt/zimbra/conf/ldap-vam.cf became the same as in 8.8.9 and the problem has disappeared.


Solution:
The regression in 8.8.10 concerning ldap filter performance used for mailing list expansion (/opt/zimbra/conf/ldap-vam.cf) should be fixed. The attribute zimbraOldMailAddress should be correctly indexed and the time of two searches (with old and new filter) should be the same. I have made a ticket (Case 00786396) at Zimbra support.

[fs.schmidt]

Hello Andrey,

Thanks for sharing the solution. We are facing the same problem.

Best regards.

[dnzktlr]

thank you for solution

[gabrieles]

Thanks for your solution, it towed us out of the mud!

[kitsanai.kmtw]

Hi Andrey,

Thanks for sharing the solution.
This solution is work !!

Best regards.
kitsanai.kmtw