Page 1 of 1

The Proxy would direct the loading of the ZWC client to the wrong mailstore host

Posted: Fri Apr 13, 2018 4:37 pm
by sistemes
Hi:

We have a zimbra installation Release 8.7.1.GA.1670.UBUNTU14.64 UBUNTU14_64 FOSS edition with three nodes:
First node named "LDAP", has these roles: Zimbra-apache Zimbra-convertd Zimbra-ldap Zimbra-logger Zimbra-snmp Zimbra-spell Zimbra-store
Second node named "MTA", has these roles: Zimbra-apache Zimbra-memcache Zimbra-mta Zimbra-proxy Zimbra-snmp
Third node is configured as an additional storage named "STORE" with these roles: Zimbra-apache Zimbra-convertd Zimbra-ldap Zimbra-snmp Zimbra-spell Zimbra-store
It worked like a charm, but the proxy to find the mailboxes of the users.

The problem we found when you login through webmail from a computer using an account which mailbox is located in the mailbox server named "LDAP".
Then you logout from that account in that computer and login in the same computer with another account which mailbox is locted in the other mailbox server named "STORE".
Then in webmail you cannot see the folders of the mailbox nor send emails.
It fails for about one hour, after one hour it works fine.
We have discovered that NGINX proxy keeps in cache the server where the mailbox of the user is located for one hour.
If we shutdown memcached service, then it works fine because nginx looks for the server where the mailbox of the user is on evey request.
If we restart all zimbra services another time in MTA/Proxy server, then the issue happens again... Until we stop memcached service.

So it seems some kind of problem related with the information that nginx keeps in memcached in order to cache the server where is located the mailbox of the user.

Any solutions to resolve this issue?
What kind or side effects we can have if we disable memcached?


Thank you advance.

Re: The Proxy would direct the loading of the ZWC client to the wrong mailstore host

Posted: Mon Apr 16, 2018 2:57 pm
by sistemes
Hi again:


The main problem with this issue is that when you access to a server with public IP through NAT, all the users from that network will share the same IP. So, some of the users in that network will have this issue.

The roles of the Zimbra servers are the same that are listed here, in the section named "Un MTA + dos Storages" (One MTA + Two Storages):
https://www.jorgedelacruz.es/2014/03/19 ... servicios/

After serveral hours of investigation, the only solution we found is to disable memcached, but it has the side effect for Zimbra that works slower.

If we check the contents of the memcached cache, we don't understand how it works, but this is the information we got:
echo "stats items" | nc localhost 11211
STAT items:3:number 2
STAT items:3:age 5
STAT items:3:evicted 0
STAT items:3:evicted_nonzero 0
STAT items:3:evicted_time 0
STAT items:3:outofmemory 0
STAT items:3:tailrepairs 0
STAT items:3:reclaimed 0
STAT items:3:expired_unfetched 0
STAT items:3:evicted_unfetched 0
STAT items:3:crawler_reclaimed 0
STAT items:3:crawler_items_checked 0
STAT items:3:lrutail_reflocked 0
END

echo "stats cachedump 3 100" | nc localhost 11211
ITEM route:proto=httpssl;id=3a031920-64fc-4661-8d21-60b46c3efa61 [14 b; 1523895465 s]
ITEM route:proto=httpssl;id=b83cecad-8d8a-4686-afa3-3d83f2cf140b [14 b; 1523895463 s]
END

We've enabled the "debug" level of logs in nginx. So we know the way nginx computes the hash to know where is the mailbox of the user is IPHASH:
2018/04/16 17:34:13 [debug] 21302#0: *45 zmauth: upstream server 127.0.1.1:8443 returned by zmlookup is not defined in upstream block, fallback to IPHASH/Fair to pick an upstream from the upstream block
...
2018/04/16 17:34:13 [debug] 21302#0: *45 zmauth: 172.16.74.92:8443 elected by iphash

Memcached with zimbra works with IPHASH by default, but we don't know if changing the way it computes the hash it make it work fine with NAT and several stores of mailboxes.



Any ideas?


Regards