zimbra proxy don't fail-over memecached

Ask questions about your setup or get help installing ZCS server (ZD section below).
Post Reply
maxyang2008
Posts: 3
Joined: Sat Apr 15, 2017 12:03 pm

zimbra proxy don't fail-over memecached

Post by maxyang2008 »

Hi buddies,

I am testing zimbra version zcs-8.7.5_GA_1764
I have a 3 node zimbra setup like below:
  • ldap: zimbra-ldap
    mailbox&proxy server 1 : zimbra-mailbox
    mailbox&proxy server 2 : zimbra-mailbox2
services running on each node are as below:

Code: Select all

[zimbra@zimbra-ldap ~]$ zmcontrol status
Host zimbra-ldap.example.com
	ldap                    Running
	stats                   Running
	zmconfigd               Running

	
[zimbra@zimbra-mailbox ~]$ zmcontrol status
Host zimbra-mailbox.example.com
	amavis                  Running
	antispam                Running
	antivirus               Running
	dnscache                Running
	logger                  Running
	mailbox                 Running
	memcached               Running
	mta                     Running
	opendkim                Running
	proxy                   Running
	service webapp          Running
	snmp                    Running
	spell                   Running
	stats                   Running
	zimbra webapp           Running
	zimbraAdmin webapp      Running
	zimlet webapp           Running
	zmconfigd               Running

	
[zimbra@zimbra-mailbox2 ~]$ zmcontrol status
Host zimbra-mailbox2.example.com
	amavis                  Running
	antispam                Running
	antivirus               Running
	dnscache                Running
	mailbox                 Running
	memcached               Running
	mta                     Running
	opendkim                Running
	proxy                   Running
	service webapp          Running
	snmp                    Running
	spell                   Running
	stats                   Running
	zimbra webapp           Running
	zimbraAdmin webapp      Running
	zimlet webapp           Running
	zmconfigd               Running
the installation order is as below:
  • 1st: zimbra-ldap
    2nd: zimbra-mailbox
    3rd: zimbra-mailbox2
when I startup all service on zimbra-mailbox and shutdown all service on zimbra-mailbox2, I cannot login https://zimbra-mailbox
nginx.log have the following error msg:

Code: Select all

2017/04/20 01:43:58 [error] 28822#0: recv() failed (111: Connection refused)
2017/04/20 01:43:58 [error] 28822#0: reconnect to memcached channel 192.168.56.102:11211 fails
2017/04/20 01:43:58 [notice] 28822#0: memcached channel:192.168.56.102:11211 down, reconnect after:60000 ms
2017/04/20 01:43:58 [error] 28821#0: recv() failed (111: Connection refused)

the output of zmprov getAllMemcachedServers shows there 2 memcached server exist:

Code: Select all

[zimbra@zimbra-mailbox ~]$ zmprov getAllMemcachedServers
zimbra-mailbox.example.com:11211 zimbra-mailbox2.example.com:11211 
and I checked memcached on zimbra-mailbox is in running mode.
It seems nginx didn't fail-over memcached connection to zimbra-mailbox.

Did you ever encounter this problem or Can you help me figure it out why?
User avatar
DualBoot
Elite member
Elite member
Posts: 1326
Joined: Mon Apr 18, 2016 8:18 pm
Location: France - Earth
ZCS/ZD Version: ZCS FLOSS - 8.8.15 Mutli servers
Contact:

Re: zimbra proxy don't fail-over memecached

Post by DualBoot »

It says connection refused, are you running a FireWall ?
maxyang2008
Posts: 3
Joined: Sat Apr 15, 2017 12:03 pm

Re: zimbra proxy don't fail-over memecached

Post by maxyang2008 »

DualBoot wrote:It says connection refused, are you running a FireWall ?
no, there is no firewall running.
It says connetion refused because I stop memcached servivce on node zimbra-mailbox2, but I have 2 memcached server configured, proxy doesn't look for the live memcached on zimbra-mailbox.
User avatar
DualBoot
Elite member
Elite member
Posts: 1326
Joined: Mon Apr 18, 2016 8:18 pm
Location: France - Earth
ZCS/ZD Version: ZCS FLOSS - 8.8.15 Mutli servers
Contact:

Re: zimbra proxy don't fail-over memecached

Post by DualBoot »

Sorry for misunderstanding.
Can you explain what you try to aim to ?
maxyang2008
Posts: 3
Joined: Sat Apr 15, 2017 12:03 pm

Re: zimbra proxy don't fail-over memecached

Post by maxyang2008 »

DualBoot wrote:Sorry for misunderstanding.
Can you explain what you try to aim to ?
Suppose I have 2 zimbra proxy server, each have its own memcached service, I will put a load-balancer in front of the 2 proxy server, when either 1 of the 2 proxy goes down, the whole system can still operate.

The problem now is that proxy server 1 always want to connect memcached on proxy server 2, when I shutdown proxy server 2, the whole system cannot operate.
User avatar
L. Mark Stone
Ambassador
Ambassador
Posts: 2796
Joined: Wed Oct 09, 2013 11:35 am
Location: Portland, Maine, US
ZCS/ZD Version: 10.0.6 Network Edition
Contact:

Re: zimbra proxy don't fail-over memecached

Post by L. Mark Stone »

I think you'll find that this behavior appears in at least one other place too: when a mailbox server is down in a proxy environment, the upstream server list (e.g. "zmprov garpu") does not get updated.

So in a 5-mailbox server environment with a separate proxy server in front, if one of the mailbox servers is down, 1/5 of the users browsing to the proxy URL to try to login to Zimbra will be greeted with a failure, as Zimbra proxy tries to get the login screen from the next random mailbox server, and chooses the one that is offline.

There are a few bugs for this, to enable Zimbra to update the upstream mailbox server list, but in the interim I would recommend the following.

First, have separate proxy/memcached servers and separate mailbox servers. This will make upgrades much easier as you can avoid the downtime from upgrading a mailbox server by just deploying a new mailbox server and moving all of the mailboxes there, then run zmdedupe and then destroy the old mailbox server. Virtually zero downtime.

Second, if you put a load balancer in front of the pair of proxy servers, you can do maintenance on each proxy server also with no downtime perceived by the end users.

Hope that helps,
Mark
___________________________________
L. Mark Stone
Mission Critical Email - Zimbra VAR/BSP/Training Partner https://www.missioncriticalemail.com/
AWS Certified Solutions Architect-Associate
Post Reply