Problem accessing ZCS upstream server. Cannot connect to the ZCS upstream server. Connection is refused.

Discuss your pilot or production implementation with other Zimbra admins or our engineers.
Post Reply
davidkillingsworth
Outstanding Member
Outstanding Member
Posts: 251
Joined: Sat Sep 13, 2014 2:26 am
ZCS/ZD Version: 8.8.15.GA.3869.UBUNTU14.64-Patch 24

Problem accessing ZCS upstream server. Cannot connect to the ZCS upstream server. Connection is refused.

Post by davidkillingsworth »

HTTP ERROR 502
Problem accessing ZCS upstream server. Cannot connect to the ZCS upstream server. Connection is refused.
Possible reasons:

upstream server is unreachable
upstream server is currently being upgraded
upstream server is down
Please contact your ZCS administrator to fix the problem.

Powered by Nginx-Zimbra://
I am getting the above error periodically on multiple servers.

Operating System: Ubuntu 16.04.7 LTS
Zimbra: Release 8.8.15.GA.3869.UBUNTU14.64 UBUNTU16_64 FOSS edition, Patch 8.8.15_P23.

I believe the issue started on P22 and upgrading to P23 did not fix the issue.

I spent 4 or 5 hours on this after upgrading to P22 and I couldn't fix the issue.

Is anybody else seeing this isue?

IMAP mail works fine, but webmail is almost unusable. These servers have been in operation for a few years and I keep them patched regularly, so the problem is definitely very recent.
davidkillingsworth
Outstanding Member
Outstanding Member
Posts: 251
Joined: Sat Sep 13, 2014 2:26 am
ZCS/ZD Version: 8.8.15.GA.3869.UBUNTU14.64-Patch 24

Re: Problem accessing ZCS upstream server. Cannot connect to the ZCS upstream server. Connection is refused.

Post by davidkillingsworth »

I'm still having this issue and I can't figure it out.

Sometimes, I have problems logging into the web interface for several minutes in a row, and then suddenly it will start working fine.

Here are some more details:
/opt/zimbra/log/nginx.log - These errors show up over and over

Code: Select all

2021/07/15 09:49:24 [error] 3465#0: reconnect to memcached channel <myserverpublicIPaddress>:11211 fails
2021/07/15 09:49:24 [notice] 3465#0: memcached channel:<myserverpublicIPaddress>:11211 down, reconnect after:60000 ms
2021/07/15 09:49:25 [notice] 3464#0: reconnect to memcached channel <myserverpublicIPaddress>:11211 (rc: -2)
2021/07/15 09:49:25 [error] 3464#0: recv() failed (111: Connection refused)
/opt/zimbra/log/nginx.access.log - These errors show up over and over when I try to access the web interface.

Code: Select all

222.166.95.4:42766 - - [15/Jul/2021:09:21:48 +0800]  "GET https://<myserverpublicDNSname>/js/JQuery_all.js.zgz?v=210621064822 HTTP/2.0" 200 32079 "https://<myserverpublicDNSname>/" "Mozilla/5.0 (Windows NT 10.0; rv:78.0) Gecko/20100101 Firefox/78.0" "<myserverpublicIPaddress>:8443" "<myserverpublicIPaddress>:443"
222.166.95.4:42766 - - [15/Jul/2021:09:21:48 +0800]  "GET https://<myserverpublicDNSname>/js/Startup1_1_all.js.zgz?v=210621064822 HTTP/2.0" 200 194878 "https://<myserverpublicDNSname>/" "Mozilla/5.0 (Windows NT 10.0; rv:78.0) Gecko/20100101 Firefox/78.0" "<myserverpublicIPaddress>:8443" "<myserverpublicIPaddress>:443"
222.166.95.4:42766 - - [15/Jul/2021:09:21:48 +0800]  "GET https://<myserverpublicDNSname>/js/Startup1_2_all.js.zgz?v=210621064822 HTTP/2.0" 502 1245 "https://<myserverpublicDNSname>/" "Mozilla/5.0 (Windows NT 10.0; rv:78.0) Gecko/20100101 Firefox/78.0" "" "<myserverpublicIPaddress>:443"
222.166.95.4:42766 - - [15/Jul/2021:09:21:49 +0800]  "GET https://<myserverpublicDNSname>/res/I18nMsg,TzMsg,AjxMsg,ZMsg,ZmMsg,AjxKeys,ZmKeys,ZdMsg,AjxTemplateMsg.js.zgz?v=210621064822&debug=false&language=en&country=US&skin=harmony HTTP/2.0" 200 72740 "https://<myserverpublicDNSname>/" "Mozilla/5.0 (Windows NT 10.0; rv:78.0) Gecko/20100101 Firefox/78.0" "<myserverpublicIPaddress>:8443" "<myserverpublicIPaddress>:443"
222.166.95.4:42766 - - [15/Jul/2021:09:21:49 +0800]  "GET https://<myserverpublicDNSname>/img/zimbra.gif?v=210621064822 HTTP/2.0" 502 1245 "https://<myserverpublicDNSname>/" "Mozilla/5.0 (Windows NT 10.0; rv:78.0) Gecko/20100101 Firefox/78.0" "" "<myserverpublicIPaddress>:443"
222.166.95.4:42766 - - [15/Jul/2021:09:21:49 +0800]  "HEAD https://<myserverpublicDNSname>/public/blank.html?_=1626312109000 HTTP/2.0" 502 77 "https://<myserverpublicDNSname>/" "Mozilla/5.0 (Windows NT 10.0; rv:78.0) Gecko/20100101 Firefox/78.0" "" "<myserverpublicIPaddress>:443"
222.166.95.4:42766 - - [15/Jul/2021:09:21:49 +0800]  "GET https://<myserverpublicDNSname>/img/zimbra.png?v=210621064822 HTTP/2.0" 502 1245 "https://<myserverpublicDNSname>/" "Mozilla/5.0 (Windows NT 10.0; rv:78.0) Gecko/20100101 Firefox/78.0" "" "<myserverpublicIPaddress>:443"
222.166.95.4:42766 - - [15/Jul/2021:09:21:49 +0800]  "GET https://<myserverpublicDNSname>/js/Startup1_2_all.js.zgz?v=210621064822 HTTP/2.0" 502 1245 "https://<myserverpublicDNSname>/" "Mozilla/5.0 (Windows NT 10.0; rv:78.0) Gecko/20100101 Firefox/78.0" "" "<myserverpublicIPaddress>:443"
222.166.95.4:42766 - - [15/Jul/2021:09:21:49 +0800]  "GET https://<myserverpublicDNSname>/img/logo/favicon.ico;jsessionid=node0al0cpwr0ccwfruuf2ws7mcwu5.node0 HTTP/2.0" 502 1245 "https://<myserverpublicDNSname>/" "Mozilla/5.0 (Windows NT 10.0; rv:78.0) Gecko/20100101 Firefox/78.0" "" "<myserverpublicIPaddress>:443"
222.166.95.4:42766 - - [15/Jul/2021:09:22:16 +0800]  "GET https://<myserverpublicDNSname>/ HTTP/2.0" 200 40737 "https://<myserverpublicDNSname>/" "Mozilla/5.0 (Windows NT 10.0; rv:78.0) Gecko/20100101 Firefox/78.0" "<myserverpublicIPaddress>:8443" "<myserverpublicIPaddress>:443"

Code: Select all

zimbra@mail:~$ lsof -i :443
COMMAND  PID   USER   FD   TYPE   DEVICE SIZE/OFF NODE NAME
nginx   3464 zimbra   11u  IPv4    49721      0t0  TCP *:https (LISTEN)
nginx   3465 zimbra   11u  IPv4    49721      0t0  TCP *:https (LISTEN)
nginx   3466 zimbra   11u  IPv4    49721      0t0  TCP *:https (LISTEN)
nginx   3466 zimbra   20u  IPv4 65269536      0t0  TCP mail.mydomain.com:https->22266622333.hkcable.com.hk:35505 (ESTABLISHED)
nginx   3467 zimbra   11u  IPv4    49721      0t0  TCP *:https (LISTEN)

Code: Select all

zimbra@mail:~$ lsof -i :11211
COMMAND    PID   USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
memcached 3442 zimbra   26u  IPv4  51334      0t0  TCP localhost:11211 (LISTEN)
memcached 3442 zimbra   27u  IPv4  49769      0t0  TCP localhost:11211->localhost:40132 (ESTABLISHED)

Code: Select all

zimbra@mail:~$ netstat -tnpl | grep memcached 
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp        0      0 127.0.0.1:11211         0.0.0.0:*               LISTEN      3442/memcached

Anybody have any ideas what's wrong?
davidkillingsworth
Outstanding Member
Outstanding Member
Posts: 251
Joined: Sat Sep 13, 2014 2:26 am
ZCS/ZD Version: 8.8.15.GA.3869.UBUNTU14.64-Patch 24

Re: Problem accessing ZCS upstream server. Cannot connect to the ZCS upstream server. Connection is refused.

Post by davidkillingsworth »

I'm still having this issue and I just noticed this message in the nginx.log.

Code: Select all

no memcache server available, cannot post request while SSL handshaking to lookup handler
Any idea how I can debug memcache?
davidkillingsworth
Outstanding Member
Outstanding Member
Posts: 251
Joined: Sat Sep 13, 2014 2:26 am
ZCS/ZD Version: 8.8.15.GA.3869.UBUNTU14.64-Patch 24

Re: Problem accessing ZCS upstream server. Cannot connect to the ZCS upstream server. Connection is refused.

Post by davidkillingsworth »

Actually, looking on the proxy wiki help page, I found a section that described my problem and had some commands.
https://wiki.zimbra.com/wiki/Zimbra_Proxy_Guide

I looked at this:

Code: Select all

$ zmprov gcf zimbraMailProxyMaxFails
which produced:

Code: Select all

zimbraMailProxyMaxFails: 1
I then ran the following:

Code: Select all

zmprov mcf zimbraMailProxyMaxFails 0
I haven't been able to re-produce the problem since.

I'm not sure what this really means, but it is working fine now and it's a low traffic server, so I guess it's fine.
Post Reply