CSFE_SVC_ERROR (502) almost constantly after upgrade to ZCS 8.7.0

Discuss your pilot or production implementation with other Zimbra admins or our engineers.
6125amartin
Advanced member
Advanced member
Posts: 63
Joined: Sat Sep 13, 2014 1:45 am

CSFE_SVC_ERROR (502) almost constantly after upgrade to ZCS 8.7.0

Postby 6125amartin » Mon Aug 29, 2016 4:06 pm

Hello,

I am running ZCS 8.7.0 on Ubuntu 14.04. I recently upgraded to 8.7.0, but now after the upgrade users are constantly being bombarded with "Network Service Error" messages when attempting to perform various actions through the WebUI.

network service error has occurred.
method: SearchRequest
msg: A network service error has occurred.
code: CSFE_SVC_ERROR
detail: HTTP response status 502


The "Show Details" button does not have any additional information. The method can change, e.g. SearchConvRequest, GetMsgRequest, SearchRequest. I do not see anything obvious in /opt/zimbra/log/zmmailboxd.out or /opt/zimbra/log/mailbox.log. Note that I am using z-push with zimbrabackend for mobile email sync, and it does appear to reduce (but not eliminate) the frequency of these errors when I turn it off. What can I do to debug this? I am guessing the 502 errors are occurring between zimbra-proxy and the backend?


6125amartin
Advanced member
Advanced member
Posts: 63
Joined: Sat Sep 13, 2014 1:45 am

Re: CSFE_SVC_ERROR (502) almost constantly after upgrade to ZCS 8.7.0

Postby 6125amartin » Mon Aug 29, 2016 4:28 pm

Could this be caused by the DoSFilter rejecting requests above the threshold? If so, could I just set zimbraHttpDosFilterDelayMillis to 0 to completely disable the DoSFilter max requests per second restriction?
6125amartin
Advanced member
Advanced member
Posts: 63
Joined: Sat Sep 13, 2014 1:45 am

Re: CSFE_SVC_ERROR (502) almost constantly after upgrade to ZCS 8.7.0

Postby 6125amartin » Tue Aug 30, 2016 4:22 pm

I was able to resolve this by disabling zimbra-proxy and just using mailboxd directly instead
phoenix
Ambassador
Ambassador
Posts: 26697
Joined: Fri Sep 12, 2014 9:56 pm
Location: Liverpool, England

Re: CSFE_SVC_ERROR (502) almost constantly after upgrade to ZCS 8.7.0

Postby phoenix » Tue Aug 30, 2016 4:26 pm

6125amartin wrote:I was able to resolve this by disabling zimbra-proxy and just using mailboxd directly instead
The proxy is a required component in ZCS 8.7 and onwards, I'm surprised you can get away with disabling it. I'd suggest you try to research the problem you're seeing and fix that rather than disable services.
Regards

Bill

Rspamd: A high performance spamassassin replacement

Per ardua ad astra
6125amartin
Advanced member
Advanced member
Posts: 63
Joined: Sat Sep 13, 2014 1:45 am

Re: CSFE_SVC_ERROR (502) almost constantly after upgrade to ZCS 8.7.0

Postby 6125amartin » Tue Aug 30, 2016 4:54 pm

I spent all day working on this problem with help from #zimbra IRC yesterday with no resolution. We made sure the proxy was properly configured but it seems to intermittently have issues connecting to mailboxd. I thought that zimbra-proxy was required for 8.7+ as well, but it seem that it is only required that it is installed, not that it is enabled. This bug is what is driving the requirement in 8.7+, but it doesn't actually say why it is required - does anyone know why?

I've researched this problem quite extensively and cannot find any more debug information for why it was happening. There's not much in the error logs, just this:

/opt/zimbra/log/nginx.log:
2016/08/29 13:50:12 [error] 28997#0: *3892 no live upstreams while connecting to upstream, client: 192.168.1.157, server: zimbra.example.com.default, request: "POST /service/soap/ConvActionRequest HTTP/1.1", host: "zimbra.example.com", referrer: "https://zimbra.example.com/zimbra/"
2016/08/29 13:50:23 [error] 28997#0: *3892 no live upstreams while connecting to upstream, client: 192.168.1.157, server: zimbra.example.com.default, request: "POST /service/soap/NoOpRequest HTTP/1.1", host: "zimbra.example.com", referrer: "https://zimbra.example.com/zimbra/"
2016/08/29 13:50:24 [error] 28997#0: *3892 no live upstreams while connecting to upstream, client: 192.168.1.157, server: zimbra.example.com.default, request: "POST /service/soap/SearchRequest HTTP/1.1", host: "zimbra.example.com", referrer: "https://zimbra.example.com/zimbra/"


/opt/zimbra/log/mailbox.log:
2016-08-29 13:50:22,421 INFO [qtp1684106402-1162:http://zimbra.example.com/service/soap/ConvActionRequest] [name=user1@example.com;mid=82;ip=192.168.1.10;port=45304;ua=ZimbraWebClient - GC52 (Win)/8.7.0_GA_1659;] mailop - Moving VirtualConversation (id=-368187) to Folder Trash (id=3). Affected message ids: 368187.
2016-08-29 13:50:22,423 INFO [qtp1684106402-1162:http://zimbra.example.com/service/soap/ConvActionRequest] [name=user1@example.com;mid=82;ip=192.168.1.10;port=45304;ua=ZimbraWebClient - GC52 (Win)/8.7.0_GA_1659;] mailop - Moving VirtualConversation (id=-368186) to Folder Trash (id=3). Affected message ids: 368186.
2016-08-29 13:50:22,425 INFO [qtp1684106402-1162:http://zimbra.example.com/service/soap/ConvActionRequest] [name=user1@example.com;mid=82;ip=192.168.1.10;port=45304;ua=ZimbraWebClient - GC52 (Win)/8.7.0_GA_1659;] mailop - Moving Conversation (id=366405) to Folder Trash (id=3). Affected message ids: 366236,366431.
2016-08-29 13:50:22,427 INFO [qtp1684106402-1162:http://zimbra.example.com/service/soap/ConvActionRequest] [name=user1@example.com;mid=82;ip=192.168.1.10;port=45304;ua=ZimbraWebClient - GC52 (Win)/8.7.0_GA_1659;] mailop - Moving VirtualConversation (id=-366239) to Folder Trash (id=3). Affected message ids: 366239.
2016-08-29 13:50:22,429 INFO [qtp1684106402-1162:http://zimbra.example.com/service/soap/ConvActionRequest] [name=user1@example.com;mid=82;ip=192.168.1.10;port=45304;ua=ZimbraWebClient - GC52 (Win)/8.7.0_GA_1659;] mailop - Moving VirtualConversation (id=-367760) to Folder Trash (id=3). Affected message ids: 367760.
2016-08-29 13:50:22,430 INFO [qtp1684106402-1162:http://zimbra.example.com/service/soap/ConvActionRequest] [name=user1@example.com;mid=82;ip=192.168.1.10;port=45304;ua=ZimbraWebClient - GC52 (Win)/8.7.0_GA_1659;] mailop - Moving VirtualConversation (id=-367761) to Folder Trash (id=3). Affected message ids: 367761.


This page talks about what the 502 error means in regards to the proxy and mailboxd communcation:
2. 502 Bad Gateway: The server was acting as a gateway or proxy and received an invalid response from the upstream server.


This is a single-node instance, so the proxy and mailboxd are on the same server. The same action will fail with the 502 error, but then if you try it again (e.g. click on the same folder or message again), it will then work. This type of intermittent behavior makes me think it's some kind of throttling or timeout problem rather than a complete misconfiguration (which I would expect to work completely or not at all).

Based on this, what would you recommend as the next step of debug?
6125amartin
Advanced member
Advanced member
Posts: 63
Joined: Sat Sep 13, 2014 1:45 am

Re: CSFE_SVC_ERROR (502) almost constantly after upgrade to ZCS 8.7.0

Postby 6125amartin » Tue Aug 30, 2016 5:07 pm

Note that I also disabled the DoSFilter by setting zimbraInvalidLoginFilterMaxFailedLogin to 0 as well as tweaking other tunables, but modifying the DoSFIlter did not seem to help

Return to “Administrators”

Who is online

Users browsing this forum: Bing [Bot] and 17 guests