CSFE_SVC_ERROR (502) almost constantly after upgrade to ZCS 8.7.0

[6125amartin]

Hello,

I am running ZCS 8.7.0 on Ubuntu 14.04. I recently upgraded to 8.7.0, but now after the upgrade users are constantly being bombarded with "Network Service Error" messages when attempting to perform various actions through the WebUI.

network service error has occurred.
method: SearchRequest
msg: A network service error has occurred.
code: CSFE_SVC_ERROR
detail: HTTP response status 502

The "Show Details" button does not have any additional information. The method can change, e.g. SearchConvRequest, GetMsgRequest, SearchRequest. I do not see anything obvious in /opt/zimbra/log/zmmailboxd.out or /opt/zimbra/log/mailbox.log. Note that I am using z-push with zimbrabackend for mobile email sync, and it does appear to reduce (but not eliminate) the frequency of these errors when I turn it off. What can I do to debug this? I am guessing the 502 errors are occurring between zimbra-proxy and the backend?

[6125amartin]

Could this be caused by the DoSFilter rejecting requests above the threshold? If so, could I just set zimbraHttpDosFilterDelayMillis to 0 to completely disable the DoSFilter max requests per second restriction?

[6125amartin]

I was able to resolve this by disabling zimbra-proxy and just using mailboxd directly instead

[phoenix]

I was able to resolve this by disabling zimbra-proxy and just using mailboxd directly instead

The proxy is a required component in ZCS 8.7 and onwards, I'm surprised you can get away with disabling it. I'd suggest you try to research the problem you're seeing and fix that rather than disable services.

[6125amartin]

I spent all day working on this problem with help from #zimbra IRC yesterday with no resolution. We made sure the proxy was properly configured but it seems to intermittently have issues connecting to mailboxd. I thought that zimbra-proxy was required for 8.7+ as well, but it seem that it is only required that it is installed, not that it is enabled. This bug is what is driving the requirement in 8.7+, but it doesn't actually say why it is required - does anyone know why?

I've researched this problem quite extensively and cannot find any more debug information for why it was happening. There's not much in the error logs, just this:

/opt/zimbra/log/nginx.log:

2016/08/29 13:50:12 [error] 28997#0: *3892 no live upstreams while connecting to upstream, client: 192.168.1.157, server: zimbra.example.com.default, request: "POST /service/soap/ConvActionRequest HTTP/1.1", host: "zimbra.example.com", referrer: "https://zimbra.example.com/zimbra/"
2016/08/29 13:50:23 [error] 28997#0: *3892 no live upstreams while connecting to upstream, client: 192.168.1.157, server: zimbra.example.com.default, request: "POST /service/soap/NoOpRequest HTTP/1.1", host: "zimbra.example.com", referrer: "https://zimbra.example.com/zimbra/"
2016/08/29 13:50:24 [error] 28997#0: *3892 no live upstreams while connecting to upstream, client: 192.168.1.157, server: zimbra.example.com.default, request: "POST /service/soap/SearchRequest HTTP/1.1", host: "zimbra.example.com", referrer: "https://zimbra.example.com/zimbra/"

/opt/zimbra/log/mailbox.log:

2016-08-29 13:50:22,421 INFO [qtp1684106402-1162:http://zimbra.example.com/service/soap/ ... ionRequest] [name=user1@example.com;mid=82;ip=192.168.1.10;port=45304;ua=ZimbraWebClient - GC52 (Win)/8.7.0_GA_1659;] mailop - Moving VirtualConversation (id=-368187) to Folder Trash (id=3). Affected message ids: 368187.
2016-08-29 13:50:22,423 INFO [qtp1684106402-1162:http://zimbra.example.com/service/soap/ ... ionRequest] [name=user1@example.com;mid=82;ip=192.168.1.10;port=45304;ua=ZimbraWebClient - GC52 (Win)/8.7.0_GA_1659;] mailop - Moving VirtualConversation (id=-368186) to Folder Trash (id=3). Affected message ids: 368186.
2016-08-29 13:50:22,425 INFO [qtp1684106402-1162:http://zimbra.example.com/service/soap/ ... ionRequest] [name=user1@example.com;mid=82;ip=192.168.1.10;port=45304;ua=ZimbraWebClient - GC52 (Win)/8.7.0_GA_1659;] mailop - Moving Conversation (id=366405) to Folder Trash (id=3). Affected message ids: 366236,366431.
2016-08-29 13:50:22,427 INFO [qtp1684106402-1162:http://zimbra.example.com/service/soap/ ... ionRequest] [name=user1@example.com;mid=82;ip=192.168.1.10;port=45304;ua=ZimbraWebClient - GC52 (Win)/8.7.0_GA_1659;] mailop - Moving VirtualConversation (id=-366239) to Folder Trash (id=3). Affected message ids: 366239.
2016-08-29 13:50:22,429 INFO [qtp1684106402-1162:http://zimbra.example.com/service/soap/ ... ionRequest] [name=user1@example.com;mid=82;ip=192.168.1.10;port=45304;ua=ZimbraWebClient - GC52 (Win)/8.7.0_GA_1659;] mailop - Moving VirtualConversation (id=-367760) to Folder Trash (id=3). Affected message ids: 367760.
2016-08-29 13:50:22,430 INFO [qtp1684106402-1162:http://zimbra.example.com/service/soap/ ... ionRequest] [name=user1@example.com;mid=82;ip=192.168.1.10;port=45304;ua=ZimbraWebClient - GC52 (Win)/8.7.0_GA_1659;] mailop - Moving VirtualConversation (id=-367761) to Folder Trash (id=3). Affected message ids: 367761.

This page talks about what the 502 error means in regards to the proxy and mailboxd communcation:

2. 502 Bad Gateway: The server was acting as a gateway or proxy and received an invalid response from the upstream server.

This is a single-node instance, so the proxy and mailboxd are on the same server. The same action will fail with the 502 error, but then if you try it again (e.g. click on the same folder or message again), it will then work. This type of intermittent behavior makes me think it's some kind of throttling or timeout problem rather than a complete misconfiguration (which I would expect to work completely or not at all).

Based on this, what would you recommend as the next step of debug?

[6125amartin]

Note that I also disabled the DoSFilter by setting zimbraInvalidLoginFilterMaxFailedLogin to 0 as well as tweaking other tunables, but modifying the DoSFIlter did not seem to help