CSFE_SVC_ERROR HTTP response status 504

Discuss your pilot or production implementation with other Zimbra admins or our engineers.
Post Reply
nisamudeen97
Posts: 11
Joined: Fri Aug 19, 2016 4:28 am

CSFE_SVC_ERROR HTTP response status 504

Post by nisamudeen97 »

With our Zimbra/8.8.15_GA webmail randomly get non responsive while trying to send email. We were getting the below error with webmail. This has been happening once in every two weeks now. When the issue is there all webmail users will have the same problem while trying to send. We have to force restart the service and everything is back normal. Our system is production one and we have 80 email users in the system. This issue has become really annoying now. Zimbra is running on ec2 instance with 2 core 16GB ram CentOS Linux release 8.5.2111. I have checked the logs in detail and below are my findings. Has some one here faced the same issue? Can some one help me to resolve the case?

A network service error has occurred.
method: SendMsgRequest
msg: A network service error has occurred.
code: CSFE_SVC_ERROR
detail: HTTP response status 504
Screenshot from 2022-12-15 22-55-25.png
Screenshot from 2022-12-15 22-55-25.png (111.79 KiB) Viewed 3027 times
I have checked the zimbra logs in detail and i could see the below information in /opt/zimbra/log/mailbox.log

Code: Select all

/opt/zimbra/log/mailbox.log

Code: Select all

2022-12-14 17:35:21,435 WARN  [qtp97652294-33101:https://mail.domain.com/service/soap/SendMsgRequest] [name=ops@domain.com;mid=58;ip=172.31.7.229;port=48182;ua=ZimbraWebClient - G
C108 (Mac)/8.8.15_GA_4372;soapId=2f6ca405;] SoapEngine - handler exception
com.zimbra.common.service.ServiceException: system failure: Unable to send message
        at com.zimbra.common.service.ServiceException.FAILURE(ServiceException.java:292) ~[zimbracommon.jar:8.8.15_GA_4372]
        at com.zimbra.cs.mailbox.MailSender.sendMimeMessage(MailSender.java:816) ~[zimbrastore.jar:8.8.15_GA_4372]
        at com.zimbra.cs.mailbox.MailSender.sendMimeMessage(MailSender.java:534) ~[zimbrastore.jar:8.8.15_GA_4372]
        at com.zimbra.cs.mailbox.MailSender.sendMimeMessage(MailSender.java:451) ~[zimbrastore.jar:8.8.15_GA_4372]
        at com.zimbra.cs.service.mail.SendMsg.getItemId(SendMsg.java:294) ~[zimbrastore.jar:8.8.15_GA_4372]
        at com.zimbra.cs.service.mail.SendMsg.doSendMessage(SendMsg.java:361) ~[zimbrastore.jar:8.8.15_GA_4372]
        at com.zimbra.cs.service.mail.SendMsg.handle(SendMsg.java:231) ~[zimbrastore.jar:8.8.15_GA_4372]
        at com.zimbra.cs.service.mail.SendMsg.handle(SendMsg.java:110) ~[zimbrastore.jar:8.8.15_GA_4372]
        at com.zimbra.soap.SoapEngine.dispatchRequest(SoapEngine.java:646) ~[zimbrastore.jar:8.8.15_GA_4372]
        at com.zimbra.soap.SoapEngine.dispatch(SoapEngine.java:491) ~[zimbrastore.jar:8.8.15_GA_4372]
        at com.zimbra.soap.SoapEngine.dispatch(SoapEngine.java:278) ~[zimbrastore.jar:8.8.15_GA_4372]
        at com.zimbra.soap.SoapServlet.doWork(SoapServlet.java:308) ~[zimbrastore.jar:8.8.15_GA_4372]
        at com.zimbra.soap.SoapServlet.doPost(SoapServlet.java:217) ~[zimbrastore.jar:8.8.15_GA_4372]
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) ~[servlet-api-3.1.jar:3.1.0]
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799) ~[jetty-servlet-9.4.46.v20220331.jar:9.4.46.v20220331]
        at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1631) ~[jetty-servlet-9.4.46.v20220331.jar:9.4.46.v20220331]
        at org.eclipse.jetty.websocket.server.WebSocketUpgradeFilter.doFilter(WebSocketUpgradeFilter.java:292) ~[websocket-server-9.4.46.v20220331.jar:9.4.46.v20220331]
        at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) ~[jetty-servlet-9.4.46.v20220331.jar:9.4.46.v20220331]
        at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) ~[jetty-servlet-9.4.46.v20220331.jar:9.4.46.v20220331]
        at com.zimbra.cs.servlet.CsrfFilter.doFilter(CsrfFilter.java:175) ~[zimbrastore.jar:8.8.15_GA_4372]
        at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) ~[jetty-servlet-9.4.46.v20220331.jar:9.4.46.v20220331]
        at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) ~[jetty-servlet-9.4.46.v20220331.jar:9.4.46.v20220331]
        at com.zimbra.cs.servlet.RequestStringFilter.doFilter(RequestStringFilter.java:54) ~[zimbrastore.jar:8.8.15_GA_4372]
        at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) ~[jetty-servlet-9.4.46.v20220331.jar:9.4.46.v20220331]
        at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) ~[jetty-servlet-9.4.46.v20220331.jar:9.4.46.v20220331]
        at com.zimbra.cs.servlet.SetHeaderFilter.doFilter(SetHeaderFilter.java:59) ~[zimbrastore.jar:8.8.15_GA_4372]
        at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) ~[jetty-servlet-9.4.46.v20220331.jar:9.4.46.v20220331]
        at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) ~[jetty-servlet-9.4.46.v20220331.jar:9.4.46.v20220331]
        at com.zimbra.cs.servlet.ETagHeaderFilter.doFilter(ETagHeaderFilter.java:47) ~[zimbrastore.jar:8.8.15_GA_4372]
        at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) ~[jetty-servlet-9.4.46.v20220331.jar:9.4.46.v20220331]
        at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) ~[jetty-servlet-9.4.46.v20220331.jar:9.4.46.v20220331]
        at com.zimbra.cs.servlet.ContextPathBasedThreadPoolBalancerFilter.doFilter(ContextPathBasedThreadPoolBalancerFilter.java:107) ~[zimbrastore.jar:8.8.15_GA_4372]
        at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) ~[jetty-servlet-9.4.46.v20220331.jar:9.4.46.v20220331]
        at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) ~[jetty-servlet-9.4.46.v20220331.jar:9.4.46.v20220331]
        at com.zimbra.cs.servlet.ZimbraQoSFilter.doFilter(ZimbraQoSFilter.java:125) ~[zimbrastore.jar:8.8.15_GA_4372]
        at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) ~[jetty-servlet-9.4.46.v20220331.jar:9.4.46.v20220331]
        at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) ~[jetty-servlet-9.4.46.v20220331.jar:9.4.46.v20220331]
        at com.zimbra.cs.servlet.ZimbraInvalidLoginFilter.doFilter(ZimbraInvalidLoginFilter.java:117) ~[zimbrastore.jar:8.8.15_GA_4372]
        at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) ~[jetty-servlet-9.4.46.v20220331.jar:9.4.46.v20220331]
        at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) ~[jetty-servlet-9.4.46.v20220331.jar:9.4.46.v20220331]
        at org.eclipse.jetty.servlets.DoSFilter.doFilterChain(DoSFilter.java:487) ~[jetty-servlets-9.4.46.v20220331.jar:9.4.46.v20220331]
        at org.eclipse.jetty.servlets.DoSFilter.doFilter(DoSFilter.java:336) ~[jetty-servlets-9.4.46.v20220331.jar:9.4.46.v20220331]
        at org.eclipse.jetty.servlets.DoSFilter.doFilter(DoSFilter.java:301) ~[jetty-servlets-9.4.46.v20220331.jar:9.4.46.v20220331]
        at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:164) [jetty-io-9.4.46.v20220331.jar:9.4.46.v20220331]
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) [jetty-io-9.4.46.v20220331.jar:9.4.46.v20220331]
        at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) [jetty-io-9.4.46.v20220331.jar:9.4.46.v20220331]
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338) [jetty-util-9.4.46.v20220331.jar:9.4.46.v20220331]
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315) [jetty-util-9.4.46.v20220331.jar:9.4.46.v20220331]
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173) [jetty-util-9.4.46.v20220331.jar:9.4.46.v20220331]
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131) [jetty-util-9.4.46.v20220331.jar:9.4.46.v20220331]
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409) [jetty-util-9.4.46.v20220331.jar:9.4.46.v20220331]
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) [jetty-util-9.4.46.v20220331.jar:9.4.46.v20220331]
        at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) [jetty-util-9.4.46.v20220331.jar:9.4.46.v20220331]
        at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: com.zimbra.cs.mailbox.MailSender$SafeMessagingException: Read timed out
        at com.zimbra.cs.mailclient.smtp.SmtpTransport.protocolConnect(SmtpTransport.java:203) ~[zimbrastore.jar:8.8.15_GA_4372]
nisamudeen97
Posts: 11
Joined: Fri Aug 19, 2016 4:28 am

Re: CSFE_SVC_ERROR HTTP response status 504

Post by nisamudeen97 »

Hi,

I am facing this issue for the past few months. A reboot fixes the issue and services are back. But i wanted to understand root cause of the problem and solve it. Like i have mentioned this is a production system and lots of live users are there.
nisamudeen97
Posts: 11
Joined: Fri Aug 19, 2016 4:28 am

Re: CSFE_SVC_ERROR HTTP response status 504

Post by nisamudeen97 »

Hi,

Yesterday again the same issue was faced. Had to do a force reboot of the instance and everything is back again working. Can Some one help me to find the root cause of the problem?
Aleksej
Posts: 15
Joined: Sun Dec 17, 2017 5:39 pm

Re: CSFE_SVC_ERROR HTTP response status 504

Post by Aleksej »

I have the same problem on 8.8.15
All is up to date.

But i can't send mail from web and also server not recieving mail.
Klug
Ambassador
Ambassador
Posts: 2746
Joined: Mon Dec 16, 2013 11:35 am
Location: France - Drôme
ZCS/ZD Version: All of them
Contact:

Re: CSFE_SVC_ERROR HTTP response status 504

Post by Klug »

This can happen when the DoSFilter gets activated (DoSFilter and/or too many login attempt with bad passwords while all users are behind the same IP).

You might want to check your DoSFilter configuration.
Also check you correctly setup Origin-IP.
Maybe you want to whitelist your IP (zimbraHttpThrottleSafeIPs).

https://wiki.zimbra.com/wiki/DoSFilter
https://www.missioncriticalemail.com/20 ... -together/

https://wiki.zimbra.com/wiki/Log_Files# ... inating_IP
https://wiki.zimbra.com/wiki/How_to_fix ... ging_issue
nisamudeen97
Posts: 11
Joined: Fri Aug 19, 2016 4:28 am

Re: CSFE_SVC_ERROR HTTP response status 504

Post by nisamudeen97 »

Hi,

In our case, I have read this before and disabled the same few months before but still the issue is there. We have to reboot the server every month. We have 91 active users and this issue is affecting our entire office. Is there a permanent solution for this?

Image
Attachments
Screenshot from 2023-09-08 17-22-01.png
Screenshot from 2023-09-08 17-22-01.png (15.67 KiB) Viewed 589 times
Post Reply