Zimbra Version: 8.6.0_GA_1153.FOSS zmmailboxdctl issue

[marskarthik]

Hi,

We have updated our Zimbra server to Zimbra Version: 8.6.0_GA_1153.FOSS about a month ago. Everything was going fine for few days then we started having issues, issue is no one can access web interface including admin but all services seems to be running as checked from command line. Only restarting zmcontrol services fixed the issue every time. Issue occurs minimum 3 times everyday at different times, it also occurs during less peak/load hours. Lately we found that restarting only zmmailboxdctl fix the issue quickly instead of restarting all services.

We are still trying to find what could be the issue for this. Any troubleshooting ideas please!

Karthik

[phoenix]

You can start by telling us how much RAM you have on this server, how many users and are they web UI or fat client users plus what's in the log files that may indicate any problem(s).

[marskarthik]

We have 64Gig RAM, total about 1100 users. About 1000 users use webclient and remaining use POP.

We only see below errors in mailbox.log regularly.

2016-10-25 12:17:51,265 ERROR [LmtpServer-112] [name=kosabel@mydomain.xxx;mid=1821;ip=abc.def.ade.cdf;] jsieve - Evaluation failed. Reason: null
2016-10-25 12:17:51,304 ERROR [LmtpServer-109] [name=figot@mydomain.xxx;mid=2482;ip=abc.def.ade.cdf;] jsieve - Evaluation failed. Reason: null
2016-10-25 12:17:51,418 ERROR [LmtpServer-113] [name=kosabel@mydomain.xxx;mid=1821;ip=abc.def.ade.cdf;] jsieve - Evaluation failed. Reason: null

Below messages from zmmailboxd.out
Caused by:
java.io.IOException: Broken pipe
at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
at sun.nio.ch.IOUtil.write(IOUtil.java:65)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:470)
at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:167)
at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.flush(SslConnection.java:783)
at org.eclipse.jetty.io.WriteFlusher.completeWrite(WriteFlusher.java:402)
at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onIncompleteFlush(SslConnection.java:391)
at org.eclipse.jetty.io.AbstractEndPoint$2.onIncompleteFlushed(AbstractEndPoint.java:53)
at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:348)
at org.eclipse.jetty.io.AbstractEndPoint.write(AbstractEndPoint.java:126)
at org.eclipse.jetty.server.HttpConnection$ContentCallback.process(HttpConnection.java:665)
at org.eclipse.jetty.util.IteratingCallback.processIterations(IteratingCallback.java:166)
at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:126)
at org.eclipse.jetty.server.HttpConnection.send(HttpConnection.java:303)
at org.eclipse.jetty.server.HttpChannel.sendResponse(HttpChannel.java:719)
at org.eclipse.jetty.server.HttpChannel.write(HttpChannel.java:752)
at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:134)
at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:127)
at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:342)
at org.eclipse.jetty.servlets.gzip.AbstractCompressedStream.write(AbstractCompressedStream.java:226)
at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:282)
at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125)
at java.io.OutputStreamWriter.write(OutputStreamWriter.java:207)
at org.eclipse.jetty.io.UncheckedPrintWriter.write(UncheckedPrintWriter.java:242)
at org.apache.jasper.runtime.JspWriterImpl.write(JspWriterImpl.java:414)
at java.io.PrintWriter.write(PrintWriter.java:426)
at org.apache.jasper.runtime.JspWriterImpl.flushBuffer(JspWriterImpl.java:189)
at org.apache.jasper.runtime.JspWriterImpl.write(JspWriterImpl.java:463)
at java.io.PrintWriter.write(PrintWriter.java:456)
at java.io.PrintWriter.write(PrintWriter.java:473)
at org.eclipse.jetty.io.WriterOutputStream.write(WriterOutputStream.java:87)
at org.eclipse.jetty.util.IO.copy(IO.java:149)
at org.eclipse.jetty.util.resource.Resource.writeTo(Resource.java:626)
at org.eclipse.jetty.servlet.DefaultServlet.sendData(DefaultServlet.java:843)
at org.eclipse.jetty.servlet.DefaultServlet.doGet(DefaultServlet.java:499)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:738)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:551)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:586)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1111)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:478)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:183)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1045)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.Dispatcher.include(Dispatcher.java:169)
at org.apache.jasper.runtime.JspRuntimeLibrary.include(JspRuntimeLibrary.java:977)
at org.apache.jsp.public_.Boot_jsp._jspService(org.apache.jsp.public_.Boot_jsp:83)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:111)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:411)
at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:473)
at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:377)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at com.zimbra.webClient.servlet.JspServlet.service(JspServlet.java:58)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:738)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:551)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:586)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1111)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:478)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:183)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1045)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.Dispatcher.include(Dispatcher.java:169)
at org.apache.jasper.runtime.JspRuntimeLibrary.include(JspRuntimeLibrary.java:977)
at org.apache.jsp.public_.launchZCS_jsp._jspService(org.apache.jsp.public_.launchZCS_jsp:523)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:111)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:411)
at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:473)
at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:377)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at com.zimbra.webClient.servlet.JspServlet.service(JspServlet.java:58)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:738)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:551)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:586)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1111)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:478)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:183)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1045)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.Dispatcher.forward(Dispatcher.java:261)
at org.eclipse.jetty.server.Dispatcher.forward(Dispatcher.java:101)
at org.apache.jasper.runtime.PageContextImpl.forward(PageContextImpl.java:792)
at org.apache.jsp.public_.login_jsp._jspx_meth_c_otherwise_7(org.apache.jsp.public_.login_jsp:2720)
at org.apache.jsp.public_.login_jsp._jspx_meth_c_choose_8(org.apache.jsp.public_.login_jsp:2466)
at org.apache.jsp.public_.login_jsp._jspx_meth_c_when_12(org.apache.jsp.public_.login_jsp:2438)
at org.apache.jsp.public_.login_jsp._jspx_meth_c_choose_7(org.apache.jsp.public_.login_jsp:2203)
at org.apache.jsp.public_.login_jsp._jspx_meth_c_otherwise_6(org.apache.jsp.public_.login_jsp:2018)
at org.apache.jsp.public_.login_jsp._jspx_meth_c_choose_4(org.apache.jsp.public_.login_jsp:1719)
at org.apache.jsp.public_.login_jsp._jspx_meth_c_if_5(org.apache.jsp.public_.login_jsp:1652)
at org.apache.jsp.public_.login_jsp._jspService(org.apache.jsp.public_.login_jsp:458)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:111)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:411)
at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:473)
at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:377)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at com.zimbra.webClient.servlet.JspServlet.service(JspServlet.java:58)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:738)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:551)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:586)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1111)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:478)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:183)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1045)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.Dispatcher.forward(Dispatcher.java:261)
at org.eclipse.jetty.server.Dispatcher.forward(Dispatcher.java:101)
at org.eclipse.jetty.servlet.DefaultServlet.doGet(DefaultServlet.java:552)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:738)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1651)
at com.zimbra.cs.servlet.RequestStringFilter.doFilter(RequestStringFilter.java:54)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
at com.zimbra.webClient.filters.ForwardFilter.doFilter(ForwardFilter.java:88)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
at com.zimbra.cs.servlet.SetHeaderFilter.doFilter(SetHeaderFilter.java:59)
at com.zimbra.webClient.filters.SetHeaderFilter.doFilter(SetHeaderFilter.java:248)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83)
at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:351)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
at com.zimbra.cs.servlet.ContextPathBasedThreadPoolBalancerFilter.doFilter(ContextPathBasedThreadPoolBalancerFilter.java:107)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
at com.zimbra.cs.servlet.ZimbraQoSFilter.doFilter(ZimbraQoSFilter.java:116)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
at org.eclipse.jetty.servlets.DoSFilter.doFilterChain(DoSFilter.java:457)
at org.eclipse.jetty.servlets.DoSFilter.doFilter(DoSFilter.java:326)
at org.eclipse.jetty.servlets.DoSFilter.doFilter(DoSFilter.java:299)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
at com.zimbra.webClient.filters.CharEncodingFilter.doFilter(CharEncodingFilter.java:37)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:549)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:544)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1111)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:478)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:183)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1045)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:199)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:109)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:309)
at org.eclipse.jetty.server.handler.DebugHandler.handle(DebugHandler.java:81)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
at org.eclipse.jetty.server.Server.handle(Server.java:462)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:279)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:232)
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:534)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
at java.lang.Thread.run(Thread.java:745)
2016-10-25 12:28:02.919:WARN:oejs.HttpChannel:qtp191382150-2425: Could not send response error 500: org.eclipse.jetty.io.RuntimeIOException: org.eclipse.jetty.io.EofException



Karthik

[vargacs]

Hello,

If you run htop as root what is your memory/cpu load?

Tapatalkkal küldve az én SM-G930F eszközömről

[marskarthik]

I have increased mailboxd java heap size and it looks it has fixed the issue. No issues for almost 48 hours now.
It was 256M before and I increased to 4G

$ zmlocalconfig -e mailboxd_java_heap_size=4096

[vargacs]

For the better performance in the my.conf the innodb_buffer_pool_size, innodb_log_file_size change to the right size.

The http://mysqltuner.com/ can help in this process. It was very helpful for me.

Tapatalkkal küldve az én SM-G930F eszközömről

[marskarthik]

Thanks, I will do the same.