Zimbra web interface timeout error.

[macglynn]

Hi, I’m using Zimbra 8.6.0 per cent 6. If I stop the service and start it again, some time after the Zimbra service is already started (about 2-3 minutes), the web interface is lost. Service status says that everything is fine. After one or two restarts of the zimbra service, everything goes back to normal. Can anyone recommend a solution. I found an error in the logs:

Code: Select all

WARN  [qtp509886383-197:https://1*.*.*.*:443/] [] webclient - system failure: com.zimbra.cs.mailbox.MailboxLock$LockFailedException: timeout
com.zimbra.common.soap.SoapFaultException: system failure: com.zimbra.cs.mailbox.MailboxLock$LockFailedException: timeout
ExceptionId:qtp509886383-194:http://127.0.0.1:80/service/soap/AuthRequest:1540199199233:47d4a3c2dd64c2df
Code:service.FAILURE
        at com.zimbra.common.soap.Soap12Protocol.soapFault(Soap12Protocol.java:90)
        at com.zimbra.common.soap.SoapTransport.extractBodyElement(SoapTransport.java:355)
        at com.zimbra.common.soap.SoapTransport.parseSoapResponse(SoapTransport.java:328)
        at com.zimbra.common.soap.SoapHttpTransport.invoke(SoapHttpTransport.java:261)
        at com.zimbra.common.soap.SoapHttpTransport.invoke(SoapHttpTransport.java:170)
        at com.zimbra.common.soap.SoapTransport.invoke(SoapTransport.java:408)
        at com.zimbra.client.ZMailbox.invoke(ZMailbox.java:721)
        at com.zimbra.client.ZMailbox.invoke(ZMailbox.java:715)
        at com.zimbra.client.ZMailbox.invokeJaxb(ZMailbox.java:703)
        at com.zimbra.client.ZMailbox.authByAuthToken(ZMailbox.java:648)
        at com.zimbra.client.ZMailbox.<init>(ZMailbox.java:519)
        at com.zimbra.client.ZMailbox.getMailbox(ZMailbox.java:465)
        at com.zimbra.cs.taglib.tag.LoginTag.doTag(LoginTag.java:158)
        at org.apache.jsp.public_.login_jsp._jspService(org.apache.jsp.public_.login_jsp:406)
        at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:111)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
        at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:411)
        at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:473)
        at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:377)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
        at com.zimbra.webClient.servlet.JspServlet.service(JspServlet.java:58)
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:738)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:551)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:586)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1111)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:478)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:183)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1045)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.Dispatcher.forward(Dispatcher.java:261)
        at org.eclipse.jetty.server.Dispatcher.forward(Dispatcher.java:101)
        at org.eclipse.jetty.servlet.DefaultServlet.doGet(DefaultServlet.java:552)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:738)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1651)
        at com.zimbra.cs.servlet.RequestStringFilter.doFilter(RequestStringFilter.java:54)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at com.zimbra.webClient.filters.ForwardFilter.doFilter(ForwardFilter.java:88)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at com.zimbra.cs.servlet.SetHeaderFilter.doFilter(SetHeaderFilter.java:59)
        at com.zimbra.webClient.filters.SetHeaderFilter.doFilter(SetHeaderFilter.java:248)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83)
        at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:351)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at com.zimbra.cs.servlet.ContextPathBasedThreadPoolBalancerFilter.doFilter(ContextPathBasedThreadPoolBalancerFilter.java:107)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at com.zimbra.cs.servlet.ZimbraQoSFilter.doFilter(ZimbraQoSFilter.java:116)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at org.eclipse.jetty.servlets.DoSFilter.doFilterChain(DoSFilter.java:457)
        at org.eclipse.jetty.servlets.DoSFilter.doFilter(DoSFilter.java:326)
        at org.eclipse.jetty.servlets.DoSFilter.doFilter(DoSFilter.java:299)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at com.zimbra.webClient.filters.CharEncodingFilter.doFilter(CharEncodingFilter.java:37)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:549)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:544)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1111)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:478)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:183)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1045)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:199)
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:109)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
        at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:309)
        at org.eclipse.jetty.server.handler.DebugHandler.handle(DebugHandler.java:81)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
        at org.eclipse.jetty.server.Server.handle(Server.java:462)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:279)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:232)
        at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:534)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
        at java.lang.Thread.run(Thread.java:745)

[DualBoot]

Hello,

looks like DoSFileter setting.

Regards,

[macglynn]

Hello,

looks like DoSFileter setting.

Regards,

Hi, I don’t see DOSPhileter error reports. In our scheme, we have 2 servers managed by a pacemaker/cman. To both servers mounted share "/zimbra/index" "/zimbra/store" from windows server via cifs. And the server configuration migrates from one node to another in case of an accident. During test migrations on the second node, the Web interface is no longer allowed to start. I see errors:
"Nov 8 09:10:21 site04 zimbramon[24615]: 24615:info: Starting zimlet via zmcontrol" after ->

In messages i see:

Code: Select all

Nov  8 09:10:35 site04 crmd[2054]:   notice: High CPU load detected: 30.969999
Nov  8 09:11:05 site04 crmd[2054]:   notice: High CPU load detected: 50.130001

Nov  8 09:13:05 site04 crmd[2054]:   notice: High CPU load detected: 130.350006
Nov  8 09:13:30 site04 kernel: INFO: task java:27331 blocked for more than 120 seconds.
Nov  8 09:13:30 site04 kernel:      Not tainted 2.6.32-696.18.7.el6.x86_64 #1
Nov  8 09:13:30 site04 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov  8 09:13:30 site04 kernel: java          D 0000000000000007     0 27331  26470 0x00000080
Nov  8 09:13:30 site04 kernel: ffff88034ad27b28 0000000000000086 0000000000000000 000000000000000b
Nov  8 09:13:30 site04 kernel: ffff88034ad27ad8 ffffffff814742f9 0002b2b3f225e5f0 ffffffff00000000
Nov  8 09:13:30 site04 kernel: ffff88034ad27aa8 000000012d3f7e7a ffff88034ac9dad8 ffff88034ad27fd8
Nov  8 09:13:30 site04 kernel: Call Trace:
Nov  8 09:13:30 site04 kernel: [<ffffffff814742f9>] ? skb_copy_datagram_iovec+0x159/0x2c0
Nov  8 09:13:30 site04 kernel: [<ffffffff8154d146>] __mutex_lock_slowpath+0x96/0x210
Nov  8 09:13:30 site04 kernel: [<ffffffff811aa398>] ? follow_managed+0x158/0x2e0
Nov  8 09:13:30 site04 kernel: [<ffffffff8154cc6b>] mutex_lock+0x2b/0x50
Nov  8 09:13:30 site04 kernel: [<ffffffff811aa63b>] do_lookup+0x11b/0x230
Nov  8 09:13:30 site04 kernel: [<ffffffff811aace0>] __link_path_walk+0x200/0x1060
Nov  8 09:13:30 site04 kernel: [<ffffffff811abdfa>] path_walk+0x6a/0xe0
Nov  8 09:13:30 site04 kernel: [<ffffffff811ac00b>] filename_lookup+0x6b/0xc0
Nov  8 09:13:30 site04 kernel: [<ffffffff811ad137>] user_path_at+0x57/0xa0
Nov  8 09:13:30 site04 kernel: [<ffffffff814eae40>] ? inet_ioctl+0x30/0xa0
Nov  8 09:13:30 site04 kernel: [<ffffffff8146612a>] ? sock_ioctl+0x7a/0x280
Nov  8 09:13:30 site04 kernel: [<ffffffff811a0410>] vfs_fstatat+0x50/0xa0
Nov  8 09:13:30 site04 kernel: [<ffffffffa003c020>] ? read_hv_clock_tsc+0x40/0x80 [hv_vmbus]
Nov  8 09:13:30 site04 kernel: [<ffffffff811a058b>] vfs_stat+0x1b/0x20
Nov  8 09:13:30 site04 kernel: [<ffffffff811a05b4>] sys_newstat+0x24/0x50
Nov  8 09:13:30 site04 kernel: [<ffffffff81556627>] ? system_call_after_swapgs+0x187/0x220
Nov  8 09:13:30 site04 kernel: [<ffffffff81556620>] ? system_call_after_swapgs+0x180/0x220
Nov  8 09:13:30 site04 kernel: [<ffffffff810eeef7>] ? audit_syscall_entry+0x1d7/0x200
Nov  8 09:13:30 site04 kernel: [<ffffffff8155660b>] ? system_call_after_swapgs+0x16b/0x220
Nov  8 09:13:30 site04 kernel: [<ffffffff81556604>] ? system_call_after_swapgs+0x164/0x220
Nov  8 09:13:30 site04 kernel: [<ffffffff815565fd>] ? system_call_after_swapgs+0x15d/0x220
Nov  8 09:13:30 site04 kernel: [<ffffffff815565f6>] ? system_call_after_swapgs+0x156/0x220
Nov  8 09:13:30 site04 kernel: [<ffffffff815565ef>] ? system_call_after_swapgs+0x14f/0x220
Nov  8 09:13:30 site04 kernel: [<ffffffff815566d6>] system_call_fastpath+0x16/0x1b
Nov  8 09:13:35 site04 crmd[2054]:   notice: High CPU load detected: 143.679993
Nov  8 09:14:35 site04 crmd[2054]:   notice: High CPU load detected: 161.279999
Nov  8 09:15:05 site04 crmd[2054]:   notice: High CPU load detected: 168.720001
Nov  8 09:15:06 site04 kernel: CIFS VFS: Unexpected lookup error -512
Nov  8 09:15:06 site04 kernel: CIFS VFS: Unexpected lookup error -512
...
Nov  8 09:15:06 site04 kernel: CIFS VFS: Unexpected lookup error -512

After that i restart service and see:

Code: Select all

2018-11-08 09:17:26,619 WARN  [] webclient - system failure: com.zimbra.cs.mailbox.MailboxLock$LockFailedException: timeout com.zimbra.common.soap.SoapFaultException: system failure: com.zimbra.cs.mailbox.MailboxLock$LockFailedException: timeout ExceptionId:qtp509886383-133:http://127.0.0.1:80/service/soap/AuthRequest:1541654246618:739e3e25e2977c64
Code:service.FAILURE

Code: Select all

Nov  8 09:16:33 site04 kernel: CIFS VFS: sends on sock ffff8803ff491780 stuck for 15 seconds
Nov  8 09:16:33 site04 kernel: CIFS VFS: Error -11 sending data on socket to server
Nov  8 09:16:35 site04 crmd[2054]:   notice: High CPU load detected: 45.860001
Nov  8 09:16:48 site04 kernel: CIFS VFS: sends on sock ffff8803ff491780 stuck for 15 seconds
Nov  8 09:16:48 site04 kernel: CIFS VFS: Error -11 sending data on socket to server
Nov  8 09:16:48 site04 kernel: CIFS VFS: Unexpected lookup error -112
Nov  8 09:17:03 site04 kernel: CIFS VFS: sends on sock ffff8803ff491780 stuck for 15 seconds
Nov  8 09:17:03 site04 kernel: CIFS VFS: Error -11 sending data on socket to server

I increased the RAM in the system to 16G, increased the innodb_buffer_pool_size, mail_box_java_heap_size. That not help. I noticed problems when displaying the directory "/zimbra/index" "/zimbra/store" on the second node with "ls", "df", that commands just hungs indefinitely. Can my problem be just because Java does not have time to reach mail files?

The topic is closed, the issue is resolved. The problem was that the CIFS share was DISCONNECTED.