After upgrade from 8.6 to 8.7 on ubuntu 12.04, the webmail hangs after few hours

Discuss your pilot or production implementation with other Zimbra admins or our engineers.
cerri
Posts: 27
Joined: Sat Sep 13, 2014 3:03 am

After upgrade from 8.6 to 8.7 on ubuntu 12.04, the webmail hangs after few hours

Post by cerri »

After the upgrade, the WebMail is unreliable: after few hours it stops working without any reasons.
The Console is still accessible and other services too, but the webmail is hanged until the proxy is restarted.

In the logs we can see:

Code: Select all

2016-09-26 16:08:30.741:WARN:oejs.HttpChannel:qtp1684106402-1767: https://HOSTNAME/service/home/~/?auth=co&loc=it&id=96359&part=2
java.io.IOException: Broken pipe
        at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
        at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
        at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
        at sun.nio.ch.IOUtil.write(IOUtil.java:51)
        at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471)
        at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:175)
        at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.flush(SslConnection.java:808)
        at org.eclipse.jetty.io.WriteFlusher.flush(WriteFlusher.java:419)
        at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:313)
        at org.eclipse.jetty.io.AbstractEndPoint.write(AbstractEndPoint.java:141)
        at org.eclipse.jetty.server.HttpConnection$SendCallback.process(HttpConnection.java:752)
        at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:241)
        at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:224)
        at org.eclipse.jetty.server.HttpConnection.send(HttpConnection.java:511)
        at org.eclipse.jetty.server.HttpChannel.sendResponse(HttpChannel.java:653)
        at org.eclipse.jetty.server.HttpChannel.write(HttpChannel.java:702)
        at org.eclipse.jetty.server.handler.gzip.GzipHttpOutputInterceptor$GzipBufferCB.process(GzipHttpOutputInterceptor.java:364)
        at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:241)
        at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:224)
        at org.eclipse.jetty.server.handler.gzip.GzipHttpOutputInterceptor.gzip(GzipHttpOutputInterceptor.java:143)
        at org.eclipse.jetty.server.handler.gzip.GzipHttpOutputInterceptor.write(GzipHttpOutputInterceptor.java:112)
        at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:177)
        at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:163)
        at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:413)
        at com.zimbra.common.util.ByteUtil.copy(ByteUtil.java:726)
        at com.zimbra.common.util.ByteUtil.copy(ByteUtil.java:698)
        at com.zimbra.cs.service.formatter.NativeFormatter.sendbackBinaryData(NativeFormatter.java:630)
        at com.zimbra.cs.service.formatter.NativeFormatter.sendbackBinaryData(NativeFormatter.java:576)
        at com.zimbra.cs.service.formatter.NativeFormatter.sendbackOriginalDoc(NativeFormatter.java:450)
        at com.zimbra.cs.service.formatter.NativeFormatter.handleMessagePart(NativeFormatter.java:275)
        at com.zimbra.cs.service.formatter.NativeFormatter.handleMessage(NativeFormatter.java:162)
        at com.zimbra.cs.service.formatter.NativeFormatter.formatCallback(NativeFormatter.java:129)
        at com.zimbra.cs.service.formatter.Formatter.format(Formatter.java:142)
        at com.zimbra.cs.service.UserServlet.doAuthGet(UserServlet.java:536)
        at com.zimbra.cs.service.UserServlet.doGet(UserServlet.java:314)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
        at com.zimbra.cs.servlet.ZimbraServlet.service(ZimbraServlet.java:206)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:821)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1685)
        at com.zimbra.cs.servlet.CsrfFilter.doFilter(CsrfFilter.java:169)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
        at com.zimbra.cs.servlet.RequestStringFilter.doFilter(RequestStringFilter.java:54)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
        at com.zimbra.common.filters.Base64Filter.doFilter(Base64Filter.java:63)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
        at com.zimbra.cs.servlet.SetHeaderFilter.doFilter(SetHeaderFilter.java:59)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
        at com.zimbra.cs.servlet.ETagHeaderFilter.doFilter(ETagHeaderFilter.java:47)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
        at com.zimbra.cs.servlet.ContextPathBasedThreadPoolBalancerFilter.doFilter(ContextPathBasedThreadPoolBalancerFilter.java:107)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
        at com.zimbra.cs.servlet.ZimbraQoSFilter.doFilter(ZimbraQoSFilter.java:116)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
        at com.zimbra.cs.servlet.ZimbraInvalidLoginFilter.doFilter(ZimbraInvalidLoginFilter.java:117)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
        at org.eclipse.jetty.servlets.DoSFilter.doFilterChain(DoSFilter.java:473)
        at org.eclipse.jetty.servlets.DoSFilter.doFilter(DoSFilter.java:318)
        at org.eclipse.jetty.servlets.DoSFilter.doFilter(DoSFilter.java:288)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1158)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1090)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:109)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:119)
        at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:318)
        at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:437)
        at org.eclipse.jetty.server.handler.DebugHandler.handle(DebugHandler.java:84)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:119)
        at org.eclipse.jetty.server.Server.handle(Server.java:517)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:306)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:242)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:261)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
        at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:192)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:261)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
        at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:75)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:213)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:147)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
        at java.lang.Thread.run(Thread.java:745)
  
INFO: [HOSTNAME]:8736 [dev] [3.4.8] processors=4, physical.memory.total=13.7G, physical.memory.free=5.0G, swap.space.total=4.0G, swap.space.free=4.0G, heap.memory.used=763.9M, heap.memory.free=209.3M, heap.memory.total=973.1M, heap.memory.max=973.1M, heap.memory.used/total=78.49%, heap.memory.used/max=78.49%, minor.gc.count=133, minor.gc.time=4060ms, major.gc.count=3, major.gc.time=349ms, load.process=1.00%, load.system=7.00%, load.systemAverage=9.00%, thread.count=130, thread.peakCount=163, event.q.size=0, executor.q.async.size=0, executor.q.client.size=0, executor.q.query.size=0, executor.q.scheduled.size=0, executor.q.io.size=0, executor.q.system.size=0, executor.q.operation.size=0, executor.q.priorityOperation.size=0, executor.q.response.size=0, operations.remote.size=0, operations.running.size=0, proxy.count=1, clientEndpoint.count=0, connection.active.count=0, client.connection.count=0, connection.count=0
Any hint?
cerri
Posts: 27
Joined: Sat Sep 13, 2014 3:03 am

Re: After upgrade from 8.6 to 8.7 on ubuntu 12.04, the webmail hangs after few hours

Post by cerri »

Waiting for a response...
User avatar
syslint
Posts: 36
Joined: Wed Feb 24, 2016 7:59 am
Contact:

Re: After upgrade from 8.6 to 8.7 on ubuntu 12.04, the webmail hangs after few hours

Post by syslint »

What is your server type. I can see the load is little high in your server.
24x7 PROACTIVE ZIMBRA SERVER MANAGEMENT | Sales : sales @ syslint.com | Skype us : SyslintSkype | Call us : (+91) 471-60 - 7799 | visit : https://syslint.com
cerri
Posts: 27
Joined: Sat Sep 13, 2014 3:03 am

Re: After upgrade from 8.6 to 8.7 on ubuntu 12.04, the webmail hangs after few hours

Post by cerri »

Thanks for the reply.
The load is normal:

sysadmin@zimbra:~$ top
top - 16:25:21 up 3 days, 5:50, 2 users, load average: 0.16, 0.30, 0.36
Tasks: 195 total, 2 running, 193 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.2%us, 0.2%sy, 0.0%ni, 99.3%id, 0.2%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 14369468k total, 9362696k used, 5006772k free, 305180k buffers
Swap: 4190204k total, 13360k used, 4176844k free, 3512248k cached

And the load cannot "kill" just the web interface (proxy)...
cerri
Posts: 27
Joined: Sat Sep 13, 2014 3:03 am

Re: After upgrade from 8.6 to 8.7 on ubuntu 12.04, the webmail hangs after few hours

Post by cerri »

Anyone to help?
Anyone from Zimbra?
ccesario
Posts: 12
Joined: Fri Sep 12, 2014 11:44 pm

Re: After upgrade from 8.6 to 8.7 on ubuntu 12.04, the webmail hangs after few hours

Post by ccesario »

Hello,

I 'm facing the same problem, I "suspect" that the problem is related to ngix/proxy service. But no solution until now.

best regards,

-Carlos
cerri
Posts: 27
Joined: Sat Sep 13, 2014 3:03 am

Re: After upgrade from 8.6 to 8.7 on ubuntu 12.04, the webmail hangs after few hours

Post by cerri »

Me too.
The solution is to restart the proxy.
The real solution is to disable it: I've got no problems after disabling it.
ccesario
Posts: 12
Joined: Fri Sep 12, 2014 11:44 pm

Re: After upgrade from 8.6 to 8.7 on ubuntu 12.04, the webmail hangs after few hours

Post by ccesario »

Hi @cerri,

Yes, I disabled it for testing and we don't have problems. (thank you for sharing it)
But I 'm looking for the root cause, maybe some special config solve it. But I 'm still looking for it :/


Regards,

-Carlos
ccesario
Posts: 12
Joined: Fri Sep 12, 2014 11:44 pm

Re: After upgrade from 8.6 to 8.7 on ubuntu 12.04, the webmail hangs after few hours

Post by ccesario »

Any update about it!?
syed.yakoob
Posts: 7
Joined: Tue Aug 29, 2017 7:56 am

Re: After upgrade from 8.6 to 8.7 on ubuntu 12.04, the webmail hangs after few hours

Post by syed.yakoob »

facing the same issue, does anyone have a permanent fix for this?
Post Reply