Issue after upgrading to 8.8.15

Ask questions about your setup or get help installing ZCS server (ZD section below).
Nix67
Posts: 7
Joined: Fri Jul 26, 2019 7:54 am

Issue after upgrading to 8.8.15

Post by Nix67 »

Hello everyone,

I updated one of my Zimbra yesterday from 8.8.12 to 8.8.15.
Everything went fine. No errors or anything unusual.
Then, I updated Zextras. Everything went well.

Made some basic tests like send/receive, imap and webclient tests - everything was actually fine.

This Morning, I do have several issue.

I cant always login - I do have sometimes a "network error has occured" error. If I restart Zimbra, it goes away for a while then it happens again.

When Im able to log, I do have an issue when I log off (see picture attached). Never seen this before and I have no idea what it can be.

I did looked a lot before writing on the forum. Tried modifying ddos values without success. I can provide logs entries to anyone who is willing to give me a hand on this.

Thanks !
Nix67
Posts: 7
Joined: Fri Jul 26, 2019 7:54 am

Re: Issue after upgrading to 8.8.15

Post by Nix67 »

So here are few things I can see in the logs - perhaps it can help :

in the mailbox.log, when login start to fail :

Code: Select all

2019-07-26 12:51:37,855 WARN  [qtp961409111-340:https://*****.*******.**/m/zmain] [] webclient - system failure: Connection pool shut down
com.zimbra.common.service.ServiceException: system failure: Connection pool shut down
ExceptionId:qtp961409111-340:https://*****.*******.**/m/zmain:1564138297853:31669e05da832d4f
Code:service.FAILURE
        at com.zimbra.common.service.ServiceException.FAILURE(ServiceException.java:288)
        at com.zimbra.client.ZMailbox.invoke(ZMailbox.java:1011)
        at com.zimbra.client.ZMailbox.invoke(ZMailbox.java:987)
        at com.zimbra.client.ZMailbox.invokeJaxbToElement(ZMailbox.java:970)
        at com.zimbra.client.ZMailbox.invokeJaxb(ZMailbox.java:975)
        at com.zimbra.client.ZMailbox.authByAuthToken(ZMailbox.java:905)
        at com.zimbra.client.ZMailbox.<init>(ZMailbox.java:739)
        at com.zimbra.client.ZMailbox.getMailbox(ZMailbox.java:673)
        at com.zimbra.cs.taglib.ZJspSession.getSession(ZJspSession.java:516)
        at com.zimbra.cs.taglib.ZJspSession.getZMailbox(ZJspSession.java:461)
        at com.zimbra.cs.taglib.tag.ZimbraSimpleTag.getMailbox(ZimbraSimpleTag.java:29)
        at com.zimbra.cs.taglib.tag.GetMailboxTag.doTag(GetMailboxTag.java:57)
        at org.apache.jsp.m.zmview._jspService(zmview.java:144)
        at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
        at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:476)
        at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:386)
        at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:330)
        at org.eclipse.jetty.jsp.JettyJspServlet.service(JettyJspServlet.java:112)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:873)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:542)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:566)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1700)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1667)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
        at org.eclipse.jetty.server.Dispatcher.include(Dispatcher.java:129)
        at org.apache.taglibs.standard.tag.common.core.ImportSupport.acquireString(ImportSupport.java:301)
        at org.apache.taglibs.standard.tag.common.core.ImportSupport.doEndTag(ImportSupport.java:151)
        at org.apache.jsp.m.zmain._jspx_meth_c_005fimport_005f1(zmain.java:5312)
        at org.apache.jsp.m.zmain.access$29(zmain.java:5293)
        at org.apache.jsp.m.zmain$Helper.invoke4(zmain.java:6768)
        at org.apache.jsp.m.zmain$Helper.invoke(zmain.java:6798)
        at org.apache.jsp.tag.web.mobile.moHandleError_tag._jspx_meth_c_005fcatch_005f0(moHandleError_tag.java:216)
        at org.apache.jsp.tag.web.mobile.moHandleError_tag.doTag(moHandleError_tag.java:138)
        at org.apache.jsp.m.zmain._jspx_meth_mo_005fhandleError_005f2(zmain.java:5286)
        at org.apache.jsp.m.zmain._jspService(zmain.java:386)
        at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
        at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:476)
        at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:386)
        at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:330)
        at org.eclipse.jetty.jsp.JettyJspServlet.service(JettyJspServlet.java:112)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:873)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1623)
        at org.eclipse.jetty.websocket.server.WebSocketUpgradeFilter.doFilter(WebSocketUpgradeFilter.java:214)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
        at com.zimbra.cs.servlet.RequestStringFilter.doFilter(RequestStringFilter.java:54)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
        at com.zimbra.webClient.filters.ForwardFilter.doFilter(ForwardFilter.java:88)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
        at com.zimbra.cs.servlet.SetHeaderFilter.doFilter(SetHeaderFilter.java:59)
        at com.zimbra.webClient.filters.SetHeaderFilter.doFilter(SetHeaderFilter.java:248)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
        at com.zimbra.cs.servlet.ContextPathBasedThreadPoolBalancerFilter.doFilter(ContextPathBasedThreadPoolBalancerFilter.java:107)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
        at com.zimbra.cs.servlet.ZimbraQoSFilter.doFilter(ZimbraQoSFilter.java:125)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
        at org.eclipse.jetty.servlets.DoSFilter.doFilterChain(DoSFilter.java:482)
        at org.eclipse.jetty.servlets.DoSFilter.doFilter(DoSFilter.java:327)
        at org.eclipse.jetty.servlets.DoSFilter.doFilter(DoSFilter.java:297)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
        at com.zimbra.webClient.filters.CharEncodingFilter.doFilter(CharEncodingFilter.java:37)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1700)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1667)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
        at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220)
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:152)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
        at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
        at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:753)
        at org.eclipse.jetty.server.handler.DebugHandler.handle(DebugHandler.java:83)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
        at org.eclipse.jetty.server.Server.handle(Server.java:505)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:370)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:267)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
        at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:427)
        at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:321)
        at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:159)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
        at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:698)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:804)
        at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.lang.IllegalStateException: Connection pool shut down
        at org.apache.http.util.Asserts.check(Asserts.java:34)
        at org.apache.http.pool.AbstractConnPool.lease(AbstractConnPool.java:191)
        at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.requestConnection(PoolingHttpClientConnectionManager.java:268)
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:176)
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
        at com.zimbra.common.soap.SoapHttpTransport.invoke(SoapHttpTransport.java:310)
        at com.zimbra.common.soap.SoapHttpTransport.invoke(SoapHttpTransport.java:197)
        at com.zimbra.common.soap.SoapTransport.invoke(SoapTransport.java:463)
        at com.zimbra.client.ZMailbox.invoke(ZMailbox.java:1002)
        ... 111 more

User avatar
axslingr
Outstanding Member
Outstanding Member
Posts: 256
Joined: Sat Sep 13, 2014 2:20 am
ZCS/ZD Version: 8.8.15.GA.3869.UBUNTU18.64 UBUNTU18

Re: Issue after upgrading to 8.8.15

Post by axslingr »

Make sure you're logging the originating ip.

https://wiki.zimbra.com/wiki/Log_Files# ... inating_IP

Lance
Nix67
Posts: 7
Joined: Fri Jul 26, 2019 7:54 am

Re: Issue after upgrading to 8.8.15

Post by Nix67 »

Hey,

Tahnks for answering and for this advice.

zimbra@mail:~$ zmprov gcf zimbraMailTrustedIP
zimbraMailTrustedIP: 127.0.0.1
zimbraMailTrustedIP: 192.168.105.207

Its done. Still, I do Believe the issue isnt linked to anything ddos related. It seems like there is something that crash at some point and once that happened, there is no way to log in the Webmail.
The error when unlogging is all the time.
I just relaized i havent put the picture I was talking about in my first message and Im not sure how i can add it, but it basically says :


???500Title???

???500Msg???
???errorTryAgainLater???
ERROR: 500
« ???errorGoBack???
User avatar
L. Mark Stone
Ambassador
Ambassador
Posts: 2796
Joined: Wed Oct 09, 2013 11:35 am
Location: Portland, Maine, US
ZCS/ZD Version: 10.0.6 Network Edition
Contact:

Re: Issue after upgrading to 8.8.15

Post by L. Mark Stone »

I believe attachments to this Board were turned off; feel free to post it someplace else and post the link here.

All the best,
Mark
___________________________________
L. Mark Stone
Mission Critical Email - Zimbra VAR/BSP/Training Partner https://www.missioncriticalemail.com/
AWS Certified Solutions Architect-Associate
Nix67
Posts: 7
Joined: Fri Jul 26, 2019 7:54 am

Re: Issue after upgrading to 8.8.15

Post by Nix67 »

Hey,

Thanks for the advice.

Little update on this topic.

When I start Zimbra, I can log with no issues.
At some point, it stop working with this error :
https://ibb.co/gSR47Fh

Restarting zimbra allows me to log again.

Switching to HTML view once logged gives mes this error (and then I can log until I restart Zimbra)
https://ibb.co/bN9TyBY


Unlogging from the admin always gives an error (but I cant log again always even if user webamil is broken) :
https://ibb.co/h7p1G6Q


Also, at the moment it breaks, and before the stack I put on my previous message that state :
com.zimbra.common.service.ServiceException: system failure: Connection pool shut down

I can see a lot of lines like that

2019-07-26 14:17:07,527 INFO [qtp961409111-151:https://mail.*********.***:8443/service/soap/GetInfoRequest] [name=admin@*********.***;mid=2;ip=192.168.105.207;port=41034;ua=zclient/8.8.15_GA_3829;soapId=140bd8fd;] session - Too many SOAP sessions (6 > 5), closing SoapSession{id=1354, authAcct=9af33fc4-45df-41c5-bc28-a0433a5c6ff6, lastAccessed=2019-07-26 14:17:07,466}
2019-07-26 14:17:07,541 INFO [qtp961409111-15:https://mail.*********.***:8443/service/soap/GetInfoRequest] [name=admin@*********.***;mid=2;ip=192.168.105.207;port=41036;ua=zclient/8.8.15_GA_3829;soapId=140bd8fe;] session - Too many SOAP sessions (6 > 5), closing SoapSession{id=1356, authAcct=9af33fc4-45df-41c5-bc28-a0433a5c6ff6, lastAccessed=2019-07-26 14:17:07,478}
2019-07-26 14:17:07,555 INFO [qtp961409111-22:https://mail.*********.***:8443/service/soap/GetInfoRequest] [name=admin@*********.***;mid=2;ip=192.168.105.207;port=41038;ua=zclient/8.8.15_GA_3829;soapId=140bd8ff;] session - Too many SOAP sessions (6 > 5), closing SoapSession{id=1358, authAcct=9af33fc4-45df-41c5-bc28-a0433a5c6ff6, lastAccessed=2019-07-26 14:17:07,491}
2019-07-26 14:17:07,570 INFO [qtp961409111-15:https://mail.*********.***:8443/service/soap/GetInfoRequest] [name=admin@*********.***;mid=2;ip=192.168.105.207;port=41040;ua=zclient/8.8.15_GA_3829;soapId=140bd900;] session - Too many SOAP sessions (6 > 5), closing SoapSession{id=1360, authAcct=9af33fc4-45df-41c5-bc28-a0433a5c6ff6, lastAccessed=2019-07-26 14:17:07,503}

So I changed the config to 15, but the same issues happens

many SOAP sessions (16 > 15), closing SoapSession{id=1718, authAcct=9af33fc4-45df-41c5-bc28-a0433a5c6ff6, lastAccessed=2019-07-26 15:58:29,091}
2019-07-26 15:58:29,262 INFO [qtp945722724-152:https://mail.*********.***:8443/service/soap/GetInfoRequest] [name=admin@*********.***;mid=2;ip=192.168.105.207;port=41430;ua=zclient/8.8.15_GA_3829;soapId=4d94a5ae;] session - Too many SOAP sessions (16 > 15), closing SoapSession{id=1720, authAcct=9af33fc4-45df-41c5-bc28-a0433a5c6ff6, lastAccessed=2019-07-26 15:58:29,103}
2019-07-26 15:58:29,278 INFO [qtp945722724-152:https://mail.*********.***:8443/service/soap/GetInfoRequest] [name=admin@*********.***;mid=2;ip=192.168.105.207;port=41432;ua=zclient/8.8.15_GA_3829;soapId=4d94a5af;] session - Too many SOAP sessions (16 > 15), closing SoapSession{id=1722, authAcct=9af33fc4-45df-41c5-bc28-a0433a5c6ff6, lastAccessed=2019-07-26 15:58:29,114}
2019-07-26 15:58:29,290 INFO [qtp945722724-135:https://mail.*********.***:8443/service/soap/GetInfoRequest] [name=admin@*********.***;mid=2;ip=192.168.105.207;port=41434;ua=zclient/8.8.15_GA_3829;soapId=4d94a5b0;] session - Too many SOAP sessions (16 > 15), closing SoapSession{id=1724, authAcct=9af33fc4-45df-41c5-bc28-a0433a5c6ff6, lastAccessed=2019-07-26 15:58:29,126}
2019-07-26 15:58:29,299 INFO [qtp945722724-159:https://mail.*********.***:8443/service/soap/GetInfoRequest] [name=admin@*********.***;mid=2;ip=192.168.105.207;port=41436;ua=zclient/8.8.15_GA_3829;soapId=4d94a5b1;] session - Too many SOAP sessions (16 > 15), closing SoapSession{id=1726, authAcct=9af33fc4-45df-41c5-bc28-a0433a5c6ff6, lastAccessed=2019-07-26 15:58:29,139}
2019-07-26 15:58:29,309 INFO [qtp945722724-14:https://mail.*********.***:8443/service/soap/GetInfoRequest] [name=admin@*********.***;mid=2;ip=192.168.105.207;port=41438;ua=zclient/8.8.15_GA_3829;soapId=4d94a5b2;] session - Too many SOAP sessions (16 > 15), closing SoapSession{id=1728, authAcct=9af33fc4-45df-41c5-bc28-a0433a5c6ff6, lastAccessed=2019-07-26 15:58:29,149}


Im I the only one who is having those weird issues on 8.8.15 ? I have no idea how I can fix this anymore :\
cubbi
Posts: 2
Joined: Sat Jul 27, 2019 9:19 am

Re: Issue after upgrading to 8.8.15

Post by cubbi »

Hi,

not sure if there is any relation to your problem. But since updating to 8.8.15 (from 8.8.12) last night, I see a high and steady increase of threads on the mailboxd-process. Looks like there is something leaking and processes get not cleaned up.

So far I did not encounter problems like your soap-limit or issues with logins, but it is weekend and not much load on the zimbra box.

See https://ibb.co/t8CLFTJ about the increase of threads.

PID 14475 is the mailboxd-process:
[root@zimbra ~]# ls /proc/14475/task/
1001 10722 1138 12136 12848 13607 1433 14821 1549 16227 16718 1744 18020 18915 19646 2037 21059 21802 22517 2319 23867 24550 25166 25993 26618 27279 28099 28713 29388 30069 30787 3138 32236 3435 4096 4780 5454 6128 6780 7415 8232 8969 9516
10015 10764 1139 12143 12849 13610 14332 14823 15496 16229 16719 17453 18030 18917 1965 20372 21065 21805 22519 23213 23888 24551 25167 26008 26619 27280 28105 28715 29392 30070 30789 31380 32238 3436 4098 4786 5455 6129 6781 742 8236 897 952
10017 10766 11390 12145 12850 13611 14334 14824 15497 16230 16720 17455 18032 18918 19650 20377 21094 21806 22520 23215 23890 24558 25196 26010 26627 27291 28106 28716 29393 30096 30790 31381 32239 3438 4099 4791 5456 6147 6782 743 8238 8973 9522
10018 10767 11392 12157 12851 13626 14335 14826 15500 16232 16722 17457 18035 18941 19655 20379 21096 21808 22521 23216 23891 24560 25202 26011 26629 27305 28107 28722 29396 3010 30806 31383 32246 3439 4111 4793 5458 6149 6783 7430 8239 898 9531
10021 10771 11393 1219 12857 13628 1435 14827 15502 16234 16741 17458 18037 18943 19657 20380 21097 21809 22523 23269 23903 24561 25204 26036 26630 27307 28108 28729 29398 30101 30808 31385 32247 344 4113 4796 5459 6150 6828 7432 8294 8997 954
10023 10773 11425 12193 12864 13629 14356 14853 15503 16235 16743 17470 18039 18944 1966 20381 21098 21813 22524 23278 23908 24567 25205 26038 26631 27312 28109 2873 29399 30102 30809 31386 32270 3445 4114 4797 5465 6173 6830 7437 8296 8999 955
10024 10774 11432 12195 12865 1364 14358 14857 1551 16236 16749 17473 18043 18989 19663 20386 21099 21815 22529 23284 23909 24569 25212 26039 26637 27339 2811 28734 29403 30116 30830 3140 32272 3447 4140 4798 5499 6175 6831 7438 8297 900 9559
1003 10786 11433 12201 12894 13646 14359 14858 15523 16237 16750 17475 18053 18991 19672 20387 21100 21816 2255 23285 23911 24570 25218 26042 26638 27341 28110 28736 29404 30118 30837 31408 32278 3448 4142 4809 5501 6176 6841 7441 830 9003 9561
10034 10788 11435 1221 12909 13648 1436 14860 15528 16238 16751 17477 18055 18995 19710 2040 21111 2182 22553 23291 23913 24581 25220 26044 26639 27342 28114 28737 29416 30119 30838 31415 32287 3451 4143 4811 5502 6204 6846 7442 832 901 9562
1004 10794 11437 1222 1291 13649 14383 14861 15537 1624 16752 17479 18056 18998 19712 20402 21113 2183 22585 23297 23914 24583 25221 26045 26643 27344 2812 28742 29417 30120 30878 31416 32306 3452 4145 4812 5547 6206 6852 745 833 9013 957
10040 10813 11438 12227 12911 13657 14384 14873 15539 16240 16768 17481 18061 19003 19713 20409 21114 21843 22587 23346 23932 24593 25222 26060 26645 27355 28146 2875 29419 30121 30880 3142 32308 3454 4147 482 5548 6211 6858 7459 8362 9015 958
1008 10815 11439 12229 12912 13659 14385 14875 1554 16241 16774 17482 18063 19010 19730 20411 21119 21850 22588 23348 23936 24607 25223 26065 26664 27362 28149 2876 29420 30122 30881 3143 32309 3455 4148 4833 5550 6220 6861 746 8369 9016 959
10097 10820 11441 12235 12929 1366 14387 14876 15540 16242 16776 17485 18064 19011 19735 20421 21125 21863 22602 23349 23938 24609 25236 26066 26666 27364 28151 28764 29426 30123 30908 3144 3231 3456 4157 4835 5551 6230 6863 747 8370 9024 960
1010 1085 11445 12253 1293 13660 14388 14920 1555 16246 16777 17490 18075 19049 19741 20425 21151 21909 22606 23350 23939 24610 25238 26068 26667 27370 28152 28766 29427 30124 30910 31454 32319 3458 4159 4836 5554 6235 6864 748 8372 9029 9601
1011 10864 11451 12267 12936 13661 14395 14927 1556 16247 16778 17496 18081 19051 19761 20438 21158 21911 22610 23352 23940 24614 25239 26069 26668 27373 28162 28771 2943 30125 30911 31456 32321 3475 4160 4844 5555 6237 6868 7492 8374 9030 9608
10126 10866 11453 12269 12937 13674 1440 14928 15585 16249 16821 17500 18083 19052 19763 20441 21163 21912 22631 23353 23941 24616 2524 26070 26672 27385 28164 28785 29430 30153 3097 31457 32322 3477 417 4846 5579 6243 6869 7494 8375 9088 9609
10128 10867 11454 12270 1294 13676 14404 14947 15609 1625 16823 17551 18084 19053 19764 20444 21176 21915 22635 23359 23944 24617 25267 26076 26676 27387 28165 28792 29436 30155 3099 31494 32329 3478 4189 4847 5585 6285 6873 7500 8417 9090 961
10132 1087 11471 1228 12940 13677 14406 14949 1561 16250 16824 17553 18085 19054 19782 20445 21178 21932 22637 23361 23946 24633 2528 26077 26684 27388 28186 28793 29442 30156 30997 31496 3233 348 419 4870 5586 6286 6879 7520 8419 9094 962
10144 10876 11473 12283 12975 13678 14407 14958 15611 16254 16835 17554 18091 19057 19793 20447 21179 21936 22640 23364 23956 24639 25297 26078 26686 27390 28188 28798 29447 3017 3100 31502 32332 3482 4190 4875 5587 6294 6902 7521 8420 9102 9628
10145 10878 11482 12285 13001 13679 14410 14960 15612 16255 16837 17628 18096 19067 19798 20486 21181 21937 22648 2337 24007 24640 25304 26080 26687 27396 28189 28800 29448 3018 31004 31508 32333 3485 4191 4881 5588 6296 6904 7527 8437 9104 9630
10167 10879 11519 12287 13003 13680 14412 14961 1562 16257 16838 17630 18103 19074 1981 20488 21183 21959 22650 23379 24009 24641 25346 26082 26702 2740 28217 28801 29452 30180 31009 31510 3234 3486 4192 4911 5592 6297 6905 7533 8441 9105 9635
10257 10883 11521 12288 13004 1371 14413 14979 15636 16258 16841 17631 18104 19075 19828 20489 21184 21981 22651 23381 24019 24643 25348 26083 26704 27402 28219 28862 29456 30182 31011 31511 32341 351 420 4913 5593 6316 6925 7538 8445 9116 9637
1026 10885 11522 12297 13019 13723 14415 14981 15638 16260 16848 1764 18116 19076 1983 20501 21196 21985 22669 23382 24020 24644 25349 26084 26705 27403 28220 28864 29457 30183 31015 31532 32346 3520 4203 4914 5612 6318 6927 7544 8450 9118 9638
10264 10888 11524 12299 13021 13725 14417 14982 15641 16264 16849 17649 18118 19078 19830 20503 21198 21986 22671 23425 24022 24651 2536 26109 2671 27404 28243 28865 29458 30201 31023 31534 32347 3527 4205 4915 5614 6319 6928 7581 8452 9119 9640
10265 10890 11526 1230 13022 13726 14418 14993 15647 16266 16869 17651 18119 19079 19831 20504 21199 21988 22672 23428 24023 24652 25360 26116 2673 27408 28245 28870 2948 30203 3103 31535 32352 3528 4206 4920 5619 6342 694 7586 8453 9120 9645
1028 10891 11527 12300 1304 13728 1442 15008 15649 16267 16870 17655 18124 19080 19839 2051 21217 21989 22680 23430 24024 24653 25362 26117 26737 2742 28246 28874 29501 30204 31038 31583 32354 353 4223 4930 5620 636 695 7613 8459 9122 9647
1029 10892 1154 12301 1306 1373 1443 15015 15655 16269 16872 1766 18126 19081 1984 20526 21219 21998 22685 23431 24036 24660 25368 2614 26739 2743 28252 28875 29503 30205 31040 31622 32355 354 4225 4943 568 6370 6953 7615 8467 9123 9648
10295 10914 11563 1231 13061 13730 1444 15016 15658 16270 16873 1767 18130 19082 19841 20533 21220 22000 22690 2344 24038 24663 25391 26140 2674 27430 28254 28876 29513 30206 31041 31624 32375 355 4226 4950 5683 6374 6955 7619 8468 9130 9678
10297 10916 11564 12329 13068 13731 14459 15018 15660 16271 16875 1768 18132 19096 19842 20535 21261 22001 22717 23451 24039 24665 25408 26142 26740 27432 28255 28878 29515 30207 3105 31628 32377 3551 4235 4951 5686 6378 6956 7621 8500 9137 9680
10298 10919 11566 12331 13069 13747 1446 15019 15661 16272 16878 17701 18139 19101 19849 20536 21273 22006 22723 23453 24043 24675 25410 26143 26755 27433 28264 28879 29519 30208 31083 31649 32378 3556 4237 4976 5688 638 697 7622 8502 9138 9690
1030 10921 11567 12335 1307 13749 14461 15020 15677 16275 16880 17703 18151 19103 19851 20538 21275 22008 22745 23454 24045 24688 25411 26166 26757 2745 28266 28887 29521 30221 31085 31650 32386 3557 4238 4978 5689 6380 6977 7625 8503 914 9704
10312 10922 11568 12336 13091 1375 14462 15021 15679 16279 16881 17707 18157 19104 19852 20539 21276 22009 22768 23471 24046 24696 2543 26173 26759 27451 28267 28889 29525 30223 31088 31656 32409 3561 4251 4984 5690 6381 6979 7636 8516 9140 9706
10314 10926 11569 12338 13093 1376 1447 15022 15680 16284 16887 17745 1816 19105 1986 2054 21305 22012 22770 23475 24063 24698 25458 26174 26762 27454 2827 28890 29530 30224 31089 31657 32411 3563 4253 4986 5692 6384 698 7639 8528 9141 9707
10315 1093 11570 12339 13094 13770 14475 15059 15697 16289 16905 17747 18162 19106 19876 20547 21307 22014 22771 23477 24065 24703 25460 26179 26763 27458 28272 28900 29533 30231 3109 31658 32412 3564 4254 4988 5693 6385 6980 7640 8530 9148 9763
10333 10943 11585 12340 1310 13789 14479 15061 1570 16292 16907 17748 18168 19107 19878 20556 21310 22016 22772 23478 24066 24716 25461 2618 26767 27459 28274 28902 29554 30237 31091 31667 32423 3565 4297 4989 5698 6386 6988 765 8531 915 9765
10335 10945 11587 12341 13107 13791 14481 15062 15702 16296 1694 17752 18172 19108 19879 20565 21313 22018 22776 23481 24105 24724 25466 26181 26824 27462 28275 28903 29559 30243 31092 31669 32437 3570 4299 5020 5700 6387 6990 7652 8534 9169 9770
10339 10946 11588 12342 13109 13797 14482 15063 15703 16302 16942 17757 18176 19109 1989 20566 21323 22019 22778 23482 24120 24729 25468 26182 26826 27463 28283 28920 2956 30261 31099 31671 32447 3571 4300 5022 5701 6388 6991 7654 8536 917 9784
10353 10975 1159 12343 13110 13800 14483 15070 15740 16307 16945 17759 1818 19110 1990 20605 21324 22026 22779 23484 24126 24732 25474 26210 26827 27465 28288 28922 29561 30266 31106 31672 32449 3573 4328 5023 5729 639 7066 7655 8537 9174 9786
10354 10977 11596 12358 1312 13802 14485 15072 15742 16312 16946 17760 18182 19112 1991 20606 21327 22037 22781 23485 24128 24734 25485 26212 2686 27466 28289 28923 29563 30267 31112 31678 32450 3579 433 5024 5731 6393 7067 767 8546 918 9787
10356 10983 11597 12360 13129 13803 14487 15073 15743 16313 16948 17766 1819 19113 1993 20621 21329 22039 22782 23487 24129 24735 25487 26213 2688 27469 2829 2894 29564 30268 31114 31680 32466 3590 4330 5026 5733 6395 7093 7677 8548 9180 9805
10357 10999 1160 12367 13131 13806 14489 15074 1576 16319 16949 17768 18265 19115 19943 20628 21330 22040 22787 23489 24154 24751 25493 26240 2689 2747 2830 2896 29575 30270 31116 31681 32477 3592 4336 5027 5735 6436 7100 7679 8549 9181 9807
10360 11001 11617 12390 13133 13816 14492 15075 1577 16321 16954 17769 18267 19117 19945 20633 21342 22047 22788 23491 24162 24753 25499 26242 26892 27470 28310 2897 29577 30271 31120 31687 32480 3593 4340 5046 5736 6438 7103 768 8578 9182 9808
10361 11002 11619 12413 13138 13854 14494 15076 15773 16322 16962 1777 18268 19118 19946 20634 21344 22049 22789 23492 24199 24754 25500 26243 26894 27472 28317 2898 29578 30274 31124 31688 32482 360 4342 505 5737 6439 7105 7680 8580 9183 9815
10363 11003 11620 12415 13144 13856 14496 15093 1578 16323 16964 1778 18341 19134 19951 20636 21345 22051 22791 23498 24201 24760 25507 2625 26895 27473 28318 29008 29597 30278 31127 31690 32488 3612 4343 507 574 6456 7106 7681 8581 9185 9817
10365 11004 11625 12418 13147 13857 14498 15094 15797 16325 16976 17791 18343 19136 19958 20642 21374 22052 22792 23500 24202 24764 25512 2626 26901 2748 28329 29010 29599 30280 31128 31691 32500 3614 4347 5071 5744 6458 7109 7688 8582 9186 9820
10366 11010 11627 12420 13149 1387 14502 15096 15802 16327 16978 17793 18344 19137 19959 20649 21376 22054 22816 23501 24228 24766 25521 26260 26903 27503 28331 29011 29605 30281 31129 31695 32502 3618 4354 5073 5746 6460 7125 7689 8583 919 9827
10395 11012 11628 12423 1315 13871 14503 15097 15803 16328 16979 17794 18350 1917 19966 20661 21377 22055 22818 23507 2423 24767 25523 26262 26904 27505 28332 29016 2961 30307 31130 31696 32503 3619 4360 5074 5748 6461 7127 7707 8606 9191 9829
10397 11014 11650 12426 13150 13872 14505 15101 15822 16329 16980 17807 18353 19179 19968 20663 2140 22063 22836 23513 24232 24768 25524 26263 26945 27506 28345 29018 2962 3031 31131 31711 32515 3621 4361 508 575 6470 7129 7708 8623 9194 9830
10407 11015 11652 12440 13154 13879 14506 15102 15829 16335 16981 17809 18359 19181 19969 20664 21400 22065 22838 23515 24236 24769 25552 26284 26947 27511 28347 29019 29624 30313 31143 31713 32518 3622 4363 5085 5750 6472 7130 7717 8625 9196 9831
10413 11021 11653 12442 13156 13881 14507 15132 15831 16336 16988 17810 18361 19182 1999 20696 21402 22066 22839 23516 24256 24774 25554 26289 26951 27513 28348 2902 29635 3033 31145 31714 32523 3625 4368 5087 5751 6473 7131 7724 8626 9197 9833
10415 11023 11668 12443 13157 13883 14508 15139 15832 16343 16989 17818 18367 19183 19999 20698 21403 22074 2284 23518 24258 24776 25555 26290 26966 27517 28350 2903 29636 3034 31146 31739 32524 3630 4372 5088 5756 6481 7132 7730 8635 9200 9834
10416 11024 11670 12454 1317 13884 14509 15140 1585 16345 1699 17820 18398 19184 20001 20699 21407 22076 22841 23519 24259 24777 25559 26293 26968 27525 28355 29042 2967 30364 31149 31741 32526 3631 4374 5103 577 6483 715 7746 8637 9207 9835
10429 11026 11671 12455 13174 13886 14511 15152 15850 16346 17022 17821 18405 19190 20007 20771 21414 22077 22842 23528 24263 2478 25561 26296 26969 27527 28356 29044 29695 30366 31151 31744 32527 3632 4375 5144 579 6484 7154 7753 8638 9209 9836
10431 11028 11674 12457 13176 13888 14514 15157 15855 16351 17024 17843 18406 1923 20013 20773 21415 22083 22843 23530 24265 24805 25562 2632 2700 27528 28357 29045 29697 30367 31152 31746 32531 3633 4392 516 580 649 7196 7754 8643 9210 9837
10437 11029 11681 12458 13177 13889 14521 15158 15857 16353 17025 17844 18408 1924 20015 20774 2142 22089 22844 23532 24266 2481 25582 26324 27019 27589 28359 29063 29698 30409 31170 31747 32536 3634 4398 5179 5807 6534 720 7760 8649 923 9855
10438 11033 11682 12459 1318 1389 14522 15170 15858 16354 17032 17845 18409 19266 20016 2079 2143 22095 22848 23533 24280 24837 25584 26338 2702 27591 28360 29065 29703 30425 31172 31750 32538 3642 4411 518 5809 6536 7202 7762 8650 9232 9861
10439 11035 11683 12463 13192 13890 14533 15172 15865 16359 17034 17850 18417 19268 20020 20809 21431 22148 22853 23534 24288 24839 25585 2634 27021 27597 28378 29066 29705 30427 31173 31751 32539 3644 4413 5181 5810 654 7204 7763 8651 9259 9863
10445 11036 11684 12484 13194 13891 14538 15173 1587 16361 17035 17851 18421 19269 20022 2081 21433 2215 22854 23537 24315 24840 25587 26340 27022 27603 28380 29090 29706 30428 31184 31752 32541 3648 4414 5182 5843 6540 7205 7799 8653 9263 9864
10459 11058 11685 12486 13198 13895 14539 15178 1589 16362 1705 17852 18423 19296 20023 20811 21439 22157 22855 23539 24319 24841 25588 26341 27025 27605 28381 29092 29720 30444 31186 31753 32544 365 4430 5185 5845 6550 7207 7801 8654 9265 9865
10466 11060 11724 12487 13222 13906 14540 1518 15912 16376 17050 17856 18424 19298 20083 20813 2145 22159 22857 23540 24320 24842 25590 2635 27027 27606 28388 29093 29722 30451 31187 31754 32545 3657 4432 5187 5846 6552 7215 7802 8656 9266 9869
10470 11061 11731 12488 13231 13907 14541 15180 15915 16378 17072 17859 18451 19299 20085 20815 21459 22160 22858 23541 24321 24843 25591 26356 27028 27607 28394 29097 29723 30452 31195 31756 32551 3658 4433 5188 5884 6556 7217 7815 8663 9322 987
10472 11069 11732 12489 13233 13908 14546 15181 15921 16384 17074 17861 18453 19316 20086 20816 2146 22164 2286 23542 24322 24868 25595 26358 2706 27608 28396 29099 29732 30456 31197 31757 32552 3659 445 519 5891 6581 7218 7820 8664 9324 9871
10473 11071 11742 12495 13234 13909 1455 15194 1593 16434 17075 17862 18454 19318 20115 20817 21461 2217 22863 23546 24330 24884 25596 2636 27078 27613 28397 2910 29734 30517 31198 31763 32553 366 4450 5192 5892 6583 7224 7821 8674 9325 9872
10474 11072 11744 12504 13235 1391 14565 15199 15953 16436 17090 17863 18508 19319 20117 2082 21465 22170 2287 23548 24333 24886 25620 26362 27080 27643 28398 29100 29735 30522 31212 31768 32559 3661 4468 5194 5904 6584 7225 7862 8676 9335 9873
10478 11084 11745 12506 13236 13916 14566 15201 15955 16437 17092 17864 18513 19418 20118 20832 21468 2218 22870 23549 24335 24887 25622 26368 27081 27645 28399 29116 2974 30525 31214 31769 32566 3662 4470 5195 5906 660 7263 7865 8677 9337 989
10480 11086 11753 12507 13237 13917 14567 15202 15959 16443 17101 17865 18514 19421 20136 20834 21475 22250 22875 23554 24336 2489 25623 2637 27085 27646 28405 2912 2975 30526 31215 31777 32594 3663 4471 5196 5907 6607 7264 7868 8703 9338 9898
10481 11087 11755 12517 1324 1392 1457 15204 15977 16445 17102 17866 18523 19422 20138 20835 21476 22252 22895 23556 24337 24895 25629 26370 27087 27647 28407 2913 29760 30529 31216 31779 32596 3664 4516 5197 5921 6609 7265 7870 8705 9340 9900
10503 11104 11756 12528 13272 13944 14573 15209 15979 16446 17107 17878 18529 19432 20140 20841 21483 22253 22902 2357 2435 24897 25631 26376 2709 27656 28408 29136 29762 30530 31221 31780 32597 3665 4518 5198 5923 6610 7266 7871 8706 9342 9906
10505 11105 1179 12529 13279 13946 14575 15210 15983 16471 17113 17880 18534 19433 20142 20843 21487 22264 22903 23586 24350 24898 25632 2638 27093 27657 28454 2914 29764 30563 31222 31797 326 3672 4519 5225 5924 6629 7269 7872 8747 9343 9928
10506 11107 11805 12609 13280 13947 14577 15212 1600 16473 17131 17881 18567 19435 20143 20849 21491 22266 22908 23588 24362 24921 25640 26399 27097 27658 28460 2916 29765 30566 31224 3181 32648 368 4520 5227 5930 6631 727 7876 8760 9351 993
1051 11108 11807 12611 13294 13976 14579 15213 16000 16474 17137 17882 18569 19436 20144 20850 21492 22267 22924 23589 24391 24923 25645 26400 27104 27659 28462 29161 29766 30568 31226 31813 32651 3686 4521 5228 5932 6632 7271 7884 8762 9353 9935
10518 11109 11808 12612 13295 13985 1458 15228 16002 16492 17138 17884 18570 19437 20145 20852 21494 22268 22926 23595 24393 24924 25646 26401 27105 27660 28463 29162 29767 30569 31227 31815 32658 3688 4523 5235 5970 664 7272 7886 8763 9354 9942
10520 11110 1181 12613 13296 13986 14580 15230 16003 16494 17139 17885 18585 19438 20159 20861 21495 22269 22927 23597 24394 24945 25663 26403 27106 27712 28475 29164 29768 30570 31256 31821 32659 3689 4524 5237 5972 6656 7274 7887 8766 9355
10521 11130 1182 12615 13298 13991 14608 15232 16008 16495 17140 17888 18587 19446 20161 20863 21496 22275 22948 23598 2440 24950 25666 26423 27107 27714 28477 29165 29815 30571 31258 31828 32683 371 4525 5243 5973 6657 7275 7907 8767 9360
10531 11132 11905 12616 13300 13993 14615 15233 16010 16519 17141 17912 18588 19448 20162 20869 21497 22295 22950 23612 24401 24952 25685 26425 27108 27715 28478 29169 29817 30582 31259 3183 32684 3718 4526 5255 6006 6658 7279 7909 8769 9362
10533 11133 11907 12618 13301 13994 14616 15234 16011 16521 17145 17914 18589 19455 20177 20870 21503 22296 22951 23614 24407 24953 25731 26426 27109 27717 28492 2917 29818 30584 31260 31830 32686 373 4582 5257 6008 6660 728 7910 8773 9363
10534 11167 11917 12620 13302 14010 14617 15237 16012 16522 17149 17915 18590 19457 20185 20871 2151 22298 22966 23615 2442 24954 2574 26494 27110 27719 28499 29233 29854 30585 31262 31831 32692 374 4584 5258 6012 6668 7290 7918 8776 938
10536 11172 11955 12625 13306 14015 14618 15289 16015 16524 17156 17926 18591 19458 20187 20872 2153 22304 22979 2368 2443 24955 25740 26496 27113 27725 28500 29235 29855 30586 31263 31838 32694 3754 4585 5267 6021 6670 7291 7920 8783 9387
10538 11174 11962 12637 13312 14017 14619 15291 16030 16526 17157 17927 18642 19459 20188 20873 21534 22306 22981 23682 24431 24976 25745 26497 27117 27758 28508 29239 29857 30587 31270 3184 32705 3756 4608 5269 6042 6675 7292 7925 8808 9388
10539 11175 11963 12638 13314 14019 14645 15292 16031 16527 1716 17928 18644 19464 20199 20874 21539 22307 22986 23693 24433 25064 25746 26506 27123 27760 28516 29242 29858 30588 31272 31845 32707 3757 4610 5271 6044 6677 7299 7926 8832 9390
10551 11176 11965 12640 13318 14020 14652 15293 16033 16528 17163 17929 18645 19466 20201 20875 21540 22315 23020 23695 24434 25066 25750 26513 27124 27761 28519 29249 29862 30603 31277 31850 32708 377 4611 5273 6045 6678 7300 7928 8833 9391
10553 11177 11968 12644 13327 14024 14653 15294 16034 16535 17173 17933 18654 19467 20202 20876 21545 22317 23022 23696 24441 25067 25755 26519 2713 27814 28521 29254 29863 30613 31278 31851 32742 378 4612 5275 6048 6686 7308 7929 8834 9392
10554 11224 11970 12650 13333 1403 14673 15295 16039 16541 17174 17934 18659 19505 20210 20877 21547 22318 23023 23720 24443 25068 25756 26523 27150 27816 28531 29255 2987 30615 3128 31861 32744 3791 4614 5276 6050 6688 7310 7934 8836 9394
1057 11226 11971 12652 13335 14030 14675 15297 16046 16555 1718 17935 18660 19506 20212 20879 21553 22325 23041 23722 24444 25069 25758 26533 27152 27817 28534 29256 29894 30616 31280 31862 32748 3803 4620 5277 6056 6689 7311 7936 8837 9396
1058 11227 11975 12658 13336 14064 14678 15298 16047 16563 1719 17936 18664 19507 20213 20880 21595 22327 23043 23724 24463 25070 25759 26537 27157 27844 28536 29257 29896 30618 31281 31868 328 3804 4622 5282 6082 6704 7312 7938 8839 9397
10591 11240 11979 12660 13341 1410 14684 15299 16068 16565 17190 17937 18666 19508 20216 20887 21600 22328 23044 23725 24465 25071 2576 2654 27159 27846 28537 29259 29897 30620 31287 31921 330 3807 4623 5286 6084 6706 7313 7939 885 940
10598 11242 11997 12664 13369 1411 14685 15301 16074 16566 17192 17938 18667 19510 20221 20889 21601 22359 23055 23741 24466 25073 25760 26542 27160 27847 28585 29275 29915 30621 31294 31923 3307 3809 4628 5301 6087 6707 7314 7947 8850 9405
10599 11243 11999 12675 13381 14117 14686 15303 16133 16584 17198 17939 18701 19511 20222 20890 21614 22361 23057 23744 24479 25074 25761 26543 27162 27864 28587 29277 29917 30627 31295 31924 3309 3810 4688 5303 6088 6710 7315 7958 8852 941
1060 11263 12000 12677 13413 14119 14692 15322 16135 16589 17201 17940 18703 19566 20226 2091 21616 22362 23058 23745 24485 25077 2577 26545 27176 27866 28588 29278 29918 30633 3130 31963 3310 3818 4690 5305 6092 6714 7316 7959 8853 9416
1061 11268 12013 12678 13415 14120 14693 15329 16137 16598 17203 17941 18704 19568 20247 2093 21617 22395 23075 2375 24494 25079 2579 26546 27178 27867 28604 29306 29921 30634 31309 31965 332 3820 4696 5306 6093 6715 7317 7967 886 942
10625 11269 12018 12680 13416 14141 14702 1534 16140 16600 17204 17942 18710 19572 20250 20937 21631 22402 23077 2376 24496 25080 25815 26547 2718 27950 28606 29308 29923 30653 3131 31966 333 3821 4709 5307 6095 6724 7318 7969 8870 9427
10627 1130 12019 12682 13442 14143 14706 15340 16141 16601 17224 17950 18712 19574 20252 20939 21633 22403 23078 23760 24497 25101 25817 26548 27184 27952 28607 29309 29924 30656 31311 31986 3336 3829 471 5312 6098 6726 7323 7970 8872 9429
10633 11301 12046 12683 13444 14144 14708 15356 16146 16604 17230 17952 18713 19575 20253 2094 21634 22416 23085 23766 24502 25103 25821 26549 2719 27953 28608 29312 29926 30658 31317 31988 3338 3836 4710 5315 6100 6727 7325 7974 8873 9430
10635 11303 12048 12696 13445 14158 14710 15378 16150 16606 17240 17957 18757 19585 20266 20940 21639 22423 23087 23769 24504 25107 25822 26554 27196 27956 28609 29319 29928 30659 31322 31994 3339 3837 4716 5318 6101 673 7327 7981 8875 9435
10636 11304 12049 12699 13468 14165 14716 15385 16152 16607 17254 17959 18758 19599 20268 20942 2164 22429 23089 2377 24505 25108 25824 26556 27198 27962 28614 29323 2993 30662 31324 32009 3340 3848 4717 5320 6102 6732 7328 7982 8877 944
10637 11313 12061 12701 13470 14193 14745 15389 16156 16610 1730 17960 18759 19601 20314 20944 21641 22449 23091 23773 2451 25110 25825 26557 27199 27975 28616 29328 29938 30664 31325 32010 3342 3851 4720 5328 6103 6734 7342 7983 888 9444
10638 11315 12063 12702 13472 14195 14751 15391 16158 16612 17302 17961 18760 19602 20316 20945 21649 2245 23092 23774 24510 25111 25843 2656 2721 27982 28618 29330 2994 30665 31336 32016 3343 3852 4722 5334 6105 6735 7365 8005 889 9446
10645 11320 12064 12706 13511 14196 14759 15392 16160 16613 17303 17965 18762 19604 20317 20946 21670 22451 23096 2382 24511 25130 25845 2657 27213 27983 28619 29331 29953 30683 31342 32022 335 3868 4725 5340 6107 6746 7367 8007 8890 9448
10646 11329 12065 12707 13544 14209 14776 15393 16169 1668 17305 17969 18763 19611 20320 20948 21681 22452 23097 23821 24512 25132 25849 26572 27214 27996 28660 29332 29960 30687 31344 32027 3371 3869 4728 5352 6108 6748 7368 8008 8892 945
10656 11331 12066 12779 13546 14218 14777 1541 16171 1670 17306 17970 18764 19612 20325 20949 21682 2247 23099 23822 24513 25138 2585 26574 2722 27998 28662 2936 29962 30689 31345 32051 3373 394 473 5354 6109 675 7369 8014 8898 9450
10658 11332 12067 1278 13547 14235 14778 15416 16172 16705 17307 17980 18820 19613 20326 21000 21712 22474 23100 23824 24515 25148 25858 26575 27251 27999 28668 29367 29963 30690 31346 32053 3374 3967 4731 5358 6110 6754 7370 8016 8900 9451
10659 11341 12080 12781 13560 14237 14780 15422 16206 16708 17310 17990 18822 19614 20339 21002 2172 22477 23117 23825 24516 25152 25860 26585 27253 28024 28692 29369 29964 30692 31347 32054 3389 3969 4733 5369 6111 676 7371 8017 8901 9452
10667 11343 12084 12782 13562 14238 14786 15424 16209 1671 17324 17992 18823 19623 20344 21003 21741 22479 23119 23826 24521 25154 25861 26587 27254 28026 28694 29372 29965 30694 31348 32127 3391 3970 4734 5371 6112 6767 7385 8109 8905 9454
10669 11344 12088 1280 13563 1425 14789 1543 1621 16710 1737 17993 18827 1963 20346 21035 21743 2248 23120 23827 24529 25155 25884 26588 27260 28030 28695 29374 30005 30700 31364 32129 3392 3972 4737 5372 6115 6769 7387 8112 8918 9456
10670 11348 1210 1281 13585 14261 14792 1544 16212 16711 1739 17996 18838 19634 20347 21037 21744 22481 23152 23828 24533 25157 25886 26600 27262 28039 28696 29375 30007 30730 3137 32130 3418 3974 474 5388 6116 6770 7388 8114 892 9483
10686 1135 12106 12812 13587 14263 14796 15441 16214 16712 1740 17997 18840 19638 20356 21038 21757 22482 23154 23829 24534 25160 25887 26605 27263 2804 28697 29376 30008 30737 31373 32145 3420 3975 4746 5390 6118 6774 7396 8115 8946 9485
10688 11357 12108 12814 13588 14264 14798 15448 16218 16713 17402 17999 18841 19641 20359 21044 21763 22487 23155 2383 24536 25161 2592 26606 27272 28046 28698 29378 30012 30738 31375 32222 3424 403 4748 5396 6119 6775 7398 8129 8964 9486
10689 1136 12114 12820 13602 14278 14799 15449 16220 16715 17404 18004 18903 19642 20360 21051 21774 22501 23166 23830 24542 25163 2593 26608 27274 28052 28701 29379 30017 30752 31376 32224 3427 4054 4750 5416 6121 6776 7408 8131 8966 9509
10719 11363 1212 12845 13603 1430 14802 15452 16221 16716 17405 18014 18905 19643 20365 21052 21775 22503 23168 23864 24546 25164 25985 26609 27276 2808 28703 29385 30019 30754 31377 32225 343 4056 4751 5435 6122 6777 7412 8132 8967 9511
10721 1137 1213 12846 13605 1432 14819 15489 16223 16717 1743 18015 18906 19644 20367 21057 2180 22513 23172 23866 24548 25165 25992 26616 27278 28097 28707 29387 30062 30755 31378 32232 3430 4062 4778 5452 6126 6779 7414 8187 8968 9512
[root@zimbra ~]#

I see not further hints in any of the logfiles of what is going wrong here.
cubbi
Posts: 2
Joined: Sat Jul 27, 2019 9:19 am

Re: Issue after upgrading to 8.8.15

Post by cubbi »

Just a quick update: my problem with increasing threads is now gone, after just restarting the zmmailboxd-process. Maybe I should have tried that earlier, but I did not bother until now, since the whole zimbrastack was just freshly started.

I will continue to monitor the system over the weekend, but for now I think that your problem does not relate to the observations I saw after my fresh upgrade.
User avatar
quanah
Zimbra Alumni
Zimbra Alumni
Posts: 1668
Joined: Fri Sep 12, 2014 10:33 pm
Contact:

Re: Issue after upgrading to 8.8.15

Post by quanah »

There are some important things to keep in mind.

In the past, a patch release was nearly always just that, a patch release. However, at some point since 8.6, Zimbra has changed what they do so "patch" releases should really be considered a "brand new" release of Zimbra (like say, 8.0 -> 8.5 was, etc). You can see this in the 8.8.12 release in which defanger was completely replaced (causing a bit of breakage).

Second, specifically relating to 8.8.15, HttpClientUtil was changed/rewritten, which causes many 3rd party applications (including some of zExtras) to break. Another one that's known to break is the Scality extension for Zimbra. This caused the public java signatures to change for 8.8.15 vs every prior releases. This of course goes against the fundamentals of modern programming itself (You never change existing public method signatures, see https://en.wikipedia.org/wiki/Single_re ... _principle and https://en.wikipedia.org/wiki/SOLID), and certainly is not something in the past that would have been allowed in something that *appears* to be a patch release. The correct way to handle this would have been to deprecate the existing HttpClientUtils methods, add new methods for the new HttpClientUtil, and remove the old HttpClientUtil and its methods in a future *major* release.

Good luck!
--
Quanah Gibson-Mount
Product Architect, Symas http://www.symas.com/
OpenLDAP Core team http://www.openldap.org/project/
Nix67
Posts: 7
Joined: Fri Jul 26, 2019 7:54 am

Re: Issue after upgrading to 8.8.15

Post by Nix67 »

Hey,
I finally decided to rollback to a previous version.
I suppose I will wait and see if someone else run into similar issues or if some kind of fix are released.
As you mentioned, it can as well be a third party soft like zextras even if the docs say they are supposed to be compatible.

Anyway. Wait and see I guess !

Thanks everyone.
Post Reply