Ok, lets remove a zero from the end...
zmprov -l ms `zmhostname` zimbraImapActiveSessionEhcacheMaxDiskSize 10737418240
zmprov -l ms `zmhostname` zimbraImapInactiveSessionEhcacheMaxDiskSize 10737418240
zmprov -l mcf zimbraImapActiveSessionEhcacheMaxDiskSize 10737418240
zmprov -l mcf zimbraImapInactiveSessionEhcacheMaxDiskSize 10737418240
zmcontrol stop
zmcontrol start
ERROR: service.PROXY_ERROR 503 service unavailable
Re: ERROR: service.PROXY_ERROR 503 service unavailable
Same issue....
Code: Select all
zmthrdump: Requested thread dump [PID 21180] at Wed Dec 12 00:44:12 2018
2018-12-12 00:44:13.878:INFO:oejs.ServerConnector:Thread-12: Stopped ServerConnector@6e2c9341{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}
2018-12-12 00:44:13.883:INFO:oejs.ServerConnector:Thread-12: Stopped ServerConnector@12028586{SSL,[ssl, http/1.1]}{0.0.0.0:8443}
2018-12-12 00:44:13.889:INFO:oejs.ServerConnector:Thread-12: Stopped ServerConnector@17776a8{SSL,[ssl, http/1.1]}{0.0.0.0:7071}
2018-12-12 00:44:13.891:INFO:oejs.ServerConnector:Thread-12: Stopped ServerConnector@69a10787{SSL,[ssl, http/1.1]}{0.0.0.0:7073}
2018-12-12 00:44:13.893:INFO:oejs.ServerConnector:Thread-12: Stopped ServerConnector@2d127a61{SSL,[ssl, http/1.1]}{0.0.0.0:7072}
2018-12-12 00:44:13.961:INFO:oejsh.ContextHandler:Thread-12: Stopped o.e.j.w.WebAppContext@3eb7fc54{/zimlet,[file:///opt/zimbra/jetty_base/webapps/zimlet/, file:///opt/zimbra/zimlets-deployed/],UNAVAILABLE}{/zimlet}
2018-12-12 00:44:13.974:INFO:oejsh.ContextHandler:Thread-12: Stopped o.e.j.w.WebAppContext@79b06cab{/zimbraAdmin,null,UNAVAILABLE}{/zimbraAdmin}
2018-12-12 00:44:13.983:INFO:oejsh.ContextHandler:Thread-12: Stopped o.e.j.w.WebAppContext@4690b489{/zimbra,null,UNAVAILABLE}{/zimbra}
2018-12-12 00:44:14.016:INFO:oejsh.ContextHandler:Thread-12: Stopped o.e.j.w.WebAppContext@4b14c583{/service,null,UNAVAILABLE}{/service}
OpenJDK 64-Bit Server VM warning: .hotspot_compiler file is present but has been ignored. Run with -XX:CompileCommandFile=.hotspot_compiler to load the file.
2018-12-12 00:46:09.815:INFO::main: Logging initialized @2805ms
JettyMonitor monitoring thread pool qtp1286783232{STOPPED,10<=0<=250,i=0,q=0}
Zimbra server reserving server socket port=7110 bindaddr=null ssl=false
Zimbra server reserving server socket port=7995 bindaddr=null ssl=false
Zimbra server reserving server socket port=7143 bindaddr=null ssl=false
Zimbra server reserving server socket port=7993 bindaddr=null ssl=false
Zimbra server reserving server socket port=7025 bindaddr=null ssl=false
2018-12-12 00:46:13.171:INFO:oejs.SetUIDListener:main: Setting umask=027
2018-12-12 00:46:13.172:INFO:oejs.SetUIDListener:main: Current rlimit_nofiles (soft=524288, hard=524288)
2018-12-12 00:46:13.172:INFO:oejs.SetUIDListener:main: Set rlimit_nofiles (soft=65536, hard=65536)
2018-12-12 00:46:13.199:INFO:oejs.SetUIDListener:main: Opened ServerConnector@6e2c9341{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}
2018-12-12 00:46:13.199:INFO:oejs.SetUIDListener:main: Opened ServerConnector@12028586{SSL,[ssl, http/1.1]}{0.0.0.0:8443}
2018-12-12 00:46:13.200:INFO:oejs.SetUIDListener:main: Opened ServerConnector@17776a8{SSL,[ssl, http/1.1]}{0.0.0.0:7071}
2018-12-12 00:46:13.201:INFO:oejs.SetUIDListener:main: Opened ServerConnector@69a10787{SSL,[ssl, http/1.1]}{0.0.0.0:7073}
2018-12-12 00:46:13.201:INFO:oejs.SetUIDListener:main: Opened ServerConnector@2d127a61{SSL,[ssl, http/1.1]}{0.0.0.0:7072}
2018-12-12 00:46:13.202:INFO:oejs.SetUIDListener:main: Setting GID=500
2018-12-12 00:46:13.204:INFO:oejs.SetUIDListener:main: Setting UID=500
2018-12-12 00:46:13.215:INFO:oejs.Server:main: jetty-9.3.5.v20151012
2018-12-12 00:46:27.694:WARN:oejs.SecurityHandler:main: ServletContext@o.e.j.w.WebAppContext@4b14c583{/service,file:///opt/zimbra/jetty_base/webapps/service/,STARTING}{/service} has uncovered http methods for path: /
Zimbra server process is running as uid=500 euid=500 gid=500 egid=500
2018-12-12 00:46:47.534:WARN:/service:main: unavailable
javax.servlet.UnavailableException: Tiering Inversion: 'Pool {107374182400 B offheap}' is not smaller than 'Pool {10737418240 B disk(persistent)}'
at com.zimbra.soap.SoapServlet.init(SoapServlet.java:134)
at javax.servlet.GenericServlet.init(GenericServlet.java:244)
at org.eclipse.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:626)
at org.eclipse.jetty.servlet.ServletHolder.initialize(ServletHolder.java:405)
at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:875)
at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:346)
at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1380)
at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1342)
at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:772)
at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:259)
at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:518)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:161)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
at org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:212)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
at org.eclipse.jetty.server.handler.DebugHandler.doStart(DebugHandler.java:143)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
at org.eclipse.jetty.server.Server.start(Server.java:405)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
at org.eclipse.jetty.server.Server.doStart(Server.java:372)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.xml.XmlConfiguration$1.run(XmlConfiguration.java:1510)
at java.security.AccessController.doPrivileged(Native Method)
at org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1435)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.eclipse.jetty.start.Main.invokeMain(Main.java:214)
at org.eclipse.jetty.start.Main.start(Main.java:457)
at org.eclipse.jetty.start.Main.main(Main.java:75)
2018-12-12 00:46:49.757:WARN:/service:main: unavailable
javax.servlet.UnavailableException: Couldn't lock rootDir: /opt/zimbra/data/mailboxd
at com.zimbra.soap.SoapServlet.init(SoapServlet.java:134)
at javax.servlet.GenericServlet.init(GenericServlet.java:244)
at org.eclipse.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:626)
at org.eclipse.jetty.servlet.ServletHolder.initialize(ServletHolder.java:405)
at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:875)
at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:346)
at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1380)
at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1342)
at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:772)
at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:259)
at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:518)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:161)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
at org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:212)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
at org.eclipse.jetty.server.handler.DebugHandler.doStart(DebugHandler.java:143)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
at org.eclipse.jetty.server.Server.start(Server.java:405)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
at org.eclipse.jetty.server.Server.doStart(Server.java:372)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.xml.XmlConfiguration$1.run(XmlConfiguration.java:1510)
at java.security.AccessController.doPrivileged(Native Method)
at org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1435)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.eclipse.jetty.start.Main.invokeMain(Main.java:214)
at org.eclipse.jetty.start.Main.start(Main.java:457)
at org.eclipse.jetty.start.Main.main(Main.java:75)
2018-12-12 00:46:49.983:WARN:oejw.WebAppContext:main: Failed startup of context o.e.j.w.WebAppContext@4b14c583{/service,file:///opt/zimbra/jetty_base/webapps/service/,STARTING}{/service}
MultiException[javax.servlet.UnavailableException: Tiering Inversion: 'Pool {107374182400 B offheap}' is not smaller than 'Pool {10737418240 B disk(persistent)}', javax.servlet.UnavailableException: Couldn't lock rootDir: /opt/zimbra/data/mailboxd]
at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:846)
at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:346)
at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1380)
at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1342)
at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:772)
at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:259)
at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:518)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:161)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
at org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:212)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
at org.eclipse.jetty.server.handler.DebugHandler.doStart(DebugHandler.java:143)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
at org.eclipse.jetty.server.Server.start(Server.java:405)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
at org.eclipse.jetty.server.Server.doStart(Server.java:372)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.xml.XmlConfiguration$1.run(XmlConfiguration.java:1510)
at java.security.AccessController.doPrivileged(Native Method)
at org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1435)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.eclipse.jetty.start.Main.invokeMain(Main.java:214)
at org.eclipse.jetty.start.Main.start(Main.java:457)
at org.eclipse.jetty.start.Main.main(Main.java:75)
Suppressed:
|javax.servlet.UnavailableException: Couldn't lock rootDir: /opt/zimbra/data/mailboxd
| at com.zimbra.soap.SoapServlet.init(SoapServlet.java:134)
| at javax.servlet.GenericServlet.init(GenericServlet.java:244)
| at org.eclipse.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:626)
| at org.eclipse.jetty.servlet.ServletHolder.initialize(ServletHolder.java:405)
| at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:875)
| at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:346)
| at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1380)
| at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1342)
| at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:772)
| at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:259)
| at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:518)
| at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
| at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
| at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
| at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
| at org.eclipse.jetty.server.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:161)
| at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
| at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
| at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
| at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
| at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
| at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
| at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
| at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
| at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
| at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
| at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
| at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
| at org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:212)
| at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
| at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
| at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
| at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
| at org.eclipse.jetty.server.handler.DebugHandler.doStart(DebugHandler.java:143)
| at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
| at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
| at org.eclipse.jetty.server.Server.start(Server.java:405)
| at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
| at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
| at org.eclipse.jetty.server.Server.doStart(Server.java:372)
| at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
| at org.eclipse.jetty.xml.XmlConfiguration$1.run(XmlConfiguration.java:1510)
| at java.security.AccessController.doPrivileged(Native Method)
| at org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1435)
| at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
| at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
| at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
| at java.lang.reflect.Method.invoke(Method.java:498)
| at org.eclipse.jetty.start.Main.invokeMain(Main.java:214)
| at org.eclipse.jetty.start.Main.start(Main.java:457)
| at org.eclipse.jetty.start.Main.main(Main.java:75)
Caused by:
javax.servlet.UnavailableException: Tiering Inversion: 'Pool {107374182400 B offheap}' is not smaller than 'Pool {10737418240 B disk(persistent)}'
at com.zimbra.soap.SoapServlet.init(SoapServlet.java:134)
at javax.servlet.GenericServlet.init(GenericServlet.java:244)
at org.eclipse.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:626)
at org.eclipse.jetty.servlet.ServletHolder.initialize(ServletHolder.java:405)
at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:875)
at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:346)
at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1380)
at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1342)
at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:772)
at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:259)
at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:518)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:161)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
at org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:212)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
at org.eclipse.jetty.server.handler.DebugHandler.doStart(DebugHandler.java:143)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
at org.eclipse.jetty.server.Server.start(Server.java:405)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
at org.eclipse.jetty.server.Server.doStart(Server.java:372)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.xml.XmlConfiguration$1.run(XmlConfiguration.java:1510)
at java.security.AccessController.doPrivileged(Native Method)
at org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1435)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.eclipse.jetty.start.Main.invokeMain(Main.java:214)
at org.eclipse.jetty.start.Main.start(Main.java:457)
at org.eclipse.jetty.start.Main.main(Main.java:75)
2018-12-12 00:46:54.207:WARN:oejpw.PlusDescriptorProcessor:main: No value for env-entry-name trustedIPs
2018-12-12 00:46:55.019:WARN:oejs.SecurityHandler:main: ServletContext@o.e.j.w.WebAppContext@4690b489{/zimbra,file:///opt/zimbra/jetty_base/webapps/zimbra/,STARTING}{/zimbra} has uncovered http methods for path: /
2018-12-12 00:46:57.127:INFO:oejsh.ContextHandler:main: Started o.e.j.w.WebAppContext@4690b489{/zimbra,file:///opt/zimbra/jetty_base/webapps/zimbra/,AVAILABLE}{/zimbra}
2018-12-12 00:47:02.829:WARN:oejpw.PlusDescriptorProcessor:main: No value for env-entry-name trustedIPs
2018-12-12 00:47:03.261:WARN:oejs.SecurityHandler:main: ServletContext@o.e.j.w.WebAppContext@79b06cab{/zimbraAdmin,file:///opt/zimbra/jetty_base/webapps/zimbraAdmin/,STARTING}{/zimbraAdmin} has uncovered http methods for path: /
2018-12-12 00:47:05.007:INFO:oejsh.ContextHandler:main: Started o.e.j.w.WebAppContext@79b06cab{/zimbraAdmin,file:///opt/zimbra/jetty_base/webapps/zimbraAdmin/,AVAILABLE}{/zimbraAdmin}
2018-12-12 00:47:06.009:WARN:oejs.SecurityHandler:main: ServletContext@o.e.j.w.WebAppContext@3eb7fc54{/zimlet,[file:///opt/zimbra/jetty_base/webapps/zimlet/, file:///opt/zimbra/zimlets-deployed/],STARTING}{/zimlet} has uncovered http methods for path: /
2018-12-12 00:47:06.012:INFO:oejsh.ContextHandler:main: Started o.e.j.w.WebAppContext@3eb7fc54{/zimlet,[file:///opt/zimbra/jetty_base/webapps/zimlet/, file:///opt/zimbra/zimlets-deployed/],AVAILABLE}{/zimlet}
2018-12-12 00:47:06.038:INFO:oejs.AbstractNCSARequestLog:main: Opened /opt/zimbra/log/access_log.2018-12-12
2018-12-12 00:47:06.049:INFO:oejm.ThreadMonitor:main: Thread Monitor started successfully
2018-12-12 00:47:06.137:INFO:oejs.ServerConnector:main: Started ServerConnector@6e2c9341{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}
2018-12-12 00:47:06.155:INFO:oejus.SslContextFactory:main: x509=X509@350da119(jetty,h=[x-ray.optiplex-networks.com],w=[]) for SslContextFactory@367d34c0(file:///opt/zimbra/jetty_base/etc/keystore,null)
2018-12-12 00:47:06.196:INFO:oejs.ServerConnector:main: Started ServerConnector@12028586{SSL,[ssl, http/1.1]}{0.0.0.0:8443}
2018-12-12 00:47:06.199:INFO:oejs.ServerConnector:main: Started ServerConnector@17776a8{SSL,[ssl, http/1.1]}{0.0.0.0:7071}
2018-12-12 00:47:06.201:INFO:oejs.ServerConnector:main: Started ServerConnector@69a10787{SSL,[ssl, http/1.1]}{0.0.0.0:7073}
2018-12-12 00:47:06.204:INFO:oejs.ServerConnector:main: Started ServerConnector@2d127a61{SSL,[ssl, http/1.1]}{0.0.0.0:7072}
2018-12-12 00:47:06.204:INFO:oejs.Server:main: Started @59194ms
Re: ERROR: service.PROXY_ERROR 503 service unavailable
I found the answer!!!
Analyzing the cache outputs that you suggested the setting was waaaaay too high:
I have got access again and everything works
I know this should be a new thread but since it was the original intention of fiddling with the Ehcache in the first place but what would I be looking at to enable access a large IMAP folder either through client or Web interface? Basically I need to run filters on this folder to sort the mail out but the 'Search' function within the web UI doesn't seem to work on the particular folder....
Analyzing the cache outputs that you suggested the setting was waaaaay too high:
Code: Select all
zmprov -l ms `zmhostname` zimbraImapInactiveSessionEhcacheSize 1048576
I know this should be a new thread but since it was the original intention of fiddling with the Ehcache in the first place but what would I be looking at to enable access a large IMAP folder either through client or Web interface? Basically I need to run filters on this folder to sort the mail out but the 'Search' function within the web UI doesn't seem to work on the particular folder....
Re: ERROR: service.PROXY_ERROR 503 service unavailable
Try this...
zmcontrol stop
sudo su - <PRESS ENTER>
As root...
mv /opt/zimbra/jetty/work/ /tmp/
mkdir /opt/zimbra/jetty/work/
chown zimbra:zimbra /opt/zimbra/jetty/work/
su - zimbra <PRESS ENTER>
zmcontrol start
zmcontrol stop
sudo su - <PRESS ENTER>
As root...
mv /opt/zimbra/jetty/work/ /tmp/
mkdir /opt/zimbra/jetty/work/
chown zimbra:zimbra /opt/zimbra/jetty/work/
su - zimbra <PRESS ENTER>
zmcontrol start
--
Rick King
Rick King
Re: ERROR: service.PROXY_ERROR 503 service unavailable
Virtually the setting that we were initially looking at was:
But it seems that the:
parameter was set over the MaxDiskSize setting causing the issue...
Though still some errors in the zmmailboxd.out file:
Again as stated above this is happening on one particular large folder that I'm getting a timeout.
Code: Select all
zimbraImapActiveSessionEhcacheMaxDiskSize
Code: Select all
zimbraImapActiveSessionEhcacheMaxSize
Though still some errors in the zmmailboxd.out file:
Code: Select all
2018-12-12 01:00:25.409:INFO:oejsh.ContextHandler:main: Started o.e.j.w.WebAppContext@3eb7fc54{/zimlet,[file:///opt/zimbra/jetty_base/webapps/zimlet/, file:///opt/zimbra/zimlets-deployed/],AVAILABLE}{/zimlet}
2018-12-12 01:00:25.446:INFO:oejs.AbstractNCSARequestLog:main: Opened /opt/zimbra/log/access_log.2018-12-12
2018-12-12 01:00:25.457:INFO:oejm.ThreadMonitor:main: Thread Monitor started successfully
2018-12-12 01:00:25.503:INFO:oejs.ServerConnector:main: Started ServerConnector@6e2c9341{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}
2018-12-12 01:00:25.540:INFO:oejus.SslContextFactory:main: x509=X509@44492c06(jetty,h=[x-ray.optiplex-networks.com],w=[]) for SslContextFactory@1c2096c6(file:///opt/zimbra/jetty_base/etc/keystore,null)
2018-12-12 01:00:25.637:INFO:oejs.ServerConnector:main: Started ServerConnector@12028586{SSL,[ssl, http/1.1]}{0.0.0.0:8443}
2018-12-12 01:00:25.639:INFO:oejs.ServerConnector:main: Started ServerConnector@17776a8{SSL,[ssl, http/1.1]}{0.0.0.0:7071}
2018-12-12 01:00:25.642:INFO:oejs.ServerConnector:main: Started ServerConnector@69a10787{SSL,[ssl, http/1.1]}{0.0.0.0:7073}
2018-12-12 01:00:25.654:INFO:oejs.ServerConnector:main: Started ServerConnector@2d127a61{SSL,[ssl, http/1.1]}{0.0.0.0:7072}
2018-12-12 01:00:25.655:INFO:oejs.Server:main: Started @79225ms
2018-12-12 01:17:22.697:INFO:oejm.ThreadMonitor:Thread-36: Thread 'Index-8'[RUNNABLE,id:45,cpu:90.72%] SPINNING
2018-12-12 01:17:22.698:WARN:oejm.ThreadMonitor:Thread-36:
org.eclipse.jetty.monitor.thread.ThreadMonitorException: Thread 'Index-8'[RUNNABLE,id:45,cpu:90.72%] STACK TRACE
at java.util.regex.Pattern$Branch.match(Pattern.java:4606)
at java.util.regex.Pattern$GroupHead.match(Pattern.java:4660)
at java.util.regex.Pattern$GroupHead.match(Pattern.java:4660)
at java.util.regex.Pattern$Branch.match(Pattern.java:4606)
at java.util.regex.Pattern$GroupHead.match(Pattern.java:4660)
at java.util.regex.Pattern$Start.match(Pattern.java:3463)
at java.util.regex.Matcher.search(Matcher.java:1248)
at java.util.regex.Matcher.find(Matcher.java:637)
at com.zimbra.cs.zimlet.handler.RegexHandler.match(RegexHandler.java:54)
at com.zimbra.cs.object.ObjectHandler.parse(ObjectHandler.java:107)
at com.zimbra.cs.mime.MimeHandler.getObjects(MimeHandler.java:301)
at com.zimbra.cs.mime.ParsedMessage.getMainBodyLuceneDocument(ParsedMessage.java:1033)
at com.zimbra.cs.mime.ParsedMessage.analyzeNonBodyParts(ParsedMessage.java:451)
at com.zimbra.cs.mime.ParsedMessage.analyzeFully(ParsedMessage.java:473)
at com.zimbra.cs.mailbox.Message.generateIndexData(Message.java:1405)
at com.zimbra.cs.mailbox.MailboxIndex.indexItemList(MailboxIndex.java:774)
at com.zimbra.cs.mailbox.MailboxIndex.indexDeferredItems(MailboxIndex.java:411)
at com.zimbra.cs.mailbox.MailboxIndex.access$600(MailboxIndex.java:85)
at com.zimbra.cs.mailbox.MailboxIndex$BatchIndexTask.exec(MailboxIndex.java:1434)
at com.zimbra.cs.mailbox.MailboxIndex$IndexTask.run(MailboxIndex.java:1413)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2018-12-12 01:17:22.707:WARN:oejm.ThreadMonitor:Thread-36:
org.eclipse.jetty.monitor.thread.ThreadMonitorException: Thread 'Index-8'[RUNNABLE,id:45,cpu:74.40%] STACK TRACE
at java.util.regex.Pattern$Branch.match(Pattern.java:4606)
at java.util.regex.Pattern$GroupHead.match(Pattern.java:4660)
at java.util.regex.Pattern$GroupHead.match(Pattern.java:4660)
at java.util.regex.Pattern$Branch.match(Pattern.java:4606)
at java.util.regex.Pattern$GroupHead.match(Pattern.java:4660)
at java.util.regex.Pattern$Start.match(Pattern.java:3463)
at java.util.regex.Matcher.search(Matcher.java:1248)
at java.util.regex.Matcher.find(Matcher.java:637)
at com.zimbra.cs.zimlet.handler.RegexHandler.match(RegexHandler.java:54)
at com.zimbra.cs.object.ObjectHandler.parse(ObjectHandler.java:107)
at com.zimbra.cs.mime.MimeHandler.getObjects(MimeHandler.java:301)
at com.zimbra.cs.mime.ParsedMessage.getMainBodyLuceneDocument(ParsedMessage.java:1033)
at com.zimbra.cs.mime.ParsedMessage.analyzeNonBodyParts(ParsedMessage.java:451)
at com.zimbra.cs.mime.ParsedMessage.analyzeFully(ParsedMessage.java:473)
at com.zimbra.cs.mailbox.Message.generateIndexData(Message.java:1405)
at com.zimbra.cs.mailbox.MailboxIndex.indexItemList(MailboxIndex.java:774)
at com.zimbra.cs.mailbox.MailboxIndex.indexDeferredItems(MailboxIndex.java:411)
at com.zimbra.cs.mailbox.MailboxIndex.access$600(MailboxIndex.java:85)
at com.zimbra.cs.mailbox.MailboxIndex$BatchIndexTask.exec(MailboxIndex.java:1434)
at com.zimbra.cs.mailbox.MailboxIndex$IndexTask.run(MailboxIndex.java:1413)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2018-12-12 01:17:22.720:WARN:oejm.ThreadMonitor:Thread-36:
org.eclipse.jetty.monitor.thread.ThreadMonitorException: Thread 'Index-8'[RUNNABLE,id:45,cpu:79.95%] STACK TRACE
at java.util.regex.Pattern$GroupHead.match(Pattern.java:4660)
at java.util.regex.Pattern$Branch.match(Pattern.java:4606)
at java.util.regex.Pattern$GroupHead.match(Pattern.java:4660)
at java.util.regex.Pattern$Start.match(Pattern.java:3463)
at java.util.regex.Matcher.search(Matcher.java:1248)
at java.util.regex.Matcher.find(Matcher.java:637)
at com.zimbra.cs.zimlet.handler.RegexHandler.match(RegexHandler.java:54)
at com.zimbra.cs.object.ObjectHandler.parse(ObjectHandler.java:107)
at com.zimbra.cs.mime.MimeHandler.getObjects(MimeHandler.java:301)
at com.zimbra.cs.mime.ParsedMessage.getMainBodyLuceneDocument(ParsedMessage.java:1033)
at com.zimbra.cs.mime.ParsedMessage.analyzeNonBodyParts(ParsedMessage.java:451)
at com.zimbra.cs.mime.ParsedMessage.analyzeFully(ParsedMessage.java:473)
at com.zimbra.cs.mailbox.Message.generateIndexData(Message.java:1405)
at com.zimbra.cs.mailbox.MailboxIndex.indexItemList(MailboxIndex.java:774)
at com.zimbra.cs.mailbox.MailboxIndex.indexDeferredItems(MailboxIndex.java:411)
at com.zimbra.cs.mailbox.MailboxIndex.access$600(MailboxIndex.java:85)
at com.zimbra.cs.mailbox.MailboxIndex$BatchIndexTask.exec(MailboxIndex.java:1434)
at com.zimbra.cs.mailbox.MailboxIndex$IndexTask.run(MailboxIndex.java:1413)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
- barrydegraaff
- Zimbra Employee
- Posts: 242
- Joined: Tue Jun 17, 2014 3:31 am
- Contact:
Re: ERROR: service.PROXY_ERROR 503 service unavailable
Since you are on CentOS6, I am guessing you did a lot of in-place upgrades. There is a high change you will run into more issues,
even if you fix the current one.
Perhaps consider migrating to a fresh install.
even if you fix the current one.
Perhaps consider migrating to a fresh install.
Re: ERROR: service.PROXY_ERROR 503 service unavailable
Thanks for the suggestion!
However, this is technically a new build as the upgrade process didn't work due to LDAP issues. The only things that remain the same with this build are the data store, db, and ldap store the rest is all completely fresh.
I ended up having to backup the /opt/zimbra dir then run install.sh -u and proceed with a migration.
However, this is technically a new build as the upgrade process didn't work due to LDAP issues. The only things that remain the same with this build are the data store, db, and ldap store the rest is all completely fresh.
I ended up having to backup the /opt/zimbra dir then run install.sh -u and proceed with a migration.