ERROR: service.PROXY_ERROR 503 service unavailable

Discuss your pilot or production implementation with other Zimbra admins or our engineers.
User avatar
king0770
Outstanding Member
Outstanding Member
Posts: 242
Joined: Fri Sep 12, 2014 10:44 pm
Contact:

Re: ERROR: service.PROXY_ERROR 503 service unavailable

Post by king0770 »

Ok, lets remove a zero from the end...

zmprov -l ms `zmhostname` zimbraImapActiveSessionEhcacheMaxDiskSize 10737418240

zmprov -l ms `zmhostname` zimbraImapInactiveSessionEhcacheMaxDiskSize 10737418240

zmprov -l mcf zimbraImapActiveSessionEhcacheMaxDiskSize 10737418240

zmprov -l mcf zimbraImapInactiveSessionEhcacheMaxDiskSize 10737418240

zmcontrol stop

zmcontrol start
--
Rick King
kayasaman
Posts: 20
Joined: Thu Nov 22, 2018 9:05 pm

Re: ERROR: service.PROXY_ERROR 503 service unavailable

Post by kayasaman »

Same issue....

Code: Select all

zmthrdump: Requested thread dump [PID 21180] at Wed Dec 12 00:44:12 2018
2018-12-12 00:44:13.878:INFO:oejs.ServerConnector:Thread-12: Stopped ServerConnector@6e2c9341{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}
2018-12-12 00:44:13.883:INFO:oejs.ServerConnector:Thread-12: Stopped ServerConnector@12028586{SSL,[ssl, http/1.1]}{0.0.0.0:8443}
2018-12-12 00:44:13.889:INFO:oejs.ServerConnector:Thread-12: Stopped ServerConnector@17776a8{SSL,[ssl, http/1.1]}{0.0.0.0:7071}
2018-12-12 00:44:13.891:INFO:oejs.ServerConnector:Thread-12: Stopped ServerConnector@69a10787{SSL,[ssl, http/1.1]}{0.0.0.0:7073}
2018-12-12 00:44:13.893:INFO:oejs.ServerConnector:Thread-12: Stopped ServerConnector@2d127a61{SSL,[ssl, http/1.1]}{0.0.0.0:7072}
2018-12-12 00:44:13.961:INFO:oejsh.ContextHandler:Thread-12: Stopped o.e.j.w.WebAppContext@3eb7fc54{/zimlet,[file:///opt/zimbra/jetty_base/webapps/zimlet/, file:///opt/zimbra/zimlets-deployed/],UNAVAILABLE}{/zimlet}
2018-12-12 00:44:13.974:INFO:oejsh.ContextHandler:Thread-12: Stopped o.e.j.w.WebAppContext@79b06cab{/zimbraAdmin,null,UNAVAILABLE}{/zimbraAdmin}
2018-12-12 00:44:13.983:INFO:oejsh.ContextHandler:Thread-12: Stopped o.e.j.w.WebAppContext@4690b489{/zimbra,null,UNAVAILABLE}{/zimbra}
2018-12-12 00:44:14.016:INFO:oejsh.ContextHandler:Thread-12: Stopped o.e.j.w.WebAppContext@4b14c583{/service,null,UNAVAILABLE}{/service}
OpenJDK 64-Bit Server VM warning: .hotspot_compiler file is present but has been ignored.  Run with -XX:CompileCommandFile=.hotspot_compiler to load the file.
2018-12-12 00:46:09.815:INFO::main: Logging initialized @2805ms
JettyMonitor monitoring thread pool qtp1286783232{STOPPED,10<=0<=250,i=0,q=0}
Zimbra server reserving server socket port=7110 bindaddr=null ssl=false
Zimbra server reserving server socket port=7995 bindaddr=null ssl=false
Zimbra server reserving server socket port=7143 bindaddr=null ssl=false
Zimbra server reserving server socket port=7993 bindaddr=null ssl=false
Zimbra server reserving server socket port=7025 bindaddr=null ssl=false
2018-12-12 00:46:13.171:INFO:oejs.SetUIDListener:main: Setting umask=027
2018-12-12 00:46:13.172:INFO:oejs.SetUIDListener:main: Current rlimit_nofiles (soft=524288, hard=524288)
2018-12-12 00:46:13.172:INFO:oejs.SetUIDListener:main: Set rlimit_nofiles (soft=65536, hard=65536)
2018-12-12 00:46:13.199:INFO:oejs.SetUIDListener:main: Opened ServerConnector@6e2c9341{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}
2018-12-12 00:46:13.199:INFO:oejs.SetUIDListener:main: Opened ServerConnector@12028586{SSL,[ssl, http/1.1]}{0.0.0.0:8443}
2018-12-12 00:46:13.200:INFO:oejs.SetUIDListener:main: Opened ServerConnector@17776a8{SSL,[ssl, http/1.1]}{0.0.0.0:7071}
2018-12-12 00:46:13.201:INFO:oejs.SetUIDListener:main: Opened ServerConnector@69a10787{SSL,[ssl, http/1.1]}{0.0.0.0:7073}
2018-12-12 00:46:13.201:INFO:oejs.SetUIDListener:main: Opened ServerConnector@2d127a61{SSL,[ssl, http/1.1]}{0.0.0.0:7072}
2018-12-12 00:46:13.202:INFO:oejs.SetUIDListener:main: Setting GID=500
2018-12-12 00:46:13.204:INFO:oejs.SetUIDListener:main: Setting UID=500
2018-12-12 00:46:13.215:INFO:oejs.Server:main: jetty-9.3.5.v20151012
2018-12-12 00:46:27.694:WARN:oejs.SecurityHandler:main: ServletContext@o.e.j.w.WebAppContext@4b14c583{/service,file:///opt/zimbra/jetty_base/webapps/service/,STARTING}{/service} has uncovered http methods for path: /
Zimbra server process is running as uid=500 euid=500 gid=500 egid=500
2018-12-12 00:46:47.534:WARN:/service:main: unavailable
javax.servlet.UnavailableException: Tiering Inversion: 'Pool {107374182400 B offheap}' is not smaller than 'Pool {10737418240 B disk(persistent)}'
	at com.zimbra.soap.SoapServlet.init(SoapServlet.java:134)
	at javax.servlet.GenericServlet.init(GenericServlet.java:244)
	at org.eclipse.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:626)
	at org.eclipse.jetty.servlet.ServletHolder.initialize(ServletHolder.java:405)
	at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:875)
	at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:346)
	at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1380)
	at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1342)
	at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:772)
	at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:259)
	at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:518)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
	at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
	at org.eclipse.jetty.server.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:161)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
	at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
	at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
	at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
	at org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:212)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
	at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
	at org.eclipse.jetty.server.handler.DebugHandler.doStart(DebugHandler.java:143)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
	at org.eclipse.jetty.server.Server.start(Server.java:405)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
	at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
	at org.eclipse.jetty.server.Server.doStart(Server.java:372)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
	at org.eclipse.jetty.xml.XmlConfiguration$1.run(XmlConfiguration.java:1510)
	at java.security.AccessController.doPrivileged(Native Method)
	at org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1435)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.eclipse.jetty.start.Main.invokeMain(Main.java:214)
	at org.eclipse.jetty.start.Main.start(Main.java:457)
	at org.eclipse.jetty.start.Main.main(Main.java:75)
2018-12-12 00:46:49.757:WARN:/service:main: unavailable
javax.servlet.UnavailableException: Couldn't lock rootDir: /opt/zimbra/data/mailboxd
	at com.zimbra.soap.SoapServlet.init(SoapServlet.java:134)
	at javax.servlet.GenericServlet.init(GenericServlet.java:244)
	at org.eclipse.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:626)
	at org.eclipse.jetty.servlet.ServletHolder.initialize(ServletHolder.java:405)
	at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:875)
	at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:346)
	at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1380)
	at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1342)
	at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:772)
	at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:259)
	at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:518)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
	at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
	at org.eclipse.jetty.server.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:161)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
	at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
	at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
	at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
	at org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:212)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
	at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
	at org.eclipse.jetty.server.handler.DebugHandler.doStart(DebugHandler.java:143)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
	at org.eclipse.jetty.server.Server.start(Server.java:405)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
	at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
	at org.eclipse.jetty.server.Server.doStart(Server.java:372)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
	at org.eclipse.jetty.xml.XmlConfiguration$1.run(XmlConfiguration.java:1510)
	at java.security.AccessController.doPrivileged(Native Method)
	at org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1435)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.eclipse.jetty.start.Main.invokeMain(Main.java:214)
	at org.eclipse.jetty.start.Main.start(Main.java:457)
	at org.eclipse.jetty.start.Main.main(Main.java:75)
2018-12-12 00:46:49.983:WARN:oejw.WebAppContext:main: Failed startup of context o.e.j.w.WebAppContext@4b14c583{/service,file:///opt/zimbra/jetty_base/webapps/service/,STARTING}{/service}
MultiException[javax.servlet.UnavailableException: Tiering Inversion: 'Pool {107374182400 B offheap}' is not smaller than 'Pool {10737418240 B disk(persistent)}', javax.servlet.UnavailableException: Couldn't lock rootDir: /opt/zimbra/data/mailboxd]
	at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:846)
	at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:346)
	at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1380)
	at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1342)
	at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:772)
	at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:259)
	at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:518)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
	at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
	at org.eclipse.jetty.server.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:161)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
	at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
	at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
	at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
	at org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:212)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
	at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
	at org.eclipse.jetty.server.handler.DebugHandler.doStart(DebugHandler.java:143)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
	at org.eclipse.jetty.server.Server.start(Server.java:405)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
	at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
	at org.eclipse.jetty.server.Server.doStart(Server.java:372)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
	at org.eclipse.jetty.xml.XmlConfiguration$1.run(XmlConfiguration.java:1510)
	at java.security.AccessController.doPrivileged(Native Method)
	at org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1435)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.eclipse.jetty.start.Main.invokeMain(Main.java:214)
	at org.eclipse.jetty.start.Main.start(Main.java:457)
	at org.eclipse.jetty.start.Main.main(Main.java:75)
Suppressed: 
	|javax.servlet.UnavailableException: Couldn't lock rootDir: /opt/zimbra/data/mailboxd
	|	at com.zimbra.soap.SoapServlet.init(SoapServlet.java:134)
	|	at javax.servlet.GenericServlet.init(GenericServlet.java:244)
	|	at org.eclipse.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:626)
	|	at org.eclipse.jetty.servlet.ServletHolder.initialize(ServletHolder.java:405)
	|	at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:875)
	|	at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:346)
	|	at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1380)
	|	at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1342)
	|	at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:772)
	|	at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:259)
	|	at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:518)
	|	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
	|	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
	|	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
	|	at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
	|	at org.eclipse.jetty.server.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:161)
	|	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
	|	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
	|	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
	|	at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
	|	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
	|	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
	|	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
	|	at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
	|	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
	|	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
	|	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
	|	at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
	|	at org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:212)
	|	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
	|	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
	|	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
	|	at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
	|	at org.eclipse.jetty.server.handler.DebugHandler.doStart(DebugHandler.java:143)
	|	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
	|	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
	|	at org.eclipse.jetty.server.Server.start(Server.java:405)
	|	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
	|	at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
	|	at org.eclipse.jetty.server.Server.doStart(Server.java:372)
	|	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
	|	at org.eclipse.jetty.xml.XmlConfiguration$1.run(XmlConfiguration.java:1510)
	|	at java.security.AccessController.doPrivileged(Native Method)
	|	at org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1435)
	|	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	|	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	|	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	|	at java.lang.reflect.Method.invoke(Method.java:498)
	|	at org.eclipse.jetty.start.Main.invokeMain(Main.java:214)
	|	at org.eclipse.jetty.start.Main.start(Main.java:457)
	|	at org.eclipse.jetty.start.Main.main(Main.java:75)
Caused by: 
javax.servlet.UnavailableException: Tiering Inversion: 'Pool {107374182400 B offheap}' is not smaller than 'Pool {10737418240 B disk(persistent)}'
	at com.zimbra.soap.SoapServlet.init(SoapServlet.java:134)
	at javax.servlet.GenericServlet.init(GenericServlet.java:244)
	at org.eclipse.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:626)
	at org.eclipse.jetty.servlet.ServletHolder.initialize(ServletHolder.java:405)
	at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:875)
	at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:346)
	at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1380)
	at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1342)
	at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:772)
	at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:259)
	at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:518)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
	at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
	at org.eclipse.jetty.server.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:161)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
	at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
	at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
	at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
	at org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:212)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
	at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
	at org.eclipse.jetty.server.handler.DebugHandler.doStart(DebugHandler.java:143)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
	at org.eclipse.jetty.server.Server.start(Server.java:405)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
	at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
	at org.eclipse.jetty.server.Server.doStart(Server.java:372)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
	at org.eclipse.jetty.xml.XmlConfiguration$1.run(XmlConfiguration.java:1510)
	at java.security.AccessController.doPrivileged(Native Method)
	at org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1435)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.eclipse.jetty.start.Main.invokeMain(Main.java:214)
	at org.eclipse.jetty.start.Main.start(Main.java:457)
	at org.eclipse.jetty.start.Main.main(Main.java:75)
2018-12-12 00:46:54.207:WARN:oejpw.PlusDescriptorProcessor:main: No value for env-entry-name trustedIPs
2018-12-12 00:46:55.019:WARN:oejs.SecurityHandler:main: ServletContext@o.e.j.w.WebAppContext@4690b489{/zimbra,file:///opt/zimbra/jetty_base/webapps/zimbra/,STARTING}{/zimbra} has uncovered http methods for path: /
2018-12-12 00:46:57.127:INFO:oejsh.ContextHandler:main: Started o.e.j.w.WebAppContext@4690b489{/zimbra,file:///opt/zimbra/jetty_base/webapps/zimbra/,AVAILABLE}{/zimbra}
2018-12-12 00:47:02.829:WARN:oejpw.PlusDescriptorProcessor:main: No value for env-entry-name trustedIPs
2018-12-12 00:47:03.261:WARN:oejs.SecurityHandler:main: ServletContext@o.e.j.w.WebAppContext@79b06cab{/zimbraAdmin,file:///opt/zimbra/jetty_base/webapps/zimbraAdmin/,STARTING}{/zimbraAdmin} has uncovered http methods for path: /
2018-12-12 00:47:05.007:INFO:oejsh.ContextHandler:main: Started o.e.j.w.WebAppContext@79b06cab{/zimbraAdmin,file:///opt/zimbra/jetty_base/webapps/zimbraAdmin/,AVAILABLE}{/zimbraAdmin}
2018-12-12 00:47:06.009:WARN:oejs.SecurityHandler:main: ServletContext@o.e.j.w.WebAppContext@3eb7fc54{/zimlet,[file:///opt/zimbra/jetty_base/webapps/zimlet/, file:///opt/zimbra/zimlets-deployed/],STARTING}{/zimlet} has uncovered http methods for path: /
2018-12-12 00:47:06.012:INFO:oejsh.ContextHandler:main: Started o.e.j.w.WebAppContext@3eb7fc54{/zimlet,[file:///opt/zimbra/jetty_base/webapps/zimlet/, file:///opt/zimbra/zimlets-deployed/],AVAILABLE}{/zimlet}
2018-12-12 00:47:06.038:INFO:oejs.AbstractNCSARequestLog:main: Opened /opt/zimbra/log/access_log.2018-12-12
2018-12-12 00:47:06.049:INFO:oejm.ThreadMonitor:main: Thread Monitor started successfully
2018-12-12 00:47:06.137:INFO:oejs.ServerConnector:main: Started ServerConnector@6e2c9341{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}
2018-12-12 00:47:06.155:INFO:oejus.SslContextFactory:main: x509=X509@350da119(jetty,h=[x-ray.optiplex-networks.com],w=[]) for SslContextFactory@367d34c0(file:///opt/zimbra/jetty_base/etc/keystore,null)
2018-12-12 00:47:06.196:INFO:oejs.ServerConnector:main: Started ServerConnector@12028586{SSL,[ssl, http/1.1]}{0.0.0.0:8443}
2018-12-12 00:47:06.199:INFO:oejs.ServerConnector:main: Started ServerConnector@17776a8{SSL,[ssl, http/1.1]}{0.0.0.0:7071}
2018-12-12 00:47:06.201:INFO:oejs.ServerConnector:main: Started ServerConnector@69a10787{SSL,[ssl, http/1.1]}{0.0.0.0:7073}
2018-12-12 00:47:06.204:INFO:oejs.ServerConnector:main: Started ServerConnector@2d127a61{SSL,[ssl, http/1.1]}{0.0.0.0:7072}
2018-12-12 00:47:06.204:INFO:oejs.Server:main: Started @59194ms
kayasaman
Posts: 20
Joined: Thu Nov 22, 2018 9:05 pm

Re: ERROR: service.PROXY_ERROR 503 service unavailable

Post by kayasaman »

I found the answer!!! :-D

Analyzing the cache outputs that you suggested the setting was waaaaay too high:

Code: Select all

zmprov -l ms `zmhostname` zimbraImapInactiveSessionEhcacheSize 1048576
I have got access again and everything works :-)

I know this should be a new thread but since it was the original intention of fiddling with the Ehcache in the first place but what would I be looking at to enable access a large IMAP folder either through client or Web interface? Basically I need to run filters on this folder to sort the mail out but the 'Search' function within the web UI doesn't seem to work on the particular folder....
User avatar
king0770
Outstanding Member
Outstanding Member
Posts: 242
Joined: Fri Sep 12, 2014 10:44 pm
Contact:

Re: ERROR: service.PROXY_ERROR 503 service unavailable

Post by king0770 »

Try this...

zmcontrol stop

sudo su - <PRESS ENTER>

As root...

mv /opt/zimbra/jetty/work/ /tmp/

mkdir /opt/zimbra/jetty/work/

chown zimbra:zimbra /opt/zimbra/jetty/work/

su - zimbra <PRESS ENTER>

zmcontrol start
--
Rick King
kayasaman
Posts: 20
Joined: Thu Nov 22, 2018 9:05 pm

Re: ERROR: service.PROXY_ERROR 503 service unavailable

Post by kayasaman »

Virtually the setting that we were initially looking at was:

Code: Select all

zimbraImapActiveSessionEhcacheMaxDiskSize
But it seems that the:

Code: Select all

zimbraImapActiveSessionEhcacheMaxSize
parameter was set over the MaxDiskSize setting causing the issue...


Though still some errors in the zmmailboxd.out file:

Code: Select all

2018-12-12 01:00:25.409:INFO:oejsh.ContextHandler:main: Started o.e.j.w.WebAppContext@3eb7fc54{/zimlet,[file:///opt/zimbra/jetty_base/webapps/zimlet/, file:///opt/zimbra/zimlets-deployed/],AVAILABLE}{/zimlet}
2018-12-12 01:00:25.446:INFO:oejs.AbstractNCSARequestLog:main: Opened /opt/zimbra/log/access_log.2018-12-12
2018-12-12 01:00:25.457:INFO:oejm.ThreadMonitor:main: Thread Monitor started successfully
2018-12-12 01:00:25.503:INFO:oejs.ServerConnector:main: Started ServerConnector@6e2c9341{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}
2018-12-12 01:00:25.540:INFO:oejus.SslContextFactory:main: x509=X509@44492c06(jetty,h=[x-ray.optiplex-networks.com],w=[]) for SslContextFactory@1c2096c6(file:///opt/zimbra/jetty_base/etc/keystore,null)
2018-12-12 01:00:25.637:INFO:oejs.ServerConnector:main: Started ServerConnector@12028586{SSL,[ssl, http/1.1]}{0.0.0.0:8443}
2018-12-12 01:00:25.639:INFO:oejs.ServerConnector:main: Started ServerConnector@17776a8{SSL,[ssl, http/1.1]}{0.0.0.0:7071}
2018-12-12 01:00:25.642:INFO:oejs.ServerConnector:main: Started ServerConnector@69a10787{SSL,[ssl, http/1.1]}{0.0.0.0:7073}
2018-12-12 01:00:25.654:INFO:oejs.ServerConnector:main: Started ServerConnector@2d127a61{SSL,[ssl, http/1.1]}{0.0.0.0:7072}
2018-12-12 01:00:25.655:INFO:oejs.Server:main: Started @79225ms
2018-12-12 01:17:22.697:INFO:oejm.ThreadMonitor:Thread-36: Thread 'Index-8'[RUNNABLE,id:45,cpu:90.72%] SPINNING
2018-12-12 01:17:22.698:WARN:oejm.ThreadMonitor:Thread-36: 
org.eclipse.jetty.monitor.thread.ThreadMonitorException: Thread 'Index-8'[RUNNABLE,id:45,cpu:90.72%] STACK TRACE
	at java.util.regex.Pattern$Branch.match(Pattern.java:4606)
	at java.util.regex.Pattern$GroupHead.match(Pattern.java:4660)
	at java.util.regex.Pattern$GroupHead.match(Pattern.java:4660)
	at java.util.regex.Pattern$Branch.match(Pattern.java:4606)
	at java.util.regex.Pattern$GroupHead.match(Pattern.java:4660)
	at java.util.regex.Pattern$Start.match(Pattern.java:3463)
	at java.util.regex.Matcher.search(Matcher.java:1248)
	at java.util.regex.Matcher.find(Matcher.java:637)
	at com.zimbra.cs.zimlet.handler.RegexHandler.match(RegexHandler.java:54)
	at com.zimbra.cs.object.ObjectHandler.parse(ObjectHandler.java:107)
	at com.zimbra.cs.mime.MimeHandler.getObjects(MimeHandler.java:301)
	at com.zimbra.cs.mime.ParsedMessage.getMainBodyLuceneDocument(ParsedMessage.java:1033)
	at com.zimbra.cs.mime.ParsedMessage.analyzeNonBodyParts(ParsedMessage.java:451)
	at com.zimbra.cs.mime.ParsedMessage.analyzeFully(ParsedMessage.java:473)
	at com.zimbra.cs.mailbox.Message.generateIndexData(Message.java:1405)
	at com.zimbra.cs.mailbox.MailboxIndex.indexItemList(MailboxIndex.java:774)
	at com.zimbra.cs.mailbox.MailboxIndex.indexDeferredItems(MailboxIndex.java:411)
	at com.zimbra.cs.mailbox.MailboxIndex.access$600(MailboxIndex.java:85)
	at com.zimbra.cs.mailbox.MailboxIndex$BatchIndexTask.exec(MailboxIndex.java:1434)
	at com.zimbra.cs.mailbox.MailboxIndex$IndexTask.run(MailboxIndex.java:1413)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
2018-12-12 01:17:22.707:WARN:oejm.ThreadMonitor:Thread-36: 
org.eclipse.jetty.monitor.thread.ThreadMonitorException: Thread 'Index-8'[RUNNABLE,id:45,cpu:74.40%] STACK TRACE
	at java.util.regex.Pattern$Branch.match(Pattern.java:4606)
	at java.util.regex.Pattern$GroupHead.match(Pattern.java:4660)
	at java.util.regex.Pattern$GroupHead.match(Pattern.java:4660)
	at java.util.regex.Pattern$Branch.match(Pattern.java:4606)
	at java.util.regex.Pattern$GroupHead.match(Pattern.java:4660)
	at java.util.regex.Pattern$Start.match(Pattern.java:3463)
	at java.util.regex.Matcher.search(Matcher.java:1248)
	at java.util.regex.Matcher.find(Matcher.java:637)
	at com.zimbra.cs.zimlet.handler.RegexHandler.match(RegexHandler.java:54)
	at com.zimbra.cs.object.ObjectHandler.parse(ObjectHandler.java:107)
	at com.zimbra.cs.mime.MimeHandler.getObjects(MimeHandler.java:301)
	at com.zimbra.cs.mime.ParsedMessage.getMainBodyLuceneDocument(ParsedMessage.java:1033)
	at com.zimbra.cs.mime.ParsedMessage.analyzeNonBodyParts(ParsedMessage.java:451)
	at com.zimbra.cs.mime.ParsedMessage.analyzeFully(ParsedMessage.java:473)
	at com.zimbra.cs.mailbox.Message.generateIndexData(Message.java:1405)
	at com.zimbra.cs.mailbox.MailboxIndex.indexItemList(MailboxIndex.java:774)
	at com.zimbra.cs.mailbox.MailboxIndex.indexDeferredItems(MailboxIndex.java:411)
	at com.zimbra.cs.mailbox.MailboxIndex.access$600(MailboxIndex.java:85)
	at com.zimbra.cs.mailbox.MailboxIndex$BatchIndexTask.exec(MailboxIndex.java:1434)
	at com.zimbra.cs.mailbox.MailboxIndex$IndexTask.run(MailboxIndex.java:1413)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
2018-12-12 01:17:22.720:WARN:oejm.ThreadMonitor:Thread-36: 
org.eclipse.jetty.monitor.thread.ThreadMonitorException: Thread 'Index-8'[RUNNABLE,id:45,cpu:79.95%] STACK TRACE
	at java.util.regex.Pattern$GroupHead.match(Pattern.java:4660)
	at java.util.regex.Pattern$Branch.match(Pattern.java:4606)
	at java.util.regex.Pattern$GroupHead.match(Pattern.java:4660)
	at java.util.regex.Pattern$Start.match(Pattern.java:3463)
	at java.util.regex.Matcher.search(Matcher.java:1248)
	at java.util.regex.Matcher.find(Matcher.java:637)
	at com.zimbra.cs.zimlet.handler.RegexHandler.match(RegexHandler.java:54)
	at com.zimbra.cs.object.ObjectHandler.parse(ObjectHandler.java:107)
	at com.zimbra.cs.mime.MimeHandler.getObjects(MimeHandler.java:301)
	at com.zimbra.cs.mime.ParsedMessage.getMainBodyLuceneDocument(ParsedMessage.java:1033)
	at com.zimbra.cs.mime.ParsedMessage.analyzeNonBodyParts(ParsedMessage.java:451)
	at com.zimbra.cs.mime.ParsedMessage.analyzeFully(ParsedMessage.java:473)
	at com.zimbra.cs.mailbox.Message.generateIndexData(Message.java:1405)
	at com.zimbra.cs.mailbox.MailboxIndex.indexItemList(MailboxIndex.java:774)
	at com.zimbra.cs.mailbox.MailboxIndex.indexDeferredItems(MailboxIndex.java:411)
	at com.zimbra.cs.mailbox.MailboxIndex.access$600(MailboxIndex.java:85)
	at com.zimbra.cs.mailbox.MailboxIndex$BatchIndexTask.exec(MailboxIndex.java:1434)
	at com.zimbra.cs.mailbox.MailboxIndex$IndexTask.run(MailboxIndex.java:1413)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
	
Again as stated above this is happening on one particular large folder that I'm getting a timeout.
User avatar
barrydegraaff
Zimbra Employee
Zimbra Employee
Posts: 242
Joined: Tue Jun 17, 2014 3:31 am
Contact:

Re: ERROR: service.PROXY_ERROR 503 service unavailable

Post by barrydegraaff »

Since you are on CentOS6, I am guessing you did a lot of in-place upgrades. There is a high change you will run into more issues,
even if you fix the current one.

Perhaps consider migrating to a fresh install.
kayasaman
Posts: 20
Joined: Thu Nov 22, 2018 9:05 pm

Re: ERROR: service.PROXY_ERROR 503 service unavailable

Post by kayasaman »

Thanks for the suggestion!

However, this is technically a new build as the upgrade process didn't work due to LDAP issues. The only things that remain the same with this build are the data store, db, and ldap store the rest is all completely fresh.

I ended up having to backup the /opt/zimbra dir then run install.sh -u and proceed with a migration.
Post Reply