zmmailboxdctl is not running (NoSuchMethodException)

fnortech · Post by **fnortech** » Tue May 26, 2020 8:30 am

Hello everyone,

I got the issue that zmmailboxd is not starting after upgrading from 8.8.12 to 8.8.15. I tried the following upgrade paths and always got the same problem:

8.8.11_Ubuntu16.04 => 8.8.12_Ubuntu16.04 => 8.8.15_Ubuntu16.04
8.8.11_Ubuntu16.04 => 8.8.12_Ubuntu16.04 => 8.8.12_Ubuntu18.04 (inplace upgrade) => 8.8.15_Ubuntu18.04
8.8.11_Ubuntu16.04 => 8.8.12_Ubuntu16.04 => 8.8.12_Ubuntu18.04 (inplace upgrade) => rsync /opt migration to fresh 18.04 installation => 8.8.15_Ubuntu18.04

After upgrading from 8.8.11 to 8.8.12 everything still works fine with all the upgrade paths. When upgrading from 8.8.12 to 8.8.15 the zmmailboxdctl ist not starting in any case:

Code: Select all

$ zmmailboxdctl start
Starting mailboxd...failed.

While starting I can see the following error in /opt/zimbra/log/zmmailboxd.out.

Code: Select all

[0.024s][info][gc] Using G1
OpenJDK 64-Bit Server VM warning: .hotspot_compiler file is present but has been ignored.  Run with -XX:CompileCommandFile=.hotspot_compiler to load the file.
[0.046s][info][gc] Periodic GC disabled
2020-05-26 08:11:41.237:INFO::main: Logging initialized @955ms to org.eclipse.jetty.util.log.StdErrLog
JettyMonitor monitoring thread pool QueuedThreadPool[qtp135184888]@80ec1f8{STOPPED,10<=0<=250,i=0,r=-1,q=0}[NO_TRY]
2020-05-26 08:11:41.877:WARN:oejx.XmlConfiguration:main: Config error at <Get name="sessionManager"><Set name="httpOnly">TRUE</Set></Get> java.lang.NoSuchMethodException: org.eclipse.jetty.server.session.SessionHandler.getSessionManager() in file:/opt/zimbra/mailboxd/etc/jetty.xml
2020-05-26 08:11:41.877:WARN:oejx.XmlConfiguration:main: Config error at <Get name="sessionHandler"><Get name="sessionManager"><Set name="httpOnly">TRUE</Set></Get></Get> java.lang.NoSuchMethodException: org.eclipse.jetty.server.session.SessionHandler.getSessionManager() in file:/opt/zimbra/mailboxd/etc/jetty.xml
2020-05-26 08:11:41.877:WARN:oejx.XmlConfiguration:main: Config error at <New id="zimbra" class="org.eclipse.jetty.webapp.WebAppContext"><Arg><Ref id="Contexts"/></Arg><Arg><SystemProperty name="jetty.base" default="."/>/webapps/zimbra</Arg><Arg>/zimbra</Arg><Set name="configurationClasses"><Ref id="plusConfig"/></Set><Set name="defaultsDescriptor"><SystemProperty name="jetty.base" default="."/>/etc/webdefault.xml</Set><Set name="tempDirectory"><SystemProperty name="jetty.base" default="."/>/work/zimbra</Set><Set name="persistTempDirectory">true</Set><Set name="compactPath">true</Set><Set name="throwUnavailableOnStartupException">true</Set><Set name="extraClasspath">/opt/zimbra/lib/jars/zimbrasoap.jar,/opt/zimbra/lib/jars/zimbraclient.jar,/opt/zimbra/lib/jars/zimbrastore.jar</Set><Get name="errorHandler"><Call name="addErrorPage"><Arg type="int">400</Arg><Arg type="int">599</Arg><Arg type="String">/public/error.jsp</Arg></Call></Get><Get name="sessionHandler"><Get name="sessionManager"><Set name="httpOnly">TRUE</Set></Get></Get><Call name="setAttribute"><Arg>org.eclipse.jetty.server.webapp.ContainerIncludeJarPattern</Arg><Arg>.*/.*jsp-api-[^/]*\.jar$|.*/.*jsp-[^/]*\.jar$|.*/.*taglibs[^/]*\.jar$</Arg></Call></New> java.lang.NoSuchMethodException: org.eclipse.jetty.server.session.SessionHandler.getSessionManager() in file:/opt/zimbra/mailboxd/etc/jetty.xml
2020-05-26 08:11:41.878:WARN:oejx.XmlConfiguration:main: 
java.security.PrivilegedActionException: java.lang.NoSuchMethodException: org.eclipse.jetty.server.session.SessionHandler.getSessionManager()
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:558)
	at org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1511)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:567)
	at org.eclipse.jetty.start.Main.invokeMain(Main.java:220)
	at org.eclipse.jetty.start.Main.start(Main.java:490)
	at org.eclipse.jetty.start.Main.main(Main.java:77)
Caused by: 
java.lang.NoSuchMethodException: org.eclipse.jetty.server.session.SessionHandler.getSessionManager()
	at java.base/java.lang.Class.getMethod(Class.java:2109)
	at org.eclipse.jetty.xml.XmlConfiguration$JettyXmlConfiguration.get(XmlConfiguration.java:744)
	at org.eclipse.jetty.xml.XmlConfiguration$JettyXmlConfiguration.configure(XmlConfiguration.java:475)
	at org.eclipse.jetty.xml.XmlConfiguration$JettyXmlConfiguration.get(XmlConfiguration.java:749)
	at org.eclipse.jetty.xml.XmlConfiguration$JettyXmlConfiguration.configure(XmlConfiguration.java:475)
	at org.eclipse.jetty.xml.XmlConfiguration$JettyXmlConfiguration.newObj(XmlConfiguration.java:875)
	at org.eclipse.jetty.xml.XmlConfiguration$JettyXmlConfiguration.configure(XmlConfiguration.java:478)
	at org.eclipse.jetty.xml.XmlConfiguration$JettyXmlConfiguration.configure(XmlConfiguration.java:413)
	at org.eclipse.jetty.xml.XmlConfiguration.configure(XmlConfiguration.java:311)
	at org.eclipse.jetty.xml.XmlConfiguration$1.run(XmlConfiguration.java:1558)
	at org.eclipse.jetty.xml.XmlConfiguration$1.run(XmlConfiguration.java:1512)
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:554)
	at org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1511)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:567)
	at org.eclipse.jetty.start.Main.invokeMain(Main.java:220)
	at org.eclipse.jetty.start.Main.start(Main.java:490)
	at org.eclipse.jetty.start.Main.main(Main.java:77)
java.lang.reflect.InvocationTargetException
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:567)
	at org.eclipse.jetty.start.Main.invokeMain(Main.java:220)
	at org.eclipse.jetty.start.Main.start(Main.java:490)
	at org.eclipse.jetty.start.Main.main(Main.java:77)
Caused by: java.security.PrivilegedActionException: java.lang.NoSuchMethodException: org.eclipse.jetty.server.session.SessionHandler.getSessionManager()
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:558)
	at org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1511)
	... 7 more
Caused by: java.lang.NoSuchMethodException: org.eclipse.jetty.server.session.SessionHandler.getSessionManager()
	at java.base/java.lang.Class.getMethod(Class.java:2109)
	at org.eclipse.jetty.xml.XmlConfiguration$JettyXmlConfiguration.get(XmlConfiguration.java:744)
	at org.eclipse.jetty.xml.XmlConfiguration$JettyXmlConfiguration.configure(XmlConfiguration.java:475)
	at org.eclipse.jetty.xml.XmlConfiguration$JettyXmlConfiguration.get(XmlConfiguration.java:749)
	at org.eclipse.jetty.xml.XmlConfiguration$JettyXmlConfiguration.configure(XmlConfiguration.java:475)
	at org.eclipse.jetty.xml.XmlConfiguration$JettyXmlConfiguration.newObj(XmlConfiguration.java:875)
	at org.eclipse.jetty.xml.XmlConfiguration$JettyXmlConfiguration.configure(XmlConfiguration.java:478)
	at org.eclipse.jetty.xml.XmlConfiguration$JettyXmlConfiguration.configure(XmlConfiguration.java:413)
	at org.eclipse.jetty.xml.XmlConfiguration.configure(XmlConfiguration.java:311)
	at org.eclipse.jetty.xml.XmlConfiguration$1.run(XmlConfiguration.java:1558)
	at org.eclipse.jetty.xml.XmlConfiguration$1.run(XmlConfiguration.java:1512)
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:554)
	... 8 more
java.lang.reflect.InvocationTargetException
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:567)
	at org.eclipse.jetty.start.Main.invokeMain(Main.java:220)
	at org.eclipse.jetty.start.Main.start(Main.java:490)
	at org.eclipse.jetty.start.Main.main(Main.java:77)
Caused by: java.security.PrivilegedActionException: java.lang.NoSuchMethodException: org.eclipse.jetty.server.session.SessionHandler.getSessionManager()
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:558)
	at org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1511)
	... 7 more
Caused by: java.lang.NoSuchMethodException: org.eclipse.jetty.server.session.SessionHandler.getSessionManager()
	at java.base/java.lang.Class.getMethod(Class.java:2109)
	at org.eclipse.jetty.xml.XmlConfiguration$JettyXmlConfiguration.get(XmlConfiguration.java:744)
	at org.eclipse.jetty.xml.XmlConfiguration$JettyXmlConfiguration.configure(XmlConfiguration.java:475)
	at org.eclipse.jetty.xml.XmlConfiguration$JettyXmlConfiguration.get(XmlConfiguration.java:749)
	at org.eclipse.jetty.xml.XmlConfiguration$JettyXmlConfiguration.configure(XmlConfiguration.java:475)
	at org.eclipse.jetty.xml.XmlConfiguration$JettyXmlConfiguration.newObj(XmlConfiguration.java:875)
	at org.eclipse.jetty.xml.XmlConfiguration$JettyXmlConfiguration.configure(XmlConfiguration.java:478)
	at org.eclipse.jetty.xml.XmlConfiguration$JettyXmlConfiguration.configure(XmlConfiguration.java:413)
	at org.eclipse.jetty.xml.XmlConfiguration.configure(XmlConfiguration.java:311)
	at org.eclipse.jetty.xml.XmlConfiguration$1.run(XmlConfiguration.java:1558)
	at org.eclipse.jetty.xml.XmlConfiguration$1.run(XmlConfiguration.java:1512)
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:554)
	... 8 more

Usage: java -jar $JETTY_HOME/start.jar [options] [properties] [configs]
       java -jar $JETTY_HOME/start.jar --help  # for more information

Since the /opt/zimbra/mailboxd/etc/jetty.xml seems to contain some sensitive information I am not pasting it fulltext here. But if you need any specific snippet from it I will provide it.

Does anyone have an idea what causes this problem? I found this two other posts:

viewtopic.php?f=13&t=68240
viewtopic.php?f=13&t=65982

but the errors are different there.

Thank you very much in advance
FT

fnortech · Post by **fnortech** » Tue Jun 16, 2020 2:40 pm

The problem persists. Does anyone have a hint into the right direction?

fs.schmidt · Post by **fs.schmidt** » Sun Jun 21, 2020 11:59 pm

Hello,

Did you find any "Unrecognized VM option ‘PrintGCDateStamps’" message in zmmailboxd.out?

Is there any error log in mailbox.log as well?

fnortech · Post by **fnortech** » Wed Jun 24, 2020 8:20 am

Hi fs.schmidt,

thank you very much for your reply. I checked the zmmailboxd.out but there is no "Unrecognized VM option" logged. Only the Logs I provided when opening this post.

Also there are no messages logged to mailbox.log when running "zmmailboxdctl start".

Do you or anyone else have some more ideas? I am really lost with this problem since I cannot upgrade the installation to the latest version.

Thank you very much!

L. Mark Stone · Post by **L. Mark Stone** » Thu Jun 25, 2020 5:19 pm

Please post the output from:

Code: Select all

zmlocalconfig mailboxd_java_options

This variable is not updated by the upgrade installer (but should be).

Mark

fnortech · Post by **fnortech** » Fri Jul 03, 2020 12:32 pm

Hi Mark,

thank you very much for your reply. Here is the information you requested:

Code: Select all

zimbra@mail:~ zmcontrol -v
Release 8.8.15.GA.3869.UBUNTU16.64 UBUNTU16_64 FOSS edition, Patch 8.8.15_P10.

zimbra@mail:~$ zmlocalconfig mailboxd_java_options
mailboxd_java_options = -server -Dhttps.protocols=TLSv1,TLSv1.1,TLSv1.2 -Djdk.tls.client.protocols=TLSv1,TLSv1.1,TLSv1.2 -Djava.awt.headless=true -Dsun.net.inetaddr.ttl= -Dorg.apache.jasper.compiler.disablejsr199=true -XX:+UseG1GC -XX:SoftRefLRUPolicyMSPerMB=1 -XX:-OmitStackTraceInFastThrow -verbose:gc -Xlog:gc*=debug,safepoint=info:file=/opt/zimbra/log/gc.log:time:filecount=20,filesize=10m -Djava.net.preferIPv4Stack=true

Best Regards

L. Mark Stone · Post by **L. Mark Stone** » Sat Jul 04, 2020 5:42 pm

This is what I am running; vetted by Zimbra Support on two different Support Cases. 8.8.15 Network Edition Patch 10, Ubuntu 16.04.

Code: Select all

zimbra@mb18:~$ zmlocalconfig mailboxd_java_options
mailboxd_java_options = -server -Dhttps.protocols=TLSv1,TLSv1.1,TLSv1.2 -Djdk.tls.client.protocols=TLSv1,TLSv1.1,TLSv1.2 -Djava.awt.headless=true -Dsun.net.inetaddr.ttl=${networkaddress_cache_ttl} -Dorg.apache.jasper.compiler.disablejsr199=true -XX:+UseG1GC -XX:SoftRefLRUPolicyMSPerMB=1 -XX:+UnlockExperimentalVMOptions -XX:G1NewSizePercent=15 -XX:G1MaxNewSizePercent=45 -XX:-OmitStackTraceInFastThrow -verbose:gc -Xlog:gc*=info,safepoint=info:file=/opt/zimbra/log/gc.log:time:filecount=20,filesize=10m -Djava.net.preferIPv4Stack=true
zimbra@mb18:~$

Hope that helps,
Mark

fnortech · Post by **fnortech** » Mon Jul 06, 2020 9:38 am

Hi Mark,

thank you very much for the tipp. I tried it in the following way:

Code: Select all

$ zmlocalconfig -e mailboxd_java_options="-server -Dhttps.protocols=TLSv1,TLSv1.1,TLSv1.2 -Djdk.tls.client.protocols=TLSv1,TLSv1.1,TLSv1.2 -Djava.awt.headless=true -Dsun.net.inetaddr.ttl=${networkaddress_cache_ttl} -Dorg.apache.jasper.compiler.disablejsr199=true -XX:+UseG1GC -XX:SoftRefLRUPolicyMSPerMB=1 -XX:+UnlockExperimentalVMOptions -XX:G1NewSizePercent=15 -XX:G1MaxNewSizePercent=45 -XX:-OmitStackTraceInFastThrow -verbose:gc -Xlog:gc*=info,safepoint=info:file=/opt/zimbra/log/gc.log:time:filecount=20,filesize=10m -Djava.net.preferIPv4Stack=true"

Code: Select all

$ zmlocalconfig mailboxd_java_options
mailboxd_java_options = -server -Dhttps.protocols=TLSv1,TLSv1.1,TLSv1.2 -Djdk.tls.client.protocols=TLSv1,TLSv1.1,TLSv1.2 -Djava.awt.headless=true -Dsun.net.inetaddr.ttl= -Dorg.apache.jasper.compiler.disablejsr199=true -XX:+UseG1GC -XX:SoftRefLRUPolicyMSPerMB=1 -XX:+UnlockExperimentalVMOptions -XX:G1NewSizePercent=15 -XX:G1MaxNewSizePercent=45 -XX:-OmitStackTraceInFastThrow -verbose:gc -Xlog:gc*=info,safepoint=info:file=/opt/zimbra/log/gc.log:time:filecount=20,filesize=10m -Djava.net.preferIPv4Stack=true

Code: Select all

$ zmcontrol stop
$ zmcontrol start

but unfortunately the error still persists. I still can not start the mailbox service. The errors in /opt/zimbra/log/zmmailboxd.out is still the same after the configuration change.

Any more Ideas?

Thank you very much everybody!

FN

L. Mark Stone · Post by **L. Mark Stone** » Mon Jul 06, 2020 11:27 am

I would say then that your problem is likely related to the Ubuntu upgrade you are doing in tandem with your Zimbra upgrade to 8.8.15.

Try the in-place upgrade to 8.8.15 without upgrading Ubuntu; if mailboxd starts then you know for sure it’s an OS issue, and I would suggest then doing a side-by-side migration to a new Ubuntu 18 server.

Hope that helps,
Mark

fnortech · Post by **fnortech** » Mon Jul 06, 2020 2:08 pm

Hi Mark,

Thank you again for your support. I already tried different upgrade paths, including the most simple (staying within same ubuntu version). The following paths were tried (as described in my initial post):

8.8.11_Ubuntu16.04 => 8.8.12_Ubuntu16.04 => 8.8.15_Ubuntu16.04
8.8.11_Ubuntu16.04 => 8.8.12_Ubuntu16.04 => 8.8.12_Ubuntu18.04 (inplace upgrade) => 8.8.15_Ubuntu18.04
8.8.11_Ubuntu16.04 => 8.8.12_Ubuntu16.04 => 8.8.12_Ubuntu18.04 (inplace upgrade) => rsync /opt migration to fresh 18.04 installation => 8.8.15_Ubuntu18.04

They all lead to the exact same result.

Is there a way to install a fresh 8.8.15_Ubuntu16.04 or 8.8.15_Ubuntu18.04 installation and migrate all data (and only the data) from an existing 8.8.11_Ubuntu16.04, 8.8.12_Ubuntu16.04 or 8.8.12_Ubuntu18.04 since these are the versions I am able to run without any problems?

Best regards,

FT

Zimbra Forums

zmmailboxdctl is not running (NoSuchMethodException)

zmmailboxdctl is not running (NoSuchMethodException)

Re: zmmailboxdctl is not running (NoSuchMethodException)

Re: zmmailboxdctl is not running (NoSuchMethodException)

Re: zmmailboxdctl is not running (NoSuchMethodException)

Re: zmmailboxdctl is not running (NoSuchMethodException)

Re: zmmailboxdctl is not running (NoSuchMethodException)

Re: zmmailboxdctl is not running (NoSuchMethodException)

Re: zmmailboxdctl is not running (NoSuchMethodException)

Re: zmmailboxdctl is not running (NoSuchMethodException)

Re: zmmailboxdctl is not running (NoSuchMethodException)