My server is still kinda broken - I upgraded on the 18th.
Apparently it has messed up the zmproxy settings and mailboxd tries to open the ports the proxy manages.
of course that way they conflict and it can't start like that.
[zimbra@zmail ~]$ zmcontrol -v
Release 8.8.15_GA_3953.RHEL8_64_20200629025823 RHEL8_64 FOSS edition.
Package changes:
2022-06-18T19:56:52+0000 DEBUG --> Starting dependency resolution
2022-06-18T19:56:52+0000 DEBUG ---> Package libretls.x86_64 3.5.1-1.el8 will be upgraded
2022-06-18T19:56:52+0000 DEBUG ---> Package libretls.x86_64 3.5.2-1.el8 will be an upgrade
2022-06-18T19:56:52+0000 DEBUG ---> Package netcat.x86_64 1.218-4.el8 will be upgraded
2022-06-18T19:56:52+0000 DEBUG ---> Package netcat.x86_64 1.218-5.el8 will be an upgrade
2022-06-18T19:56:52+0000 DEBUG ---> Package zimbra-clamav.x86_64 0.103.2-1zimbra8.8b3.el8 will be upgraded
2022-06-18T19:56:52+0000 DEBUG ---> Package zimbra-clamav.x86_64 0.103.3-1zimbra8.8b3.el8 will be an upgrade
2022-06-18T19:56:52+0000 DEBUG ---> Package zimbra-clamav-libs.x86_64 0.103.2-1zimbra8.8b3.el8 will be upgraded
2022-06-18T19:56:52+0000 DEBUG ---> Package zimbra-clamav-libs.x86_64 0.103.3-1zimbra8.8b3.el8 will be an upgrade
2022-06-18T19:56:52+0000 DEBUG ---> Package zimbra-common-core-jar.x86_64 8.8.15.1647325929-1.r8 will be upgraded
2022-06-18T19:56:52+0000 DEBUG ---> Package zimbra-common-core-jar.x86_64 8.8.15.1655458176-1.r8 will be an upgrade
2022-06-18T19:56:52+0000 DEBUG ---> Package zimbra-common-core-libs.x86_64 8.8.15.1647230035-1.r8 will be upgraded
2022-06-18T19:56:52+0000 DEBUG ---> Package zimbra-common-core-libs.x86_64 8.8.15.1654854265-1.r8 will be an upgrade
2022-06-18T19:56:52+0000 DEBUG ---> Package zimbra-common-mbox-conf-attrs.x86_64 8.8.15.1571124020-1.r8 will be upgraded
2022-06-18T19:56:52+0000 DEBUG ---> Package zimbra-common-mbox-conf-attrs.x86_64 8.8.15.1652767386-1.r8 will be an upgrade
2022-06-18T19:56:52+0000 DEBUG ---> Package zimbra-common-mbox-conf-msgs.x86_64 8.8.15.1556130968-1.r8 will be upgraded
2022-06-18T19:56:52+0000 DEBUG ---> Package zimbra-common-mbox-conf-msgs.x86_64 8.8.15.1652703447-1.r8 will be an upgrade
2022-06-18T19:56:52+0000 DEBUG ---> Package zimbra-core-components.x86_64 2.0.14-1zimbra8.8b1.el8 will be upgraded
2022-06-18T19:56:52+0000 DEBUG ---> Package zimbra-core-components.x86_64 2.0.16-1zimbra8.8b1.el8 will be an upgrade
2022-06-18T19:56:52+0000 DEBUG ---> Package zimbra-jetty-distribution.x86_64 9.4.18.v20190429-2.r8 will be upgraded
2022-06-18T19:56:52+0000 DEBUG ---> Package zimbra-jetty-distribution.x86_64 9.4.46.v20220331-2.r8 will be an upgrade
2022-06-18T19:56:52+0000 DEBUG ---> Package zimbra-ldap-components.x86_64 1.0.14-1zimbra8.8b1.el8 will be upgraded
2022-06-18T19:56:52+0000 DEBUG ---> Package zimbra-ldap-components.x86_64 1.0.16-1zimbra8.8b1.el8 will be an upgrade
2022-06-18T19:56:52+0000 DEBUG ---> Package zimbra-mbox-admin-console-war.x86_64 8.8.15.1643880751-1.r8 will be upgraded
2022-06-18T19:56:52+0000 DEBUG ---> Package zimbra-mbox-admin-console-war.x86_64 8.8.15.1653031987-1.r8 will be an upgrade
2022-06-18T19:56:52+0000 DEBUG ---> Package zimbra-mbox-store-libs.x86_64 8.8.15.1647230035-1.r8 will be upgraded
2022-06-18T19:56:52+0000 DEBUG ---> Package zimbra-mbox-store-libs.x86_64 8.8.15.1654854265-1.r8 will be an upgrade
2022-06-18T19:56:52+0000 DEBUG ---> Package zimbra-mbox-war.x86_64 8.8.15.1647325929-1.r8 will be upgraded
2022-06-18T19:56:52+0000 DEBUG ---> Package zimbra-mbox-war.x86_64 8.8.15.1655458176-1.r8 will be an upgrade
2022-06-18T19:56:52+0000 DEBUG ---> Package zimbra-mbox-webclient-war.x86_64 8.8.15.1647334948-1.r8 will be upgraded
2022-06-18T19:56:52+0000 DEBUG ---> Package zimbra-mbox-webclient-war.x86_64 8.8.15.1654769776-1.r8 will be an upgrade
2022-06-18T19:56:52+0000 DEBUG ---> Package zimbra-mta-components.x86_64 1.0.14-1zimbra8.8b1.el8 will be upgraded
2022-06-18T19:56:52+0000 DEBUG ---> Package zimbra-mta-components.x86_64 1.0.15-1zimbra8.8b1.el8 will be an upgrade
2022-06-18T19:56:52+0000 DEBUG ---> Package zimbra-mta-patch.x86_64 8.8.15.1647332084.p31-1.r8 will be upgraded
2022-06-18T19:56:52+0000 DEBUG ---> Package zimbra-mta-patch.x86_64 8.8.15.1655471268.p32-1.r8 will be an upgrade
2022-06-18T19:56:52+0000 DEBUG ---> Package zimbra-nginx.x86_64 1.20.0-1zimbra8.8b2.el8 will be upgraded
2022-06-18T19:56:52+0000 DEBUG ---> Package zimbra-nginx.x86_64 1.20.0-1zimbra8.8b3.el8 will be an upgrade
2022-06-18T19:56:52+0000 DEBUG ---> Package zimbra-openjdk.x86_64 13.0.1-1zimbra8.8b1.el8 will be upgraded
2022-06-18T19:56:52+0000 DEBUG ---> Package zimbra-openjdk.x86_64 17.0.2-1zimbra8.8b1.el8 will be an upgrade
2022-06-18T19:56:52+0000 DEBUG ---> Package zimbra-openssl.x86_64 1.1.1l-1zimbra8.7b4.el8 will be upgraded
2022-06-18T19:56:52+0000 DEBUG ---> Package zimbra-openssl.x86_64 1.1.1n-1zimbra8.7b4.el8 will be an upgrade
2022-06-18T19:56:52+0000 DEBUG ---> Package zimbra-openssl-libs.x86_64 1.1.1l-1zimbra8.7b4.el8 will be upgraded
2022-06-18T19:56:52+0000 DEBUG ---> Package zimbra-openssl-libs.x86_64 1.1.1n-1zimbra8.7b4.el8 will be an upgrade
2022-06-18T19:56:52+0000 DEBUG ---> Package zimbra-patch.x86_64 8.8.15.1648259498.p31-1.r8 will be upgraded
2022-06-18T19:56:52+0000 DEBUG ---> Package zimbra-patch.x86_64 8.8.15.1655471268.p32-1.r8 will be an upgrade
2022-06-18T19:56:52+0000 DEBUG ---> Package zimbra-perl-mail-spamassassin.x86_64 3.4.5-1zimbra8.8b4.el8 will be upgraded
2022-06-18T19:56:52+0000 DEBUG ---> Package zimbra-perl-mail-spamassassin.x86_64 3.4.6-1zimbra8.8b4.el8 will be an upgrade
2022-06-18T19:56:52+0000 DEBUG ---> Package zimbra-proxy-components.x86_64 1.0.9-1zimbra8.8b1.el8 will be upgraded
2022-06-18T19:56:52+0000 DEBUG ---> Package zimbra-proxy-components.x86_64 1.0.10-1zimbra8.8b1.el8 will be an upgrade
2022-06-18T19:56:52+0000 DEBUG ---> Package zimbra-proxy-patch.x86_64 8.8.15.1648259498.p31-1.r8 will be upgraded
2022-06-18T19:56:52+0000 DEBUG ---> Package zimbra-proxy-patch.x86_64 8.8.15.1655471268.p32-1.r8 will be an upgrade
2022-06-18T19:56:52+0000 DEBUG ---> Package zimbra-spamassassin-rules.x86_64 1.0.0-1zimbra8.8b5.el8 will be upgraded
2022-06-18T19:56:52+0000 DEBUG ---> Package zimbra-spamassassin-rules.x86_64 1.0.0-1zimbra8.8b6.el8 will be an upgrade
2022-06-18T19:56:52+0000 DEBUG --> Finished dependency resolution
symptom:
Jun 19 03:48:01 zmail sudo[220050]: pam_unix(sudo:session): session opened for user root by (uid=0)
Jun 19 03:48:01 zmail zmmailboxdmgr[220052]: stale pid 21616 found in /opt/zimbra/log/zmmailboxd_manager.pid: No such process
Jun 19 03:48:01 zmail zmmailboxdmgr[220052]: assuming no other instance is running
Jun 19 03:48:01 zmail zmmailboxdmgr[220052]: file /opt/zimbra/log/zmmailboxd.pid does not exist
Jun 19 03:48:01 zmail zmmailboxdmgr[220052]: assuming no other instance is running
Jun 19 03:48:01 zmail zmmailboxdmgr[220052]: no manager process is running
Jun 19 03:48:01 zmail sudo[220050]: pam_unix(sudo:session): session closed for user root
Jun 19 03:48:01 zmail zmconfigd[216832]: Tracking service mailbox
Jun 19 03:48:01 zmail sudo[220065]: zimbra : TTY=unknown ; PWD=/opt/zimbra ; USER=root ; COMMAND=/opt/zimbra/libexec/zmmailboxdmgr status
Jun 19 03:48:01 zmail sudo[220065]: pam_systemd(sudo:session): Cannot create session: Already running in a session or user slice
Jun 19 03:48:01 zmail sudo[220065]: pam_unix(sudo:session): session opened for user root by (uid=0)
Jun 19 03:48:01 zmail zmmailboxdmgr[220066]: stale pid 21616 found in /opt/zimbra/log/zmmailboxd_manager.pid: No such process
Jun 19 03:48:01 zmail zmmailboxdmgr[220066]: assuming no other instance is running
Jun 19 03:48:01 zmail zmmailboxdmgr[220066]: file /opt/zimbra/log/zmmailboxd.pid does not exist
Jun 19 03:48:01 zmail zmmailboxdmgr[220066]: assuming no other instance is running
Jun 19 03:48:01 zmail zmmailboxdmgr[220066]: no manager process is running
actual issue:
==> /opt/zimbra/log/zmmailboxd.out <==
2022-06-18 23:26:02.000:INFO::main: Logging initialized @930ms to org.eclipse.jetty.util.log.StdErrLog
JettyMonitor monitoring thread pool QueuedThreadPool[qtp93314457]@58fdd99{STOPPED,10<=0<=250,i=0,r=-1,q=0}[NO_TRY]
2022-06-18 23:26:02.484:WARN:oejx.XmlConfiguration:main: Deprecated constructor public org.eclipse.jetty.util.ssl.SslContextFactory() in file:///opt/zimbra/mailboxd/etc/jetty.xml
2022-06-18 23:26:03.119:WARN:oejx.XmlConfiguration:main: Deprecated constructor public java.lang.Integer(java.lang.String) throws java.lang.NumberFormatException in file:///opt/zimbra/mailboxd/etc/jetty-setuid.xml
2022-06-18 23:26:03.120:WARN:oejx.XmlConfiguration:main: Deprecated constructor public java.lang.Boolean(java.lang.String) in file:///opt/zimbra/mailboxd/etc/jetty-setuid.xml
2022-06-18 23:26:03.120:WARN:oejx.XmlConfiguration:main: Deprecated constructor public java.lang.Boolean(java.lang.String) in file:///opt/zimbra/mailboxd/etc/jetty-setuid.xml
2022-06-18 23:26:03.121:WARN:oejx.XmlConfiguration:main: Deprecated constructor public java.lang.Integer(java.lang.String) throws java.lang.NumberFormatException in file:///opt/zimbra/mailboxd/etc/jetty-setuid.xml
2022-06-18 23:26:03.121:WARN:oejx.XmlConfiguration:main: Deprecated constructor public java.lang.Boolean(java.lang.String) in file:///opt/zimbra/mailboxd/etc/jetty-setuid.xml
2022-06-18 23:26:03.122:WARN:oejx.XmlConfiguration:main: Deprecated constructor public java.lang.Integer(java.lang.String) throws java.lang.NumberFormatException in file:///opt/zimbra/mailboxd/etc/jetty-setuid.xml
2022-06-18 23:26:03.126:WARN:oejx.XmlConfiguration:main: Deprecated constructor public java.lang.Boolean(java.lang.String) in file:///opt/zimbra/mailboxd/etc/jetty-setuid.xml
2022-06-18 23:26:03.127:WARN:oejx.XmlConfiguration:main: Deprecated constructor public java.lang.Integer(java.lang.String) throws java.lang.NumberFormatException in file:///opt/zimbra/mailboxd/etc/jetty-setuid.xml
2022-06-18 23:26:03.127:WARN:oejx.XmlConfiguration:main: Deprecated constructor public java.lang.Boolean(java.lang.String) in file:///opt/zimbra/mailboxd/etc/jetty-setuid.xml
2022-06-18 23:26:03.127:WARN:oejx.XmlConfiguration:main: Deprecated constructor public java.lang.Integer(java.lang.String) throws java.lang.NumberFormatException in file:///opt/zimbra/mailboxd/etc/jetty-setuid.xml
2022-06-18 23:26:03.128:WARN:oejx.XmlConfiguration:main: Deprecated constructor public java.lang.Boolean(java.lang.String) in file:///opt/zimbra/mailboxd/etc/jetty-setuid.xml
Fatal error: exception while binding to ports
java.lang.NoClassDefFoundError: org/apache/logging/log4j/core/appender/ConsoleAppender$Target
at com.zimbra.common.soap.W3cDomUtil.<clinit>(W3cDomUtil.java:69)
at com.zimbra.common.localconfig.LocalConfig.<init>(LocalConfig.java:174)
at com.zimbra.common.localconfig.LocalConfig.load(LocalConfig.java:387)
at com.zimbra.common.localconfig.LocalConfig.<clinit>(LocalConfig.java:392)
at com.zimbra.common.localconfig.LC.get(LC.java:1539)
at com.zimbra.common.localconfig.KnownKey.booleanValue(KnownKey.java:158)
at com.zimbra.common.handlers.PrivilegedHandler.openPorts(PrivilegedHandler.java:70)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at org.eclipse.jetty.xml.XmlConfiguration$JettyXmlConfiguration.invokeMethod(XmlConfiguration.java:765)
at org.eclipse.jetty.xml.XmlConfiguration$JettyXmlConfiguration.invokeMethod(XmlConfiguration.java:760)
at org.eclipse.jetty.xml.XmlConfiguration$JettyXmlConfiguration.call(XmlConfiguration.java:979)
at org.eclipse.jetty.xml.XmlConfiguration$JettyXmlConfiguration.call(XmlConfiguration.java:942)
at org.eclipse.jetty.xml.XmlConfiguration$JettyXmlConfiguration.configure(XmlConfiguration.java:515)
at org.eclipse.jetty.xml.XmlConfiguration$JettyXmlConfiguration.configure(XmlConfiguration.java:468)
at org.eclipse.jetty.xml.XmlConfiguration.configure(XmlConfiguration.java:380)
at org.eclipse.jetty.xml.XmlConfiguration.lambda$main$3(XmlConfiguration.java:1893)
at java.base/java.security.AccessController.doPrivileged(AccessController.java:569)
at org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1857)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at org.eclipse.jetty.start.Main.invokeMain(Main.java:218)
at org.eclipse.jetty.start.Main.start(Main.java:491)
at org.eclipse.jetty.start.Main.main(Main.java:77)
Caused by: java.lang.ClassNotFoundException: org.apache.logging.log4j.core.appender.ConsoleAppender$Target
at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:445)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:587)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:520)
... 28 more
gonna try to burn down the server and reinstall some of the packages.
i had to go the very long route - cannot say if i had the wrong approach or what was going on.
I did the long postponed/doubted upgrade to zextras OSS v9, with proxy disabled (just to be sure), then did the downgrade of the packages via
Code: Select all
yum downgrade zimbra-core-components-3.0.11-1zimbra8.8b1.el8.x86_64 zimbra-jetty-distribution-9.4.18.v20190429-2.r8.x86_64 zimbra-ldap-components-2.0.5-1zimbra8.8b1.el8.x86_64 zimbra-openjdk-13.0.1-1zimbra8.8b1.el8.x86_64
For that command to work well I also had to disable the other repos in /etc/yum.repos.d/zimbra.repo
Code: Select all
[zimbra]
name=Zimbra RPM Repository
baseurl=https://repo.zimbra.com/rpm/87/rhel8
gpgcheck=1
enabled=0
[zimbra-8815-oss]
name=Zimbra New RPM Repository
baseurl=https://repo.zimbra.com/rpm/8815/rhel8
gpgcheck=1
enabled=0
[zimbra-90-oss]
name=Zimbra New RPM Repository
baseurl=https://repo.zimbra.com/rpm/90/rhel8
gpgcheck=1
enabled=1
My zmmproxy is now half-broken / half-enabled / not installed, but that's really a harmless issue compared to the imaginary port listener of before.
Running version now:
Code: Select all
Release 9.0.0_ZEXTRAS_20220402.RHEL8_64_20220419075929 RHEL8_64 FOSS edition.
I also switched from CentOS8 (the system had an issue with -STREAM-) to Oracle Linux 8 using centos2oel.
My lessons learned are:
- should not have missed this system had fallen out of centos-stream and its repos were dead
- should have switched to OEL right after the centos stream announcement
- a preferred multi-server setup would limit cross-service impact
- should have noticed zmbackup had been dead due to an ip change. point things at 127.0.0.1 unless you have a reason
- it was a massive mistake that this system also didn't have yum-transactions plugin - at least it had etckeeper running
- it does not really appear as if zextras has sufficient build infra / turnaround to deliver better builds than the upstream
if i had a wish, or someone cares about advice:
- it would help iif the header comments in the jetty xml files were a bit more up to date
- disabling/enabling the proxy and switching the ports could use better encapsulation