Replying to my own thread to provide learnings to others ...
It appears as though this particular issue might have been caused by a temp table not being cleared quickly enough in mysql on my test build. I re-started the whole upgrade process a couple of times and each time ran into this.
Finally, I decided to see what would happen if I just re-ran the zcs-10... installer a second time without reverting to zimbra 8. The second time the installer went through the basic re-install again, and then recognised it had already updated the schema to 112, and continued from there. The temp table was obviously cleared already so that didn't cause a problem this time. The installation completed up to the point where it could not re-install my ESDCA commercial certificate due to the bug in zmcertmgr that does not handle non-RSA certs. This is a known issue and I hope will be fixed by the time 10.0.6 is released. See this thread for details
viewtopic.php?t=72284
I applied the patch to zmcertmgr and re-deployed my commercial certificate. All went well.
However, upon starting the services, mailbox and the related webapps failed to start.
Looking in /opt/zimbra/log/zmmailboxd.out I found the problem
Code: Select all
2023-10-25 17:16:29.432:INFO::main: Logging initialized @5114ms to org.eclipse.jetty.util.log.StdErrLog
2023-10-25 17:16:30.419:WARN:oejx.XmlParser:main: FATAL@null line:989 col:13 : org.xml.sax.SAXParseException; lineNumber: 989; columnNumber: 13; The string "--" is not permitted within comments.
2023-10-25 17:16:30.420:WARN:oejx.XmlConfiguration:main:
java.security.PrivilegedActionException: org.xml.sax.SAXParseException; lineNumber: 989; columnNumber: 13; The string "--" is not permitted within comments.
at java.base/java.security.AccessController.doPrivileged(AccessController.java:573)
at org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1857)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at org.eclipse.jetty.start.Main.invokeMain(Main.java:218)
at org.eclipse.jetty.start.Main.start(Main.java:491)
at org.eclipse.jetty.start.Main.main(Main.java:77)
Caused by:
org.xml.sax.SAXParseException; lineNumber: 989; columnNumber: 13; The string "--" is not permitted within comments.
It doesn't specifically call out the offending XML file but I started with what I thought would be the obvious one - /opt/zimbra/jetty_base/etc/jetty.xml
Sure enough jetty.xml has "--" in line 989 at column 13.
Line 989
Code: Select all
<!-- Modern UI uses build time compression -->
The problem arises because the comment marks around this line are contained in a larger comment block - commenting out HTTPCOMPRESSION
Code: Select all
<!-- HTTPCOMPRESSIONBEGIN
<Get id="next" name="handler" />
<Set name="handler">
<New id="GzipHandler" class="org.eclipse.jetty.server.handler.gzip.GzipHandler">
<Set name="handler"><Ref refid="next" /></Set>
<Set name="minGzipSize"><Property name="jetty.gzip.minGzipSize" deprecated="gzip.minGzipSize" default="2048"/></Set>
<Set name="checkGzExists"><Property name="jetty.gzip.checkGzExists" deprecated="gzip.checkGzExists" default="false"/></Set>
<Set name="compressionLevel"><Property name="jetty.gzip.compressionLevel" deprecated="gzip.compressionLevel" default="-1"/></Set>
<Set name="excludedAgentPatterns">
<Array type="String">
<Item><Property name="jetty.gzip.excludedUserAgent" deprecated="gzip.excludedUserAgent" default=".*MSIE.6\.0.*"/></Item>
</Array>
</Set>
<Set name="includedMethods">
<Array type="String">
<Item>GET</Item>
<Item>POST</Item>
</Array>
</Set>
<!-- Modern UI uses build time compression -->
<Set name="excludedPaths">
<Array type="String">
<Item>/modern/*</Item>
</Array>
</Set>
</New>
</Set>
HTTPCOMPRESSIONEND -->
but the presense of comment marks nested inside a comment block breaks the parser rules. See screenshot of code editor - you can see that the presense of the nested comment breaks the larger comment block.
- ErrantCommentAtLine989ofJettyXML.png (71.17 KiB) Viewed 10653 times
Code: Select all
XML Comments Rules
Following rules should be followed for XML comments −
Comments cannot appear before XML declaration.
Comments may appear anywhere in a document.
Comments must not appear within attribute values.
Comments cannot be nested inside the other comments.
Deleting the nested comment from line 989 and saving the file allowed mailbox and the webapps to start.
It looks like this jetty.xml fix will not survive a restart though - I assume the file is geting exploded from a WAR file periodically. This needs Synacor to fix it.
In the meantime, if anyone can tell me where the XML file comes from it would be appreciated.
UPDATE: Looks like zmconfigd periodically re-writes jetty.xml based on jetty.xml.in in the same directory with local configurations added. Deleting line 976 of jetty.xml.in allows the update to persist (until the next build) Still needs a Synacor fix.
Other than these two small issues (zmcertmgr and jetty.xml comment) the whole process went quite well.
As noted elsewhere in the forum (perhaps Ian's thread) you need at least 4GB of RAM in your build system to build zcs-10... If you try with less the build will randomly stop, and in my experience it is an all or nothing build. It either runs all the way through from start to finish successfully in one go, or it fails in a terminal way. In my environment I could not restart a failed build and have it complete. I had to delete the build folder and start the process again.
