Simultaneous ZCS 8.6 > 8.8.12 & Ubuntu 14.04LTS to 16.04 to 18.04 Upgrade
Posted: Thu Apr 04, 2019 5:53 pm
So let's get it out there- this was really dumb, caused by a miscommunication, and a junior admin not snapping the virtual server prior to attempting anything. Ugh.
So, the do-release-upgrade command was run twice in succession on an Ubuntu 14.04 box, updating it first to 16.04 LTS and then to 18.04 LTS. This broke lots of things, including Zimbra. Here's what we've accomplished so far:
- We moved the /opt/zimbra folder to zimbra.old, and installed the 8.8.12 version of ZCS to /opt/zimbra using the -s switch. (Since 8.6 was not released for Ubuntu 18.04)
- We moved this new installation to /opt/zimbra.new and moved /opt/zimbra.old back to /opt/zimbra.
- We removed the contents of the [/opt/zimbra/] bin common/bin common/lib common/libexec common/sbin common/share lib libexec directories.
- We copied all of the contents of [/opt/zimbra.new] bin common/bin common/lib common/libexec common/sbin common/share lib libexec directories to [/opt/zimbra] respectively.
- We ran and seemingly successfully upgraded the ZCS install using the zmsetup.pl script. It walked through all the steps and patches very nicely.
Amazingly, all of the services started, mail is flowing in and out, and users can login to the web interface, IMAP daemon, and admins can access the admin portal!
Output of zmcontrol -v is: Release 8.8.12.GA.3794.UBUNTU18.64 UBUNTU18_64 FOSS edition.
zmcontrol start looks like this:
HOWEVER, I still have 2 unresolved issues that I know about, and I'm sure countless others that I don't know about. I wondered if the gurus here could help me isolate these issues.
Issue 1:
When attempting to run the 8.8.12 installer to do a 'cleaner' in-place upgrade I receive an error as follows:
If I leave Zimbra running while executing the install.sh above, the install.log file looks like this:
..and if Zimbra is not running, then the install.log looks like this:
So I'm not sure how to diagnose that farther. I'm concerned about it more for future patching and upgrades, not so much to run it now since the server and all services are running.
Issue 2:
When logged in as an admin, if I attempt to access anything related to stats, I receive a pop up box like this:
Then /opt/zimbra/log/mailbox.log shows this:
I've run the zmsyslogsetup command, but it errors out as follows:
I've also tried reinstalling rsyslog, no change. I also tried zmupdateauthkeys.
Any ideas?
So, the do-release-upgrade command was run twice in succession on an Ubuntu 14.04 box, updating it first to 16.04 LTS and then to 18.04 LTS. This broke lots of things, including Zimbra. Here's what we've accomplished so far:
- We moved the /opt/zimbra folder to zimbra.old, and installed the 8.8.12 version of ZCS to /opt/zimbra using the -s switch. (Since 8.6 was not released for Ubuntu 18.04)
- We moved this new installation to /opt/zimbra.new and moved /opt/zimbra.old back to /opt/zimbra.
- We removed the contents of the [/opt/zimbra/] bin common/bin common/lib common/libexec common/sbin common/share lib libexec directories.
- We copied all of the contents of [/opt/zimbra.new] bin common/bin common/lib common/libexec common/sbin common/share lib libexec directories to [/opt/zimbra] respectively.
- We ran and seemingly successfully upgraded the ZCS install using the zmsetup.pl script. It walked through all the steps and patches very nicely.
Amazingly, all of the services started, mail is flowing in and out, and users can login to the web interface, IMAP daemon, and admins can access the admin portal!
Output of zmcontrol -v is: Release 8.8.12.GA.3794.UBUNTU18.64 UBUNTU18_64 FOSS edition.
zmcontrol start looks like this:
Code: Select all
zimbra@servername:~$ zmcontrol start
Host mail.servername.com
Starting zmconfigd...Done.
Starting logger...Done.
Starting mailbox...Done.
Starting memcached...Done.
Starting amavis...Done.
Starting antispam...Done.
Starting antivirus...Done.
Starting opendkim...Done.
Starting snmp...Done.
Starting spell...Done.
Starting mta...Done.
Starting stats...Done.
Starting service webapp...Done.
Starting zimbra webapp...Done.
Starting zimbraAdmin webapp...Done.
Starting zimlet webapp...Done.
zimbra@servername:~$
Issue 1:
When attempting to run the 8.8.12 installer to do a 'cleaner' in-place upgrade I receive an error as follows:
Code: Select all
root@servername:~/zcs-8.8.12_GA_3794.UBUNTU18_64.20190329045002# ./install.sh
Operations logged to /tmp/install.log.jcDtcnRP
Checking for existing installation...
zimbra-drive...FOUND zimbra-drive-1.0.12.1542291479-1.u18
zimbra-imapd...NOT FOUND
zimbra-patch...FOUND zimbra-patch-8.8.12.1552477867.p0-1.u18
zimbra-mta-patch...FOUND zimbra-mta-patch-8.8.12.1552427139.p0-1.u18
zimbra-proxy-patch...FOUND zimbra-proxy-patch-8.8.12.1552427139.p0-1.u18
zimbra-license-tools...NOT FOUND
zimbra-license-extension...NOT FOUND
zimbra-network-store...NOT FOUND
zimbra-network-modules-ng...NOT FOUND
zimbra-chat...FOUND zimbra-chat-2.0.2.1546498111-1.u18
zimbra-talk...NOT FOUND
zimbra-ldap...FOUND zimbra-ldap-8.8.12.GA.3794.UBUNTU18.64
zimbra-logger...FOUND zimbra-logger-8.8.12.GA.3794.UBUNTU18.64
zimbra-mta...FOUND zimbra-mta-8.8.12.GA.3794.UBUNTU18.64
zimbra-dnscache...NOT FOUND
zimbra-snmp...FOUND zimbra-snmp-8.8.12.GA.3794.UBUNTU18.64
zimbra-store...FOUND zimbra-store-8.8.12.GA.3794.UBUNTU18.64
zimbra-apache...FOUND zimbra-apache-8.8.12.GA.3794.UBUNTU18.64
zimbra-spell...FOUND zimbra-spell-8.8.12.GA.3794.UBUNTU18.64
zimbra-convertd...NOT FOUND
zimbra-memcached...FOUND zimbra-memcached-1:1.4.37-2.u18
zimbra-proxy...FOUND zimbra-proxy-8.8.12.GA.3794.UBUNTU18.64
zimbra-archiving...NOT FOUND
zimbra-core...FOUND zimbra-core-8.8.12.GA.3794.UBUNTU18.64
ZCS upgrade from 8.8.12 to 8.8.12 will be performed.
Validating ldap configuration
Unknown Error: It should be impossible to reach this statement.
Code: Select all
COMMAND: ldap start
slapd already running: pid 20115
Code: Select all
COMMAND: ldap start
Started slapd: pid 38209
Issue 2:
When logged in as an admin, if I attempt to access anything related to stats, I receive a pop up box like this:
Then /opt/zimbra/log/mailbox.log shows this:
Code: Select all
2019-04-04 12:40:17,200 WARN [qtp573673894-198:https:https://mail.servername.com:7071/service/admin/soap/GetLoggerStatsRequest] [name=packetdog@servername.com;mid=5;ip=192.168.10.254;port=51235;ua=ZimbraWebClient
- GC73 (Win);soapId=22abf379;] SoapEngine - handler exception
com.zimbra.common.service.ServiceException: system failure: Unable to read logger stats
ExceptionId:qtp573673894-198:https:https://mail.servername.com:7071/service/admin/soap/GetLoggerStatsRequest:1554399617200:d7dbbc4b7271db7b
Code:service.FAILURE
at com.zimbra.common.service.ServiceException.FAILURE(ServiceException.java:288)
at com.zimbra.cs.service.admin.GetLoggerStats.execfetch(GetLoggerStats.java:453)
at com.zimbra.cs.service.admin.GetLoggerStats.fetchColumnData(GetLoggerStats.java:268)
at com.zimbra.cs.service.admin.GetLoggerStats.handle(GetLoggerStats.java:175)
at com.zimbra.soap.SoapEngine.dispatchRequest(SoapEngine.java:646)
at com.zimbra.soap.SoapEngine.dispatch(SoapEngine.java:491)
at com.zimbra.soap.SoapEngine.dispatch(SoapEngine.java:278)
at com.zimbra.soap.SoapServlet.doWork(SoapServlet.java:307)
at com.zimbra.soap.SoapServlet.doPost(SoapServlet.java:216)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at com.zimbra.cs.servlet.ZimbraServlet.service(ZimbraServlet.java:211)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:821)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1685)
at com.zimbra.cs.servlet.CsrfFilter.doFilter(CsrfFilter.java:169)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
at com.zimbra.cs.servlet.RequestStringFilter.doFilter(RequestStringFilter.java:54)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
at com.zimbra.cs.servlet.SetHeaderFilter.doFilter(SetHeaderFilter.java:59)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
at com.zimbra.cs.servlet.ETagHeaderFilter.doFilter(ETagHeaderFilter.java:47)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
at com.zimbra.cs.servlet.ContextPathBasedThreadPoolBalancerFilter.doFilter(ContextPathBasedThreadPoolBalancerFilter.java:107)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
at com.zimbra.cs.servlet.ZimbraQoSFilter.doFilter(ZimbraQoSFilter.java:125)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
at com.zimbra.cs.servlet.ZimbraInvalidLoginFilter.doFilter(ZimbraInvalidLoginFilter.java:117)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
at org.eclipse.jetty.servlets.DoSFilter.doFilterChain(DoSFilter.java:473)
at org.eclipse.jetty.servlets.DoSFilter.doFilter(DoSFilter.java:318)
at org.eclipse.jetty.servlets.DoSFilter.doFilter(DoSFilter.java:288)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1158)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1090)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:109)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:119)
at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:318)
at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:437)
at org.eclipse.jetty.server.handler.DebugHandler.handle(DebugHandler.java:84)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:119)
at org.eclipse.jetty.server.Server.handle(Server.java:517)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:306)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:242)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:261)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:192)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:261)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:75)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:213)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:147)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)
at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:399)
at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:242)
at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:224)
at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:403)
at java.base/java.net.Socket.connect(Socket.java:591)
at java.base/java.net.Socket.connect(Socket.java:540)
at java.base/java.net.Socket.<init>(Socket.java:436)
at java.base/java.net.Socket.<init>(Socket.java:246)
at com.zimbra.cs.service.admin.GetLoggerStats.execfetch(GetLoggerStats.java:394)
... 60 more
2019-04-04 12:40:17,205 INFO [qtp573673894-198:https:https://mail.servername.com:7071/service/admin/soap/GetLoggerStatsRequest] [name=packetdog@servername.com;mid=5;ip=192.168.10.254;port=51235;ua=ZimbraWebClient
- GC73 (Win);soapId=22abf379;] soap - GetLoggerStatsRequest elapsed=6
I've also tried reinstalling rsyslog, no change. I also tried zmupdateauthkeys.
Any ideas?