8.7.11 -> 8.8.5 upgrade problem

Ask questions about your setup or get help installing ZCS server (ZD section below).
siliconalchemy
Posts: 11
Joined: Wed Jan 03, 2018 8:39 am

8.7.11 -> 8.8.5 upgrade problem

Post by siliconalchemy »

Hi, I've been trying to upgrade 8.7.11 to 8.8 for a couple of months now, always with the same 503 proxy errors. Looking in zmmailboxd.out, I see these errors:

Code: Select all

2018-01-03 08:19:38.104:WARN:oejs.SecurityHandler:main: ServletContext@o.e.j.w.WebAppContext@65466a6a{/service,file:///opt/zimbra/jetty_base/webapps/service/,STARTING}{/serv
ice} has uncovered http methods for path: /
Zimbra server process is running as uid=675 euid=675 gid=674 egid=674
2018-01-03 08:19:39.782:WARN:/service:main: unavailable
javax.servlet.UnavailableException: Implementing class
	at com.zimbra.soap.SoapServlet.init(SoapServlet.java:131)
	at javax.servlet.GenericServlet.init(GenericServlet.java:244)
	at org.eclipse.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:626)
	at org.eclipse.jetty.servlet.ServletHolder.initialize(ServletHolder.java:405)
	at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:875)
	at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:346)
	at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1380)
	at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1342)
	at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:772)
	at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:259)
	at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:518)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
	at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
	at org.eclipse.jetty.server.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:161)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
	at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
	at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
	at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
	at org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:212)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
	at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
	at org.eclipse.jetty.server.handler.DebugHandler.doStart(DebugHandler.java:143)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
	at org.eclipse.jetty.server.Server.start(Server.java:405)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
	at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
	at org.eclipse.jetty.server.Server.doStart(Server.java:372)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
	at org.eclipse.jetty.xml.XmlConfiguration$1.run(XmlConfiguration.java:1510)
	at java.security.AccessController.doPrivileged(Native Method)
	at org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1435)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.eclipse.jetty.start.Main.invokeMain(Main.java:214)
	at org.eclipse.jetty.start.Main.start(Main.java:457)
	at org.eclipse.jetty.start.Main.main(Main.java:75)
I've tried setting log4j to DEBUG for more info but nothing more comes up. Help please :)
siliconalchemy
Posts: 11
Joined: Wed Jan 03, 2018 8:39 am

Re: 8.7.11 -> 8.8.5 upgrade problem

Post by siliconalchemy »

Don't know why this was moved to the administrator forum, it's an issue with an upgrade..
siliconalchemy
Posts: 11
Joined: Wed Jan 03, 2018 8:39 am

Re: 8.7.11 -> 8.8.5 upgrade problem

Post by siliconalchemy »

I'm surprised by the lack of interest from Zimbra in what appears to be quite a few 8.8 upgrade issues. It's clear that there's been a lot of changes around the (now mandatory) proxy component and the upgrade process is not yet complete. I would consider 8.8.5 (still) a beta until these are better ironed out. I'm in the process of installing a new instance of 8.8 and migrating old data over using zextras migration tool - thank goodness for the excellent 3rd party software, as such a basic tool is still lacking from zimbra after all these years.
phoenix
Ambassador
Ambassador
Posts: 27272
Joined: Fri Sep 12, 2014 9:56 pm
Location: Liverpool, England

Re: 8.7.11 -> 8.8.5 upgrade problem

Post by phoenix »

siliconalchemy wrote:Don't know why this was moved to the administrator forum, it's an issue with an upgrade..
Sorry about that it was my fault, I moved a bunch of other posts to that forum and I must have included yours at the same time.
siliconalchemy wrote:I'm surprised by the lack of interest from Zimbra in what appears to be quite a few 8.8 upgrade issues. It's clear that there's been a lot of changes around the (now mandatory) proxy component and the upgrade process is not yet complete. I would consider 8.8.5 (still) a beta until these are better ironed out. I'm in the process of installing a new instance of 8.8 and migrating old data over using zextras migration tool - thank goodness for the excellent 3rd party software, as such a basic tool is still lacking from zimbra after all these years.
This comment isn't going to be of much help I'm afraid but I've been complaining long and hard about the lack of interaction from Zimbra (the company) on these forums and I wouldn't expect anything to change any time soon. The lack of interaction of the company with the Community Forums just shows their disinterest in anything that happens here. Comments and questions remain unanswered, serious and long term bugs get no attention in the forums or on bugzilla and after many years on these forums I'm beginning to think it's a waste of time posting anything here. Occasionally we get a post from a staff member saying how much they support the forums but their lack of interaction with an important and large customer base tells me otherwise.
Regards

Bill

Rspamd: A high performance spamassassin replacement

Per ardua ad astra
User avatar
jorgedlcruz
Zimbra Alumni
Zimbra Alumni
Posts: 2782
Joined: Thu May 22, 2014 4:47 pm

Re: 8.7.11 -> 8.8.5 upgrade problem

Post by jorgedlcruz »

Hello,
When do you see this message? Once the process finishes completely? Or during the upgrade process? I have upgraded without any problem from ZCS 8.7.11 to 8.8.5 multiple times during the last couple of weeks, and I have hit this issue, which doesn't mean anything of course :)

It seems you are going to perform a migration finally from one server to another using ZX, do you have OSE or NE? Also which OS do you have? So I can try to reproduce it?

Best regards
Jorge de la Cruz https://jorgedelacruz.es
Systems Engineer at Veeam Software https://www.veeam.com/
siliconalchemy
Posts: 11
Joined: Wed Jan 03, 2018 8:39 am

Re: 8.7.11 -> 8.8.5 upgrade problem

Post by siliconalchemy »

This comment isn't going to be of much help I'm afraid but I've been complaining long and hard about the lack of interaction from Zimbra (the company) on these forums and I wouldn't expect anything to change any time soon. The lack of interaction of the company with the Community Forums just shows their disinterest in anything that happens here. Comments and questions remain unanswered, serious and long term bugs get no attention in the forums or on bugzilla and after many years on these forums I'm beginning to think it's a waste of time posting anything here. Occasionally we get a post from a staff member saying how much they support the forums but their lack of interaction with an important and large customer base tells me otherwise.
@phoenix Sorry to hear that. The lack of interest in early adopters problems with 8.8 is concerning - most professional engineering teams would find this invaluable testing and be all over it, especially as this is the fifth point release and supposedly a GA. The handling of 8.8 release has been rather amateur, and the glacial farce that is universal ui is not boding well for the future of Zimbra.
When do you see this message? Once the process finishes completely? Or during the upgrade process? I have upgraded without any problem from ZCS 8.7.11 to 8.8.5 multiple times during the last couple of weeks, and I have hit this issue, which doesn't mean anything of course :)
It seems you are going to perform a migration finally from one server to another using ZX, do you have OSE or NE? Also which OS do you have? So I can try to reproduce it?
@jorgedlcruz - After the upgrade process is completed - which goes without any apparent problem - the services all restart. I get the dreaded network service error from the web ui when trying to login, and 503 errors in the nginx proxy, so nothing works including zmprov. Eventually found the zmmailboxd log which shows that the main /service endpoint is not starting and is probably the root of all problems. CentOS 7, running OSE and zextras (although the notes say this is removed during the upgrade process?). I've searched long and hard for related issues and worked through a lot of the suggestions with no luck. A lot of people seem to have just given up and restored a previous version from backup. It's quite clear the proxy component needs more work.
User avatar
jorgedlcruz
Zimbra Alumni
Zimbra Alumni
Posts: 2782
Joined: Thu May 22, 2014 4:47 pm

Re: 8.7.11 -> 8.8.5 upgrade problem

Post by jorgedlcruz »

Hello siliconalchemy,
Might be that you are hitting the bug https://bugzilla.zimbra.com/show_bug.cgi?id=108299 ?

What happens if you try to open the web client using the port https://IP:8443? Which is jetty, also, are you trying a valid virtual host? If not, please add it to the Admin console or by CLI as mentioned in the bug.

Best regards
Jorge de la Cruz https://jorgedelacruz.es
Systems Engineer at Veeam Software https://www.veeam.com/
siliconalchemy
Posts: 11
Joined: Wed Jan 03, 2018 8:39 am

Re: 8.7.11 -> 8.8.5 upgrade problem

Post by siliconalchemy »

jorgedlcruz wrote:Hello siliconalchemy,
Might be that you are hitting the bug https://bugzilla.zimbra.com/show_bug.cgi?id=108299 ?

What happens if you try to open the web client using the port https://IP:8443? Which is jetty, also, are you trying a valid virtual host? If not, please add it to the Admin console or by CLI as mentioned in the bug.

Best regards
Hi @jorgedlcruz

No, I've been through that already and added zimbraVirtualHostname for any potential fqdn. As others have pointed out, this is poor new behaviour of the proxy component - it is very common to use multiple fqdns/identities. I've also been through changing the proxy lookup handlers.

If you look at the log output, I think something much more fundamental is wrong - the jetty /service endpoint is failing to start up correctly. But I don't know how to debug this as there is no failure reason that I can find in the logs, despite making changes to log4j properties.
User avatar
msquadrat
Advanced member
Advanced member
Posts: 183
Joined: Mon Oct 14, 2013 10:09 am

Re: 8.7.11 -> 8.8.5 upgrade problem

Post by msquadrat »

If you look at the log output, I think something much more fundamental is wrong - the jetty /service endpoint is failing to start up correctly. But I don't know how to debug this as there is no failure reason that I can find in the logs, despite making changes to log4j properties.
I think I've seen a similar issue while hacking on Zimbra but can't remember the exact root cause. There should be some more helpful information in mailbox.log though. There should first be an INFO log message "Servlet X starting up" (I'd like to know the X) later followed by a FATAL error with the message "Unable to start servlet" and hopefully a stacktrace.
siliconalchemy
Posts: 11
Joined: Wed Jan 03, 2018 8:39 am

Re: 8.7.11 -> 8.8.5 upgrade problem

Post by siliconalchemy »

I think I've seen a similar issue while hacking on Zimbra but can't remember the exact root cause. There should be some more helpful information in mailbox.log though. There should first be an INFO log message "Servlet X starting up" (I'd like to know the X) later followed by a FATAL error with the message "Unable to start servlet" and hopefully a stacktrace.
There wasn't anything like that, but while I was looking in mailbox.log at the startup I noticed reference to two versions of the zm-ssdb-ephemeral-store extensions - 8.7.10 and 8.8.6. Looking in /opt/zimbra/lib/ext/com_zimbra_ssdb_ephemeral_store there was an old jar file lying around:
/opt/zimbra/lib/ext/com_zimbra_ssdb_ephemeral_store]# ls -l
total 720

Code: Select all

-r--r--r--. 1 root root 111969 Nov 30 10:29 commons-pool2-2.4.2.jar
-r--r--r--. 1 root root 553762 Nov 30 10:29 jedis-2.9.0.jar
-r--r--r--. 1 root root  25166 Jan 18 15:49 zm-ssdb-ephemeral-store-8.7.10.1494922903.jar
-r--r--r--. 1 root root  32924 Nov 30 10:29 zm-ssdb-ephemeral-store-8.8.6.1510867717.jar
Removing this old jar file now allows zimbra to start, hurrah! This is presumably a bug in the installer.
Post Reply