Upgrade attempt from NE 6.0.10 to NE 7.0.0 hung

goo0h · Post by **goo0h** » Wed Feb 09, 2011 9:23 pm

On a test system, attempted to upgrade it from 6.0.10 to 7.0.0, and it seems to have hung. It was all progressing fine....
[QUOTE]

Restoring existing configuration file from /opt/zimbra/.saveconfig/localconfig.xml...done

Installing /opt/zimbra/conf/ZCSLicense.xml

Operations logged to /tmp/zmsetup.02092011-181519.log

Upgrading from 6.0.10_GA_2692 to 7.0.0_GA_3077

Stopping zimbra services...done.

Verifying /opt/zimbra/conf/my.cnf

Starting mysql...done.

This appears to be 6.0.10_GA

Checking ldap status...not running.

Running zmldapapplyldif...done.

Checking ldap status...already running.

Running mysql_upgrade...done.

Redolog version update required.

Wed Feb 9 18:16:14 2011: Verified redolog version 1.28.

Wed Feb 9 18:16:14 2011: Updating Redolog schema version from 1.28 to 1.30.

Redolog version update finished.

Schema upgrade required from version 64 to 65.

Running /opt/zimbra/libexec/scripts/migrate20100926-Dumpster.pl

Wed Feb 9 18:16:20 2011: Verified schema version 64.

Wed Feb 9 18:16:34 2011: Verified schema version 64.

Wed Feb 9 18:16:34 2011: Updating DB schema version from 64 to 65.

Stopping mysql...done.

Updating from 6.0.11_GA

Stopping ldap...done.

[/QUOTE]
but then it hung at this point, and there has been no further apparent activity for hours. This is on SUSE 10, 64-bit. Upgrades have been fine in the past.
[QUOTE]

> zmcontrol status

Unable to determine enabled services from ldap.

Enabled services read from cache. Service list may be inaccurate.

Host tzem1.x.org

antispam Stopped

zmamavisdctl is not running

antivirus Stopped

zmamavisdctl is not running

zmclamdctl is not running

convertd Stopped

convertd is not running.

imapproxy Stopped

zmnginxctl is not running

ldap Stopped

logger Stopped

zmlogswatchctl is not running

mailbox Stopped

mysql.server is not running.

zmmailboxdctl is not running.

memcached Stopped

memcached is not running.

mta Stopped

postfix is not running

zmsaslauthdctl is not running

snmp Stopped

zmswatch is not running.

spell Stopped

zmapachectl is not running

stats Stopped

zmconfigd Stopped

zmconfigd is not running.

[/QUOTE]
Fortunately this is only a test system, but would be nice to know how to recover from this. I haven't hit ^C yet, but figure I'll have to eventually.

goo0h · Post by **goo0h** » Thu Feb 10, 2011 12:23 am

I didn't see anything in the logs, at least not anything obvious. I checked to see what was running, and got:
[QUOTE]

zimbra@tzem1:~/log> ps -ft pts/1

UID PID PPID C STIME TTY TIME CMD

root 6796 18187 0 18:15 pts/1 00:00:00 /usr/bin/perl /opt/zimbra/libexec/zmsetup.pl

root 11026 6796 0 18:17 pts/1 00:00:00 sh -c su - zimbra -c "/opt/zimbra/openldap/sbin/slapindex -b '' -q -F /opt/zimbra/data/ldap/co

zimbra 11027 11026 0 18:17 pts/1 00:00:00 su - zimbra -c /opt/zimbra/openldap/sbin/slapindex -b '' -q -F /opt/zimbra/data/ldap/config zim

zimbra 11028 11027 0 18:17 pts/1 00:00:00 /opt/zimbra/openldap/sbin/slapindex -b -q -F /opt/zimbra/data/ldap/config zimbraMailHost

root 18187 20874 0 18:06 pts/1 00:00:00 /bin/bash ./install.sh -x -l ../ZCSLicense.xml

amos 20822 20815 0 Feb08 pts/1 00:00:00 -bash

root 20866 20822 0 Feb08 pts/1 00:00:00 su -

root 20874 20866 0 Feb08 pts/1 00:00:00 -bash

[/QUOTE]
so I did an strace to see what that process was doing:
[QUOTE]

zimbra@tzem1:~> strace -p 11028

Process 11028 attached - interrupt to quit

futex(0x6a8e04, FUTEX_WAIT, 1, NULL

[/QUOTE]
since it didn't appear that it would ever finish doing what it was waiting for, i killed it
[QUOTE]

zimbra@tzem1:~> kill 11028

[/QUOTE]
the install then proceeded....
[QUOTE]

Checking ldap status...not running.

Running zmldapapplyldif...done.

Checking ldap status...already running.

Updating from 7.0.0_BETA1

Updating from 7.0.0_BETA2

Updating from 7.0.0_BETA3

Updating from 7.0.0_RC1

Updating from 7.0.0_GA

Updating global config and COS's with attributes introduced after 6.0.10_GA...done.

Stopping ldap...done.

Upgrade complete.
Running bdb db_recover...done.
.....

[/QUOTE]
Upgrade appears to have finished, and I'm able to login to this test system. Will have to poke around more later to see that everything is working OK.

y@w · Post by **y@w** » Thu Feb 10, 2011 5:57 pm

I did the upgrade last night and had the same thing happen to me (blogged here)
Mine also seemed fine after the upgrade.
After the upgrade was all finished, I did a restore to snapshot on the VM to try it again, but it didn't happen the second time around. Since I couldn't reproduce I didn't file a bug. Perhaps this needs a second look.

y@w · Post by **y@w** » Thu Feb 10, 2011 5:59 pm

Should add - mine was Ubuntu 10.04 x64 from 6.0.9 -> 7.0.

Labsy · Post by **Labsy** » Mon Feb 14, 2011 2:35 am

Hi,

similar problem here with 7.0.0GA_3077 on Ubuntu 8.04 64-bit. Mine box seems to run fine after manually running the ERROR-ed line from /opt/zimbra/log/zmconfigd.log (postfix config line with a lot of config settings), but despite of that, I get this repeated in /opt/zimbra/log/zmconfigd.log every 1 minute:

Starting zmconfigd...zmconfigd is already running.

2011-02-14 09:24:28,861 zmconfigd WARNING [8394-MainThread] Fetching All configs

2011-02-14 09:24:29,107 zmconfigd WARNING [8394-MainThread] All configs fetched in 0.24 seconds

2011-02-14 09:24:29,342 zmconfigd WARNING [8394-MainThread] All rewrite threads completed in 0.01 sec

2011-02-14 09:24:29,342 zmconfigd WARNING [8394-MainThread] All restarts completed in 0.00 sec

Starting zmconfigd...zmconfigd is already running.

2011-02-14 09:25:29,354 zmconfigd WARNING [8394-MainThread] Fetching All configs

2011-02-14 09:25:29,585 zmconfigd WARNING [8394-MainThread] All configs fetched in 0.23 seconds

2011-02-14 09:25:29,786 zmconfigd WARNING [8394-MainThread] All rewrite threads completed in 0.00 sec

2011-02-14 09:25:29,786 zmconfigd WARNING [8394-MainThread] All restarts completed in 0.00 sec

Starting zmconfigd...zmconfigd is already running.

2011-02-14 09:26:29,798 zmconfigd WARNING [8394-MainThread] Fetching All configs

2011-02-14 09:26:30,132 zmconfigd WARNING [8394-MainThread] All configs fetched in 0.34 seconds

2011-02-14 09:26:30,338 zmconfigd WARNING [8394-MainThread] All rewrite threads completed in 0.00 sec

2011-02-14 09:26:30,339 zmconfigd WARNING [8394-MainThread] All restarts completed in 0.00 sec

phoenix · Post by **phoenix** » Mon Feb 14, 2011 9:17 am

[quote user="Labsy"]Hi,

similar problem here with 7.0.0GA_3077 on Ubuntu 8.04 64-bit. Mine box seems to run fine after manually running the ERROR-ed line from /opt/zimbra/log/zmconfigd.log (postfix config line with a lot of config settings), but despite of that, I get this repeated in /opt/zimbra/log/zmconfigd.log every 1 minute:[/QUOTE]Why have you posted this again when I've already answered your other post that these messages are normal and just warnings? What does it have to do with the content of this thread?

liverpoolfcfan · Post by **liverpoolfcfan** » Mon Feb 14, 2011 3:15 pm

I had the same issue with an upgrade of my test box from 7.0 Beta to 7.0 Release on CentOS_64 FOSS.
It hung at the
Stopping ldap...done.
and when I eventually hit Ctrl-C the install proceeded and finished.
After it completed, I re-ran the upgrade install again in-place, and this time it went all the way through without stopping.
Definitely seems to be a pattern here for upgrades.

Rich Graves · Post by **Rich Graves** » Wed Mar 02, 2011 10:15 pm

The potentially long delay for upgrades to 7.0 and 6.0.11 seems to be caused by Bug 54901 - Add index for LDAP zimbraMailHost attribute.
Why that indexing takes forever, I'm not sure, but the index certainly isn't going to be very efficient at sites with only one zimbraMailHost.

Rich Graves · Post by **Rich Graves** » Thu Mar 03, 2011 1:16 pm

Bug 57491 - slapindex of zimbraMailHost might hang during install