Findings: Upgrade 8.0.7 to 8.7.n (Centos 6.5)

cyber7 · Post by **cyber7** » Mon Jan 23, 2017 12:17 pm

I am starting this topic to assist other users needing to upgrade from 8.0.7 to 8.7.n (As of this writing 8.7.1).

I have asked other users to assist and write their success stories, like in the olden/golden Zimbra days, but no-one have come forward. I am now in the process of upgrading and these are the steps I have had to go through. I will update the topic as I go along.

Preparation - Part one:

My server's license expired, so I had to upgrade this to get my system to 8.7.1 So, my first step was to get my license up to date.
I had an error in my configuration of my Virtual Machine Name for one of my domains and had to get the Virtual Server Name fixed to continue.
NOTE: I got this by running "zmproxyconfgen"

Preparation - Part two:

I had to install "Zimbra Proxy" and "Zimbra Memcache".
To do this I downloaded "zcs-NETWORK-8.0.7_GA_6021.RHEL6_64.20140408123937.tgz" file, extracted it.
As "root" I ran from the extracted directory "./install.sh", just to get an error that the Zimbra Licencing Server could not be reached.
I then ran "./install.sh --skip-activation-check", which allowed me to continue.
NOTE: The funny thing here is that the installation script reported the server was already licensed when it was bringing the server up.
NOTE TO ZIMBRA: Why can you not do this at the start of the installation? Seems like someone forgot to do a check for the license on installation?

The (re)installation took 45 minutes, this was to get Zimbra Proxy installed.

I am now on Zimbra 8.0.7 with Zimbra Proxy enabled. I will let the server run for 10 days prior to upgrading to 8.7.1 and will continue the writing then.

Code: Select all

Host mail.<HOSTNAME>
        antispam                Running
        antivirus               Running
        archiving               Running
        convertd                Running
        ldap                    Running
        logger                  Running
        mailbox                 Running
        mta                     Running
        opendkim                Running
        proxy                   Running
        snmp                    Running
        spell                   Running
        stats                   Running
        zmconfigd               Running

UPGRADE from 8.0.7 to 8.7.1
19:30
Started upgrade. Once again had to run "./install.sh --skip-activation-check" to get past the license check problem, but this happened later.

I got a lot of errors, the first one that memcache was not running. To fix this I had to do:
Code: Select all
```
zmprov ms `zmhostname` +zimbraServiceEnabled memcached
zmcontrol restart
```

I then got a lot of errors regarding the proxy server and ngix (The services did not want to come up). I then did the following:

Code: Select all

zmprov ms `zmhostname` zimbraReverseProxyMailEnabled TRUE zimbraReverseProxyHttpEnabled TRUE
/opt/zimbra/libexec/zmproxyconfig -e -w -H `zmhostname`
/opt/zimbra/libexec/zmproxyconfig -e -m -H `zmhostname`
./libexec/zmproxyconfig -e -w -o -a 8080:80:8443:443 -x https  -H `zmhostname`
./libexec/zmproxyconfig -e -m -o -i 7143:143:7993:993 -p 7110:110:7995:995 -H `zmhostname`

This fixed the errors I got with proxy and restarting "zmcontrol restart" showed everything was up and running.

I restarted the installation and then ran into some other problems:
./install.sh --skip-activation-check

Operations logged to /tmp/install.log.OfsNN1tN
Checking for existing installation...
zimbra-ldap...FOUND zimbra-ldap-8.0.7_GA_6021
zimbra-logger...FOUND zimbra-logger-8.0.7_GA_6021
zimbra-mta...FOUND zimbra-mta-8.0.7_GA_6021
zimbra-dnscache...NOT FOUND
zimbra-snmp...FOUND zimbra-snmp-8.0.7_GA_6021
zimbra-store...FOUND zimbra-store-8.0.7_GA_6021
zimbra-apache...FOUND zimbra-apache-8.0.7_GA_6021
zimbra-spell...FOUND zimbra-spell-8.0.7_GA_6021
zimbra-convertd...FOUND zimbra-convertd-8.0.7_GA_6021
zimbra-memcached...FOUND zimbra-memcached-8.0.7_GA_6021
zimbra-proxy...FOUND zimbra-proxy-8.0.7_GA_6021
zimbra-archiving...FOUND zimbra-archiving-8.0.7_GA_6021
zimbra-core...FOUND zimbra-core-8.0.7_GA_6021
ZCS upgrade from 8.0.7 to 8.7.1 will be performed.
Checking for existing proxy service in your environment
Checking for existing memcached service in your environment
Validating ldap configuration
Error: Unable to create a successful TLS connection to the ldap masters.
Fix cert configuration prior to upgrading.
Now I KNOW for a fact my SSL CERTS are 100% correct as I renewed them late last year. This is now starting to piss me off! I read that Jorge wrote something about the internal and external FQHN not being the same. Now this is bullshit! I followed a strict setup with this server and BILL (phoenix) still assisted me WAY BACK WHEN.

Read somewhere that I could disable the local ldap tls support and did that with:
Code: Select all
```
zmlocalconfig -e ldap_starttls_supported=0
zmlocalconfig -e ldap_starttls_required=false
zmcontrol restart
```

21:56
Started the installation again. This time I got to the point where I have to check my database.

22:12
Completed the DB check!

22:17
Failed with a PGP KEY check. Realized that I did not open my firewall. Did so and installation continued. As a matter of fact, the installation continued without having to skip the activation check...

23:34
Install script bombed with:

ERROR 1050 (42S01) at line 1: Table 'mboxgroup20/#sql-ib7060-2677863574' already exists
Fri Jan 27 23:35:26 2017: Error while running '/opt/zimbra/bin/mysql --user=zimbra --password=<EDIT> --database=zimbra --batch --skip-column-names'.
Script failed with code 256: - exiting
UPGRADE FAILED - exiting.

I am slowly but surely starting to get to the point to toss this server for a Microsoft server. The fault here is the Zimbra database is corrupt, although the script (Right at the start of this writing) checked the database and did not find any errors.
NOTE: WTF ZIMBRA??? It is your script that is causing all these errors. You should do better testing. This kind of cr@p will cause your company to loose a lot of customers!!!

I then Followed:
viewtopic.php?t=60289
and commented:

Code: Select all

#    Migrate::myquit(1, "Error while running '$command'.")
#      if ($? != 0);

from /opt/zimbra/libexec/scripts/Migrate.pm
restarted and hopefully this will complete.

02:22
Lots of java errors. Mailbox does not want to start!

02:40
Started rolling the system back. Total disaster!

In Conclusion:

Make sure you have an open view to the internet on installation. Do not filter any firewall traffic while installing.
Do not (like me) leave your installation for too late. I had the opportunity to upgrade through the entire 8.1/2/3/4/5/6/7 releases, but chose to stay on 8.0.7. The release was stable and I did not want to spend money on licenses.
Take your time and read. If you get stuck with an error, read other people's posts regarding and related to your error. Someone would have posted something somewhere about your problem.
Use the forum. I can not stress this more. The forums is a great source of knowledge.

dbayer · Post by **dbayer** » Tue Jan 24, 2017 11:16 pm

Hi Cyber7,

Thanks for sharing your efforts at upgrading. I would like to do the same.

I'm in the process of researching my upgrade from 8.6 to 8.7.1 on Ubuntu 16.04. I've been advised that Zextras Migration tool will make this much easier. So I've been reading their documentation as well. Their advice is to do a fresh install on the new server, but use the config.1234 file that is generated from the old 8.6 server install.sh script. This should get the server close to the original server settings. The one problem I have, is that I know the server has changed some since then. Do you know a way to generate a new config.1234 file?

Thanks,
Daniel

cyber7 · Post by **cyber7** » Wed Jan 25, 2017 7:38 am

dbayer wrote:Hi Cyber7,
<snip>Their advice is to do a fresh install on the new server, but use the config.1<end snip>
Thanks,
Daniel

Hi Daniel
I am sorry to say, but this is never viable for anyone in my position. I have a single server (VM) with 14TB of storage, so no, this is not an option. I have done this before and dread the day I have to roll-over to a newer OS as the problems and hassles of transferring this amount of data is just not an easy task. DAUNTING!

timgman · Post by **timgman** » Wed Jan 25, 2017 3:42 pm

About to do this same upgrade on a Centos 6.8 zimbra 8.07.x to 8.71. I'll keep you'all posted.. It should be in the next few days...
I have about 1.4 Tb of data so I hope it moves fairly quickly?
Any Idea how long this UG might take?

cyber7 · Post by **cyber7** » Thu Jan 26, 2017 6:33 am

timgman wrote:About to do this same upgrade on a Centos 6.8 zimbra 8.07.x to 8.71. I'll keep you'all posted.. It should be in the next few days...
I have about 1.4 Tb of data so I hope it moves fairly quickly?
Any Idea how long this UG might take?

As I had to do an "upgrade" from 8.0.7 to 8.0.7 to get the "Proxy" and "Memcache" options added, I take it for granted it should take the same time to upgrade from 8.0.7 to 8.7.1. In my case, 45 minutes...

timgman · Post by **timgman** » Thu Jan 26, 2017 4:31 pm

Cyber,
Did you do the upgrade already?

my system ihas 2 Intel(R) Xeon(R) CPU E5-2640 0 @ 2.50GHz cpu's and 80 gigs of memory..
I guess I'll report back as to how long it takes to do the UG...

I'll install proxy and memcache first.. Thanks

cyber7 · Post by **cyber7** » Fri Jan 27, 2017 6:12 am

timgman wrote:Cyber,
Did you do the upgrade already?

my system ihas 2 Intel(R) Xeon(R) CPU E5-2640 0 @ 2.50GHz cpu's and 80 gigs of memory..
I guess I'll report back as to how long it takes to do the UG...

I'll install proxy and memcache first.. Thanks

Hi Timgman
Nope, the physical update will take place tonight. I have only done the preparation to get the system ready. I will update post #1 as I continue, so I should have a full report on Monday

timgman · Post by **timgman** » Fri Jan 27, 2017 4:25 pm

OK, I upgraded 8.07 to 8.07 including the proxy and memcache............. all went well..
I Upgraded to 8.7 and it seemed OK... Until I realized that all accounts that were previously renamed reverted to their old original name... and I can't change it...
Ugh.. I'm opening a ticket.

JDunphy · Post by **JDunphy** » Fri Jan 27, 2017 9:22 pm

I am going to share a method from my personal blog on my centos 6+ and any zimbra upgrade process. I have done a variation of this since zimbra verson 6 if memory serves. It's intentionally simple (not too automated) with only a few commands. The technique also provides me with a method should we have a catastrophic loss of datacenter since I can pull from our other backups fairly quickly with this method.

http://silentadmin.gsans.com/everything ... ew-server/

I test my updates on a new machine that I rent for a few hours or days depending on need. Because of ldap, I choose to do tar (about 2-3 seconds with sparse file vs rsync which takes much too long for me). I can do this fairly hot. The method allows me to learn as much about the upgrade problems before touching the production machine. If I am moving to new hardware, I will run the last rsync with zimbra down to verify we don't have any db corruption problems.

There is a order to my process that I start after following a few tricks with /etc/hosts and your resolver mentioned in the blog entry. The process goes basically like this:

# Get the new host to same zimbra version that you will rsync with

Code: Select all

./install.sh -s --skip-activation-check

followed by getting that existing environment over by pushing or pulling it. I generally push it with something like this.

Code: Select all

#!/bin/sh
 
#
# stage a backup to make writing tar faster between upgrades
# of zimbra releases. Just some protection against a castrophic failure
#
 
# create portable ldap file to copy
cd /opt/zimbra/data/ldap/mdb/db
tar cvzSpf data.mdb.tar.gz data.mdb
 
# Backup to X.X.X.X (newhost or temporary testing host)
rsync --progress --delete --archive --verbose --hard-links --exclude=zimbra/data/ldap/mdb/db/data\.mdb --exclude=zimbra/backup /opt/zimbra X.X.X.X:/opt

Repeat this rsync as often as necessary. Next I move to the configuration phase by doing this on the staging machine:

Code: Select all

extract the ldap file from tar image and then proceed to zimbra install which fixes everything.
 ./install.sh --skip-activation-check

Then I change the ip address and work through any issues associated with that. If I need to upgrade I proceed and gain some valuable rehearsal/learning experience. Not a lot of steps but this is the fastest method I have found to migrate zimbra to a new machine so I can experiment with actual user data. I tend to work in 100GB chunks so it pays to use SSD and GigE with your providers which is now really cheap. The one I list in my blog works well enough for me with 200GB of SSD, 12GB ram and 8 cpu's, etc that you can rent by the hour/day/etc . I think I pay about $1.50/day and see my rsync sustained at over 100Mb/s with my production servers as I setup the environment.

Once I have learned what I need, I queue up the commands that I will use in some text files... go to my production machine and basically cut/paste these commands the morning of the upgrade. So lots of rehearsal to gain some operational experiences. Nothing worse than being up at 3:00am and having Zimbra broken and that sick feeling in the pit of your stomach.

That documented process has kept me sane because there have been some really bad upgrades... like moving from 32 bit to 64 bit that took days of practice to get right.

Hope this helps others.

Jim

cyber7 · Post by **cyber7** » Fri Jan 27, 2017 9:34 pm

Hi Jim
Thanx for joining the conversation.

I am in the process of upgrading and am at the last stage of the installation. (Migrating mboxgroupXX) You will see that I have had to get over a couple of hurdles, but your input will be valuable for anyone stuck with the dilemma of having to upgrade.

kind regards
cyber7 (aka Aubrey Kloppers; Cape town; south Africa)

Zimbra Forums

Findings: Upgrade 8.0.7 to 8.7.n (Centos 6.5)

Findings: Upgrade 8.0.7 to 8.7.n (Centos 6.5)

Re: Findings: Upgrade 8.0.7 to 8.7.n (Centos 6.5)

Re: Findings: Upgrade 8.0.7 to 8.7.n (Centos 6.5)

Re: Findings: Upgrade 8.0.7 to 8.7.n (Centos 6.5)

Re: Findings: Upgrade 8.0.7 to 8.7.n (Centos 6.5)

Re: Findings: Upgrade 8.0.7 to 8.7.n (Centos 6.5)

Re: Findings: Upgrade 8.0.7 to 8.7.n (Centos 6.5)

Re: Findings: Upgrade 8.0.7 to 8.7.n (Centos 6.5)

Re: Findings: Upgrade 8.0.7 to 8.7.n (Centos 6.5)

Re: Findings: Upgrade 8.0.7 to 8.7.n (Centos 6.5)