Request for Advice - OS upgrade/migration

Discuss your pilot or production implementation with other Zimbra admins or our engineers.
DennisT
Posts: 37
Joined: Fri May 19, 2017 11:07 pm

Request for Advice - OS upgrade/migration

Post by DennisT »

Hi,
I have a single server ZCS NE install (8.8.15) on a Ubuntu 14 VM. Zimbra is dropping support for v14 so I need to move to a newer OS version. Here's what I tried so far:
In place OS upgrade to v16 - lost network settings + other stuff. Restored entire system from backup.
Opened a professional services case w/Zimbra for them to do the migration. After one acknowledgement email they went dark.
Tried a migration to a v18 VM per the instructions here:
https://wiki.zimbra.com/wiki/How_to_mov ... 0the%20old
This seemed to work up to when I tried to access the management console or ZWC (neither worked). Opened a support case where they said to use this instead:
https://wiki.zimbra.com/wiki/Steps_To_R ... tion_steps
While I was waiting for that response I made a couple attempts with the first procedure. One failed due to a hash issue with the repository and another with a ldap root problem.

I'm thinking of giving the support recommended migration a try this weekend. Before that though it would be nice to get some feedback here on what path I should take. Eventually I'd like to be on Ubuntu v18 and Zimbra v9.
TIA
7224jobe
Outstanding Member
Outstanding Member
Posts: 283
Joined: Sat Sep 13, 2014 1:55 am
ZCS/ZD Version: 8.8.15_FOSS Patch38

Re: Request for Advice - OS upgrade/migration

Post by 7224jobe »

Hi, since you have Network Edition you could try using NG Backup: https://wiki.zimbra.com/wiki/Zimbra_NG_ ... -migration

It is surprising that neither Zimbra support and Professional services helped you in an useful way -.-
DennisT
Posts: 37
Joined: Fri May 19, 2017 11:07 pm

Re: Request for Advice - OS upgrade/migration

Post by DennisT »

Thanks for posting. I took a look at the page you linked and it appears that won't work for us as we're running HSM.

Yeah, my feelings towards Zimbra paid support has taken quite a hit this year.
Klug
Ambassador
Ambassador
Posts: 2746
Joined: Mon Dec 16, 2013 11:35 am
Location: France - Drôme
ZCS/ZD Version: All of them
Contact:

Re: Request for Advice - OS upgrade/migration

Post by Klug »

My suggestion:

1. install new VM, install same Zimbra version with "./install.sh -s" (will install software only, no data/configuration), choose the same modules than on old server
2. rsync data from old VM to new VM (see command below) while zimbra is running
3. rsync again, while running, just before the move
4. stop zimbra on old server
5. rsync again
6. export LDAP on old server then import on new one, see: https://wiki.zimbra.com/wiki/LDAP_data_import_export
7. unmount HSM/backup volumes/path from old server
8. stop old server
9. change IP and hostname on new server so they're the ones from the old server
10. re-install zimbra's binary (./install.sh -s again) on new server
11 import LDAP dumps on new server
12. mount HSM/backup volumes/paths on new server
13. check ownerhip and permisson on HSM/backup
14. recreate the crontab on new server (https://wiki.zimbra.com/wiki/Step_to_re ... imbra_user)

I've been doing that a lot lately (Centos6 -> Ubuntu 18 or Centos7) and it just works.

Here's the rsync command I use, from old server:

Code: Select all

rsync --progress --delete --archive --verbose --hard-links --exclude=zimbra/data/ldap/mdb/db/data.mdb --exclude zimbra/data/ldap/accesslog/db/data.mdb --exclude zimbra/data/ldap/config/db/data.mdb /opt/zimbra newserver.domain.tld:/opt/
If you're using HSM exclude HSM from rsync (of course), you'll unmount from old VM and mount on new VM

Edited a couple of months later to add step 14.
DennisT
Posts: 37
Joined: Fri May 19, 2017 11:07 pm

Re: Request for Advice - OS upgrade/migration

Post by DennisT »

Thanks. I'll give this a go this coming weekend.
User avatar
JDunphy
Outstanding Member
Outstanding Member
Posts: 889
Joined: Fri Sep 12, 2014 11:18 pm
Location: Victoria, BC
ZCS/ZD Version: 9.0.0_P39 NETWORK Edition

Re: Request for Advice - OS upgrade/migration

Post by JDunphy »

Exactly as Klug has said... I do something very similar after evaluating all the methods we have as Network customers and this is by far the best method IMHO as it still retains any files if you have created special transportfiles, etc. You will still need to apply any patches if you modified any of the config files (ie. amavisd.conf.in, bugs that Zimbra won't fix, etc)... I create a patch script anytime I modify anything of zimbra. This allows me to verify if any of my changes or bug fixes were removed during patch updates, upgrades, etc.

The zimbra user id is created during the first package installation so I do this as part of my migration script to match uid's with the old machine before issuing my first install.sh -s on the new machine. This gives me other options of mounting volumes and saving a step of fixing perms, etc. Another future use of this is that if you have 2 machines and want to use Backup NG's files you won't have to chown/chgrp the millions of files they create during an export for your import when you remount that volume.

Code: Select all

create_zimbra_uid() {
   grep -q zimbra /etc/passwd
   if [ $? -eq 1 ]; then
      groupadd -g $zimbra_gid zimbra
      useradd --system -M -d /opt/zimbra -s /bin/bash -u $zimbra_uid -g $zimbra_gid zimbra
      usermod -a -G tty zimbra
   fi
}
I did an install.sh -s followed by install.sh -u and then looked at /etc/passwd and /etc/group and recreated those steps to understand what was being created by their first package installation. I probably spent too much time trying to understand install.sh and zmsetup.pl to see what methods could be used to speed up the install process. I am quite confident of this rsync approach mentioned having rehearsed this so many times in the last month and verifying the results against the old server.

I spend a little more time up front creating the illusion that the new machine is the old machine.. ie) /etc/hosts with same names but new ip's, authoritative but non delegated zone for zmhostname, and creating the same hostname. I have a single bash function that does all this including installing the packages necessary. You just give it zmhostname and it will do all the rest on your new machine to build this new clone. I use here documents to automate my install.sh's with my keystrokes so I don't install zimbra's resolver in our configs as it is set to N, etc.

With most things, the more you practice this stuff the easier it becomes. The trick is to practice/rehearse on a staging server and gain the confidence and trust before your migration. I go from RHEL6 (Centos6) directly to RHEL8 (Centos8 to oracle 8 (centos2ol.sh) with our migrations. Automate as much as you can as you rehearse to make this reproducible and put less stress on yourself during final cutover.

Another hint: modify /etc/hosts on your client machine so that you can login into the new server to verify it works before final cutover and your last rsysnc. I will also update the ip address on the new machine with zimbra if I need to test sending email out/in. The last step of moving the ldap back will overwrite that change so you don't have to worry about having the new ip's should you be swapping ip's and using the previous machines ip address.

Jim
User avatar
wentum
Advanced member
Advanced member
Posts: 53
Joined: Fri Apr 04, 2014 10:49 am
Location: Pforzheim (Germany)
ZCS/ZD Version: Release 9.0.0.GA.3924 _P30
Contact:

Re: Request for Advice - OS upgrade/migration

Post by wentum »

Klug wrote:My suggestion:

1. install new VM, install same Zimbra version with "./install.sh -s" (will install software only, no data/configuration), choose the same modules than on old server
2. rsync data from old VM to new VM (see command below) while zimbra is running
3. rsync again, while running, just before the move
4. stop zimbra on old server
5. rsync again
6. export LDAP on old server then import on new one, see: https://wiki.zimbra.com/wiki/LDAP_data_import_export
7. unmount HSM/backup volumes/path from old server
8. stop old server
9. change IP and hostname on new server so they're the ones from the old server
10. re-install zimbra's binary (./install.sh -s again) on new server
11 import LDAP dumps on new server
12. mount HSM/backup volumes/paths on new server
13. check ownerhip and permisson on HSM/backup

I've been doing that a lot lately (Centos6 -> Ubuntu 18 or Centos7) and it just works.

Here's the rsync command I use, from old server:

Code: Select all

rsync --progress --delete --archive --verbose --hard-links --exclude=zimbra/data/ldap/mdb/db/data.mdb --exclude zimbra/data/ldap/accesslog/db/data.mdb --exclude zimbra/data/ldap/config/db/data.mdb /opt/zimbra newserver.domain.tld:/opt/
If you're using HSM exclude HSM from rsync (of course), you'll unmount from old VM and mount on new VM
Hello Klug,

this is really a nice little howto and your way to migrate seems best for me, too!
But let me ask one question: why do you re-install again at point 10? Isn't that probematic if there are patches on the source server?

Regards
Joerg
Klug
Ambassador
Ambassador
Posts: 2746
Joined: Mon Dec 16, 2013 11:35 am
Location: France - Drôme
ZCS/ZD Version: All of them
Contact:

Re: Request for Advice - OS upgrade/migration

Post by Klug »

The rsync command will rsync all the binaries/libraries too.
So if you do not reinstall again (step 10), the new server ends up with the binaries of the source server (bad OS version).
I found this method easier than adding lots of exclusion in the rsync command (so the binaries/libraries are not rsynced).

About the patches, you have to have the very same version (base +patch) on both servers.
That means you might have to update the source one before the rsync the the two servers are at the same level.
DennisT
Posts: 37
Joined: Fri May 19, 2017 11:07 pm

Re: Request for Advice - OS upgrade/migration

Post by DennisT »

Klug wrote:My suggestion:

1. install new VM, install same Zimbra version with "./install.sh -s" (will install software only, no data/configuration), choose the same modules than on old server
2. rsync data from old VM to new VM (see command below) while zimbra is running
3. rsync again, while running, just before the move
4. stop zimbra on old server
5. rsync again
6. export LDAP on old server then import on new one, see: https://wiki.zimbra.com/wiki/LDAP_data_import_export
7. unmount HSM/backup volumes/path from old server
8. stop old server
9. change IP and hostname on new server so they're the ones from the old server
10. re-install zimbra's binary (./install.sh -s again) on new server
11 import LDAP dumps on new server
12. mount HSM/backup volumes/paths on new server
13. check ownerhip and permisson on HSM/backup

I've been doing that a lot lately (Centos6 -> Ubuntu 18 or Centos7) and it just works.

Here's the rsync command I use, from old server:

Code: Select all

rsync --progress --delete --archive --verbose --hard-links --exclude=zimbra/data/ldap/mdb/db/data.mdb --exclude zimbra/data/ldap/accesslog/db/data.mdb --exclude zimbra/data/ldap/config/db/data.mdb /opt/zimbra newserver.domain.tld:/opt/
If you're using HSM exclude HSM from rsync (of course), you'll unmount from old VM and mount on new VM
Thanks. That got me there. I had problems with rsync - it would partially sync and then hang. I ended up running this instead:

rsync --progress --rsync-path="sudo rsync" --delete --archive --verbose --hard-links --exclude=zimbra/data/ldap/mdb/db/data.mdb --exclude zimbra/data/ldap/accesslog/db/data.mdb --exclude zimbra/data/ldap/config/db/data.mdb /opt/zimbra root@newserver:/opt/

Before that would work I had to modify /etc/ssh/sshd_config by adding PermitRootLogin yes

After that the rsync was fast and I stopped getting permission errors. After the migration I removed the line from sshd_config.

The NG modules weren't appearing on the admin console. I ended up doing the following:
apt update
apt upgrade

After that the NG modules were visible in the admin console.

Note I didn't do step 11 as it was already done in step 6.
Klug
Ambassador
Ambassador
Posts: 2746
Joined: Mon Dec 16, 2013 11:35 am
Location: France - Drôme
ZCS/ZD Version: All of them
Contact:

Re: Request for Advice - OS upgrade/migration

Post by Klug »

Sorry about the root issue with rsync, I forgot about it.
You're right, rsync must be run a root (permission and rights) so you have to allow root login (or play with sudo).

You were able to import LDAP dumps on step 6 but it might not work depending on your source and destination server.
When upgrading from 32 to 64 bits (that's old), you have to use the 64 bits tools to import on the destination server, so you need to do step 11 first.
It might also not work (I haven't tried) when upgrading with an OS change (not just a version change), that's why there's the step 11.

NG tools not showing is a running gag (quite an old bug, still not fixed), there are several threads here and a page in the wiki about this.
apt update ; apt dist-upgrade (or yum upgrade) might fix it.
You might also have to remove and reinstall the network zimlets.
Post Reply