Zimbra to Zimbra migration/upgrade, options?

Discuss your pilot or production implementation with other Zimbra admins or our engineers.
Post Reply
User avatar
gabrieles
Outstanding Member
Outstanding Member
Posts: 233
Joined: Tue Feb 14, 2017 9:40 am

Zimbra to Zimbra migration/upgrade, options?

Post by gabrieles »

The title is just provocative. We do mainly this, i know basically all the migration options, and i would just discuss with you which option remains after the last disastrous releases.

Let's say we have to migrate some old ZCS installations to the brand new 8.8.15 (now p4).
Let's make it easy, let's talk only about of single server installations, and only Network Edition.
We have manly these options:

Legacy backup/restore.

Basically we do a zmbackup -f -a all --noZip from the source VM on a shared storage and a zmrestore of all the accounts on the destination VM.
Clean and fast, just some downtime due to zmrestore. Only usable with reasonable size, let's say for migrations under 100GB.
But we know that with the advent of NG modules zmbackup is on the deprecation road: despite being retro-compatible to 5.0.12, and solid as rock until trhough centuries, lately it had some flaws.
First 8.8. releases had it totally broken due to the zad, and had it fixed immediately.
Last 8.8.15 P4 began to fail with "Error read timed out" leaving disastrous holes in the /opt/zimbra/store folder structure, and behaviouring strange with NG modules (sometimes, I did't understand based on what, they want to purge the restored mailboxes beacuse they define them incomplete)
Let's say that this method is actually not safe.

NG Incremental backup
Basically described here: https://wiki.zimbra.com/wiki/Zimbra_NG_ ... _NG_Backup
You install the legacy zextras backup on your old infrastructure, make a backup on a shared storage, and restore that backup with the NG external restore.
Clean and fast, and you have zero-like downtime, you can begin working directly on the new zimbra vm with empty mailbox, but just the time your content are injected and you are done.
Clean and fast if it wasn't for the fact that the legacy zextras modules aren't available from a month and they didn't noticed: https://www.zextras.com/download-legacy
Zimbra after being informed says, correctly, "it's not our website".
Funny, the only ZCS that could be migrated are the 8.8.x and in that case an upgrade is enough, not a migration.
Update: as zimbra support said: "[...] feedback from zextras team and they are not supporting the legacy tool anymore."
Let's say that this method is now clearly impossible.

Rolling upgrade with zmmboxmove

Basically you add the new VM (after some steps, temporary VMs, ldap promotions, and intermediate upgrades) to the old VM transforming your single server infrastructure in a multi-server.
Then you zmmboxmove the accounts from the old mailstore to the new, switch the two VM ip addresses (or in more complex infrastructures, switch the flows), adjust some configs, remove the old server, and you're done.
It's a pity that zmmboxmove is based on zmbackup, and in 8.8.15 P4 is starting to suffer the same "Error read timed out".
Update: I haven't anymore encountered the "Error read timed out", and successfully migrated 8.8.15 p5. Otherwise, we found it suffering the "million module bug" (https://wiki.zimbra.com/wiki/Account_ma ... _structure): restore of accounts that have more than 1048575 elements results in loss of data. In this case must be used the Rolling upgrade with rsync method.
Let's say that this method is actually safe under the correct environment.

Rolling upgrade with NG modules

Basically you perform the same steps of the classic rolling upgrade, but you move the mailboxes with zxsuite hsm doMailboxMove.
Unlike the zmmboxmove, the doMailboxMove must be launched on the source server, that means that the source server must be a 8.8.x with NG modules, or have some legacy module installed (I don't know, there's no documentation and legacy download page can't be accessed).
Let's say that even this method is actually impossible.

NEW: Rolling upgrade with rsync

In this case you first disable the blobfile move from the primary and secondary volume putting zimbraMailboxMoveSkipBlobs and zimbraMailboxMoveSkipHsmBlobs TRUE at serverConfig.
Then do a classic zmmboxmove from source to destination. This will perform a complete move just without the blobfiles.
Then you have to retrieve the mailbox id on the source and the destination server and rsync the /opt/zimbra/store/0/[mailbox id]/msg directories between the two servers.
If there are hsm or in general other volumes, even them must be rsynced according to the locator structure. I've not tried but I think that the volume topology (zmvolume -l) must be the same on the two servers.
This zmmboxmove does not seems to suffer the "Error read timed out" bug, probably because the actual move is far less than 1 GB.
Let's say that this method is actually working.
Last edited by gabrieles on Fri Jan 10, 2020 2:36 pm, edited 5 times in total.
User avatar
axslingr
Outstanding Member
Outstanding Member
Posts: 256
Joined: Sat Sep 13, 2014 2:20 am
ZCS/ZD Version: 8.8.15.GA.3869.UBUNTU18.64 UBUNTU18

Re: Zimbra to Zimbra migration/upgrade, options?

Post by axslingr »

Upgrade Zimbra on the old server to the latest version its OS will support.

Install supporting OS on new server of that version of Zimbra and rsync /opt/zimbra to new server.

Install same version of Zimbra on new server then upgrade Zimbra to latest version.

Hopefully you aren't so far behind on upgrades that this is not a 10 step process.

Lance
User avatar
gabrieles
Outstanding Member
Outstanding Member
Posts: 233
Joined: Tue Feb 14, 2017 9:40 am

Re: Zimbra to Zimbra migration/upgrade, options?

Post by gabrieles »

axslingr wrote: Install supporting OS on new server of that version of Zimbra and rsync /opt/zimbra to new server.
Install same version of Zimbra on new server then upgrade Zimbra to latest version.
I'ts a good technique we used until 8.7. /opt/zimbra structure changes in 8.7 and rsync from a previous version will not work.
axslingr wrote:Hopefully you aren't so far behind
There are customers on SUSE and they will not go over 8.6.
There are others on Ubuntu 10, not over 8.0.9.

Thank you, but rsync is not an option we take anymore.
User avatar
axslingr
Outstanding Member
Outstanding Member
Posts: 256
Joined: Sat Sep 13, 2014 2:20 am
ZCS/ZD Version: 8.8.15.GA.3869.UBUNTU18.64 UBUNTU18

Re: Zimbra to Zimbra migration/upgrade, options?

Post by axslingr »

In that case I'd be using Zextras.

Lance
7224jobe
Outstanding Member
Outstanding Member
Posts: 283
Joined: Sat Sep 13, 2014 1:55 am
ZCS/ZD Version: 8.8.15_FOSS Patch38

Re: Zimbra to Zimbra migration/upgrade, options?

Post by 7224jobe »

Good analysis, the results are woeful.

For old installations (<= Zimbra 8.5) you can use ZxMigration Tool, that was removed from Zextras website but still available thanks to Imanudin: https://imanudin.com/2019/09/04/link-do ... ion-tools/

But it is a workaround, good for Open Source customers, a paying Network Edition customer should have an official solution provided by Zimbra.

Last solution: imapsync... :roll:
User avatar
JDunphy
Outstanding Member
Outstanding Member
Posts: 889
Joined: Fri Sep 12, 2014 11:18 pm
Location: Victoria, BC
ZCS/ZD Version: 9.0.0_P39 NETWORK Edition

Re: Zimbra to Zimbra migration/upgrade, options?

Post by JDunphy »

Thank you for a great topic discussion. I will have this in front of me also. I am currently at 8.7.11 which is end of life Dec 31, 2019 but 8.8+ appears to raise stability issues that we don't want our users to experience according to various reports in these forums. As a result, I am probably going to be a little late to the upgrade party and will probably stay with technical guidance until I see real stability in that 8.8.15 release. Every patch and subsequent bug fix seems to break something new from what others are reporting here. Shortly after 8.8 first came out, I did an in place upgrade and while it worked the product was too buggy and I reverted back and took a wait and see approach.

Having said that, I have been watching what Mark Stone and what other have been saying and tracking various ideas. This wiki link below looks promising but I have not tested it.

https://wiki.zimbra.com/wiki/Steps_To_R ... ZCS_Server

I generally do an rsync method to a test server and then test my upgrades before trying anything on the production server. We are under 1TB so that is fairly fast with 10G links. That method doesn't catch every problem when it comes time to hit the production server but it captures the big stuff and I find I am able to solve most things when I don't have the stress of a failed production server and users calling for an ETA while learning about the upgrade process. Not to mention, upgrade to production servers are early morning hours to minimize downtime so I am already on heightened stress levels. :-)

Maybe Mark will write another one of his best practices papers on upgrades from 8.7.11 to give us some options.

Jim
User avatar
L. Mark Stone
Ambassador
Ambassador
Posts: 2796
Joined: Wed Oct 09, 2013 11:35 am
Location: Portland, Maine, US
ZCS/ZD Version: 10.0.6 Network Edition
Contact:

Re: Zimbra to Zimbra migration/upgrade, options?

Post by L. Mark Stone »

JDunphy wrote:
Maybe Mark will write another one of his best practices papers on upgrades from 8.7.11 to give us some options.

Jim
Challenge accepted!

But let me do a bit of a brain dump here to add my experience to the thread...

To be fair I haven't yet experienced the "Error read time out" issue. If it's widespread, then hopefully a few NE customers have reported it so it's getting some attention.

First thing to clear the air is that once you install the NG kit on the new 8.8.15 servers, you cannot/should not use any of the zmmboxmove or other legacy tools. I'm working on one large multi-server rolling migration now, and we just upgraded the new servers to P4 and haven't had the "Error read timeout issue" yet. What we did was install the 8.8.15 servers without the NG modules so we could safely use the classic zmmailboxmove, and this has worked well except where the source mailboxes have some latent corruption, which we are addressing now with help from Zimbra Support (they are NE Archive mailboxes that have been in existence for a decade or so...).

Second thing is that the NG backups should not go on NFS volumes, according to ZeXtras. Zimbra however have not yet updated their official documentation to indicate that classic backups work fine with a good NFS target, but that NG backups should not use NFS for a backend store. Out of about ten NG Incremental backups with NFS for the source server backup store, two failed with data loss -- and no errors reported by ZeXtras/NG. It was just that the new servers' mailboxes were missing data. One of the failed systems had really excellent networking/storage infrastructure: all high-end Cisco/EMC kit. As a result, I refuse to do migrations now where the customer insists on using NFS, and I've lost a few prospects over this, but I just won't put my name to anything that I don't feel is data-safe.

Third thing as regards Rolling Upgrades is to be mindful of LDAP when doing upgrades from older Zimbra systems. ZCS 8.5 and later have many more global config variables than earlier versions. Installing a new 8.8.15 LDAP replica into an LDAP pool of older Zimbra servers sometimes (not always) results in these new global config variables not being populated into LDAP. You'll then find that when you go to add new 8.8.15 MTA or mailbox servers, the installer will complete just fine, but pieces of Zimbra won't start during the zmsetup.pl phase. The fix is to manually create these new global config variables by hand (script actually), by copying them from a virgin 8.8.15 installation. Rerunning the 8.8.15 installer on the 8.8.15 LDAP server will not update the LDAP schema and create the missing gcf's Zimbra Support and I have found (although that was the expectation...).

Fourth, the rsync method does still work, but you have to have the target system running the same version of Zimbra as the source system (albeit on a new OS), and you can't rsync all of the directories (zimbramon for example needs to be OS-specific). And then you have to do an in-place ZCS upgrade. If you moving from a really old Zimbra system on, say, Ubuntu 10, you may have to do this twice depending on which Zimbra binaries are available for your OS.

At the end of the day, for smaller systems I typically use the NG Incremental Backup method, and for larger multi-server environments I'm using the Rolling Upgrade with zmmboxmove. I also try to avoid in-place upgrades from older Zimbra systems so I don't have to worry about any "creative" customizations previous admins may have made to the system in years past.

For the Rolling Upgrades, I use LDAP MMR as needed with older systems first, to be able to deploy "new" LDAP servers at the same ZCS version as the existing LDAP servers, but on newer operating systems, so I can (eventually) do in-place upgrades of the LDAP servers to get them up to 8.8.15. It's a lot of fussing with LDAP replication agreements, and lots of Zimbra restarts as you progressively change ldap_url and ldap_master_url localconfig variables. Once you've got all-new 8.8.15 LDAP servers on current operating systems, then I do the new proxy and mta servers, and then the new mailbox servers.

Rolling upgrades take A LOT of planning and work behind the scenes, but create near-zero downtime for end users -- except when you have to bounce a server to use the new ldap_url and ldap_master_url localconfig variables.

I have also referred several customers to Audriga, and the results have been good. These customers were either resource constrained, or their old Zimbra systems were in a, let's say, "fragile" state., Some customers did not have for example enough storage they could provision for an NG Incremental migration; other customers were leery of rebooting/restarting their Zimbra servers because they didn't always restart cleanly.

I also maintain an IMAPSYNC server on AWS for these kinds of outlier migrations; I do the mail moves via IMAPSYNC, and the customer's team exports calendars and contacts to import into the new system. Distribution lists and sharing needs to be recreated on the new system, but "fragile" older systems often are in need of significant "cleanup" anyway, so creating sharing and distribution lists fresh can be a welcome, if time-consuming exercise.

Thanks Jim for the idea for a blog post; I've got to finish my CJIS/HIPAA Compliance blog post first however, and then I'll try and tackle this.

Thanks to gabrieles for a very comprehensive summary.

Best regards to all,
Mark
___________________________________
L. Mark Stone
Mission Critical Email - Zimbra VAR/BSP/Training Partner https://www.missioncriticalemail.com/
AWS Certified Solutions Architect-Associate
User avatar
gabrieles
Outstanding Member
Outstanding Member
Posts: 233
Joined: Tue Feb 14, 2017 9:40 am

Re: Zimbra to Zimbra migration/upgrade, options?

Post by gabrieles »

Thanks to all for the answers. I'll try to respond in line
7224jobe wrote:still available thanks to Imanudin: https://imanudin.com/2019/09/04/link-do ... ion-tools/
a paying Network Edition customer should have an official solution provided by Zimbra.
Nothing to add, It was my first thought: having to rely to an external blog (however a good and usegful blog) for a required piece of software is not a fun thing.
L. Mark Stone wrote: once you install the NG kit on the new 8.8.15 servers, you cannot/should not use any of the zmmboxmove or other legacy tools.
This has been said much times, but where is effectivley stated? As far that the legacy tools continued to work, many customers prefer to rely on them, than on the NGs.
If I remember well, the 8.8.0 broke the zmmboxmove, and was fixed in one of the first minor release just to meet these cases.
L. Mark Stone wrote: Second thing is that the NG backups should not go on NFS volumes, according to ZeXtras.
This is far much more important. Can you provide a documentation link where is this stated? Maybe I missed a so important requirement, but as far as I remember, I've not seen it anywhere. I've read that NG modules does not support CIFS, because they make a large use of hashes and CIFS, and other case-insensitive filesystems cannot support it. I've even read (honestly don't rememmber exactly where) that the destination filesystem for backups and hsm had to be fast, due to the near-realtime backups. But as far as i remember, I've never found any requirements that forbids NFS. If so, we have to rethink 99% of our customer's architecture.
L. Mark Stone wrote: Third thing as regards Rolling Upgrades is to be mindful of LDAP when doing upgrades from older Zimbra systems.
ZCS 8.5 and later have many more global config variables than earlier versions.
...for larger multi-server environments I'm using the Rolling Upgrade with zmmboxmove.
Rolling upgrades take A LOT of planning and work behind the scenes, but create near-zero downtime for end users
Agree, we have mainly big installations and that's actually the main method. I have still to troubleshoot where that "Error read timeout" goes from.
L. Mark Stone wrote: Fourth, the rsync method does still work,
We use it rarely, mainly for recovering damaged or violated installations, rsyncing as you said on the same identical version.


Last thing: I feared NFS was the cause for the "Error read time out" given the discriminant is the NFS use or not, so i made a test moving all on a local storage, and I found the same error.
Last edited by gabrieles on Mon Dec 09, 2019 3:14 pm, edited 1 time in total.
User avatar
wentum
Advanced member
Advanced member
Posts: 53
Joined: Fri Apr 04, 2014 10:49 am
Location: Pforzheim (Germany)
ZCS/ZD Version: Release 9.0.0.GA.3924 _P30
Contact:

Re: Zimbra to Zimbra migration/upgrade, options?

Post by wentum »

L. Mark Stone wrote:
Second thing is that the NG backups should not go on NFS volumes, according to ZeXtras. Zimbra however have not yet updated their official documentation to indicate that classic backups work fine with a good NFS target, but that NG backups should not use NFS for a backend store. Out of about ten NG Incremental backups with NFS for the source server backup store, two failed with data loss -- and no errors reported by ZeXtras/NG. It was just that the new servers' mailboxes were missing data. One of the failed systems had really excellent networking/storage infrastructure: all high-end Cisco/EMC kit. As a result, I refuse to do migrations now where the customer insists on using NFS, and I've lost a few prospects over this, but I just won't put my name to anything that I don't feel is data-safe.
Hello Mark,

reading your statement about zextras backup and nfs I remembered me doing some pre 8.8. update research...
At that time I came across this statement from a zextras employee which claims that it can be done...

https://forums.zextras.com/zxbackup/310 ... ackup.html

But I have to admit it is pretty old...

Is your information more recent or from technical staff?

Regards
Joerg
User avatar
gabrieles
Outstanding Member
Outstanding Member
Posts: 233
Joined: Tue Feb 14, 2017 9:40 am

Re: Zimbra to Zimbra migration/upgrade, options?

Post by gabrieles »

Here's a little updates.
First the bad one, directly from zimbra support:

"Hello,
I have received a feedback from zextras team and they are not supporting the legacy tool anymore."

Well actually you can migrate to zimbra 8.8.x with NG move only if you are already on 8.8.x. Nice

Then the good one.
Thanks to zimbra support (yes, they are not always bearer of bad news, like this time) we are able standardize a rsync migration. I'll edit the first post with it
Post Reply