Zimbra to Zimbra migration/upgrade, options?
Posted: Fri Dec 06, 2019 11:52 am
The title is just provocative. We do mainly this, i know basically all the migration options, and i would just discuss with you which option remains after the last disastrous releases.
Let's say we have to migrate some old ZCS installations to the brand new 8.8.15 (now p4).
Let's make it easy, let's talk only about of single server installations, and only Network Edition.
We have manly these options:
Legacy backup/restore.
Basically we do a zmbackup -f -a all --noZip from the source VM on a shared storage and a zmrestore of all the accounts on the destination VM.
Clean and fast, just some downtime due to zmrestore. Only usable with reasonable size, let's say for migrations under 100GB.
But we know that with the advent of NG modules zmbackup is on the deprecation road: despite being retro-compatible to 5.0.12, and solid as rock until trhough centuries, lately it had some flaws.
First 8.8. releases had it totally broken due to the zad, and had it fixed immediately.
Last 8.8.15 P4 began to fail with "Error read timed out" leaving disastrous holes in the /opt/zimbra/store folder structure, and behaviouring strange with NG modules (sometimes, I did't understand based on what, they want to purge the restored mailboxes beacuse they define them incomplete)
Let's say that this method is actually not safe.
NG Incremental backup
Basically described here: https://wiki.zimbra.com/wiki/Zimbra_NG_ ... _NG_Backup
You install the legacy zextras backup on your old infrastructure, make a backup on a shared storage, and restore that backup with the NG external restore.
Clean and fast, and you have zero-like downtime, you can begin working directly on the new zimbra vm with empty mailbox, but just the time your content are injected and you are done.
Clean and fast if it wasn't for the fact that the legacy zextras modules aren't available from a month and they didn't noticed: https://www.zextras.com/download-legacy
Zimbra after being informed says, correctly, "it's not our website".
Funny, the only ZCS that could be migrated are the 8.8.x and in that case an upgrade is enough, not a migration.
Update: as zimbra support said: "[...] feedback from zextras team and they are not supporting the legacy tool anymore."
Let's say that this method is now clearly impossible.
Rolling upgrade with zmmboxmove
Basically you add the new VM (after some steps, temporary VMs, ldap promotions, and intermediate upgrades) to the old VM transforming your single server infrastructure in a multi-server.
Then you zmmboxmove the accounts from the old mailstore to the new, switch the two VM ip addresses (or in more complex infrastructures, switch the flows), adjust some configs, remove the old server, and you're done.
It's a pity that zmmboxmove is based on zmbackup, and in 8.8.15 P4 is starting to suffer the same "Error read timed out".
Update: I haven't anymore encountered the "Error read timed out", and successfully migrated 8.8.15 p5. Otherwise, we found it suffering the "million module bug" (https://wiki.zimbra.com/wiki/Account_ma ... _structure): restore of accounts that have more than 1048575 elements results in loss of data. In this case must be used the Rolling upgrade with rsync method.
Let's say that this method is actually safe under the correct environment.
Rolling upgrade with NG modules
Basically you perform the same steps of the classic rolling upgrade, but you move the mailboxes with zxsuite hsm doMailboxMove.
Unlike the zmmboxmove, the doMailboxMove must be launched on the source server, that means that the source server must be a 8.8.x with NG modules, or have some legacy module installed (I don't know, there's no documentation and legacy download page can't be accessed).
Let's say that even this method is actually impossible.
NEW: Rolling upgrade with rsync
In this case you first disable the blobfile move from the primary and secondary volume putting zimbraMailboxMoveSkipBlobs and zimbraMailboxMoveSkipHsmBlobs TRUE at serverConfig.
Then do a classic zmmboxmove from source to destination. This will perform a complete move just without the blobfiles.
Then you have to retrieve the mailbox id on the source and the destination server and rsync the /opt/zimbra/store/0/[mailbox id]/msg directories between the two servers.
If there are hsm or in general other volumes, even them must be rsynced according to the locator structure. I've not tried but I think that the volume topology (zmvolume -l) must be the same on the two servers.
This zmmboxmove does not seems to suffer the "Error read timed out" bug, probably because the actual move is far less than 1 GB.
Let's say that this method is actually working.
Let's say we have to migrate some old ZCS installations to the brand new 8.8.15 (now p4).
Let's make it easy, let's talk only about of single server installations, and only Network Edition.
We have manly these options:
Legacy backup/restore.
Basically we do a zmbackup -f -a all --noZip from the source VM on a shared storage and a zmrestore of all the accounts on the destination VM.
Clean and fast, just some downtime due to zmrestore. Only usable with reasonable size, let's say for migrations under 100GB.
But we know that with the advent of NG modules zmbackup is on the deprecation road: despite being retro-compatible to 5.0.12, and solid as rock until trhough centuries, lately it had some flaws.
First 8.8. releases had it totally broken due to the zad, and had it fixed immediately.
Last 8.8.15 P4 began to fail with "Error read timed out" leaving disastrous holes in the /opt/zimbra/store folder structure, and behaviouring strange with NG modules (sometimes, I did't understand based on what, they want to purge the restored mailboxes beacuse they define them incomplete)
Let's say that this method is actually not safe.
NG Incremental backup
Basically described here: https://wiki.zimbra.com/wiki/Zimbra_NG_ ... _NG_Backup
You install the legacy zextras backup on your old infrastructure, make a backup on a shared storage, and restore that backup with the NG external restore.
Clean and fast, and you have zero-like downtime, you can begin working directly on the new zimbra vm with empty mailbox, but just the time your content are injected and you are done.
Clean and fast if it wasn't for the fact that the legacy zextras modules aren't available from a month and they didn't noticed: https://www.zextras.com/download-legacy
Zimbra after being informed says, correctly, "it's not our website".
Funny, the only ZCS that could be migrated are the 8.8.x and in that case an upgrade is enough, not a migration.
Update: as zimbra support said: "[...] feedback from zextras team and they are not supporting the legacy tool anymore."
Let's say that this method is now clearly impossible.
Rolling upgrade with zmmboxmove
Basically you add the new VM (after some steps, temporary VMs, ldap promotions, and intermediate upgrades) to the old VM transforming your single server infrastructure in a multi-server.
Then you zmmboxmove the accounts from the old mailstore to the new, switch the two VM ip addresses (or in more complex infrastructures, switch the flows), adjust some configs, remove the old server, and you're done.
It's a pity that zmmboxmove is based on zmbackup, and in 8.8.15 P4 is starting to suffer the same "Error read timed out".
Update: I haven't anymore encountered the "Error read timed out", and successfully migrated 8.8.15 p5. Otherwise, we found it suffering the "million module bug" (https://wiki.zimbra.com/wiki/Account_ma ... _structure): restore of accounts that have more than 1048575 elements results in loss of data. In this case must be used the Rolling upgrade with rsync method.
Let's say that this method is actually safe under the correct environment.
Rolling upgrade with NG modules
Basically you perform the same steps of the classic rolling upgrade, but you move the mailboxes with zxsuite hsm doMailboxMove.
Unlike the zmmboxmove, the doMailboxMove must be launched on the source server, that means that the source server must be a 8.8.x with NG modules, or have some legacy module installed (I don't know, there's no documentation and legacy download page can't be accessed).
Let's say that even this method is actually impossible.
NEW: Rolling upgrade with rsync
In this case you first disable the blobfile move from the primary and secondary volume putting zimbraMailboxMoveSkipBlobs and zimbraMailboxMoveSkipHsmBlobs TRUE at serverConfig.
Then do a classic zmmboxmove from source to destination. This will perform a complete move just without the blobfiles.
Then you have to retrieve the mailbox id on the source and the destination server and rsync the /opt/zimbra/store/0/[mailbox id]/msg directories between the two servers.
If there are hsm or in general other volumes, even them must be rsynced according to the locator structure. I've not tried but I think that the volume topology (zmvolume -l) must be the same on the two servers.
This zmmboxmove does not seems to suffer the "Error read timed out" bug, probably because the actual move is far less than 1 GB.
Let's say that this method is actually working.