Zextra 9.0.0 FOSS => Zimbra 10 NE (missing mails / zmmailbox import issues)

Looking to migrate to ZCS? Ask here. Got a great tip or script that helped you migrate? Post it here.
Post Reply
jurriaan
Posts: 9
Joined: Mon Sep 18, 2023 7:39 am

Zextra 9.0.0 FOSS => Zimbra 10 NE (missing mails / zmmailbox import issues)

Post by jurriaan »

Hi,

How can we upgrade from:

Zextras Release 9.0.0_ZEXTRAS_20221203.RHEL8 FOSS edition.
==>
Zimbra 10 Network edition (licenced)
Server: Rocky8 with 16Gb RAM - 1 core 12 sockets RAID10

We have tried:
A.) Upgrading the version from 9 to 10. Zimbra didn't want to start properly after the update, not clear why. We puzzled 2 evenings with it. Reinstalled certificates etc etc.

B.) Migration tool in Zimbra 10 NE
==> fails at the mailbox data import (couldn't find any logs to see why)

C.) Using migrate "method 1" from Zextras:
https://community.zextras.com/migrate-z ... -method-1/
==> Also here it fails at the mailbox stage. All steps like account import etc without any trouble.
Though when updating the timeout most of the mailboxes seems to get imported:

Code: Select all

zmlocalconfig -e socket_so_timeout=3000000
https://community.zextras.com/forum/zim ... esnt-work/

Though some smaller mailboxes got "IO error java net connect errors", with these I can live with not to get imported. The most important mailboxes got imported with zmmailbox -z -m, including the biggest one all without any import errors.
Only it seems half of the mail is just not there after importing? The 40Gb zipfile contains all email between 2015 and 2023. When logging in into the webapp it shows only email from 2015 and 2016. The zip-file is build of sub-dir's per directory, some 500 mails per directory. INBOX1, INBOX2, INBOX3 etc.

Questions:
- It seems not all subdirs are being restored? Restoring took some 1,5-2 hour with the biggest mailbox.
- With this part of the zmmailbox command:

Code: Select all

"/?fmt=zip&resolve=skip"
can it be that a large part of the to be restored mails are being skipped? If so, how to solve it?
- Any other thoughts with a different upgrade path to Zimbra 10 NE?
Klug
Ambassador
Ambassador
Posts: 2796
Joined: Mon Dec 16, 2013 11:35 am
Location: France - Drôme
ZCS/ZD Version: All of them
Contact:

Re: Zextra 9.0.0 FOSS => Zimbra 10 NE (missing mails / zmmailbox import issues)

Post by Klug »

Are you sure the issue is in the import and not in the export?

There's a very old bug about tgz export through the zimbra-proxy.
You should try do export by connecting directly to mailboxd and then try to import this export.
jurriaan
Posts: 9
Joined: Mon Sep 18, 2023 7:39 am

Re: Zextra 9.0.0 FOSS => Zimbra 10 NE (missing mails / zmmailbox import issues)

Post by jurriaan »

The "Zextras method 1" described above uses zip instead of tgz.

All email is there in the export.zip file coming from Zimbra 9 FOSS.
The file gets imported "succesfully" (no errors) into Zimbra 10 with the zmmailbox command,
but when logging in into webapp, it seems 80% is being skipped to import.
User avatar
L. Mark Stone
Ambassador
Ambassador
Posts: 2836
Joined: Wed Oct 09, 2013 11:35 am
Location: Portland, Maine, US
ZCS/ZD Version: 10.0.9 Network Edition
Contact:

Re: Zextra 9.0.0 FOSS => Zimbra 10 NE (missing mails / zmmailbox import issues)

Post by L. Mark Stone »

For these kinds of migrations I typically use IMAPSYNC to move the emails and then use the tgz method to export/import users' Calendars, Contacts, Briefcase and Tasks. The Zextras recommendations for getting Distribution Lists etc. in my experience work well, but not when you export whole mailboxes, for the reason Klug mentioned above but also because I find there is often latent corruption with older emails that can cause issues, so IMAPSYNC works well to not migrate the garbage emails.

Those garbage emails typically comprise either/both of emails with latent corruption from old server crashes, bad email clients etc, as well as emails which for example, when they arrived three years ago contained what was then a zero-day, but which if the email arrived today would be rejected.

If you elect to use IMAPSYNC to move the mail over, and you continue to use the Zextras guide to get like Distribution Lists, then you just need to get users' Calendars, Contacts, Briefcase and Tasks. I do the following:

Get all the users from the domains you want to migrate:

Code: Select all

zmprov -l gaa | grep "domain1.com\|domain5.com" | sort > /opt/zimbra/backup3/zcs/emails1.txt
(Use this list also to populate the IMAPSYNC input file)

Get all their data:

Code: Select all

for i in `cat /opt/zimbra/backup3/zcs/emails1.txt`; do zmmailbox -z -t 300 -m $i getRestURL 'Calendar/?fmt=tgz' > /opt/zimbra/backup3/zcs/userdata/$i-calendar.tgz
for i in `cat /opt/zimbra/backup3/zcs/emails1.txt`; do zmmailbox -z -t 600 -m $i getRestURL 'Briefcase/?fmt=tgz' > /opt/zimbra/backup3/zcs/userdata/$i-briefcase.tgz
for i in `cat /opt/zimbra/backup3/zcs/emails1.txt`; do zmmailbox -z -t 300 -m $i getRestURL 'Contacts/?fmt=tgz' > /opt/zimbra/backup3/zcs/userdata/$i-contacts.tgz
for i in `cat /opt/zimbra/backup3/zcs/emails1.txt`; do zmmailbox -z -t 300 -m $i getRestURL 'Tasks/?fmt=tgz' > /opt/zimbra/backup3/zcs/userdata/$i-tasks.tgz
You can then copy the data over to the new server and then import the per-account tgz files.

Hope that helps,
Mark
___________________________________
L. Mark Stone
Mission Critical Email - Zimbra VAR/BSP/Training Partner https://www.missioncriticalemail.com/
AWS Certified Solutions Architect-Associate
Klug
Ambassador
Ambassador
Posts: 2796
Joined: Mon Dec 16, 2013 11:35 am
Location: France - Drôme
ZCS/ZD Version: All of them
Contact:

Re: Zextra 9.0.0 FOSS => Zimbra 10 NE (missing mails / zmmailbox import issues)

Post by Klug »

jurriaan wrote: Mon Sep 18, 2023 11:40 am The "Zextras method 1" described above uses zip instead of tgz.
Same issue with ZIP or TGZ iirc.
It's an issue with the export being a huge file going through the proxy.

I remember doing three exports of the same (not used) mailbox in a row, the three exports each had a different size, none of them containing all emails.
jurriaan wrote: Mon Sep 18, 2023 11:40 am All email is there in the export.zip file coming from Zimbra 9 FOSS.
The file gets imported "succesfully" (no errors) into Zimbra 10 with the zmmailbox command, but when logging in into webapp, it seems 80% is being skipped to import.
This is exactly what happens because of the issue I'm talking about.
The import works OK.
But you're importing an incomplete export, you are missing emails in the export archive and then in the webmail.

Mark's suggestion works much better (imapsync + exports) than full export of the mailboxes.
jurriaan
Posts: 9
Joined: Mon Sep 18, 2023 7:39 am

Re: Zextra 9.0.0 FOSS => Zimbra 10 NE (missing mails / zmmailbox import issues)

Post by jurriaan »

Mmm, pity is that I'm having 2 servers (OLD 9.0 FOSS + NEW 10.0 NE) both setup both with same DNS settings in my local network. No problem to change the old servers IP-address to sth else, but it would be problematic with resolving the same hostname on the internal network.
Does IMAPSYNC require both fully running Zimbra servers A->B?

Maybe IMAPSYNC can export data on Zimbra 9 FOSS to a tgz file, and on the new server import with IMAPSYNC from file to Zimbra 10NE?

Edit: IMAP works on ports 143/993. The question is can IMAP run, with wrong hostname on one of the servers? (thinking about LDAP, which is pretty sensitive I've noticed)

Edit2:
This what I want to try: (right now restoring with Proxmox backup sever Zimbra 9 for the 27th time)
./imapsync --host1 172.22.22.100 --user1 name@domain.com --password 123456 --host2 172.22.22.222 --user name@domain.com --password 123456

Would it work?
User avatar
L. Mark Stone
Ambassador
Ambassador
Posts: 2836
Joined: Wed Oct 09, 2013 11:35 am
Location: Portland, Maine, US
ZCS/ZD Version: 10.0.9 Network Edition
Contact:

Re: Zextra 9.0.0 FOSS => Zimbra 10 NE (missing mails / zmmailbox import issues)

Post by L. Mark Stone »

You can use IP addresses with imapsync.

You can also use the Global Admin account to auth for each user.

The imapsync website has a number of useful guides.
___________________________________
L. Mark Stone
Mission Critical Email - Zimbra VAR/BSP/Training Partner https://www.missioncriticalemail.com/
AWS Certified Solutions Architect-Associate
jurriaan
Posts: 9
Joined: Mon Sep 18, 2023 7:39 am

Re: Zextra 9.0.0 FOSS => Zimbra 10 NE (missing mails / zmmailbox import issues)

Post by jurriaan »

edit:
Thanks for all help! have tried IMAPSYNC, including tricks to speed it up. (Zimbra/wiki) but I don't believe it would be ready synced this week. :-(

Will try another approach:
- removed all "NG" modules from Zextras installed on the Zextra 9.0.0. FOSS version 202212/RHEL8
- remove remaining Zextras zimlets like drive etc.
- backup server with Proxmox
- try to install the lastest build of https://techfiles.online/zimbra/ on top of it.
(have read that some people had succes with it.)
- After that backup server with Proxmox again
- try upgrading with Zimbra 10 NE
jurriaan
Posts: 9
Joined: Mon Sep 18, 2023 7:39 am

Re: Zextra 9.0.0 FOSS => Zimbra 10 NE (missing mails / zmmailbox import issues)

Post by jurriaan »

I can confirm the following steps to be working out for Zimbra 10 NE:

Nightmare edition: Zextras Release 9.0.0_ZEXTRAS_20221203.RHEL8 FOSS edition
==> uninstall all Zextras Suite stuff and remove from webapp all left over Zextras Zimlets like drive etc.
==> make server snapshot
==> force install:
https://cdn.techfiles.online/rocky/zcs- ... 142943.tgz
from: https://techfiles.online/zimbra/
==> make server snapshot
==> force install:
https://files.zimbra.com/downloads/10.0 ... 065514.tgz
from: https://www.zimbra.com/product/download ... k-edition/

Force install of Zimbra 10 NE because of:
package zimbra-timezone-data-4.0.0.1678342256-1.r8.x86_64 (which is newer than zimbra-timezone-data-4.0.0.1675193012-1.r8.x86_64) is already installed

Optional upgrade step could have been to install first:
https://cdn.techfiles.online/rocky/zcs- ... 154520.tgz
after that:
https://files.zimbra.com/downloads/10.0 ... 065514.tgz

Tip: Keep the Yum repo's right, disable old Repo's, I saw I had Zimbra 8.7 still enabled... how? Seems not to be updated automatically on the way.

After 50 hours of headache I'm all happy again, just like in 8.8.15 times :-)
jurriaan
Posts: 9
Joined: Mon Sep 18, 2023 7:39 am

Re: Zextra 9.0.0 FOSS => Zimbra 10 NE (missing mails / zmmailbox import issues)

Post by jurriaan »

Right, and a big thumbs upp for Ian Walker, with his builds on https://techfiles.online/
It seems he cleaned up the **** between/for Zextras and Synacor.
Post Reply