Page 1 of 1

Full restore, same 2 accounts give system failure: Error executing redoOp

Posted: Thu Nov 28, 2013 5:47 pm
by batfastad
Hi everyone
I'm migrating a Zimbra NE starter edition server from on-site to a dedicated server, CentOS 5 64bit to CentOS 6 64bit. I assumed this would be ok because I've installed exactly the same version of the software on each side... 7.2.3_GA_2872
I took a full backup, compressed it with 7z into 300MB chunks, then transferred it using rsync. The transfer of 70GB of data took 10 days over their ADSL connection. This was acceptable as since then I have been running daily incrementals which I have been transferring separately. However the longer it takes me to get the new server up and running, the higher the chance of an invalid backup.
So I ran a test restore with zmrestoreoffline and the -c option, running through these steps... Network Edition Disaster Recovery - Zimbra :: Wiki

Everything appeared to be ok, apart from 2 mailboxes appeared empty. I thought this was an index issue, tried re-indexing through admin interface and CLI but both just appear to complete really quickly. No noticeable errors.
All other mailboxes show the expected behaviour when re-indexing.
So I thought I would just try restoring one of these accounts using zmrestoreoffline to see if I noticed anything in the logs, and sure enough:

com.zimbra.common.service.ServiceException: system failure: Error executing redoOp

Full log here... zmrestoreoffline redoOp - Pastebin.com
Then I went through the mailbox.log from the full restore that I ran the other night and noticed that both these two account shows the same redoOp message. All other accounts were ok.
There doesn't seem to be much out there for this error message and even the stack trace doesn't show anything too obvious.
1) Is there anything in particular I need to be aware of when trying to restore from a full backup from CentOS 5 to CentOS 6? Or should that work so long as I use the same version/build of Zimbra on both sides?
2) Is it worth me running full backups of these problem mailboxes and trying to restore these separately?
3) What happens to future incrementals of all accounts after this full backup of selected accounts?
4) Should I restore this newer selective full backup over the full and incremental restore of the previous "all accounts" full backup?
Anyone else got any comments/suggestions/ideas on this?

I don't understand why this would be a problem for just 2 accounts.
Cheers, B

Full restore, same 2 accounts give system failure: Error executing redoOp

Posted: Thu Nov 28, 2013 6:04 pm
by batfastad
I'd also like to add... thank god I've still got the original server! Having to deal with this in an actual DR situation would be very bad news.

If I am able to restore this individual account from a fresh backup, I will be incredibly worried about the validity of the backups that Zimbra is producing in the future!

Full restore, same 2 accounts give system failure: Error executing redoOp

Posted: Wed Dec 11, 2013 5:16 pm
by batfastad
Just as an update, the only way I was able to complete this migration was by running full backups of the failing accounts and restoring those after the main DR restore.
This has got me really worried about the Zimbra full backups though. At least on this occasion I had access to a working server. I would think though that most times when backups are needed are for DR where the original server is down or no longer functioning. I'm just not sure I will be able to trust Zimbra NE backups again without finding out what went wrong.
If anyone ever has a similar situation and manages to recover the empty mailboxes then please let me know.
Cheers, B

Full restore, same 2 accounts give system failure: Error executing redoOp

Posted: Thu Mar 27, 2014 7:46 am
by Petrolej
Hello,
we are also seeing this exact issue when testing DR (hope we will never need it). We are trying to restore from full+incremental backup set from 8.0.3 ova appliance to Ubuntu 12.04 server (same Zimbra version 8.0.3). The full backup only restore is OK, but any incremental restore fails.
BR

Petr Olejník

Full restore, same 2 accounts give system failure: Error executing redoOp

Posted: Thu Mar 27, 2014 7:51 am
by phoenix
[quote user="9224petrolej"]we are also seeing this exact issue when testing DR (hope we will never need it). We are trying to restore from full+incremental backup set from 8.0.3 ova appliance to Ubuntu 12.04 server (same Zimbra version 8.0.3). The full backup only restore is OK, but any incremental restore fails.[/QUOTE]I'd suggest you upgrade your server to the most recent version of ZCS because of this and also read this.

Full restore, same 2 accounts give system failure: Error executing redoOp

Posted: Thu Mar 27, 2014 8:39 am
by Petrolej
Thank you, that is a "helpfull" post. I understand that we should upgrade. And in fact this restore testing is part of our upgrade process. If you see the upgrade instructions, on the first page there is a sentence:
[QUOTE]Make sure you have a good backup for all users on ZCA!

[/QUOTE]
Well, we cannot say that we have a good backup. I am realist and I do not believe that the upgrade process goes smoothly, frankly I do expect big troubles. Taking this into account there is no way for us to upgrade before we have a solid and tested backup.
Best regards

Petr Olejník

Full restore, same 2 accounts give system failure: Error executing redoOp

Posted: Thu Mar 27, 2014 8:52 am
by batfastad
I hadn't even noticed this issue until it came to migrate to a new server. Fortunately we still had the original source server available and we could run another backup for the affected users and restore them separately. However it's completely destroyed my faith in the NE full backup.
I would love to have an explanation as to what actually causes this issue.
Cheers, B