Zimbra Restore With Veeam

Running our Appliance (ZCA), ZCS on VMware, or any other virtual machine software? Post your thoughts here.
Post Reply
guilhermehnog
Posts: 12
Joined: Fri Aug 02, 2019 1:56 pm

Zimbra Restore With Veeam

Post by guilhermehnog »

Hi,

I created this topic in case my problem can help another person.

A few days ago, our server went down, taking the disks with it.
so we had to restore from backup.

Here we use veeam to backup the entire VM.
Our zimbra runs on Centos7.

After the restore, the xfs file system was corrupted and the S.O. was stuck in Recovery Mode.

to solve that error:

Code: Select all

xfs_repair -L /dev/mapper/centos_disk
then a few mailboxes started to log errors on mailbox.log and buffer overflow crashes on the SQL Database.

the problem was a few corrputed indexes of mail_item.
to solve that i had to identify the mboxgroupid of all the accounts with problem, drop the corrputed indexes and recreate them.

Enter zimbra account:

Code: Select all

su - zimbra
Get Mbox Group id from user:

Code: Select all

zmprov gmi usuario@dominio.com
Enter Mysql, select de correct database and check the table :

Code: Select all

mysql
MariaDB [(none)]> use mboxgroupXX;
MariaDB [mboxgroupXX]> check table mail_item;
+-----------------------+-------+----------+---------------------------------------------------------+
| Table                 | Op    | Msg_type | Msg_text                                                |
+-----------------------+-------+----------+---------------------------------------------------------+
| mboxgroupXX.mail_item | check | Warning  | InnoDB: Index "i_folder_id_date" is marked as corrupted |
| mboxgroupXX.mail_item | check | Warning  | InnoDB: Index "i_date" is marked as corrupted           |
| mboxgroupXX.mail_item | check | Warning  | InnoDB: Index "i_mod_metadata" is marked as corrupted   |
| mboxgroupXX.mail_item | check | error    | Corrupt                                                 |
+-----------------------+-------+----------+---------------------------------------------------------+
Get Table indexes:

Code: Select all

MariaDB [mboxgroupXX]>show index from mail_item;
+-----------+------------+------------------+--------------+--------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| Table     | Non_unique | Key_name         | Seq_in_index | Column_name  | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
+-----------+------------+------------------+--------------+--------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| mail_item |          0 | PRIMARY          |            1 | mailbox_id   | A         |           2 |     NULL | NULL   |      | BTREE      |         |               |
| mail_item |          0 | PRIMARY          |            2 | id           | A         |       79784 |     NULL | NULL   |      | BTREE      |         |               |
| mail_item |          0 | i_name_folder_id |            1 | mailbox_id   | A         |           2 |     NULL | NULL   |      | BTREE      |         |               |
| mail_item |          0 | i_name_folder_id |            2 | folder_id    | A         |          46 |     NULL | NULL   | YES  | BTREE      |         |               |
| mail_item |          0 | i_name_folder_id |            3 | name         | A         |         124 |     NULL | NULL   | YES  | BTREE      |         |               |
| mail_item |          1 | i_type           |            1 | mailbox_id   | A         |           2 |     NULL | NULL   |      | BTREE      |         |               |
| mail_item |          1 | i_type           |            2 | type         | A         |          18 |     NULL | NULL   |      | BTREE      |         |               |
| mail_item |          1 | i_parent_id      |            1 | mailbox_id   | A         |           2 |     NULL | NULL   |      | BTREE      |         |               |
| mail_item |          1 | i_parent_id      |            2 | parent_id    | A         |           2 |     NULL | NULL   | YES  | BTREE      |         |               |
| mail_item |          1 | i_folder_id_date |            1 | mailbox_id   | A         |           2 |     NULL | NULL   |      | BTREE      |         |               |
| mail_item |          1 | i_folder_id_date |            2 | folder_id    | A         |          48 |     NULL | NULL   | YES  | BTREE      |         |               |
| mail_item |          1 | i_folder_id_date |            3 | date         | A         |       79784 |     NULL | NULL   |      | BTREE      |         |               |
| mail_item |          1 | i_index_id       |            1 | mailbox_id   | A         |           2 |     NULL | NULL   |      | BTREE      |         |               |
| mail_item |          1 | i_index_id       |            2 | index_id     | A         |       79784 |     NULL | NULL   | YES  | BTREE      |         |               |
| mail_item |          1 | i_date           |            1 | mailbox_id   | A         |           2 |     NULL | NULL   |      | BTREE      |         |               |
| mail_item |          1 | i_date           |            2 | date         | A         |       79784 |     NULL | NULL   |      | BTREE      |         |               |
| mail_item |          1 | i_mod_metadata   |            1 | mailbox_id   | A         |           2 |     NULL | NULL   |      | BTREE      |         |               |
| mail_item |          1 | i_mod_metadata   |            2 | mod_metadata | A         |       79784 |     NULL | NULL   |      | BTREE      |         |               |
| mail_item |          1 | i_uuid           |            1 | mailbox_id   | A         |           2 |     NULL | NULL   |      | BTREE      |         |               |
| mail_item |          1 | i_uuid           |            2 | uuid         | A         |          96 |     NULL | NULL   | YES  | BTREE      |         |               |
+-----------+------------+------------------+--------------+--------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
Drop them:

Code: Select all

MariaDB [mboxgroupXX]>alter table mail_item drop index i_mod_metadata;
MariaDB [mboxgroupXX]>alter table mail_item drop index i_date;
MariaDB [mboxgroupXX]>alter table mail_item drop index i_folder_id_date;
Recreate them:

Code: Select all

MariaDB [mboxgroupXX]>create index i_date on mail_item(mailbox_id);
MariaDB [mboxgroupXX]>create index i_folder_id_date on mail_item(mailbox_id);
MariaDB [mboxgroupXX]>create index i_mod_metadata on mail_item(mailbox_id);
One of our accounts had the problem on i_parent_id. That index is needed in a foreign key and couldn't be deleted.
so this code, fixed it:

Code: Select all

MariaDB [mboxgroupXX]>OPTIMIZE TABLE mail_item;
Here are all the links i've used to fix the problem:

Code: Select all

https://forums.zimbra.org/viewtopic.php?t=38127
https://wiki.zimbra.com/wiki/How_to_recreate_corrupted_index_(mysql)
https://www.mariadbtutorial.com/mariadb-basics/mariadb-select-database/
https://stackoverflow.com/questions/5213339/how-to-see-indexes-for-a-database-or-table-in-mysql
https://www.mariadbtutorial.com/mariadb-index/mariadb-create-index/
https://blog.christosoft.de/2017/03/zimbra-corrupt-index-open_conversation-mysql/
Then you need to reindex each account so they can work again:

Change the account to Maintenance mode, save it.
click on the gear and select reindex mailbox.
after is done, activate the account again.

Thanks.
User avatar
jholder
Ambassador
Ambassador
Posts: 4824
Joined: Fri Sep 12, 2014 10:00 pm

Re: Zimbra Restore With Veeam

Post by jholder »

I know this is old, but I wanted to reply so that people see this.

1) Don't use XFS.
XFS is great for performance but disaster recovery with Zimbra MySQL data is awful.. I've had multiple personal experiences with it.

2) When backing up always use a method that extracts the data from the application. Veem (while great) doesn't.. Errors that exist will exist in your veem copy.
When you extract the data, it forces a read. So in your case, a mysqldump would have been most helpful.

3) Zimbra MySQL uses self-referencing foreign keys. This is to say, you're going to have a difficult time if the wrong key is busted.
I would try innodb_recovery_mode = 4
Once you do this, you can never use the data again. Just dump it. This ignores reading indexes, but will recreate them if it can export the data.
If this fails, you have to use a tool like undrop. You must compile it, because it's unique to the schema.
Post Reply