(SOLVED) Backup fails for single account

Discuss your pilot or production implementation with other Zimbra admins or our engineers.
Post Reply
yellowhousejake
Advanced member
Advanced member
Posts: 129
Joined: Tue Sep 09, 2014 9:57 am
ZCS/ZD Version: Release 10.0.1.GA.4518.UBUNTU20_64

(SOLVED) Backup fails for single account

Post by yellowhousejake »

Zimbra 8.0.7 NE on Ubuntu 12.0.4 LTS

Good afternoon,
We received an error message after our last two inc backups as follows.

Server: zimbra.greenfieldin.org

Label: full-20180131.060123.442
Type: full
Status: completed (with errors)
Started: Wed, 2018/01/31 01:01:23.442 EST
Ended: Wed, 2018/01/31 01:01:45.281 EST
Redo log sequence range: 1922 .. 1922
Number of accounts: 1
Number of errors: 1


ERRORS

lbrese@greenfieldin.org: Input/output error
java.io.IOException: Input/output error
at java.io.FileInputStream.readBytes(Native Method)
at java.io.FileInputStream.read(FileInputStream.java:243)
at com.zimbra.cs.backup.util.ZipUtil.computeCRC32(ZipUtil.java:24)
at com.zimbra.cs.backup.util.ParallelZipCopier$ZipCopierThread.run(ParallelZipCopier.java:185)

The rest of the backup completes, but when Zimbra sees that the mailbox in question failed it's last backup it tries to complete a full backup on that account. The full backup fails. We see nothing in the logs beyond the information included in the notification. We did verify the mailbox indexes and ran a blob check, both returned no errors.

I suspect this is something simple, so I thought I would ask here before opening a support ticket. What other information can I provide?

DAve
Last edited by yellowhousejake on Thu Mar 15, 2018 1:39 pm, edited 1 time in total.
yellowhousejake
Advanced member
Advanced member
Posts: 129
Joined: Tue Sep 09, 2014 9:57 am
ZCS/ZD Version: Release 10.0.1.GA.4518.UBUNTU20_64

Re: Backup fails for single account

Post by yellowhousejake »

I removed the offending account and recreated it. Problem went away. Then, two weeks later it happens again. This time it is multiple accounts and not the same account as the first time.

This is now looking like a Zimbra problem, possibly with the DB backend. This is a mix of high traffic and low traffic accounts being affected. The server is experiencing no disk or network issues so I do not think it is an impending hardware failure, the problem is only occurring during full backup runs.

XXX@XXX.XXX: system failure: getting map of message id to blob digest
com.zimbra.common.service.ServiceException: system failure: getting map of message id to blob digest
ExceptionId:FullBackupThread:1519624834473:770080fc7765080e
Code:service.FAILURE
at com.zimbra.common.service.ServiceException.FAILURE(ServiceException.java:258)
at com.zimbra.cs.db.DbBackup.getBlobDigestMap(DbBackup.java:230)
at com.zimbra.cs.backup.BackupAccountSession$BlobDigests.preload(BackupAccountSession.java:684)
at com.zimbra.cs.backup.BackupAccountSession.loadBlobDigestMap(BackupAccountSession.java:641)
at com.zimbra.cs.backup.FileBackupTarget$BackupAcctSession.storeBlobsBeforeMaintenanceMode(FileBackupTarget.java:1466)
at com.zimbra.cs.backup.BackupAccountSession.startFullBackup(BackupAccountSession.java:146)
at com.zimbra.cs.backup.BackupSet.startFullBackup(BackupSet.java:395)
at com.zimbra.cs.backup.FileBackupTarget$FileBackupSet.startFullBackup(FileBackupTarget.java:983)
at com.zimbra.cs.backup.BackupManager.doBackupFull(BackupManager.java:847)
at com.zimbra.cs.backup.BackupManager$FullBackupThread.run(BackupManager.java:235)
Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: Communications link failure

The last packet successfully received from the server was 11,812 milliseconds ago. The last packet sent successfully to the server was 12,086 milliseconds ago.

Query being executed when exception was thrown:
SELECT id, mod_content, blob_digest FROM mboxgroup22.mail_item_dumpster WHERE mailbox_id = 422


at sun.reflect.GeneratedConstructorAccessor314.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1116)
at com.mysql.jdbc.MysqlIO.nextRowFast(MysqlIO.java:1678)
at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1434)
at com.mysql.jdbc.MysqlIO.readSingleRowSet(MysqlIO.java:2936)
at com.mysql.jdbc.MysqlIO.getResultSet(MysqlIO.java:477)
at com.mysql.jdbc.MysqlIO.readResultsForQueryOrUpdate(MysqlIO.java:2631)
at com.mysql.jdbc.MysqlIO.readAllResults(MysqlIO.java:1800)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2221)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2624)
at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2127)
at com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:2293)
at com.zimbra.cs.db.DebugPreparedStatement.executeQuery(DebugPreparedStatement.java:163)
at org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:96)
at org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:96)
at com.zimbra.cs.db.DbBackup.getBlobDigestMap(DbBackup.java:217)
... 8 more

Still looking for suggestions.

DAve
yellowhousejake
Advanced member
Advanced member
Posts: 129
Joined: Tue Sep 09, 2014 9:57 am
ZCS/ZD Version: Release 10.0.1.GA.4518.UBUNTU20_64

Re: Backup fails for single account

Post by yellowhousejake »

Support ticket #00700163 opened.

DAve
yellowhousejake
Advanced member
Advanced member
Posts: 129
Joined: Tue Sep 09, 2014 9:57 am
ZCS/ZD Version: Release 10.0.1.GA.4518.UBUNTU20_64

Re: Backup fails for single account (solved)

Post by yellowhousejake »

Working with Support, we went through several tests with no effect on the backups. This week I was faced with multiple accounts having problems and saw that the mysql_error.log file was filling up as mysql crashed every time one of the three problem accounts attempted to use email. Looking at the log I saw where the mboxgroup22.mail_item_dumpster table was corrupted. As all three of the problem accounts were in mboxgroup22, it certainly looked like my problem.

With just a 30 minute window to fix the issue without taking mail offline during the day, I logged in and shutdown mailboxd. Opened a connection to mysql and selected the mboxgroup22 database. Fixed it and restarted mailboxd. This solved the problem and full backups completed.

This is written off the top of my head, DO NOT copy these commands to a terminal.

# source ~/bin/zmshutil ; zmsetvars
# mysql
# USE mboxgroup22;
# SELECT * FROM mail_item_DUMPSTER;
( MYSQL CRASHED)
# ~/mysql/bin/mysqldump mboxgroup22 -S $mysql_socket -u root --password=$mysql_root_password > mboxgroup22.sql
# mysql
# USE mboxgroup22;
# CREATE TABLE sav LIKE mail_item_dumpster;
# DROP TABLE mail_item_dumpster;
# RENAME TABLE sav TO mail_item_dumpster;
# quit

I then restarted mailboxd and watched the mysql_error.log while accessing the problem accounts. No issues! Running a full backup on each of those accounts completed as well without error.

I could, if needed, reload the table from the dumped data as it should be cleaned up once dumped to text. As this is the dumpster folder, I chose not to bother. The accounts are light users of email and will likely never notice or need the data.

After completing the fix I added my steps to the support ticket and support responded that they were heading in the same direction and supplied a url outlining the same steps. https://wiki.zimbra.com/wiki/Crash_Reco ... ific_Table

DAve
phoenix
Ambassador
Ambassador
Posts: 27272
Joined: Fri Sep 12, 2014 9:56 pm
Location: Liverpool, England

Re: Backup fails for single account

Post by phoenix »

Thanks for the update to your thread, could you mark it as "Solved" by editing the title?
Regards

Bill

Rspamd: A high performance spamassassin replacement

Per ardua ad astra
yellowhousejake
Advanced member
Advanced member
Posts: 129
Joined: Tue Sep 09, 2014 9:57 am
ZCS/ZD Version: Release 10.0.1.GA.4518.UBUNTU20_64

Re: (SOLVED) Backup fails for single account

Post by yellowhousejake »

Yes, a pet peeve of mine to spend an hour searching the web for a solution to a problem and to find a thread with the same issue, and after three pages of discussion the last post is "I fixed it" and nothing more.

Ahh, I put it in the wrong place. That is corrected thank you.

DAve
Post Reply