BackupNG fails on restore
Posted: Thu Nov 15, 2018 2:46 pm
Good day,
I am testing a migration from our old Zimbra server to a new Zimbra installation. The backup source was written to a NFS mount from the new Zimbra server using Zextras as an export. Pretty much exactly as suggested by this forum.
The export ran without error and the restore began fine and ran for a good while before failing. The notification message is below, I also included a Java Heap error that was found in the restore log and not displayed in the error notification.
Any thoughts on what may be the cause? It seems to me I need to increase the memory of the restore process.
DAve
------------------------------------------------
Zimbra 8.8.9_GA_3019.NETWORK
Ubuntu 16 LTS
Running as a VM - 8GB Memory, 4 vCPU
264 accounts
2 domains
6 COS
------------------------------- ERROR FROM JOB NOTIFICATION -------------------------------------------
This is an automated notification from ZxBackup about External Restore.
Operation External Restore failed.
Operation Id: 9b3ae8a5-3769-4001-a036-6f251505fcf5
Operation Host: zimbra2.greenfieldin.org
Operation requested by: admin@zimbra2.greenfieldin.org
Network Modules NG Version: 2.9.0
commit: 423b5076c550c71168b23f8b8cc649363bea2d78
Zal Version: 2.4.0
Zal commit: 93807f76531868754de673a94725162b2b2a0edc
Zimbra version: 8.8.9_GA_3019 20180809160254 20180823-0614 NETWORK
- exception -
com.zextras.lib.Error.OperationBlockingError: Blocking operation error
at com.zextras.modules.backup.operations.ZEExternalRestoreOperation.restoreItemsFromExternal (ZEExternalRestoreOperation.java:1206)
at com.zextras.modules.backup.operations.ZEExternalRestoreOperation$1._visit (ZEExternalRestoreOperation.java:886)
at com.zextras.modules.backup.operations.ZEExternalRestoreOperation$1._visit (ZEExternalRestoreOperation.java:870)
at com.zextras.lib.visitors.AbstractVisitor.visit (AbstractVisitor.java:32)
at com.zextras.lib.visitors.MultiThreadVisitor$1.run (MultiThreadVisitor.java:72)
at com.zextras.lib.activities.ThreadLimiter$1.run (ThreadLimiter.java:60)
at com.zextras.lib.activities.ActivityThread.run (ActivityThread.java:138)
Caused by: java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf (Arrays.java:3236)
at java.io.ByteArrayOutputStream.toByteArray (ByteArrayOutputStream.java:191)
at com.sun.mail.util.ASCIIUtility.getBytes (ASCIIUtility.java:271)
at javax.mail.internet.MimeMessage.parse (MimeMessage.java:339)
at javax.mail.internet.MimeMessage. (MimeMessage.java:209)
at com.zextras.modules.backup.ZEBItem.restoreCalendarItem (ZEBItem.java:1504)
at com.zextras.modules.backup.ZEBItem.createZimbraItem (ZEBItem.java:2222)
at com.zextras.modules.backup.ZEBItem.restore (ZEBItem.java:735)
at com.zextras.modules.backup.operations.ZEExternalRestoreOperation.restoreItemsFromExternal (ZEExternalRestoreOperation.java:1124)
at com.zextras.modules.backup.operations.ZEExternalRestoreOperation$1._visit (ZEExternalRestoreOperation.java:886)
at com.zextras.modules.backup.operations.ZEExternalRestoreOperation$1._visit (ZEExternalRestoreOperation.java:870)
at com.zextras.lib.visitors.AbstractVisitor.visit (AbstractVisitor.java:32)
at com.zextras.lib.visitors.MultiThreadVisitor$1.run (MultiThreadVisitor.java:72)
at com.zextras.lib.activities.ThreadLimiter$1.run (ThreadLimiter.java:60)
at com.zextras.lib.activities.ActivityThread.run (ActivityThread.java:138)
Last progress info:
--
Operation log path: /opt/zimbra/log/op_ExternalRestore_9b3ae8a5-3769-4001-a036-6f251505fcf5.log
Warnings/Errors: 0/0
Restored Accounts(only attributes): 273/273
Restored Accounts(common items): 10/273
Restored Accounts(only shared & datasources): 0/273
Current Account: gdewitt@greenfieldin.org
Restored/Skipped Items: 46690/0
Unrestored Items: 0
------------------------ ERROR FROM RESTORE LOG ----------------------------------
2018-11-14 07:35:57,369 jkoch@greenfieldin.org ERR Exception: java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf (Arrays.java:3236)
at java.io.ByteArrayOutputStream.toByteArray (ByteArrayOutputStream.java:191)
at com.sun.mail.util.ASCIIUtility.getBytes (ASCIIUtility.java:271)
at javax.mail.internet.MimeMessage.parse (MimeMessage.java:339)
at javax.mail.internet.MimeMessage.<init> (MimeMessage.java:209)
at com.zextras.modules.backup.ZEBItem.restoreCalendarItem (ZEBItem.java:1504)
at com.zextras.modules.backup.ZEBItem.createZimbraItem (ZEBItem.java:2222)
at com.zextras.modules.backup.ZEBItem.restore (ZEBItem.java:735)
at com.zextras.modules.backup.operations.ZEExternalRestoreOperation.restoreItemsFromExternal (ZEExternalRestoreOperation.java:1124)
at com.zextras.modules.backup.operations.ZEExternalRestoreOperation$1._visit (ZEExternalRestoreOperation.java:886)
at com.zextras.modules.backup.operations.ZEExternalRestoreOperation$1._visit (ZEExternalRestoreOperation.java:870)
at com.zextras.lib.visitors.AbstractVisitor.visit (AbstractVisitor.java:32)
at com.zextras.lib.visitors.MultiThreadVisitor$1.run (MultiThreadVisitor.java:72)
at com.zextras.lib.activities.ThreadLimiter$1.run (ThreadLimiter.java:60)
at com.zextras.lib.activities.ActivityThread.run (ActivityThread.java:138)
I am testing a migration from our old Zimbra server to a new Zimbra installation. The backup source was written to a NFS mount from the new Zimbra server using Zextras as an export. Pretty much exactly as suggested by this forum.
The export ran without error and the restore began fine and ran for a good while before failing. The notification message is below, I also included a Java Heap error that was found in the restore log and not displayed in the error notification.
Any thoughts on what may be the cause? It seems to me I need to increase the memory of the restore process.
DAve
------------------------------------------------
Zimbra 8.8.9_GA_3019.NETWORK
Ubuntu 16 LTS
Running as a VM - 8GB Memory, 4 vCPU
264 accounts
2 domains
6 COS
------------------------------- ERROR FROM JOB NOTIFICATION -------------------------------------------
This is an automated notification from ZxBackup about External Restore.
Operation External Restore failed.
Operation Id: 9b3ae8a5-3769-4001-a036-6f251505fcf5
Operation Host: zimbra2.greenfieldin.org
Operation requested by: admin@zimbra2.greenfieldin.org
Network Modules NG Version: 2.9.0
commit: 423b5076c550c71168b23f8b8cc649363bea2d78
Zal Version: 2.4.0
Zal commit: 93807f76531868754de673a94725162b2b2a0edc
Zimbra version: 8.8.9_GA_3019 20180809160254 20180823-0614 NETWORK
- exception -
com.zextras.lib.Error.OperationBlockingError: Blocking operation error
at com.zextras.modules.backup.operations.ZEExternalRestoreOperation.restoreItemsFromExternal (ZEExternalRestoreOperation.java:1206)
at com.zextras.modules.backup.operations.ZEExternalRestoreOperation$1._visit (ZEExternalRestoreOperation.java:886)
at com.zextras.modules.backup.operations.ZEExternalRestoreOperation$1._visit (ZEExternalRestoreOperation.java:870)
at com.zextras.lib.visitors.AbstractVisitor.visit (AbstractVisitor.java:32)
at com.zextras.lib.visitors.MultiThreadVisitor$1.run (MultiThreadVisitor.java:72)
at com.zextras.lib.activities.ThreadLimiter$1.run (ThreadLimiter.java:60)
at com.zextras.lib.activities.ActivityThread.run (ActivityThread.java:138)
Caused by: java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf (Arrays.java:3236)
at java.io.ByteArrayOutputStream.toByteArray (ByteArrayOutputStream.java:191)
at com.sun.mail.util.ASCIIUtility.getBytes (ASCIIUtility.java:271)
at javax.mail.internet.MimeMessage.parse (MimeMessage.java:339)
at javax.mail.internet.MimeMessage. (MimeMessage.java:209)
at com.zextras.modules.backup.ZEBItem.restoreCalendarItem (ZEBItem.java:1504)
at com.zextras.modules.backup.ZEBItem.createZimbraItem (ZEBItem.java:2222)
at com.zextras.modules.backup.ZEBItem.restore (ZEBItem.java:735)
at com.zextras.modules.backup.operations.ZEExternalRestoreOperation.restoreItemsFromExternal (ZEExternalRestoreOperation.java:1124)
at com.zextras.modules.backup.operations.ZEExternalRestoreOperation$1._visit (ZEExternalRestoreOperation.java:886)
at com.zextras.modules.backup.operations.ZEExternalRestoreOperation$1._visit (ZEExternalRestoreOperation.java:870)
at com.zextras.lib.visitors.AbstractVisitor.visit (AbstractVisitor.java:32)
at com.zextras.lib.visitors.MultiThreadVisitor$1.run (MultiThreadVisitor.java:72)
at com.zextras.lib.activities.ThreadLimiter$1.run (ThreadLimiter.java:60)
at com.zextras.lib.activities.ActivityThread.run (ActivityThread.java:138)
Last progress info:
--
Operation log path: /opt/zimbra/log/op_ExternalRestore_9b3ae8a5-3769-4001-a036-6f251505fcf5.log
Warnings/Errors: 0/0
Restored Accounts(only attributes): 273/273
Restored Accounts(common items): 10/273
Restored Accounts(only shared & datasources): 0/273
Current Account: gdewitt@greenfieldin.org
Restored/Skipped Items: 46690/0
Unrestored Items: 0
------------------------ ERROR FROM RESTORE LOG ----------------------------------
2018-11-14 07:35:57,369 jkoch@greenfieldin.org ERR Exception: java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf (Arrays.java:3236)
at java.io.ByteArrayOutputStream.toByteArray (ByteArrayOutputStream.java:191)
at com.sun.mail.util.ASCIIUtility.getBytes (ASCIIUtility.java:271)
at javax.mail.internet.MimeMessage.parse (MimeMessage.java:339)
at javax.mail.internet.MimeMessage.<init> (MimeMessage.java:209)
at com.zextras.modules.backup.ZEBItem.restoreCalendarItem (ZEBItem.java:1504)
at com.zextras.modules.backup.ZEBItem.createZimbraItem (ZEBItem.java:2222)
at com.zextras.modules.backup.ZEBItem.restore (ZEBItem.java:735)
at com.zextras.modules.backup.operations.ZEExternalRestoreOperation.restoreItemsFromExternal (ZEExternalRestoreOperation.java:1124)
at com.zextras.modules.backup.operations.ZEExternalRestoreOperation$1._visit (ZEExternalRestoreOperation.java:886)
at com.zextras.modules.backup.operations.ZEExternalRestoreOperation$1._visit (ZEExternalRestoreOperation.java:870)
at com.zextras.lib.visitors.AbstractVisitor.visit (AbstractVisitor.java:32)
at com.zextras.lib.visitors.MultiThreadVisitor$1.run (MultiThreadVisitor.java:72)
at com.zextras.lib.activities.ThreadLimiter$1.run (ThreadLimiter.java:60)
at com.zextras.lib.activities.ActivityThread.run (ActivityThread.java:138)