BackupNG fails on restore

Forum to discuss, report issues, or provide feedback about Zimbra Admin Plus
yellowhousejake
Advanced member
Advanced member
Posts: 129
Joined: Tue Sep 09, 2014 9:57 am
ZCS/ZD Version: Release 10.0.1.GA.4518.UBUNTU20_64

BackupNG fails on restore

Post by yellowhousejake »

Good day,

I am testing a migration from our old Zimbra server to a new Zimbra installation. The backup source was written to a NFS mount from the new Zimbra server using Zextras as an export. Pretty much exactly as suggested by this forum.

The export ran without error and the restore began fine and ran for a good while before failing. The notification message is below, I also included a Java Heap error that was found in the restore log and not displayed in the error notification.

Any thoughts on what may be the cause? It seems to me I need to increase the memory of the restore process.

DAve

------------------------------------------------

Zimbra 8.8.9_GA_3019.NETWORK
Ubuntu 16 LTS
Running as a VM - 8GB Memory, 4 vCPU
264 accounts
2 domains
6 COS

------------------------------- ERROR FROM JOB NOTIFICATION -------------------------------------------


This is an automated notification from ZxBackup about External Restore.

Operation External Restore failed.

Operation Id: 9b3ae8a5-3769-4001-a036-6f251505fcf5
Operation Host: zimbra2.greenfieldin.org

Operation requested by: admin@zimbra2.greenfieldin.org

Network Modules NG Version: 2.9.0
commit: 423b5076c550c71168b23f8b8cc649363bea2d78

Zal Version: 2.4.0
Zal commit: 93807f76531868754de673a94725162b2b2a0edc
Zimbra version: 8.8.9_GA_3019 20180809160254 20180823-0614 NETWORK

- exception -
com.zextras.lib.Error.OperationBlockingError: Blocking operation error
at com.zextras.modules.backup.operations.ZEExternalRestoreOperation.restoreItemsFromExternal (ZEExternalRestoreOperation.java:1206)
at com.zextras.modules.backup.operations.ZEExternalRestoreOperation$1._visit (ZEExternalRestoreOperation.java:886)
at com.zextras.modules.backup.operations.ZEExternalRestoreOperation$1._visit (ZEExternalRestoreOperation.java:870)
at com.zextras.lib.visitors.AbstractVisitor.visit (AbstractVisitor.java:32)
at com.zextras.lib.visitors.MultiThreadVisitor$1.run (MultiThreadVisitor.java:72)
at com.zextras.lib.activities.ThreadLimiter$1.run (ThreadLimiter.java:60)
at com.zextras.lib.activities.ActivityThread.run (ActivityThread.java:138)
Caused by: java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf (Arrays.java:3236)
at java.io.ByteArrayOutputStream.toByteArray (ByteArrayOutputStream.java:191)
at com.sun.mail.util.ASCIIUtility.getBytes (ASCIIUtility.java:271)
at javax.mail.internet.MimeMessage.parse (MimeMessage.java:339)
at javax.mail.internet.MimeMessage. (MimeMessage.java:209)
at com.zextras.modules.backup.ZEBItem.restoreCalendarItem (ZEBItem.java:1504)
at com.zextras.modules.backup.ZEBItem.createZimbraItem (ZEBItem.java:2222)
at com.zextras.modules.backup.ZEBItem.restore (ZEBItem.java:735)
at com.zextras.modules.backup.operations.ZEExternalRestoreOperation.restoreItemsFromExternal (ZEExternalRestoreOperation.java:1124)
at com.zextras.modules.backup.operations.ZEExternalRestoreOperation$1._visit (ZEExternalRestoreOperation.java:886)
at com.zextras.modules.backup.operations.ZEExternalRestoreOperation$1._visit (ZEExternalRestoreOperation.java:870)
at com.zextras.lib.visitors.AbstractVisitor.visit (AbstractVisitor.java:32)
at com.zextras.lib.visitors.MultiThreadVisitor$1.run (MultiThreadVisitor.java:72)
at com.zextras.lib.activities.ThreadLimiter$1.run (ThreadLimiter.java:60)
at com.zextras.lib.activities.ActivityThread.run (ActivityThread.java:138)

Last progress info:
--
Operation log path: /opt/zimbra/log/op_ExternalRestore_9b3ae8a5-3769-4001-a036-6f251505fcf5.log
Warnings/Errors: 0/0
Restored Accounts(only attributes): 273/273
Restored Accounts(common items): 10/273
Restored Accounts(only shared & datasources): 0/273
Current Account: gdewitt@greenfieldin.org
Restored/Skipped Items: 46690/0
Unrestored Items: 0

------------------------ ERROR FROM RESTORE LOG ----------------------------------

2018-11-14 07:35:57,369 jkoch@greenfieldin.org ERR Exception: java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf (Arrays.java:3236)
at java.io.ByteArrayOutputStream.toByteArray (ByteArrayOutputStream.java:191)
at com.sun.mail.util.ASCIIUtility.getBytes (ASCIIUtility.java:271)
at javax.mail.internet.MimeMessage.parse (MimeMessage.java:339)
at javax.mail.internet.MimeMessage.<init> (MimeMessage.java:209)
at com.zextras.modules.backup.ZEBItem.restoreCalendarItem (ZEBItem.java:1504)
at com.zextras.modules.backup.ZEBItem.createZimbraItem (ZEBItem.java:2222)
at com.zextras.modules.backup.ZEBItem.restore (ZEBItem.java:735)
at com.zextras.modules.backup.operations.ZEExternalRestoreOperation.restoreItemsFromExternal (ZEExternalRestoreOperation.java:1124)
at com.zextras.modules.backup.operations.ZEExternalRestoreOperation$1._visit (ZEExternalRestoreOperation.java:886)
at com.zextras.modules.backup.operations.ZEExternalRestoreOperation$1._visit (ZEExternalRestoreOperation.java:870)
at com.zextras.lib.visitors.AbstractVisitor.visit (AbstractVisitor.java:32)
at com.zextras.lib.visitors.MultiThreadVisitor$1.run (MultiThreadVisitor.java:72)
at com.zextras.lib.activities.ThreadLimiter$1.run (ThreadLimiter.java:60)
at com.zextras.lib.activities.ActivityThread.run (ActivityThread.java:138)
User avatar
pup_seba
Outstanding Member
Outstanding Member
Posts: 687
Joined: Sat Sep 13, 2014 2:43 am
Location: Tarragona - Spain
Contact:

Re: BackupNG fails on restore

Post by pup_seba »

HI,

Did you see this line?

"OutOfMemoryError: Java heap space"

what's the output of these 2 commands on your new server:
free -g
zmlocalconfig | grep -i heap
yellowhousejake
Advanced member
Advanced member
Posts: 129
Joined: Tue Sep 09, 2014 9:57 am
ZCS/ZD Version: Release 10.0.1.GA.4518.UBUNTU20_64

Re: BackupNG fails on restore

Post by yellowhousejake »

Yep, saw the memory error.

(note that backup NG is not running when I checked free memory)
zimbra@zimbra2:~$ free -g
total used free shared buff/cache available
Mem: 7 2 0 0 5 5
Swap: 0 0 0


zimbra@zimbra2:~$ zmlocalconfig | grep -i heap
imapd_java_heap_new_size_percent = ${mailboxd_java_heap_new_size_percent}
imapd_java_heap_size =
mailboxd_java_heap_new_size_percent = 25
mailboxd_java_heap_size = 256
zimbra_activesync_syncstate_item_cache_heap_size = 10M

DAve
User avatar
pup_seba
Outstanding Member
Outstanding Member
Posts: 687
Joined: Sat Sep 13, 2014 2:43 am
Location: Tarragona - Spain
Contact:

Re: BackupNG fails on restore

Post by pup_seba »

Did you changed manually the size of mailboxd_java_heap_size or did this machine had only 1GB of RAM when zimbra was installed?

That value is way too low. For an 8 GB machine (like your case), that value should be at least 2GB. Then, depending on what other services you have running, you should take care of other memory values.

What is the output of these 2 commands:
zmprov gas
zmcontrol status
yellowhousejake
Advanced member
Advanced member
Posts: 129
Joined: Tue Sep 09, 2014 9:57 am
ZCS/ZD Version: Release 10.0.1.GA.4518.UBUNTU20_64

Re: BackupNG fails on restore

Post by yellowhousejake »

The installation is still a default installation, I have made no adjustments at this point. I believe I created the VM with the current memory and cpu values prior to installation of Zimbra. I generally do.

zimbra@zimbra2:~$ zmprov gas
zimbra2.greenfieldin.org
zimbra@zimbra2:~$ zmcontrol status
Host zimbra2.greenfieldin.org
amavis Running
antispam Running
antivirus Running
convertd Running
dnscache Running
ldap Running
logger Running
mailbox Running
memcached Running
mta Running
opendkim Running
proxy Running
service webapp Running
spell Running
stats Running
zimbra webapp Running
zimbraAdmin webapp Running
zimlet webapp Running
zmconfigd Running
User avatar
JDunphy
Outstanding Member
Outstanding Member
Posts: 889
Joined: Fri Sep 12, 2014 11:18 pm
Location: Victoria, BC
ZCS/ZD Version: 9.0.0_P39 NETWORK Edition

Re: BackupNG fails on restore

Post by JDunphy »

yellowhousejake wrote: ...
Caused by: java.lang.OutOfMemoryError: Java heap space
...
mailboxd_java_heap_size = 256
DAve
This looks too small to me... I generally allocate 30% of memory to the java virtual machine, and 25% to the mysql service for an 8GB instance.

As for contrast, my personal 8GB zimbra server that has 3 users is configured as:

Code: Select all

% zmlocalconfig mailboxd_java_heap_size                         
mailboxd_java_heap_size = 2969
Set it like this: (I think 2458 is recommended value for general purpose 8GB cases if memory serves)

Code: Select all

zmlocalconfig -e mailboxd_java_heap_size=2458
yellowhousejake
Advanced member
Advanced member
Posts: 129
Joined: Tue Sep 09, 2014 9:57 am
ZCS/ZD Version: Release 10.0.1.GA.4518.UBUNTU20_64

Re: BackupNG fails on restore

Post by yellowhousejake »

JDunphy,

I thought the heap was too low as well, and was not surprised f it was the cause. I had thought that the system would automatically set the heap size based on available memory and the size percent?

Looks like I need to set it manually. I can believe that the performance tuning documents are still current?
https://wiki.zimbra.com/wiki/Performanc ... eployments

DAve
User avatar
pup_seba
Outstanding Member
Outstanding Member
Posts: 687
Joined: Sat Sep 13, 2014 2:43 am
Location: Tarragona - Spain
Contact:

Re: BackupNG fails on restore

Post by pup_seba »

That's weird.

Change the heap size to 2048 and the innodb_buffer_pool_size to 3G (you can use this form in /opt/zimbra/conf/my.cnf)

The restart services and give it another try.
yellowhousejake
Advanced member
Advanced member
Posts: 129
Joined: Tue Sep 09, 2014 9:57 am
ZCS/ZD Version: Release 10.0.1.GA.4518.UBUNTU20_64

Re: BackupNG fails on restore

Post by yellowhousejake »

zimbra@zimbra2:~$zmlocalconfig -e mailboxd_java_heap_size=2048

Checked my.conf, found buffer pool at 312mb. Opened in vi and edited to 3G

Restarted services, we will try another restore and I will report back with results.

Thanks!

DAve
User avatar
pup_seba
Outstanding Member
Outstanding Member
Posts: 687
Joined: Sat Sep 13, 2014 2:43 am
Location: Tarragona - Spain
Contact:

Re: BackupNG fails on restore

Post by pup_seba »

Those values are because zimbra was installed when the vm had 1gb of ram. By default during install zimbra qill configure those values at 25(heap) and 30 (innodb buffer) of the configured ram of the vm. Or at least that's the theory of how it should be.

Restore will work just fine now.
Post Reply