java.lang.OutOfMemoryError

Discuss your pilot or production implementation with other Zimbra admins or our engineers.
Post Reply
babitts
Posts: 19
Joined: Tue May 28, 2019 3:49 am

java.lang.OutOfMemoryError

Post by babitts »

Hi,

We are having an error on zimbra Version 8.8.10, the access via webmail and admin are down and in the log /opt/zimbra/log/zmmailboxd.out it shows for example


Exception in thread "FileLogWriter.FsyncThread-1585931471909" java.lang.OutOfMemoryError: Java heap space
Exception in thread "mailboxd.csv" java.lang.OutOfMemoryError: Java heap space
Exception in thread "Health Check Thread for LDAPConnectionPool(serverSet=SingleServerSet(server=XX.XX:389), maxConnections=50)" java.lang.OutOfMemoryError: Java heap space
OpenJDK 64-Bit Server VM warning: .hotspot_compiler file is present but has been ignored. Run with -XX:CompileCommandFile=.hotspot_compiler to load the file.


2020-04-03 13:40:17.597:WARN:oejs.ServletHandler:qtp1286783232-220:https:https://XX.XX/service/home/~/?auth=co&loc=es_ES&id=33280&part=2.4&t=1585931650414: Error for /service/home/~/
java.lang.OutOfMemoryError: Java heap space


Exception in thread "imap.csv" java.lang.OutOfMemoryError: Java heap space
2020-04-03 13:45:44.404:WARN:oejs.HttpChannel:qtp1286783232-187: https://XX.XXl/service/soap/ZxChatRequest


Exception in thread "threads.csv" java.lang.OutOfMemoryError: Java heap space
Exception in thread "mailboxd.csv"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "NioProcessor-0"


Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "pool-11-thread-1"
OpenJDK 64-Bit Server VM warning: .hotspot_compiler file is present but has been ignored. Run with -XX:CompileCommandFile=.hotspot_compiler to load the file.



We have incresead the memory for java by (the server has 12G of RAM) but the issue continues


zmlocalconfig -e zimbra_zmjava_options="-Xmx2048m -Djava.net.preferIPv4Stack=true"



Thanks in advance
User avatar
L. Mark Stone
Ambassador
Ambassador
Posts: 2796
Joined: Wed Oct 09, 2013 11:35 am
Location: Portland, Maine, US
ZCS/ZD Version: 10.0.6 Network Edition
Contact:

Re: java.lang.OutOfMemoryError

Post by L. Mark Stone »

I would recommend restoring the LC zimbra_zmjava_options settings to as it was originally.

Please post the output from running the command:

Code: Select all

 zmlocalconfig mailboxd_java_heap_size
Please also consider upgrading (once Zimbra is running OK) to a supported version, currently 8.8.15 Patch 8.

Once you are on 8.8.15, you can check that your LC mailboxd_java_options are up to date; you should see:

Code: Select all

zmlocalconfig mailboxd_java_options
mailboxd_java_options = -server -Dhttps.protocols=TLSv1,TLSv1.1,TLSv1.2 -Djdk.tls.client.protocols=TLSv1,TLSv1.1,TLSv1.2 -Djava.awt.headless=true -Dsun.net.inetaddr.ttl=${networkaddress_cache_ttl} -Dorg.apache.jasper.compiler.disablejsr199=true -XX:+UseG1GC -XX:SoftRefLRUPolicyMSPerMB=1 -XX:+UnlockExperimentalVMOptions -XX:G1NewSizePercent=15 -XX:G1MaxNewSizePercent=45 -XX:-OmitStackTraceInFastThrow -verbose:gc -Xlog:gc*=info,safepoint=info:file=/opt/zimbra/log/gc.log:time:filecount=20,filesize=10m -Djava.net.preferIPv4Stack=true
Again, do NOT use the above settings until you upgrade from 8.8.10; the above settings were appropriate to use only from 8.8.12 onwards.

Mark
___________________________________
L. Mark Stone
Mission Critical Email - Zimbra VAR/BSP/Training Partner https://www.missioncriticalemail.com/
AWS Certified Solutions Architect-Associate
babitts
Posts: 19
Joined: Tue May 28, 2019 3:49 am

Re: java.lang.OutOfMemoryError

Post by babitts »

Thanks for your reply, the command shows

mailboxd_java_heap_size = 460


Is there any risk to update the ZCS version, and is it posible to do something without updating?

Thanks in advance
User avatar
L. Mark Stone
Ambassador
Ambassador
Posts: 2796
Joined: Wed Oct 09, 2013 11:35 am
Location: Portland, Maine, US
ZCS/ZD Version: 10.0.6 Network Edition
Contact:

Re: java.lang.OutOfMemoryError

Post by L. Mark Stone »

babitts wrote:Thanks for your reply, the command shows

mailboxd_java_heap_size = 460


Is there any risk to update the ZCS version, and is it posible to do something without updating?

Thanks in advance
mailboxd_java_heap_size is set by the Zimbra installer to be equal to 30 percent of system RAM at the time the installer is run. Zimbra requires a minimum of 8GB (and IMHO 16GB is much better), so looking at your mailboxd_java_heap_size setting to me says you initially installed Zimbra with way too little RAM.

Similarly, the innodb_buffer_pool_size setting in ~/conf/my.cnf is also configured during the initial install as a percentage of RAM. I would suspect you have a good chance that your innodb database size exceeds the buffer pool size, which can lead to memory and disk thrashing. You can run (as the Zimbra user) a tool like mysqltuner.pl to compare the size of your innodb databases against the innodb buffer pool; the buffer pool should be ~1.25x the size of your databases.

Suggest you configure the server with 16GB RAM and then increase the mailboxd_java_heap_size to 4192 as a start.

Hope that helps,
Mark
___________________________________
L. Mark Stone
Mission Critical Email - Zimbra VAR/BSP/Training Partner https://www.missioncriticalemail.com/
AWS Certified Solutions Architect-Associate
williamholding
Posts: 1
Joined: Tue Sep 14, 2021 6:36 am

Re: java.lang.OutOfMemoryError

Post by williamholding »

The java.lang.OutOfMemoryError means that your program needs more memory than your Java Virtual Machine (JVM) allowed it to use.

How to Track the error?

Increase the default memory your program is allowed to use using the -Xmx option (for instance for 1024 MB: -Xmx1024m). By default, the values are based on the JRE version and system configuration. NOTE: Increasing the heap size is a bad solution, 100% temporary, because you will hit the same issue if you get several parallel requests or when you try to process a bigger file.

Find the root cause of memory leaks with help of profiling tools like MAT, Visual VM , jconsole etc. Once you find the root cause, You can fix this memory leaks.

Optimize your code so that it needs less memory, using less big data structures and getting rid of objects that are not any more used at some point in your program.

How to avoid this issue?

Use local variables wherever possible.
Release those objects which you think shall not be needed further.
Avoid creation of objects in your loop each time.
Try to use caches.
Try to move with Multy Threading.
User avatar
L. Mark Stone
Ambassador
Ambassador
Posts: 2796
Joined: Wed Oct 09, 2013 11:35 am
Location: Portland, Maine, US
ZCS/ZD Version: 10.0.6 Network Edition
Contact:

Re: java.lang.OutOfMemoryError

Post by L. Mark Stone »

Hi William and welcome to the Zimbra forums!

Not sure if you saw that this thread is more than a year old, but also current versions of Zimbra use Java 13, and so no longer specify the Xmx option:

Code: Select all

zimbra@mb21:~$ zmlocalconfig mailboxd_java_options
mailboxd_java_options = -server -Dhttps.protocols=TLSv1,TLSv1.1,TLSv1.2,TLSv1.3 -Djdk.tls.client.protocols=TLSv1,TLSv1.1,TLSv1.2 -Djava.awt.headless=true -Dsun.net.inetaddr.ttl=${networkaddress_cache_ttl} -Dorg.apache.jasper.compiler.disablejsr199=true -XX:+UseG1GC -XX:SoftRefLRUPolicyMSPerMB=1 -XX:+UnlockExperimentalVMOptions -XX:G1NewSizePercent=15 -XX:G1MaxNewSizePercent=45 -XX:-OmitStackTraceInFastThrow -verbose:gc -Xlog:gc*=info,safepoint=info:file=/opt/zimbra/log/gc.log:time:filecount=20,filesize=10m -Djava.net.preferIPv4Stack=true
zimbra@mb21:~$ 
Depending on the patch level, admins may have modified the above (check the release notes) to implement TLS 1.3 and remove TLS 1 and 1.1 as well.

All the best,
Mark
___________________________________
L. Mark Stone
Mission Critical Email - Zimbra VAR/BSP/Training Partner https://www.missioncriticalemail.com/
AWS Certified Solutions Architect-Associate
Post Reply