java.lang.OutOfMemoryError

Discuss your pilot or production implementation with other Zimbra admins or our engineers.
babitts
Posts: 8
Joined: Tue May 28, 2019 3:49 am

java.lang.OutOfMemoryError

Postby babitts » Fri Apr 03, 2020 4:58 pm

Hi,

We are having an error on zimbra Version 8.8.10, the access via webmail and admin are down and in the log /opt/zimbra/log/zmmailboxd.out it shows for example


Exception in thread "FileLogWriter.FsyncThread-1585931471909" java.lang.OutOfMemoryError: Java heap space
Exception in thread "mailboxd.csv" java.lang.OutOfMemoryError: Java heap space
Exception in thread "Health Check Thread for LDAPConnectionPool(serverSet=SingleServerSet(server=XX.XX:389), maxConnections=50)" java.lang.OutOfMemoryError: Java heap space
OpenJDK 64-Bit Server VM warning: .hotspot_compiler file is present but has been ignored. Run with -XX:CompileCommandFile=.hotspot_compiler to load the file.


2020-04-03 13:40:17.597:WARN:oejs.ServletHandler:qtp1286783232-220:https:https://XX.XX/service/home/~/?auth=co&loc=es_ES&id=33280&part=2.4&t=1585931650414: Error for /service/home/~/
java.lang.OutOfMemoryError: Java heap space


Exception in thread "imap.csv" java.lang.OutOfMemoryError: Java heap space
2020-04-03 13:45:44.404:WARN:oejs.HttpChannel:qtp1286783232-187: https://XX.XXl/service/soap/ZxChatRequest


Exception in thread "threads.csv" java.lang.OutOfMemoryError: Java heap space
Exception in thread "mailboxd.csv"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "NioProcessor-0"


Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "pool-11-thread-1"
OpenJDK 64-Bit Server VM warning: .hotspot_compiler file is present but has been ignored. Run with -XX:CompileCommandFile=.hotspot_compiler to load the file.



We have incresead the memory for java by (the server has 12G of RAM) but the issue continues


zmlocalconfig -e zimbra_zmjava_options="-Xmx2048m -Djava.net.preferIPv4Stack=true"



Thanks in advance


User avatar
L. Mark Stone
Elite member
Elite member
Posts: 2141
Joined: Wed Oct 09, 2013 11:35 am
Location: Portland, Maine, US
ZCS/ZD Version: 8.8.15 Network Edition
Contact:

Re: java.lang.OutOfMemoryError

Postby L. Mark Stone » Fri Apr 03, 2020 5:50 pm

I would recommend restoring the LC zimbra_zmjava_options settings to as it was originally.

Please post the output from running the command:

Code: Select all

 zmlocalconfig mailboxd_java_heap_size


Please also consider upgrading (once Zimbra is running OK) to a supported version, currently 8.8.15 Patch 8.

Once you are on 8.8.15, you can check that your LC mailboxd_java_options are up to date; you should see:

Code: Select all

zmlocalconfig mailboxd_java_options
mailboxd_java_options = -server -Dhttps.protocols=TLSv1,TLSv1.1,TLSv1.2 -Djdk.tls.client.protocols=TLSv1,TLSv1.1,TLSv1.2 -Djava.awt.headless=true -Dsun.net.inetaddr.ttl=${networkaddress_cache_ttl} -Dorg.apache.jasper.compiler.disablejsr199=true -XX:+UseG1GC -XX:SoftRefLRUPolicyMSPerMB=1 -XX:+UnlockExperimentalVMOptions -XX:G1NewSizePercent=15 -XX:G1MaxNewSizePercent=45 -XX:-OmitStackTraceInFastThrow -verbose:gc -Xlog:gc*=info,safepoint=info:file=/opt/zimbra/log/gc.log:time:filecount=20,filesize=10m -Djava.net.preferIPv4Stack=true


Again, do NOT use the above settings until you upgrade from 8.8.10; the above settings were appropriate to use only from 8.8.12 onwards.

Mark
___________________________________
L. Mark Stone
Mission Critical Email - Zimbra VAR/BSP/Training Partner https://www.missioncriticalemail.com/
Zeta Alliance http://www.zetalliance.org/
babitts
Posts: 8
Joined: Tue May 28, 2019 3:49 am

Re: java.lang.OutOfMemoryError

Postby babitts » Fri Apr 03, 2020 6:04 pm

Thanks for your reply, the command shows

mailboxd_java_heap_size = 460


Is there any risk to update the ZCS version, and is it posible to do something without updating?

Thanks in advance
User avatar
L. Mark Stone
Elite member
Elite member
Posts: 2141
Joined: Wed Oct 09, 2013 11:35 am
Location: Portland, Maine, US
ZCS/ZD Version: 8.8.15 Network Edition
Contact:

Re: java.lang.OutOfMemoryError

Postby L. Mark Stone » Fri Apr 03, 2020 7:03 pm

babitts wrote:Thanks for your reply, the command shows

mailboxd_java_heap_size = 460


Is there any risk to update the ZCS version, and is it posible to do something without updating?

Thanks in advance


mailboxd_java_heap_size is set by the Zimbra installer to be equal to 30 percent of system RAM at the time the installer is run. Zimbra requires a minimum of 8GB (and IMHO 16GB is much better), so looking at your mailboxd_java_heap_size setting to me says you initially installed Zimbra with way too little RAM.

Similarly, the innodb_buffer_pool_size setting in ~/conf/my.cnf is also configured during the initial install as a percentage of RAM. I would suspect you have a good chance that your innodb database size exceeds the buffer pool size, which can lead to memory and disk thrashing. You can run (as the Zimbra user) a tool like mysqltuner.pl to compare the size of your innodb databases against the innodb buffer pool; the buffer pool should be ~1.25x the size of your databases.

Suggest you configure the server with 16GB RAM and then increase the mailboxd_java_heap_size to 4192 as a start.

Hope that helps,
Mark
___________________________________
L. Mark Stone
Mission Critical Email - Zimbra VAR/BSP/Training Partner https://www.missioncriticalemail.com/
Zeta Alliance http://www.zetalliance.org/

Return to “Administrators”

Who is online

Users browsing this forum: No registered users and 9 guests