Page 1 of 2

load average goes to 80, 90 , 120 for java

Posted: Thu Jul 19, 2018 2:11 pm
by indunil75
Hi,

I have been running zimbra Release 8.6.0_GA_1153.RHEL7_64_20141215151110 RHEL7_64 FOSS edition for long time with 2200 users on CentOS Linux release 7.2.1511. But, today, load average increased suddenly. top command shows this.

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4790 zimbra 20 0 21.829g 3.547g 22068 S 701.7 11.3 1151:20 java


this 4790 process is given below

ps aux |grep 4790

zimbra 4790 1115 11.3 22889732 3719096 ? Sl 17:43 1157:05 /opt/zimbra/java/bin/java -Dfile.encoding=UTF-8 -server -Dhttps.protocols=TLSv1,TLSv1.1,TLSv1.2 -Djdk.tls.client.protocols=TLSv1,TLSv1.1,TLSv1.2 -Djava.awt.headless=true -Dsun.net.inetaddr.ttl=60 -Dorg.apache.jasper.compiler.disablejsr199=true -XX:+UseConcMarkSweepGC -XX:PermSize=128m -XX:MaxPermSize=350m -XX:SoftRefLRUPolicyMSPerMB=1 -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCApplicationStoppedTime -XX:-OmitStackTraceInFastThrow -Xloggc:/opt/zimbra/log/gc.log -XX:-UseGCLogFileRotation -XX:NumberOfGCLogFiles=20 -XX:GCLogFileSize=4096K -Djava.net.preferIPv4Stack=true -Xss256k -Xms4096m -Xmx4096m -Xmn1638m -Djava.io.tmpdir=/opt/zimbra/mailboxd/work -Djava.library.path=/opt/zimbra/lib -Djava.endorsed.dirs=/opt/zimbra/mailboxd/common/endorsed -Dzimbra.config=/opt/zimbra/conf/localconfig.xml -Djetty.home=/opt/zimbra/mailboxd -DSTART=/opt/zimbra/mailboxd/etc/start.config -jar /opt/zimbra/mailboxd/start.jar --module=zimbra,server,servlet,servlets,jsp,jmx,resources,websocket,ext,plus,rewrite,monitor,continuation,webapp,setuid jetty.home=/opt/zimbra/mailboxd /opt/zimbra/mailboxd/etc/jetty.xml

sometimes load average goes to 80, 90 , 120 etc. it never happened before.

everything worked perfectly well until this morning.

here's bit of config settings

zmlocalconfig | grep java_heap

mailboxd_java_heap_new_size_percent = 20
mailboxd_java_heap_size = 6410


zmprov gs `zmhostname` zimbraHttpNumThreads

zimbraHttpNumThreads: 500

and

zmprov gs `zmhostname` zimbraLmtpNumThreads

zimbraLmtpNumThreads: 40


What has gone wrong? Hope to hear from you.

Re: load average goes to 80, 90 , 120 for java

Posted: Thu Jul 19, 2018 3:41 pm
by gabrieles
-what does dmesg says?
-have you mounted some NFS shares? are they alive? df -h does works?

Re: load average goes to 80, 90 , 120 for java

Posted: Thu Jul 19, 2018 5:33 pm
by indunil75
-have you mounted some NFS shares? are they alive? df -h does works?[/quote]

No NFS shares.. df -h works.

Re: load average goes to 80, 90 , 120 for java

Posted: Fri Jul 20, 2018 3:22 am
by indunil75
indunil75 wrote:Hi,
What has gone wrong? Hope to hear from you.
my zimbra version is zimbra Release 8.6.0_GA_1153.RHEL7_64_20141215151110 RHEL7_64 FOSS edition

This URl says ( https://www.zimbra.com/downloads/zimbra-collaboration/ )

Patch-10 has been issued on 10 May 2018 for 8.6.0 GA which includes fixes for reported bugs listed in the release notes.


I want to know if I Installed Patch-10, Does this patch 10 cover all the patches from patch 1 to patch 10 or Do I need to install separately all patches?

Hope to hear from you. is there patch to reduce load average and Cpu usage in zimbra Release 8.6.0? if available, pls update me since it's the most important thing..

Re: load average goes to 80, 90 , 120 for java

Posted: Fri Jul 20, 2018 4:17 am
by indunil75
What about this patch? i.e - Patch 9

the below Url says

All Zimbra Collaboration 8.6.0 sites are recommended to install this patch. Patch 9 is cumulative with all the previous eight patches, so only Patch 9 is required in case you didn’t install the previous ones.

it has this below fix.

CPU load & latency when open mail with data:image/png:base64 inline image

https://blog.zimbra.com/2018/02/zimbra- ... 2017-8802/

Has anyone applied this patch?

Re: load average goes to 80, 90 , 120 for java

Posted: Fri Jul 20, 2018 4:54 am
by phoenix
indunil75 wrote:Has anyone applied this patch?
Why don't you apply it after taking a suitable backup and find out if it solves your problem, after all it is a 'recommended' patch. ;)

Re: load average goes to 80, 90 , 120 for java

Posted: Fri Jul 20, 2018 5:36 am
by indunil75
Why don't you apply it after taking a suitable backup and find out if it solves your problem, after all it is a 'recommended' patch. ;)[/quote]


Thanks for your inputs.

May I ask a few questions before applying the patch?

8.6 Patches can be found here

https://wiki.zimbra.com/wiki/Zimbra_Col ... _-_Patches

patch 9 url

https://files.zimbra.com/downloads/8.6. ... A_1224.tgz

patch 10 url

https://files.zimbra.com/downloads/8.6. ... A_1229.tgz

I think applying patch 10 is also OK since it covers patch 9.

Am I right? Can I go with patch 10?

Is my thinking process OK?

If I am right, I am going to take a back up and apply patch 10 since it covers patch 9.

hope to hear from you.

Re: load average goes to 80, 90 , 120 for java

Posted: Fri Jul 20, 2018 5:43 am
by phoenix
As all the patches are cumulative (it tells you that in the documentation) it would make sense to install the most recent patch.

Re: load average goes to 80, 90 , 120 for java

Posted: Fri Jul 20, 2018 6:22 am
by indunil75
phoenix wrote:As all the patches are cumulative (it tells you that in the documentation) it would make sense to install the most recent patch.
Sir, Thanks for your update.

I will apply patch 10 and UPDATE you. It's very likely to happen on Saturday.

Thanks for your wonderful support.

Have a nice day....

Re: load average goes to 80, 90 , 120 for java

Posted: Fri Jul 20, 2018 9:57 am
by indunil75
phoenix wrote:
indunil75 wrote:Has anyone applied this patch?
Why don't you apply it after taking a suitable backup and find out if it solves your problem, after all it is a 'recommended' patch. ;)
mailbox.log shows this

misc - Exceeded the max requests limit. Suspending org.eclipse.jetty.continuation.Servlet3Continuation@1b66838c

So, I ran below command

zmlocalconfig -e servlet_max_concurrent_http_requests_per_account=20

about 2 hours later, both load average and cpu usage decreased a lot.

Now, load average has gone to 8 , 9

cpu usage (%Cpu) has also gone down.

This is just an update..... Perhaps, It may help zimbra users.

Anyway, I am going to apply patch 10 ASAP.