load average goes to 80, 90 , 120 for java

Post feedback about our hosted demo or your local install. Tell us what you love and/or what you’d like to see added in the future.
indunil75
Advanced member
Advanced member
Posts: 97
Joined: Sat Sep 13, 2014 12:35 am

load average goes to 80, 90 , 120 for java

Post by indunil75 »

Hi,

I have been running zimbra Release 8.6.0_GA_1153.RHEL7_64_20141215151110 RHEL7_64 FOSS edition for long time with 2200 users on CentOS Linux release 7.2.1511. But, today, load average increased suddenly. top command shows this.

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4790 zimbra 20 0 21.829g 3.547g 22068 S 701.7 11.3 1151:20 java


this 4790 process is given below

ps aux |grep 4790

zimbra 4790 1115 11.3 22889732 3719096 ? Sl 17:43 1157:05 /opt/zimbra/java/bin/java -Dfile.encoding=UTF-8 -server -Dhttps.protocols=TLSv1,TLSv1.1,TLSv1.2 -Djdk.tls.client.protocols=TLSv1,TLSv1.1,TLSv1.2 -Djava.awt.headless=true -Dsun.net.inetaddr.ttl=60 -Dorg.apache.jasper.compiler.disablejsr199=true -XX:+UseConcMarkSweepGC -XX:PermSize=128m -XX:MaxPermSize=350m -XX:SoftRefLRUPolicyMSPerMB=1 -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCApplicationStoppedTime -XX:-OmitStackTraceInFastThrow -Xloggc:/opt/zimbra/log/gc.log -XX:-UseGCLogFileRotation -XX:NumberOfGCLogFiles=20 -XX:GCLogFileSize=4096K -Djava.net.preferIPv4Stack=true -Xss256k -Xms4096m -Xmx4096m -Xmn1638m -Djava.io.tmpdir=/opt/zimbra/mailboxd/work -Djava.library.path=/opt/zimbra/lib -Djava.endorsed.dirs=/opt/zimbra/mailboxd/common/endorsed -Dzimbra.config=/opt/zimbra/conf/localconfig.xml -Djetty.home=/opt/zimbra/mailboxd -DSTART=/opt/zimbra/mailboxd/etc/start.config -jar /opt/zimbra/mailboxd/start.jar --module=zimbra,server,servlet,servlets,jsp,jmx,resources,websocket,ext,plus,rewrite,monitor,continuation,webapp,setuid jetty.home=/opt/zimbra/mailboxd /opt/zimbra/mailboxd/etc/jetty.xml

sometimes load average goes to 80, 90 , 120 etc. it never happened before.

everything worked perfectly well until this morning.

here's bit of config settings

zmlocalconfig | grep java_heap

mailboxd_java_heap_new_size_percent = 20
mailboxd_java_heap_size = 6410


zmprov gs `zmhostname` zimbraHttpNumThreads

zimbraHttpNumThreads: 500

and

zmprov gs `zmhostname` zimbraLmtpNumThreads

zimbraLmtpNumThreads: 40


What has gone wrong? Hope to hear from you.
User avatar
gabrieles
Outstanding Member
Outstanding Member
Posts: 233
Joined: Tue Feb 14, 2017 9:40 am

Re: load average goes to 80, 90 , 120 for java

Post by gabrieles »

-what does dmesg says?
-have you mounted some NFS shares? are they alive? df -h does works?
indunil75
Advanced member
Advanced member
Posts: 97
Joined: Sat Sep 13, 2014 12:35 am

Re: load average goes to 80, 90 , 120 for java

Post by indunil75 »

-have you mounted some NFS shares? are they alive? df -h does works?[/quote]

No NFS shares.. df -h works.
indunil75
Advanced member
Advanced member
Posts: 97
Joined: Sat Sep 13, 2014 12:35 am

Re: load average goes to 80, 90 , 120 for java

Post by indunil75 »

indunil75 wrote:Hi,
What has gone wrong? Hope to hear from you.
my zimbra version is zimbra Release 8.6.0_GA_1153.RHEL7_64_20141215151110 RHEL7_64 FOSS edition

This URl says ( https://www.zimbra.com/downloads/zimbra-collaboration/ )

Patch-10 has been issued on 10 May 2018 for 8.6.0 GA which includes fixes for reported bugs listed in the release notes.


I want to know if I Installed Patch-10, Does this patch 10 cover all the patches from patch 1 to patch 10 or Do I need to install separately all patches?

Hope to hear from you. is there patch to reduce load average and Cpu usage in zimbra Release 8.6.0? if available, pls update me since it's the most important thing..
indunil75
Advanced member
Advanced member
Posts: 97
Joined: Sat Sep 13, 2014 12:35 am

Re: load average goes to 80, 90 , 120 for java

Post by indunil75 »

What about this patch? i.e - Patch 9

the below Url says

All Zimbra Collaboration 8.6.0 sites are recommended to install this patch. Patch 9 is cumulative with all the previous eight patches, so only Patch 9 is required in case you didn’t install the previous ones.

it has this below fix.

CPU load & latency when open mail with data:image/png:base64 inline image

https://blog.zimbra.com/2018/02/zimbra- ... 2017-8802/

Has anyone applied this patch?
phoenix
Ambassador
Ambassador
Posts: 27272
Joined: Fri Sep 12, 2014 9:56 pm
Location: Liverpool, England

Re: load average goes to 80, 90 , 120 for java

Post by phoenix »

indunil75 wrote:Has anyone applied this patch?
Why don't you apply it after taking a suitable backup and find out if it solves your problem, after all it is a 'recommended' patch. ;)
Regards

Bill

Rspamd: A high performance spamassassin replacement

Per ardua ad astra
indunil75
Advanced member
Advanced member
Posts: 97
Joined: Sat Sep 13, 2014 12:35 am

Re: load average goes to 80, 90 , 120 for java

Post by indunil75 »

Why don't you apply it after taking a suitable backup and find out if it solves your problem, after all it is a 'recommended' patch. ;)[/quote]


Thanks for your inputs.

May I ask a few questions before applying the patch?

8.6 Patches can be found here

https://wiki.zimbra.com/wiki/Zimbra_Col ... _-_Patches

patch 9 url

https://files.zimbra.com/downloads/8.6. ... A_1224.tgz

patch 10 url

https://files.zimbra.com/downloads/8.6. ... A_1229.tgz

I think applying patch 10 is also OK since it covers patch 9.

Am I right? Can I go with patch 10?

Is my thinking process OK?

If I am right, I am going to take a back up and apply patch 10 since it covers patch 9.

hope to hear from you.
phoenix
Ambassador
Ambassador
Posts: 27272
Joined: Fri Sep 12, 2014 9:56 pm
Location: Liverpool, England

Re: load average goes to 80, 90 , 120 for java

Post by phoenix »

As all the patches are cumulative (it tells you that in the documentation) it would make sense to install the most recent patch.
Regards

Bill

Rspamd: A high performance spamassassin replacement

Per ardua ad astra
indunil75
Advanced member
Advanced member
Posts: 97
Joined: Sat Sep 13, 2014 12:35 am

Re: load average goes to 80, 90 , 120 for java

Post by indunil75 »

phoenix wrote:As all the patches are cumulative (it tells you that in the documentation) it would make sense to install the most recent patch.
Sir, Thanks for your update.

I will apply patch 10 and UPDATE you. It's very likely to happen on Saturday.

Thanks for your wonderful support.

Have a nice day....
indunil75
Advanced member
Advanced member
Posts: 97
Joined: Sat Sep 13, 2014 12:35 am

Re: load average goes to 80, 90 , 120 for java

Post by indunil75 »

phoenix wrote:
indunil75 wrote:Has anyone applied this patch?
Why don't you apply it after taking a suitable backup and find out if it solves your problem, after all it is a 'recommended' patch. ;)
mailbox.log shows this

misc - Exceeded the max requests limit. Suspending org.eclipse.jetty.continuation.Servlet3Continuation@1b66838c

So, I ran below command

zmlocalconfig -e servlet_max_concurrent_http_requests_per_account=20

about 2 hours later, both load average and cpu usage decreased a lot.

Now, load average has gone to 8 , 9

cpu usage (%Cpu) has also gone down.

This is just an update..... Perhaps, It may help zimbra users.

Anyway, I am going to apply patch 10 ASAP.
Post Reply