Insane cpu usage by zmmailboxd
Posted: Thu Feb 08, 2018 4:50 pm
Zimbra Release 8.7.6_GA_1776.RHEL7_64_20170326144124 RHEL7_64 FOSS edition.
CentOS Linux release 7.4.1708 (Core)
2 servers (VMs on ESXi - HP DL380G6). Mailstore on VM with 6vCPU and 16Gb RAM. 650 Users, and more or less 350 always active sessions.
Since few months we are facing an huge CPU usage by zmmailboxd and in the last days it becames unsustainable.
CPU and Load average grows up to up over 40/60 due to java starts many threads that nevers ends. CPU usage still grows also during nightime or in the weekend, when the server usage is likely to be near null. The ESXi hosts has 2 socket with 4 cores (8 cores and 16 with HT) but Zimbra is eating all the CPU time.
We followed the guidelines for large deployment and increased the "zimbraHttpNumThreads" up to 1000: It gives us just few hours before the maillog fills up again of messages like this:
2018-02-07 09:23:35,497 WARN [qtp1286783232-91802:http://webmail.ourdomain.it/service/soap/] [ip=172.16.39.222;port=43978;] misc - Exceeded the max requests limit. Suspending org.eclipse.jetty.continuation.Servlet3Continuation@5ce9070e
At this point we can only restart the zmmailboxd: The LA goes down and log became clear but it starts grows again. We grow from 4/5 of LA to > 40 in about 15 hours.
The other machine (4vCPU, 6Gb RAM on another ESXi host with same hardware) own the proxy and MTA role but is absolutley idle (Load average around 0,40 for 15 minutes).
The proxy machine:
zmcontrol status
amavis Running
antispam Running
antivirus Running
dnscache Running
memcached Running
mta Running
proxy Running
snmp Running
stats Running
zmconfigd Running
The mailstore/LDAP server:
zmcontrol status
ldap Running
logger Running
mailbox Running
service webapp Running
snmp Running
spell Running
stats Running
zimbra webapp Running
zimbraAdmin webapp Running
zimlet webapp Running
zmconfigd Running
Please, any suggestions?
CentOS Linux release 7.4.1708 (Core)
2 servers (VMs on ESXi - HP DL380G6). Mailstore on VM with 6vCPU and 16Gb RAM. 650 Users, and more or less 350 always active sessions.
Since few months we are facing an huge CPU usage by zmmailboxd and in the last days it becames unsustainable.
CPU and Load average grows up to up over 40/60 due to java starts many threads that nevers ends. CPU usage still grows also during nightime or in the weekend, when the server usage is likely to be near null. The ESXi hosts has 2 socket with 4 cores (8 cores and 16 with HT) but Zimbra is eating all the CPU time.
We followed the guidelines for large deployment and increased the "zimbraHttpNumThreads" up to 1000: It gives us just few hours before the maillog fills up again of messages like this:
2018-02-07 09:23:35,497 WARN [qtp1286783232-91802:http://webmail.ourdomain.it/service/soap/] [ip=172.16.39.222;port=43978;] misc - Exceeded the max requests limit. Suspending org.eclipse.jetty.continuation.Servlet3Continuation@5ce9070e
At this point we can only restart the zmmailboxd: The LA goes down and log became clear but it starts grows again. We grow from 4/5 of LA to > 40 in about 15 hours.
The other machine (4vCPU, 6Gb RAM on another ESXi host with same hardware) own the proxy and MTA role but is absolutley idle (Load average around 0,40 for 15 minutes).
The proxy machine:
zmcontrol status
amavis Running
antispam Running
antivirus Running
dnscache Running
memcached Running
mta Running
proxy Running
snmp Running
stats Running
zmconfigd Running
The mailstore/LDAP server:
zmcontrol status
ldap Running
logger Running
mailbox Running
service webapp Running
snmp Running
spell Running
stats Running
zimbra webapp Running
zimbraAdmin webapp Running
zimlet webapp Running
zmconfigd Running
Please, any suggestions?