Suddenly, Horrendously Slow Server - Errors in zmmailboxd.out

guitardood · Post by **guitardood** » Sat May 25, 2019 7:30 am

Hello,
I have a vm server running ZCS Release 8.8.10.GA.3039.UBUNTU16.64 UBUNTU16_64 FOSS edition, Patch 8.8.10_P8 on Ubuntu 16.04.6 LTS with Z-PUSH 2.5.0+0-0 with zimbrabackend68. I have Apache running as the front door door on the server which either reverse proxies to Zimbra's proxy server or runs Z-Push depending on the request.

The ESXi server is a dual 8-core Xeon with 128gb and has 2 NUMA's. It is running the latest patched ESXi 6.5 and the VM is configured with 4-cores and 32gig of memory. The system/Zimbra had been running just fine for the last few months with nary an issue, but this week started exhibiting this bad behavior.

I'm experiencing a huge CPU load on both the primary java process and it's children as well as the mysqld process. Also, I've noticed a huge number of java exceptions in the zmmailboxd.out log file similar to:

Code: Select all

2019-05-25 00:04:38.754:INFO:oejm.ThreadMonitor:Thread-39: Thread 'qtp66233253-10172:https:https://smtp.XXXX.com/home/XXXX@XXXX.com/?fmt=tgz'[RUNNABLE,id:10172,cpu:90.26%] SPINNING
2019-05-25 00:04:38.755:WARN:oejm.ThreadMonitor:Thread-39: 
org.eclipse.jetty.monitor.thread.ThreadMonitorException: Thread 'qtp66233253-10172:https:https://smtp.XXXX.com/home/XXXX@XXXX.com/?fmt=tgz'[RUNNABLE,id:10172,cpu:90.26%] STACK TRACE
        at java.util.zip.Deflater.deflateBytes(Native Method)
        at java.util.zip.Deflater.deflate(Deflater.java:444)
        at java.util.zip.Deflater.deflate(Deflater.java:366)

and

Code: Select all

2019-05-25 01:13:38.224:WARN:oejs.HttpChannel:qtp66233253-10695: https://smtp.XXXX.com:8443/service/soap/
java.lang.NullPointerException
        at org.eclipse.jetty.security.SecurityHandler.checkSecurity(SecurityHandler.java:387)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:443)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1158)

During the day, I'm seeing load averages of 36.0 and ESXi's esxitop is showing this vm using 480% cpu with 0.9% idle. Something is definitely not right. From an ignorant, "I don't know really know what to do" approach, I'm trying a run of "zmdbintegrityreport -o -r -v", hoping that optimizing the database will help.

Any help would be greatly appreciated.

Best,
guitardood