I have a vm server running ZCS Release 8.8.10.GA.3039.UBUNTU16.64 UBUNTU16_64 FOSS edition, Patch 8.8.10_P8 on Ubuntu 16.04.6 LTS with Z-PUSH 2.5.0+0-0 with zimbrabackend68. I have Apache running as the front door door on the server which either reverse proxies to Zimbra's proxy server or runs Z-Push depending on the request.
The ESXi server is a dual 8-core Xeon with 128gb and has 2 NUMA's. It is running the latest patched ESXi 6.5 and the VM is configured with 4-cores and 32gig of memory. The system/Zimbra had been running just fine for the last few months with nary an issue, but this week started exhibiting this bad behavior.
I'm experiencing a huge CPU load on both the primary java process and it's children as well as the mysqld process. Also, I've noticed a huge number of java exceptions in the zmmailboxd.out log file similar to:
Code: Select all
2019-05-25 00:04:38.754:INFO:oejm.ThreadMonitor:Thread-39: Thread 'qtp66233253-10172:https:https://smtp.XXXX.com/home/XXXX@XXXX.com/?fmt=tgz'[RUNNABLE,id:10172,cpu:90.26%] SPINNING
2019-05-25 00:04:38.755:WARN:oejm.ThreadMonitor:Thread-39:
org.eclipse.jetty.monitor.thread.ThreadMonitorException: Thread 'qtp66233253-10172:https:https://smtp.XXXX.com/home/XXXX@XXXX.com/?fmt=tgz'[RUNNABLE,id:10172,cpu:90.26%] STACK TRACE
at java.util.zip.Deflater.deflateBytes(Native Method)
at java.util.zip.Deflater.deflate(Deflater.java:444)
at java.util.zip.Deflater.deflate(Deflater.java:366)
Code: Select all
2019-05-25 01:13:38.224:WARN:oejs.HttpChannel:qtp66233253-10695: https://smtp.XXXX.com:8443/service/soap/
java.lang.NullPointerException
at org.eclipse.jetty.security.SecurityHandler.checkSecurity(SecurityHandler.java:387)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:443)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1158)
Any help would be greatly appreciated.
Best,
guitardood