Huge CPU Java
Huge CPU Java
I have a 8.8.15 P33 installation and experiencing a huge CPU usage
Does anyone experiencing this?
Any solution would be much appreciated.
Does anyone experiencing this?
Any solution would be much appreciated.
Re: Huge CPU Java
Hey weblike. I've been experiencing this problem for several months now and can't find a solution on my own. For all available logs stored along the /opt/zimbra/log path, they do not give a direction vector in the search for a solution. Judging by the search for this thread, the problem is systematic, and there are many topics where the java process consumes 100 percent or more of server resources, and no one offers an unambiguous solution, even developers.weblike wrote:I have a 8.8.15 P33 installation and experiencing a huge CPU usage
Does anyone experiencing this?
Any solution would be much appreciated.
If you happen to find a solution, let me know.
Re: Huge CPU Java
This solution worked for me. I have followed the topic for large deployments https://wiki.zimbra.com/wiki/Performanc ... eployments
After applying recommendations in this article, users does not experience issues anymore.
what I want to optimize further is MySql performance (I see CPU stress from MySQL frequently) ...For this case I want to try this script https://github.com/major/MySQLTuner-perl (I mention that I still study this part). Further more I don't want to make many changes on the server at once...I want to give a time for the first phase and to see how user a experiencing the usability of the web-client.
Hope this helps
After applying recommendations in this article, users does not experience issues anymore.
what I want to optimize further is MySql performance (I see CPU stress from MySQL frequently) ...For this case I want to try this script https://github.com/major/MySQLTuner-perl (I mention that I still study this part). Further more I don't want to make many changes on the server at once...I want to give a time for the first phase and to see how user a experiencing the usability of the web-client.
Hope this helps
Re: Huge CPU Java
Thank you for keeping up to date. Please tell me, from all the points of that article, will it be enough to apply the "Recommended Options" settings from the "JVM Options" section? In order to reduce the consumption of java.weblike wrote:This solution worked for me. I have followed the topic for large deployments https://wiki.zimbra.com/wiki/Performanc ... eployments
After applying recommendations in this article, users does not experience issues anymore.
- L. Mark Stone
- Ambassador
- Posts: 2799
- Joined: Wed Oct 09, 2013 11:35 am
- Location: Portland, Maine, US
- ZCS/ZD Version: 10.0.7 Network Edition
- Contact:
Re: Huge CPU Java
Much of the data in that article is for older versions of Zimbra, and a number of the tuning suggestions are now the default.Se7enPRO wrote:Thank you for keeping up to date. Please tell me, from all the points of that article, will it be enough to apply the "Recommended Options" settings from the "JVM Options" section? In order to reduce the consumption of java.weblike wrote:This solution worked for me. I have followed the topic for large deployments https://wiki.zimbra.com/wiki/Performanc ... eployments
After applying recommendations in this article, users does not experience issues anymore.
I think you would be better off looking for the root cause, rather than just trying some changes from that article, many of which are specific to older versions of Zimbra.
So first, let's see what the Garbage Collector is doing. Please post 20 concurrent lines from the output of the following command run as the Zimbra user:
Code: Select all
tail -f ~/log/zmmailboxd.out | grep -v "at \|PROCESSING\|native formatter failure\|Broken pipe"
Lastly, please tell us how many mailboxes you have, approximately how big the mail store is, and the output from this command, also run as the Zimbra user:
Code: Select all
cat ~/conf/my.cnf | grep innodb_buffer_pool_size
Hope that helps,
Mark
___________________________________
L. Mark Stone
Mission Critical Email - Zimbra VAR/BSP/Training Partner https://www.missioncriticalemail.com/
AWS Certified Solutions Architect-Associate
L. Mark Stone
Mission Critical Email - Zimbra VAR/BSP/Training Partner https://www.missioncriticalemail.com/
AWS Certified Solutions Architect-Associate
Re: Huge CPU Java
Here is the result of this command both under Zimbra user and root userL. Mark Stone wrote:So first, let's see what the Garbage Collector is doing. Please post 20 concurrent lines from the output of the following command run as the Zimbra user:
Next, please run "atop" as the root user and attach a screencap.Code: Select all
tail -f ~/log/zmmailboxd.out | grep -v "at \|PROCESSING\|native formatter failure\|Broken pipe"
Number of mailboxes: 560L. Mark Stone wrote:Lastly, please tell us how many mailboxes you have, approximately how big the mail store is, and the output from this command, also run as the Zimbra user:Code: Select all
cat ~/conf/my.cnf | grep innodb_buffer_pool_size
Mail storage size: 276 GB
Code: Select all
zimbra@zbs:~$ cat ~/conf/my.cnf | grep innodb_buffer_pool_size
innodb_buffer_pool_size = 1240429363
Code: Select all
root@zbs:~# cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=16.04
DISTRIB_CODENAME=xenial
DISTRIB_DESCRIPTION="Ubuntu 16.04.2 LTS"
Code: Select all
zimbra@zbs:~$ zmcontrol -v
Release 8.8.4.BETA.1158.UBUNTU16.64 UBUNTU16_64 FOSS edition.
- L. Mark Stone
- Ambassador
- Posts: 2799
- Joined: Wed Oct 09, 2013 11:35 am
- Location: Portland, Maine, US
- ZCS/ZD Version: 10.0.7 Network Edition
- Contact:
Re: Huge CPU Java
You are running an older version of Zimbra with known security exploits. I would recommend you immediately perform a side-by-side migration to a current 8.8.15 version of Zimbra, ideally on a newer operating system.
FWIW, the tail command of zmmailboxd.out as wriiten only works as the Zimbra user, and what I was looking for was lines like this:
I need about 20 of those kinds of lines. Alternatively, you could run:
But if your CPU is pegged inexplicably, I suspect your server has been exploited.
FWIW, the tail command of zmmailboxd.out as wriiten only works as the Zimbra user, and what I was looking for was lines like this:
Code: Select all
[506804.442s][info][gc] GC(45684) Pause Young (Prepare Mixed) (G1 Evacuation Pause) 5863M->1200M(8192M) 42.326ms
[506808.598s][info][gc] GC(45685) Pause Young (Mixed) (G1 Evacuation Pause) 3268M->1134M(8192M) 36.974ms
[506820.899s][info][gc] GC(45686) Pause Young (Normal) (G1 Evacuation Pause) 5882M->1223M(8192M) 47.275ms
[506821.063s][info][gc] GC(45687) Pause Young (Concurrent Start) (G1 Humongous Allocation) 1301M->1104M(8192M) 19.245ms
[506821.063s][info][gc] GC(45688) Concurrent Mark Cycle
[506821.517s][info][gc] GC(45688) Pause Remark 1231M->1231M(8192M) 34.825ms
[506821.770s][info][gc] GC(45688) Pause Cleanup 1320M->1320M(8192M) 0.383ms
[506821.777s][info][gc] GC(45688) Concurrent Mark Cycle 713.174ms
Code: Select all
tail -f ~/log/zmmailboxd.out | grep "\[info\]\[gc\]"
___________________________________
L. Mark Stone
Mission Critical Email - Zimbra VAR/BSP/Training Partner https://www.missioncriticalemail.com/
AWS Certified Solutions Architect-Associate
L. Mark Stone
Mission Critical Email - Zimbra VAR/BSP/Training Partner https://www.missioncriticalemail.com/
AWS Certified Solutions Architect-Associate
Re: Huge CPU Java
I have a high CPU consumption of a java process after some time. And when I notice this through "htop", I highlight this process, press F9 and then Enter. After a while, consumption returns to normal.
I promise to provide you with the output of that command as soon as consumption rises again.
Regarding migration: Where can I find step-by-step instructions for correctly migrating to a new version with a complete transfer of mailboxes ?
And if there is a suspicion that I might have been hacked, would it be enough to change the passwords for the database ? This refers to the "mysql_root_password" and "zimbra_mysql_password" parameters.
I promise to provide you with the output of that command as soon as consumption rises again.
Regarding migration: Where can I find step-by-step instructions for correctly migrating to a new version with a complete transfer of mailboxes ?
And if there is a suspicion that I might have been hacked, would it be enough to change the passwords for the database ? This refers to the "mysql_root_password" and "zimbra_mysql_password" parameters.
- L. Mark Stone
- Ambassador
- Posts: 2799
- Joined: Wed Oct 09, 2013 11:35 am
- Location: Portland, Maine, US
- ZCS/ZD Version: 10.0.7 Network Edition
- Contact:
Re: Huge CPU Java
If your server has been breached with the current exploit, the attacker likely already has root shell access. You need to move a current supported and patched version of Zimbra ASAP. Data loss is possible.
There are plenty of guides as to how to do this with the OSE version of Zimbra. For example, you can use IMAPSYNC to copy the emails over, and then export the user's calendars and contacts. and import them into the new server.
You can also use for example:
https://rants.tech/migrating-opensource ... -downtime/
or
https://syslint.com/blog/tutorial/zimbr ... ct-method/
There are plenty of guides as to how to do this with the OSE version of Zimbra. For example, you can use IMAPSYNC to copy the emails over, and then export the user's calendars and contacts. and import them into the new server.
You can also use for example:
https://rants.tech/migrating-opensource ... -downtime/
or
https://syslint.com/blog/tutorial/zimbr ... ct-method/
___________________________________
L. Mark Stone
Mission Critical Email - Zimbra VAR/BSP/Training Partner https://www.missioncriticalemail.com/
AWS Certified Solutions Architect-Associate
L. Mark Stone
Mission Critical Email - Zimbra VAR/BSP/Training Partner https://www.missioncriticalemail.com/
AWS Certified Solutions Architect-Associate
Re: Huge CPU Java
I understand all the responsibility and risks, thanks for the links to the manuals.
Regarding checking the Garbage Collector: as soon as there is a high CPU load, I will report the results in this thread.
Regarding checking the Garbage Collector: as soon as there is a high CPU load, I will report the results in this thread.