zmswatch high cpu usage

Discuss your pilot or production implementation with other Zimbra admins or our engineers.
Post Reply
GlooM
Advanced member
Advanced member
Posts: 127
Joined: Sat Sep 13, 2014 12:50 am

zmswatch high cpu usage

Post by GlooM »

Hello!

Release 8.7.11.GA.1854.UBUNTU16.64 UBUNTU16_64 FOSS edition.

Yesterday, I noticed that a virtual machine consumes a lot of CPU resources. When executing a command "top" in operating system, I see:
"7209 zimbra 20 0 420652 11772 1128 S 399,7 0,0 2342:20 zmswatch". This is 400% CPU = 4 permanently full loaded cores from 8 cores of this virtual machine.

After executing the command "zmcontrol stop", this process does not stop and the load is 400%.
After a full reboot, everything becomes normal, but today, checking the status, I again see 400% load zmswatch. Please tell, how can i fix this ?
GlooM
Advanced member
Advanced member
Posts: 127
Joined: Sat Sep 13, 2014 12:50 am

Re: zmswatch high cpu usage

Post by GlooM »

I want to add information.
The zmswatch process was launched from /opt/zimbra/log/zmswatch

Is that normal? Binary file placed in logs folder?
phoenix
Ambassador
Ambassador
Posts: 27263
Joined: Fri Sep 12, 2014 9:56 pm
Location: Liverpool, England

Re: zmswatch high cpu usage

Post by phoenix »

I can't be certain as I haven't run that version of ZCS for a long time but I doubt that was it's original location or even that name. The file I have on my current ZCS 8.8.12 is in this folder and, as you can see, it's named differently:

Code: Select all

ls -l /opt/zimbra/bin/zmswatchctl
-rwxr-xr-x 1 root root 3111 2019-03-29 10:57 /opt/zimbra/bin/zmswatchctl
I suggest you take a look at that file and see what it contains, also check if there's another file in the location I've mentioned - as far as I know, it's always been named 'zmswatchctl' but I could be wrong. :)

[EDIT}Sorry, I wasn't too clear on what I posted earlier. The process that runs is named zmswatch but the file that controls it is name zmswatchctl and it's in the location I gave. Run the following and I get the shown output:

Code: Select all

find /opt/zimbra -name zmswatch*

/opt/zimbra/bin/zmswatchctl
/opt/zimbra/log/zmswatch.out
/opt/zimbra/log/zmswatch.out-20190418.gz
/opt/zimbra/log/zmswatch.out-20190417.gz
Regards

Bill

Rspamd: A high performance spamassassin replacement

Per ardua ad astra
GlooM
Advanced member
Advanced member
Posts: 127
Joined: Sat Sep 13, 2014 12:50 am

Re: zmswatch high cpu usage

Post by GlooM »

phoenix wrote:I can't be certain as I haven't run that version of ZCS for a long time but I doubt that was it's original location or even that name. The file I have on my current ZCS 8.8.12 is in this folde and, as you can see, it's named differently:
Thanks for the answer! I see that this file was created on April 16, 2019. My backup created on April 14th does not contain this file!
This file is probably not from the original Zimbra!!!!

-rwxr-x--- 1 zimbra zimbra 999088 apr 16 19:17 zmswatch

/opt/zimbra/log/zmswatch: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), tatically linked, stripped

Is that hackers?

I check viewtopic.php?f=15&t=65932
but my /tmp folder dont contain "zmcat" and "l.sh" or "s.sh"
GlooM
Advanced member
Advanced member
Posts: 127
Joined: Sat Sep 13, 2014 12:50 am

Re: zmswatch high cpu usage

Post by GlooM »

Unfortunately, downtime is critical.
Will it be enough to install Patch-11 for version 8.7.11?

I read this instructions : https://lorenzo.mile.si/zimbra-cve-2019 ... ction/961/
And find modified file: 93393769 4 -rw-r----- 1 zimbra zimbra 332 apr 16 19:17 /opt/zimbra/jetty/webapps/zimbra/public/Ajax.jsp !!!

19:17 16 april 2019
This is exactly up to the minute /zimbra/log/zmswatch file creation time!

Can I replace Ajax.jsp from backup after patching and delete zmswatch?
GlooM
Advanced member
Advanced member
Posts: 127
Joined: Sat Sep 13, 2014 12:50 am

Re: zmswatch high cpu usage

Post by GlooM »

Update!

viewtopic.php?t=66005

Ajax.jsp - The content of the file is exactly like the author of the question!
ans
Posts: 1
Joined: Thu May 09, 2019 7:23 pm

Re: zmswatch high cpu usage

Post by ans »

Hello,

I had the same problem with this version.
I believe this happened because I upgraded the OS without applying patch fixes.
I updated with patch 11 and resolved.
Patch download link: https://files.zimbra.com/downloads/8.7. ... A_3800.tgz
bisi
Posts: 21
Joined: Sat Sep 13, 2014 2:43 am
ZCS/ZD Version: many versions&clients from 6.x up

Re: zmswatch high cpu usage

Post by bisi »

Just in case you haven't realized it yet, your machine was infected with a coin miner.

see
https://cve.mitre.org/cgi-bin/cvename.c ... -2019-9670

lots of discussion of post-patch cleanup here (warning - there are a couple of variants, so not everything applies)
https://lorenzo.mile.si/zimbra-cve-2019 ... ction/961/
and here
viewtopic.php?t=65932
User avatar
SamueleERAL
Posts: 7
Joined: Tue Jun 11, 2019 12:26 pm
Location: Via Europa, Vazzola, 31028 (TV) Italy
Contact:

Re: zmswatch high cpu usage

Post by SamueleERAL »

The patch is very simple and fast to apply, you just need to extract the archive in the right directory and then restart the services, should be only few seconds of down time.
The patch applies only to mailboxes servers.
It obviously doesn't clean up all the files infected and doesn't close other backdors the attacker could have placed.
What I found on my server was a lot of .jsp files modified or created, the two executable files on the /opt/zimbra/log directory, permissions of folder /opt/zimbra/data/tmp/upload changed and the crontab of the zimbra user where changed.
They uses cron to lounch directly or indirectly zmswatch.
The best way to clean up your server is to have a similar server not infected or a previous backup and match the differences betwen the two crontabs.
Note somwhere the lines that differ and look at the executables lounched, you need to check if they differ in the two servers.
Than there are some very useful commands.

find /opt/zimbra/jetty/ -name "*.jsp" -mtime -15 -ls
it will list all .jsp files created or modified in the last 15 days.

find bin common conf contrib docs extensions-extra fbqueue jetty-distribution-9.3.5.v20151012 lib libexec log logger redolog ssl zimbramon zimlets zimlets-deployed zmstat -name *.jsp | sort
it will list all .jsp files, so you can check if somthing differ in the two server

grep "if.*equals(" -R /opt/zimbra/mailboxd/
it was suggester in another topic about this infection, if you find a very long no-sense alfanumeric string they are probably files infected and you should check them between the two servers

This is the link to the topic viewtopic.php?t=66213&start=120#p291256

Check your zimbra management console for global-administrator accounts, when I had been infected I found a newly created account.
Last edited by SamueleERAL on Wed Jun 26, 2019 7:22 am, edited 1 time in total.
Post Reply