Hello!
Release 8.7.11.GA.1854.UBUNTU16.64 UBUNTU16_64 FOSS edition.
Yesterday, I noticed that a virtual machine consumes a lot of CPU resources. When executing a command "top" in operating system, I see:
"7209 zimbra 20 0 420652 11772 1128 S 399,7 0,0 2342:20 zmswatch". This is 400% CPU = 4 permanently full loaded cores from 8 cores of this virtual machine.
After executing the command "zmcontrol stop", this process does not stop and the load is 400%.
After a full reboot, everything becomes normal, but today, checking the status, I again see 400% load zmswatch. Please tell, how can i fix this ?
zmswatch high cpu usage
Re: zmswatch high cpu usage
I want to add information.
The zmswatch process was launched from /opt/zimbra/log/zmswatch
Is that normal? Binary file placed in logs folder?
The zmswatch process was launched from /opt/zimbra/log/zmswatch
Is that normal? Binary file placed in logs folder?
Re: zmswatch high cpu usage
I can't be certain as I haven't run that version of ZCS for a long time but I doubt that was it's original location or even that name. The file I have on my current ZCS 8.8.12 is in this folder and, as you can see, it's named differently:
I suggest you take a look at that file and see what it contains, also check if there's another file in the location I've mentioned - as far as I know, it's always been named 'zmswatchctl' but I could be wrong.
[EDIT}Sorry, I wasn't too clear on what I posted earlier. The process that runs is named zmswatch but the file that controls it is name zmswatchctl and it's in the location I gave. Run the following and I get the shown output:
Code: Select all
ls -l /opt/zimbra/bin/zmswatchctl
-rwxr-xr-x 1 root root 3111 2019-03-29 10:57 /opt/zimbra/bin/zmswatchctl
[EDIT}Sorry, I wasn't too clear on what I posted earlier. The process that runs is named zmswatch but the file that controls it is name zmswatchctl and it's in the location I gave. Run the following and I get the shown output:
Code: Select all
find /opt/zimbra -name zmswatch*
/opt/zimbra/bin/zmswatchctl
/opt/zimbra/log/zmswatch.out
/opt/zimbra/log/zmswatch.out-20190418.gz
/opt/zimbra/log/zmswatch.out-20190417.gz
Re: zmswatch high cpu usage
Thanks for the answer! I see that this file was created on April 16, 2019. My backup created on April 14th does not contain this file!phoenix wrote:I can't be certain as I haven't run that version of ZCS for a long time but I doubt that was it's original location or even that name. The file I have on my current ZCS 8.8.12 is in this folde and, as you can see, it's named differently:
This file is probably not from the original Zimbra!!!!
-rwxr-x--- 1 zimbra zimbra 999088 apr 16 19:17 zmswatch
/opt/zimbra/log/zmswatch: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), tatically linked, stripped
Is that hackers?
I check viewtopic.php?f=15&t=65932
but my /tmp folder dont contain "zmcat" and "l.sh" or "s.sh"
Re: zmswatch high cpu usage
Unfortunately, downtime is critical.
Will it be enough to install Patch-11 for version 8.7.11?
I read this instructions : https://lorenzo.mile.si/zimbra-cve-2019 ... ction/961/
And find modified file: 93393769 4 -rw-r----- 1 zimbra zimbra 332 apr 16 19:17 /opt/zimbra/jetty/webapps/zimbra/public/Ajax.jsp !!!
19:17 16 april 2019
This is exactly up to the minute /zimbra/log/zmswatch file creation time!
Can I replace Ajax.jsp from backup after patching and delete zmswatch?
Will it be enough to install Patch-11 for version 8.7.11?
I read this instructions : https://lorenzo.mile.si/zimbra-cve-2019 ... ction/961/
And find modified file: 93393769 4 -rw-r----- 1 zimbra zimbra 332 apr 16 19:17 /opt/zimbra/jetty/webapps/zimbra/public/Ajax.jsp !!!
19:17 16 april 2019
This is exactly up to the minute /zimbra/log/zmswatch file creation time!
Can I replace Ajax.jsp from backup after patching and delete zmswatch?
Re: zmswatch high cpu usage
Update!
viewtopic.php?t=66005
Ajax.jsp - The content of the file is exactly like the author of the question!
viewtopic.php?t=66005
Ajax.jsp - The content of the file is exactly like the author of the question!
Re: zmswatch high cpu usage
Hello,
I had the same problem with this version.
I believe this happened because I upgraded the OS without applying patch fixes.
I updated with patch 11 and resolved.
Patch download link: https://files.zimbra.com/downloads/8.7. ... A_3800.tgz
I had the same problem with this version.
I believe this happened because I upgraded the OS without applying patch fixes.
I updated with patch 11 and resolved.
Patch download link: https://files.zimbra.com/downloads/8.7. ... A_3800.tgz
Re: zmswatch high cpu usage
Just in case you haven't realized it yet, your machine was infected with a coin miner.
see
https://cve.mitre.org/cgi-bin/cvename.c ... -2019-9670
lots of discussion of post-patch cleanup here (warning - there are a couple of variants, so not everything applies)
https://lorenzo.mile.si/zimbra-cve-2019 ... ction/961/
and here
viewtopic.php?t=65932
see
https://cve.mitre.org/cgi-bin/cvename.c ... -2019-9670
lots of discussion of post-patch cleanup here (warning - there are a couple of variants, so not everything applies)
https://lorenzo.mile.si/zimbra-cve-2019 ... ction/961/
and here
viewtopic.php?t=65932
- SamueleERAL
- Posts: 7
- Joined: Tue Jun 11, 2019 12:26 pm
- Location: Via Europa, Vazzola, 31028 (TV) Italy
- Contact:
Re: zmswatch high cpu usage
The patch is very simple and fast to apply, you just need to extract the archive in the right directory and then restart the services, should be only few seconds of down time.
The patch applies only to mailboxes servers.
It obviously doesn't clean up all the files infected and doesn't close other backdors the attacker could have placed.
What I found on my server was a lot of .jsp files modified or created, the two executable files on the /opt/zimbra/log directory, permissions of folder /opt/zimbra/data/tmp/upload changed and the crontab of the zimbra user where changed.
They uses cron to lounch directly or indirectly zmswatch.
The best way to clean up your server is to have a similar server not infected or a previous backup and match the differences betwen the two crontabs.
Note somwhere the lines that differ and look at the executables lounched, you need to check if they differ in the two servers.
Than there are some very useful commands.
find /opt/zimbra/jetty/ -name "*.jsp" -mtime -15 -ls
it will list all .jsp files created or modified in the last 15 days.
find bin common conf contrib docs extensions-extra fbqueue jetty-distribution-9.3.5.v20151012 lib libexec log logger redolog ssl zimbramon zimlets zimlets-deployed zmstat -name *.jsp | sort
it will list all .jsp files, so you can check if somthing differ in the two server
grep "if.*equals(" -R /opt/zimbra/mailboxd/
it was suggester in another topic about this infection, if you find a very long no-sense alfanumeric string they are probably files infected and you should check them between the two servers
This is the link to the topic viewtopic.php?t=66213&start=120#p291256
Check your zimbra management console for global-administrator accounts, when I had been infected I found a newly created account.
The patch applies only to mailboxes servers.
It obviously doesn't clean up all the files infected and doesn't close other backdors the attacker could have placed.
What I found on my server was a lot of .jsp files modified or created, the two executable files on the /opt/zimbra/log directory, permissions of folder /opt/zimbra/data/tmp/upload changed and the crontab of the zimbra user where changed.
They uses cron to lounch directly or indirectly zmswatch.
The best way to clean up your server is to have a similar server not infected or a previous backup and match the differences betwen the two crontabs.
Note somwhere the lines that differ and look at the executables lounched, you need to check if they differ in the two servers.
Than there are some very useful commands.
find /opt/zimbra/jetty/ -name "*.jsp" -mtime -15 -ls
it will list all .jsp files created or modified in the last 15 days.
find bin common conf contrib docs extensions-extra fbqueue jetty-distribution-9.3.5.v20151012 lib libexec log logger redolog ssl zimbramon zimlets zimlets-deployed zmstat -name *.jsp | sort
it will list all .jsp files, so you can check if somthing differ in the two server
grep "if.*equals(" -R /opt/zimbra/mailboxd/
it was suggester in another topic about this infection, if you find a very long no-sense alfanumeric string they are probably files infected and you should check them between the two servers
This is the link to the topic viewtopic.php?t=66213&start=120#p291256
Check your zimbra management console for global-administrator accounts, when I had been infected I found a newly created account.
Last edited by SamueleERAL on Wed Jun 26, 2019 7:22 am, edited 1 time in total.