UPDATE 30.05.2019 Found addition malicious process and file. Tutorial updated.
Hello guys
Thanks for the plenty of info. It seems we all are under the same umrella now
The case is not that bad. According to my basic linux experience there are far more nasty ways the situation could develop. It seems that so far the attacker relies on a backdoor in zimbra suite to escalate his privileges and install malicious miner of monero coins. Yesterday i spent the whole day till 11 pm. dealing with this situation. Following are the steps i performed and so far they work(meaning i don`t have 100% cpu usage and suspicious processes). The services are working fine. This does not mean that the system is totally clean and there are no backdoors left..Here is what i did:
=====START=================================================================================================================================================================================================
#Stop Zimbra suite
su - zimbra
zmcontrol stop
exit
#List suspicious processes and then kill them
ps -ax | grep zmswatch
ps -ax | grep zmlogswatch
ps -ax | grep zmmailboxdwatch
ps -ax | grep zmsstore
ps -ax | grep zmstorewatch
ps -ax | grep zmmailboxwatch
ps -ax | grep .kthrotlds
ps -aux
kill -9 viral pids, be suspicious about processes of zimbra user like "sleep 30" and all strange names
#List crontabs and delete suspicious lines. If the whole crontab is replaced then use the following tutorial to recreate it https://wiki.zimbra.com/wiki/Step_to_re ... imbra_user
crontab -l -u zimbra
crontab -l -u root
crontab -e -u zimbra
#Lock crontab files so they cannot be eddited even by root user
chattr +i /var/spool/cron/zimbra
chattr +i /var/spool/cron/root
#Find the last 50 day created files which are known so far to be malicious. Delete them step by step and also check in /opt/zimbra/log for suspicious .sh scripts and so on
find / -name "*zmswatch*" -mtime -50 -ls
find / -name "*zmlogswatch*" -mtime -50 -ls
find / -name "*zmmailboxdwatch*" -mtime -50 -ls
find / -name "*zmsstore*" -mtime -50 -ls
find / -name "*zmstorewatch*" -mtime -50 -ls
find / -name "*zmmailboxwatch*" -mtime -50 -ls
find / -name "*zmcat*" -mtime -50 -ls
# Remove the known shitty scripts files location and also check in /opt/zimbra/log for suspicious .sh scripts and so on
rm -rf /tmp/.cache
rm -rf /tmp/.scr/
rm -rf /opt/zimbra/conf/zmsstore.conf
rm -rf /opt/zimbra/log/.editorinfo
#Find all *.jsp * _jsp.java and *.class files and inspect them for malicious code or compare them with the installation sources. Have in mind that some files may be already modified by different patches(extra skills required to recognize). If you need delete them or replace them with originals
find / -name "*.jsp" -mtime -50 -ls
find / -name "*_jsp.java" -mtime -50 -ls
find / -name "*.class" -mtime -50 -ls
#Find all *.jsp files containing exec function and compare them with the installation sources. Have in mind that some files may be already modified by different patches(extra skills required to recognize). If you need delete them or replace them with originals.
find /opt/zimbra -name \*.jsp -exec grep --with-filename LlSqsDmOgh {} \;
find /opt/zimbra -name \*.jsp -exec grep --with-filename exec {} \;
#Some of the found malicious *.jsp files are recreated immediately after deletion. Check which process creates them with the following commands
##Create a trap and monitor files activity in the pointed path
auditctl -w /opt/zimbra/jetty-distribution-9.1.5.v20140505/webapps/zimbra/public/jsp/ -p war -k whatsgoingon
##List the activity
tail -f /var/log/audit/audit.log
##Remove the monitor trap
auditctl -W /opt/zimbra/jetty-distribution-9.1.5.v20140505/webapps/zimbra/public/jsp/ -p war -k whatsgoingon
#Download and apply the Zimbra patch. Use the correct version for your Zimbra version.
wget https://files.zimbra.com/downloads/8.6. ... A_1242.tgz
tar xzf zcs-patch-8.6.0_GA_1242.tgz
cd zcs-patch-8.6.0_GA_1242
./installPatch.sh
#Deinstall wget. Kind of useless because the new modifications continue to use curl
yum -f remove wget
reboot
#After patching check the system again
ps -aux
crontab -l -u zimbra
crontab -l -u root
#Change the password
passwd root
passwd zimbra
change webadmin password web panel
#Delete additional created admin accounts looking strange like 34gds@yourdomain.com You can doi it through the web panel or in console. Following is the command to list them in console.
su - zimbra
zmprov gaaa
#Stop your mail server being listed by web search engines. This is precautious measure for future scans.
zmprov mcf zimbraMailKeepOutWebCrawlers TRUE +zimbraResponseHeader "X-Robots-Tag: noindex"
#Recreate ssh keys
zmsshkeygen
zmupdateauthkeys
#Restore correct function of Webmail and uploading of files
cd /opt/zimbra/mailboxd
find webapps -type d -exec chmod 0755 {} \;
find webapps -type f -exec chmod 0644 {} \;
chmod 755 /opt/zimbra/data/tmp/
chmod 755 /opt/zimbra/data/tmp/upload/
#Inspect the original checksums of zimbra package with the new ones.(r)
rpm -qa "zimbra*" | xargs rpm -qV - | egrep -E '^.{2}5'
Output codes:
S = File size changed
M = File mode changed
5 = MD5 checksum changed
L = Symlink changed
U = Owner changed
G = Group changed
T = Modification time changed
c = it is a configuration file that has changed
missing = file is gone.
=====END=================================================================================================================================================================================================
P.S. My system is with Zimbra 8.6 on CentOS 7. Most of the commands are paste only but some require modification according to your needs(the url adress of the patch which is for 8.6 version as example) so be good sys admins and dont copy/paste without thinking first. Good Luck.
