CVE-2019-9670 being actively exploited (Hacked Server)

[Media]

Addtionally...

Check for existence of

Code: Select all

/opt/zimbra/lib/zmlogswatch

If it exists, it'll be a recently created binary file. Delete it.

However, there would likely be multiple instances already running. To find them, run (then kill all instances):

Code: Select all

top -p $(pgrep -d ',' zmlogswatch)

Looks like that /opt/zimbra/lib/zmlogswatch binary was actively adding itself back to crontab, so you should then cleanup, or regenerate your zimbra crontab.

My Zimbra is 8.6.0_GA_1242 with the last patch 14.
I've found /opt/zimbra/lib/zmlogswatch but it's not a recent one. His date is 2014-12-15. I don't think it's a part of the malware.

[elby]

Thanks to Drake for support

Zimbra 8.6.0 GA Network Editions , CentOS 6.6

Patch Zimbra :

wget https://files.zimbra.com/downloads/8.6. ... A_1242.tgz
tar xzf zcs-patch-8.6.0_GA_1242.tgz
cd /tar xzf zcs-patch-8.6.0_GA_1242


Delete global admin accounts

Change password

# Zimbra AJAX Webmail not loading

cd /opt/zimbra/mailboxd
find webapps -type d -exec chmod 0755 {} \;
find webapps -type f -exec chmod 0644 {} \;

# restart Zimbra;
su - zimbra
zmcontrol restart


change the files or delete them

files:

/opt/zimbra/log/.editorinfo --> /opt/zimbra/log/.editorinfo--

/opt/zimbra/log/zmswatch

/opt/zimbra/log/zmswatch.sh

/opt/zimbra/jetty-distribution-9.1.5.v20140505/webapps/zimbra/public/Ajax.jsp

/opt/zimbra/jetty-distribution-9.1.5.v20140505/webapps/zimbra/public/jsp/CryptCore.jsp

/opt/zimbra/jetty-distribution-9.1.5.v20140505/webapps/zimbra/portals/example/static.jsp

/opt/zimbra/jetty-distribution-9.1.5.v20140505/webapps/zimbra/portals/example/static.jsp

/tmp/.cache/.ntp

/tmp/.cache/.kthrotlds

Or :

touch /opt/zimbra/log/zmswatch
chattr +i /opt/zimbra/log/zmswatch

Do the same with other files ...

Replace with the original installation files:

wget https://files.zimbra.com/downloads/8.6. ... 151258.tgz
tar xzf zcs-NETWORK-8.6.0_GA_1153.RHEL6_64.20141215151258.tgz
cd zcs-NETWORK-8.6.0_GA_1153.RHEL6_64.20141215151258/packages/
rpm2cpio zimbra-store-8.6.0_GA_1153.RHEL6_64-20141215151258.x86_64.rpm | cpio -idmv


find /opt/zimbra -name \*.jsp -exec grep --with-filename exec {} \;

/opt/zimbra/jetty-distribution-9.1.5.v20140505/webapps/service/error/403.jsp
/opt/zimbra/jetty-distribution-9.1.5.v20140505/webapps/service/error/sfdc_preauth.jsp
/opt/zimbra/jetty-distribution-9.1.5.v20140505/webapps/service/error/attachment_blocked.jsp
/opt/zimbra/jetty-distribution-9.1.5.v20140505/webapps/zimbra/public/login.jsp
/opt/zimbra/jetty-distribution-9.1.5.v20140505/webapps/zimbraAdmin/public/jsp/Debug.jsp
/opt/zimbra/jetty-distribution-9.1.5.v20140505/webapps/zimbra/public/jsp/Alert.jsp


top

Kill PID 400%

fix error upload files :

chmod 755 /opt/zimbra/data/tmp/
chmod 755 /opt/zimbra/data/tmp/upload


regenerate crontab

crontab -l -u zimbra
crontab -e -u zimbra

replace with:
===================

# ZIMBRASTART -- DO NOT EDIT ANYTHING BETWEEN THIS LINE AND ZIMBRAEND
SHELL=/bin/bash
#
# Log pruning
#
30 2 * * * find /opt/zimbra/log/ -type f -name \*.log\* -mtime +8 -exec rm {} \; > /dev/null 2>&1
35 2 * * * find /opt/zimbra/log/ -type f -name \*.out.???????????? -mtime +8 -exec rm {} \; > /dev/null 2>&1
#
# compress logs manually to avoid application pauses when
# handled through the log4j thread
#
50 2 * * * /opt/zimbra/libexec/zmcompresslogs > /dev/null 2>&1
#
# tmp dir cleaning
#
40 2 * * * /opt/zimbra/libexec/zmcleantmp
#
# Status logging
#
*/2 * * * * /opt/zimbra/libexec/zmstatuslog > /dev/null 2>&1
#*/10 * * * * /opt/zimbra/libexec/zmdisklog > /dev/null 2>&1
#
# SSL Certificate Expiration Checks
#
0 0 1 * * /opt/zimbra/libexec/zmcheckexpiredcerts -days 30 -email
#
# Backups
#
# BACKUP BEGIN
# BACKUP END
#
# crontab.store
#
# Log pruning
#
30 2 * * * find /opt/zimbra/mailboxd/logs/ -type f -name \*log\* -mtime +8 -exec rm {} \; > /dev/null 2>&1
30 2 * * * find /opt/zimbra/log/ -type f -name stacktrace.\* -mtime +8 -exec rm {} \; > /dev/null 2>&1
#
# Report on any database inconsistencies
#
0 23 * * 7 /opt/zimbra/libexec/zmdbintegrityreport -m
#
# Monitor for multiple mysqld to prevent corruption
#
#*/5 * * * * /opt/zimbra/libexec/zmcheckduplicatemysqld -e > /dev/null 2>&1
#
# Check zimbraVersionCheckURL for new update.
# Only runs if this server matches zimbraVersionCheckServer
# Only executes on zimbraVersionCheckInterval. min 2h interval
#
18 */2 * * * /opt/zimbra/libexec/zmcheckversion -c >> /dev/null 2>&1
#
# Invoke "ComputeAggregateQuotaUsageRequest" periodically
#
15 2 * * * /opt/zimbra/libexec/zmcomputequotausage > /dev/null 2>&1
#
# Invoke "client_usage_report.py" periodically to process /opt/zimbra/log/access_log* files
#
55 1 * * * /opt/zimbra/libexec/client_usage_report.py > /dev/null 2>&1
#
# Run zmgsaupdate util to trickeSync galsync accounts
#
49 0 * * 7 /opt/zimbra/libexec/zmgsaupdate > /dev/null 2>&1
#
# crontab.logger
#
# process logs
#
00,10,20,30,40,50 * * * * /opt/zimbra/libexec/zmlogprocess > /tmp/logprocess.out 2>&1
#
# Graph generation
#
#10 * * * * /opt/zimbra/libexec/zmgengraphs >> /tmp/gengraphs.out 2>&1
#
# Daily reports
#
30 23 * * * /opt/zimbra/libexec/zmdailyreport -m
#
# crontab.mta
#
#
# Queue logging
#
0,10,20,30,40,50 * * * * /opt/zimbra/libexec/zmqueuelog
#
# Spam training
#
0 22 * * * /opt/zimbra/bin/zmtrainsa >> /opt/zimbra/log/spamtrain.log 2>&1
#
# Spam training cleanup
#
45 23 * * * /opt/zimbra/bin/zmtrainsa --cleanup >> /opt/zimbra/log/spamtrain.log 2>&1
#
# Spam rule updates
#
45 0 * * * . /opt/zimbra/.bashrc; /opt/zimbra/libexec/zmsaupdate
#
# Dspam cleanup
#
0 1 * * * [ -d /opt/zimbra/data/dspam/data/z/i/zimbra/zimbra.sig ] && find /opt/zimbra/data/dspam/data/z/i/zimbra/zimbra.sig/ -type f -name \*sig -mtime +7 -exec rm {} \; > /dev/null 2>&1
8 4 * * * [ -f /opt/zimbra/data/dspam/system.log ] && /opt/zimbra/dspam/bin/dspam_logrotate -a 60 -l /opt/zimbra/data/dspam/system.log
8 8 * * * [ -f /opt/zimbra/data/dspam/data/z/i/zimbra/zimbra.log ] && /opt/zimbra/dspam/bin/dspam_logrotate -a 60 -l /opt/zimbra/data/dspam/data/z/i/zimbra/zimbra.log
#
# Spam Bayes auto-expiry
#
20 23 * * * /opt/zimbra/libexec/sa-learn --dbpath /opt/zimbra/data/amavisd/.spamassassin --force-expire --sync > /dev/null 2>&1
#
# Clean up amavisd/tmp
#
15 5,20 * * * find /opt/zimbra/data/amavisd/tmp -maxdepth 1 -type d -name 'amavis-*' -mtime +1 -exec rm -rf {} \; > /dev/null 2>&1
#
# Clean up the quarantine dir
#
0 1 * * * find /opt/zimbra/data/amavisd/quarantine -type f -mtime +7 -exec rm -f {} \; > /dev/null 2>&1

#ZIMBRAEND -- DO NOT EDIT ANYTHING BETWEEN THIS LINE AND ZIMBRASTART

===================

or use file /tmp/crontab.zimbra

#help
zmschedulebackup -h

#To Verify Backup Schedules:
zmschedulebackup -q

#To Configure Default Backup Schedules:
zmschedulebackup -D

#To Configure Customized Backup Schedules:
zmschedulebackup -R f "0 1 * * 6" --mail-report i "0 1 * * 0-5" --mail-report d 14d "0 0 * * *" --mail-report

crontab -l |grep -i backup

Lock up crontab

chattr +i /var/spool/cron/zimbra


reboot

[patsybrian]

Hi all!
I have followed all steps in Lorenzo's blog (https://lorenzo.mile.si/zimbra-cve-2019 ... ction/961/).
But still, the l.sh and wgets keep appearing a second after killing the processes.
Any ideas what I can do?

Thanks so much!

[Drake]

Fast food, fast dates, fast sex, fast Internet, fast info, fast solutions .......
Take some time to read the topic and the posts before yours pls....
You may also try to replicate the steps described and then ask constructive questions or even propose some solutions.

Good Luck

[fcourtaud]

Hi everyone,

Below are the steps I followed

First delete the following files

rm /opt/zimbra.log/zmswatch
rm /opt/zimbra.log/zmswatch.sh
/opt/zimbra/lib/zmlogswatch

Kill all the processes you'll get with :

top -p $(pgrep -d ',' zmlogswatch)
top -p $(pgrep -d ',' zmswatch)

zmlogswatch seems to be the one updating the crontab

Clean all of the .jsp, .java and .class you'll find with

grep -R '(request\.getParameter.*' /opt/zimbra/mailboxd
grep -R '(request\.getParameter.*' /opt/zimbra/jetty

cleanup zimbra's crontab

Reset permissions
chmod 1777 /opt/zimbra/data/tmp
chmod 755 /opt/zimbra/data/tmp/upload
chmod 755 /opt/zimbra/data/tmp/nginx
chmod 755 /opt/zimbra/data/tmp/nginx/client
chmod 755 /opt/zimbra/data/tmp/nginx/proxy
chmod 755 /opt/zimbra/data/tmp/nginx/fastcgi

cd /opt/zimbra/mailboxd
find webapps -type d -exec chmod 0755 {} \;
find webapps -type f -exec chmod 0644 {} \;
/opt/zimbra/libexec/zmfixperms -verbose

Update ssh keys as zimra user
zmsshkeygen
zmupdateauthkeys

reboot

Hope this helps

[zimbraxtc]

Hi all!

I have a old 5.0 installation running on an old hp-server, dont ask me why. Are the 5.0 affected by the virus?

Anyone knows?

Thanks!

[cmgonzalez]

Caution, this MF, is cloacking the files now using same dates as regular ones....He may be reading this

[halfgaar]

You can always use 'stat' to look at the ctime: you can't change that.

[jaca_sv]

Clean all of the .jsp, .java and .class you'll find with

grep -R '(request\.getParameter.*' /opt/zimbra/mailboxd
grep -R '(request\.getParameter.*' /opt/zimbra/jetty

Hello fcourtaud,
By "clean all of the .jsp, .java and .class" do you mean delete the suspicious lines or completely delete the files?
All the files found with "request\.getParameter" are not supposed to be in the server?

I already restored the zimbra crontab and locked it with chattr +i (Also removed and touched .kthrotlds and .editorinfo and lock them)
I also blocked python-requests user agent in my nginx and obviously patched my zimbra 8.7.11 to the latest patch.

My next steps will be to remove the zmswatch.sh and delete the logs the script create

-rw-r--r-- 1 zimbra zimbra 100 May 26 03:45 zmswatch.out-20190526.gz
-rw-r--r-- 1 zimbra zimbra 100 May 27 03:38 zmswatch.out-20190527.gz
-rw-r--r-- 1 zimbra zimbra 304 May 28 03:28 zmswatch.out-20190528.gz
-rw-r--r-- 1 zimbra zimbra 236 May 29 03:06 zmswatch.out-20190529.gz
-rw-r--r-- 1 zimbra zimbra 3.0K May 30 03:21 zmswatch.out-20190530
-rwxr-x--- 1 zimbra zimbra 225 May 30 14:11 zmswatch.sh

Thanks everyone for the help!

[SPAMSAM]

Having this exact issue.

I noticed that it would blank out the webmail page after someone logged in. Steps above helped me fix it, though the admin page still loads a blank page. not sure if related