CVE-2019-9670 being actively exploited (Hacked Server)

Discuss your pilot or production implementation with other Zimbra admins or our engineers.
Klug
Ambassador
Ambassador
Posts: 2747
Joined: Mon Dec 16, 2013 11:35 am
Location: France - Drôme
ZCS/ZD Version: All of them
Contact:

Re: CVE-2019-9670 being actively exploited

Post by Klug »

I've always and only set it up per domain (this also alows me to have a different value per domain).
If the wiki page is up-to-date and right, it should work the way you're set it up.
tin
Posts: 11
Joined: Wed Jan 17, 2018 2:32 am

Re: CVE-2019-9670 being actively exploited

Post by tin »

403 could be damaged webapps (cant think of the exact path off the top of my head) folder. We had to replace ours from a backup after mystery jsp files appeared.

Rename the current folder (while zimbra is stopped) and put the backup from before it broke in place (remember to check permissions too).

I also uninstalled wget and curl on our server to stop the attack scripts from working, but this is a bandaid solution while rebuilding... Something is still wrong with our system after the attack and I suspect that is true for everyone who has just cleaned out files.
opalomo
Posts: 6
Joined: Sat Sep 13, 2014 1:11 am

Re: CVE-2019-9670 being actively exploited

Post by opalomo »

Hi guys, thanks for the great work on this issue.

I want to find out if this version has the same vulnerability... 8.7.6_GA_1776.FOSS I am not quite sure because there are no patches released for this version.

I have found this messages in the log, however there is nothing else, like the zmcat folder in /tmp If you can provide more information, it will be greatly appreciated.

[34503787.231413] zmcat[17529]: segfault at 63 ip 00007f4287bee60d sp 00007f428d9eb4e0 error 4 in libnss_files-2.17.so[7f4287beb000+c000]
[34504435.249023] zmcat[19796]: segfault at 63 ip 00007fd7f732e60d sp 00007fd7f8f584e0 error 4 in libnss_files-2.17.so[7fd7f732b000+c000]
[34505111.037873] zmcat[27039]: segfault at 63 ip 00007fb2df42460d sp 00007fb2e004c4e0 error 4 in libnss_files-2.17.so[7fb2df421000+c000]
[34505749.562160] zmcat[1021]: segfault at 63 ip 00007f7148e4460d sp 00007f714a26d4e0 error 4 in libnss_files-2.17.so[7f7148e41000+c000]
[34510806.461213] zmswatch[17284]: segfault at 63 ip 00007f147456460d sp 00007f147618e4e0 error 4 in libnss_files-2.17.so[7f1474561000+c000]
[34524462.289291] zmswatch[17818]: segfault at 63 ip 00007fb75166560d sp 00007fb7528884e0 error 4 in libnss_files-2.17.so[7fb751662000+c000]
[34526442.670988] zmswatch[2829]: segfault at 63 ip 00007fdc6425060d sp 00007fdc65e7a4e0 error 4 in libnss_files-2.17.so[7fdc6424d000+c000]
[34539744.039257] zmswatch[23340]: segfault at 63 ip 00007ff08801b60d sp 00007ff089a3f4e0 error 4 in libnss_files-2.17.so[7ff088018000+c000]



Thanks in advance.

Oscar.
halfgaar
Advanced member
Advanced member
Posts: 171
Joined: Sat Sep 13, 2014 12:54 am
Location: Netherlands
ZCS/ZD Version: Ubuntu 18.04, 8.8.15_P43
Contact:

Re: CVE-2019-9670 being actively exploited

Post by halfgaar »

I want to find out if this version has the same vulnerability... 8.7.6_GA_1776.FOSS I am not quite sure because there are no patches released for this version.
There are: you need to update to 8.7.11 and then apply patch 11.
I have found this messages in the log, however there is nothing else, like the zmcat folder in /tmp If you can provide more information, it will be greatly appreciated.
They may be doiing very different things by now (of which a lot is mentioned in this discussion). At least also check the crontab in /var/spool/cron/crontab/zimbra (off the top of my head; path my differ).
lucadevac
Posts: 7
Joined: Fri May 17, 2019 10:01 am

Re: CVE-2019-9670 being actively exploited

Post by lucadevac »

tin wrote:403 could be damaged webapps (cant think of the exact path off the top of my head) folder. We had to replace ours from a backup after mystery jsp files appeared.

Rename the current folder (while zimbra is stopped) and put the backup from before it broke in place (remember to check permissions too).

I also uninstalled wget and curl on our server to stop the attack scripts from working, but this is a bandaid solution while rebuilding... Something is still wrong with our system after the attack and I suspect that is true for everyone who has just cleaned out files.
I have solved by reinstalling a brand new zimbra on the same version, on another server, and then copying back the jetty* content.

Thank you all.
halfgaar
Advanced member
Advanced member
Posts: 171
Joined: Sat Sep 13, 2014 12:54 am
Location: Netherlands
ZCS/ZD Version: Ubuntu 18.04, 8.8.15_P43
Contact:

Re: CVE-2019-9670 being actively exploited

Post by halfgaar »

Smart.

Are you still in a position to do 'diff -r' on the dirs? I'd like to see the difference.
lucadevac
Posts: 7
Joined: Fri May 17, 2019 10:01 am

Re: CVE-2019-9670 being actively exploited

Post by lucadevac »

halfgaar wrote:Smart.

Are you still in a position to do 'diff -r' on the dirs? I'd like to see the difference.
sorry i'm not. because the hacked server was already removed.
By the way from what I have seen, there was some file missing: hostedlogin.jsp, authorize.jsp and access.jsp for sure.
tin
Posts: 11
Joined: Wed Jan 17, 2018 2:32 am

Re: CVE-2019-9670 being actively exploited

Post by tin »

I just checked our backups from the day our web interface was broken... Not sure if it's a backup from while it was broken or not, but I found this:
Only in webapps/zimbra/downloads: 05x6.jsp
Only in webapps/zimbra/downloads: 51Qi.jsp
Only in webapps/zimbra/downloads: jfyJ.jsp
Only in webapps/zimbra/downloads: Od6g.jsp
Only in webapps/zimbra/downloads: test.jsp
Only in webapps/zimbra/img: Pwg4.jsp
Only in webapps/zimbra/portals/example: times.jsp
Only in webapps/zimbra/public: 404.jsp
Only in webapps/zimbra/public: b3mx5q.jsp
Only in webapps/zimbra/public: hostedlogin.jsp
Only in webapps/zimbra/public/jsp: H5Tp.jsp
Only in webapps/zimbra/public/jsp: Leak1U.jsp
Only in webapps/zimbra/public/jsp: LeakkB.jsp
Only in webapps/zimbra/public/jsp: LeakOE.jsp
Only in webapps/zimbra/public/jsp: xHK0.jsp
Only in webapps/zimbra/public: login.jsp

Most of those don't look like legit files names. I assume the legit looking ones are either also not meant to be there, or were removed from the folder I compared to during an attempted cleanup.

Edit: I tarred up the files that matched - if they don't exactly match, it's because I copied them by hand and might have missed one or put it into the wrong folder :P
Attachments
busted-zimbra.tar.gz
(40.94 KiB) Downloaded 696 times
naivsupr
Posts: 1
Joined: Wed Mar 08, 2017 1:41 pm

Re: CVE-2019-9670 being actively exploited

Post by naivsupr »

Thank you all for this thread!

My case:
zmswatch in /opt/zimbra/log, giving high cpu load. On a monitored cpu.. I killed the binary and moved it off to a safe location until I found this thread.

zmswatch.sh:

Code: Select all

#!/bin/sh
AGENT_FILE='/opt/zimbra/log/zmswatch'
if ps cax | grep -v grep | grep -v "zmswatch.sh" | grep "zmswatch" > /dev/null; then
  echo "running"
else
  echo "nohup"
  nohup /opt/zimbra/log/zmswatch > /dev/null 2>&1 &
fi
sed -i '/Ajax\.jsp/d' /opt/zimbra/log/*_log.2019*
sed -i '/XZimbra\.jsp/d' /opt/zimbra/log/*_log.2019*
sed -i '/login\.jsp/d' /opt/zimbra/log/*_log.2019*
sed -i '/ZimbraCore\.jsp/d' /opt/zimbra/log/*_log.2019*
Launched from /var/spool/cron/crontab/zimbra:

Code: Select all

*/15 * * * * sh /opt/zimbra/log/zmswatch.sh;
Cleaned and patched to 8.8.11P4, but will move to a new VM.

Thanks again!
Klug
Ambassador
Ambassador
Posts: 2747
Joined: Mon Dec 16, 2013 11:35 am
Location: France - Drôme
ZCS/ZD Version: All of them
Contact:

Re: CVE-2019-9670 being actively exploited

Post by Klug »

Additional threads about the fake zmswatch:
viewtopic.php?t=66031
viewtopic.php?f=15&t=66213

Actually, the forum is full (as of yesterday, sunday) of new threads about this issue.
Post Reply