CVE-2019-9670 being actively exploited (Hacked Server)

Discuss your pilot or production implementation with other Zimbra admins or our engineers.
User avatar
JDunphy
Outstanding Member
Outstanding Member
Posts: 897
Joined: Fri Sep 12, 2014 11:18 pm
Location: Victoria, BC
ZCS/ZD Version: 9.0.0_P39 NETWORK Edition

Re: CVE-2019-9670 being actively exploited

Post by JDunphy »

scrubudu wrote: I'm searching for a exhaustive url that list every binary file that should be in zimbra, per version : specially binary files : does it exists ? (like zmbackup that is born with 8.7 --> Disaster recovery Tech Zimbra Post )..

Would be ideal if list of access made by each binary about r, w, or x too, in /opt/zimbra, and in other filesystem too ( /tmp etc... )
A few things... use sha256sum or sha1sum to see if the contents change.

Code: Select all

% sha1sum /tmp/somefile
da39a3ee5e6b4b0d3255bfef95601890afd80709  /tmp/somefile
% echo hello >> /tmp/somefile
% sha1sum /tmp/somefile
f572d396fae9206628714fb2ce00f72e94f2258f  /tmp/somefile
You can run the above with find to build a list and compare it against the previous run...

Code: Select all

# su - zimbra
% find /opt/zimbra -type f -exec sha1sum {} \; | head -5
9e1146751bc76bb2a96fa663768e7f4c03c96e4e  /opt/zimbra/jetty-distribution-7.6.2.z4/webapps/zimlet/WEB-INF/web.xml
21082b6caa310fc9faf60997d8c071ea48294d95  /opt/zimbra/jetty-distribution-7.6.2.z4/etc/jetty.properties
e4b8f7148fdd1483d2d26591ce64d7d96d26bddd  /opt/zimbra/jetty-distribution-7.6.2.z4/etc/mailboxd.der
...
but if you want more, then investigate a tool like aide or tripwire. That would allow you to generate your list of signatures, permissions and compare it against the database to know when something changed. You can determine what you want it to look at... The resulting database is a zip of a text file so you can use zcat/zmore/etc to look at it and you can move them around from machine to machine.... So you could do the following.

install a pristine zimbra + patches on some VM but don't worry about the data, etc. You are after the patched binaries, etc.
run aide against /opt/zimbra to generate a database of signatures for each file and directory, etc.
copy the resulting aide database to your production machine
verify the production system against that list of signatures (permissions + sha256) which will spit out a list of changes

Here is something quick and dirty to give you an idea how the configuration file looks

Code: Select all

#just sha256 + permission + user + group + number links
CONTENT = sha256+p+n+u+g

/opt/zimbra    CONTENT

!/opt/zimbra/index
!/opt/zimbra/redolog
!/opt/zimbra/zmstat
!/opt/zimbra/data
!/opt/zimbra/zmstat
!/opt/zimbra/store
!/opt/zimbra/backup
!/opt/zimbra/db
!/opt/zimbra/log
This would look at every file under /opt/zimbra with the exception of the paths with '!'.

The other way is to look at the packages to get a list and use the package manager to help you( I don't know what OS you have so I show centos/RHEL). You are probably interested in jetty which is in zimbra-store and you can run these commands as normal user (don't need to be root or zimbra) so you can't mess anything up.

Code: Select all

% rpm -q zimbra-store
zimbra-store-8.7.11_GA_1854.RHEL6_64-20170531151956.x86_64
% rpm -ql zimbra-store | grep jetty |head -5
/opt/zimbra/jetty-distribution-9.3.5.v20151012
/opt/zimbra/jetty-distribution-9.3.5.v20151012/README.TXT
/opt/zimbra/jetty-distribution-9.3.5.v20151012/VERSION.txt
/opt/zimbra/jetty-distribution-9.3.5.v20151012/bin
/opt/zimbra/jetty-distribution-9.3.5.v20151012/bin/jetty.sh
% rpm -V zimbra-store |head -1
SM5....T.    /opt/zimbra/conf/templates/templates/calendar/Appointment.template
Where fields are explained by man rpm ...

Code: Select all

       S file Size differs
       M Mode differs (includes permissions and file type)
       5 digest (formerly MD5 sum) differs
       D Device major/minor number mismatch
       L readLink(2) path mismatch
       U User ownership differs
       G Group ownership differs
       T Time differs
       P caPabilities differ
 
Hint: if you want to know more... add -vv like rpm -Vvv zimbra-store
the problem with the rpm verify is that subsequent patches, normal install from Zimbra with install.sh, changes like fixperms, etc have changed things a bit as has the normal operations but it is a start if you don't have a aide/tripwire database.
Run a tripwire daily and it will teach you about how your system changes so you know what is normal and what is not... it will teach you what the patches and updates do also.

Having said all that... I need to caution if they obtained root then a determined hacker can hide in places that would make me never trust that server again. At this point in time everyone should have either locked down access to trusted ip's or applied the patches.

HTH,

Jim
scrubudu
Posts: 4
Joined: Mon Jun 03, 2019 9:12 pm

Re: CVE-2019-9670 being actively exploited

Post by scrubudu »

Thank you ! Yes system went really bad with the days !

Finally restore process went fine ! (via chapter C rsync process (page 19 url i talked about ) - was faster with tar pczvf the 3 directories before real transfer to the new server. )

Process from On UBUNTU14_64 FOSS edition, Patch 8.6.0_P4 Patched too late to : On UBUNTU14_64 FOSS edition, Patch 8.6.0_P14.
Rebuild machine (old vm backup) from P4 to P14 and migrate into it.
Added :
I added every password changes too before starting, on old and new server --> system compromised. As well as for zimbra pam.d authentification.
#rm -rf jetty-distribution-7.6.12.v20130726
ln -s pointing on /opt/zimbra/jetty-distribution-9.1.5.v20140505 since patched.
A backup and copy the ssl directory,if that is up.
( at least to start the server, before renew the ssl certificate)

Thank you all for your posts !!
Regards,
calandacomp
Posts: 9
Joined: Tue Jun 20, 2017 7:32 am

Re: CVE-2019-9670 being actively exploited

Post by calandacomp »

May be i found a way to remove the infected files without re-installation. For me, reinstallation was no option because of to much work behind...

in my case: 3 of around 50 managed zimbra servers where affected. all 3 without newest patches and 8.7.xxx versions. These are the first steps i've done:

-block https access
-remove /opt/zimbra/log/zmswatch and zmswatcher
-kill all zmswatch process with ps aux | grep zmswatch
-cleanup crontab (last line and described in post above)
-change folder chmod 755 /opt/zimbra/data/tmp and /opt/zimbra/data/tmp/upload
-update zimbra to latest version
-change all account passwords (<<-- may be not realy required...)
-renewed certficates with -> zmsshkeygen and zmupdateauthkeys
-opened https again

on 2 of 3 server's this was solution. on the other one, i had zmswatch and zmswatcher after around 30minutes back on the system... i was almost reinstalling but then i found yesterday this solution ->

do the same like above without update zimbra (because it's updated now) and without renewing certificates and open https. leave https blocked during deleting zmswatcher, cleanup crontab etc... and then do this (i've copied from earlier post) ->

grep "if.*equals(" -R /opt/zimbra/mailboxd/ > /tmp/result

in the file /tmp/result you will find a lot of lines. search or look for the ones looking like
:if ( "YuJb8NsE6pVFNish3_leYERZRwt4Za27GVdS4H2lNZM" .equals(
the string YuJb8NsE6pVFNish3_leYERZRwt4Za27GVdS4H2lNZM won't be the same (iv'e searched for :if (" and then i scanned with my eyes)
in my case i found 2 'bad looking' strings like above.

then i've made

Then grep "strings found earlier" -R /opt/zimbra/mailboxd
grep "string found earlier" -R /opt/zimbra/jetty

i've opened each file and if it was only cryptic looking stuf between function < and > iv'e moved the file into my admin folder. if it was normal zimbra code i had to look where the hacker's code is (between < and >) and deleted those lines. helpfull was another clean system to compare.
after i've cleaned up restart of server and open https

after more than 24 hours, the server seems to be not infected or hacked anymore.
if you are affected and have any questions - feel free to contact me

best regards from switzerland

Chrigi Hartmann
CEO http://www.calandacomp.ch
mqaroush
Posts: 42
Joined: Sun Aug 03, 2014 4:31 am

Re: CVE-2019-9670 being actively exploited

Post by mqaroush »

According to hack... there are files in /tmp directory .. if any one can help us about these files :

drwxr-x--- 2 root root 4096 Jun 9 08:14 hsperfdata_root
drwxr-x--- 2 zimbra zimbra 4096 Jun 9 08:38 hsperfdata_zimbra
-r--r--r-- 1 root root 11 May 18 09:26 .X0-lock
drwxrwxrwt 2 root root 4096 May 26 12:17 .X11-unix
-r--r--r-- 1 root root 11 May 26 12:17 .X1-lock
drwxrwxrwt 3 root root 4096 Feb 8 2015 .xrdp
-rw-rw---- 1 zimbra zimbra 6 Dec 21 2014 .UUID_NODEID
-rw-r----- 1 zimbra zimbra 16 Jun 9 08:38 .UUID_STATE
User avatar
maxxer
Outstanding Member
Outstanding Member
Posts: 224
Joined: Fri Oct 04, 2013 2:12 am
Contact:

Re: CVE-2019-9670 being actively exploited

Post by maxxer »

mqaroush wrote:According to hack... there are files in /tmp directory .. if any one can help us about these files
what kind of help you need? they're unlikely zimbra files, but without contents we cannot know. you should inspect them yourself, you can try moving them out to a location non accessible to zimbra user and see if everything still work
User avatar
maxxer
Outstanding Member
Outstanding Member
Posts: 224
Joined: Fri Oct 04, 2013 2:12 am
Contact:

Re: CVE-2019-9670 being actively exploited

Post by maxxer »

calandacomp wrote:if you are affected and have any questions - feel free to contact me
good to know there's still a chance to clean up the system :) thanks for sharing your cleanup steps!
freshfitz
Posts: 31
Joined: Fri Sep 12, 2014 10:25 pm

Re: CVE-2019-9670 being actively exploited

Post by freshfitz »

I have not been able to clean up, the hack has been coming back about every 12 hrs. I have ssh and 7071 blocked to public and always have. I've changed all my admin pws 3x. ssh, zimbra, and webmail admin.
The hack script still comes back. I finally blocked the attacking IP but soon enough it will probably come from another IP.


Release 8.7.10.GA.1829.UBUNTU12.64 UBUNTU12_64 FOSS edition.

2019-06-09 08:53:55,784 INFO [qtp1798286609-8914:https:https://x.x.x.x/service/soap] [name=zimbra;ip=89.248.171.57;port=35122;ua=ZimbraWebClient - SAF3 (Win)/5.0.15_GA_2851.RHEL5_64;] security - cmd=Auth; account=zimbra; protocol=soap;
2019-06-09 08:59:34,078 INFO [qtp1798286609-8960:https:https://x.x.x.x/service/soap] [name=zimbra;ip=89.248.171.57;port=36036;ua=ZimbraWebClient - SAF3 (Win)/5.0.15_GA_2851.RHEL5_64;] security - cmd=Auth; account=zimbra; protocol=soap;

89.248.171.57 - - [09/Jun/2019:12:59:33 +0000] "POST https://x.x.x.x./service/autodiscover/Autodiscover.xml HTTP/1.1" 503 13691 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0" 193
89.248.171.57 - - [09/Jun/2019:12:59:34 +0000] "POST https://x.x.x.x./service/soap HTTP/1.1" 200 586 "-" "python-requests/2.21.0" 3
89.248.171.57 - - [09/Jun/2019:12:59:34 +0000] "POST https://local:7071/service/proxy?target ... admin/soap HTTP/1.1" 200 587 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0" 6
89.248.171.57 - - [09/Jun/2019:12:59:35 +0000] "POST https://x.x.x.x/service/extension/clientUploader/upload HTTP/1.1" 200 99 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0" 697
89.248.171.57 - - [09/Jun/2019:12:59:36 +0000] "GET https://x.x.x.x/downloads/spam_filter.jsp HTTP/1.1" 200 6 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0" 977

I've followed this to cleanup - viewtopic.php?t=65932&start=120
User avatar
maxxer
Outstanding Member
Outstanding Member
Posts: 224
Joined: Fri Oct 04, 2013 2:12 am
Contact:

Re: CVE-2019-9670 being actively exploited

Post by maxxer »

freshfitz wrote:I have not been able to clean up, the hack has been coming back about every 12 hrs. I have ssh and 7071 blocked to public and always have. I've changed all my admin pws 3x. ssh, zimbra, and webmail admin.
The hack script still comes back. I finally blocked the attacking IP but soon enough it will probably come from another IP.
The hack comes from port 443, so your best pratices aren't enough this time.
There's likely a JSP file or something like that left around which allows them to come back in. If you cannot move your server with ZeXtras to a new one try installing a new server and copy over the jetty webapps folder.
And see this other post for other cleanup steps
calandacomp
Posts: 9
Joined: Tue Jun 20, 2017 7:32 am

Re: CVE-2019-9670 being actively exploited

Post by calandacomp »

yes, i confirm. it's IMPORTANT to have 443 CLOSED during my steps explained above. i really think this should be solution when you work 'clean' and take the time to identify the infected files.

good luck and feel free to contact me. would be interesting for me to help if i can!

maxxer wrote:
freshfitz wrote:I have not been able to clean up, the hack has been coming back about every 12 hrs. I have ssh and 7071 blocked to public and always have. I've changed all my admin pws 3x. ssh, zimbra, and webmail admin.
The hack script still comes back. I finally blocked the attacking IP but soon enough it will probably come from another IP.
The hack comes from port 443, so your best pratices aren't enough this time.
There's likely a JSP file or something like that left around which allows them to come back in. If you cannot move your server with ZeXtras to a new one try installing a new server and copy over the jetty webapps folder.
And see this other post for other cleanup steps
freshfitz
Posts: 31
Joined: Fri Sep 12, 2014 10:25 pm

Re: CVE-2019-9670 being actively exploited

Post by freshfitz »

I agree but I think there is a jsp file that they are accessing but it still looks like an attack though soap. I also have another server that I think did not get attacked but zimbra services have locked up twice due to the amount of soap attacks / requests.
Can't I just disable soap or have it goto page not foind, it's not needed is it? They are just single servers
Post Reply