Exploit Questions

Discuss your pilot or production implementation with other Zimbra admins or our engineers.
Post Reply
mhammett
Advanced member
Advanced member
Posts: 133
Joined: Sat Jul 19, 2014 7:07 am
ZCS/ZD Version: Release 8.6.0.GA.1153.UBUNTU14.64 U

Exploit Questions

Post by mhammett »

I have been reading other threads and blogs, but I've come up with a few questions.

To start off with, I had 8.6. I have a multi-server setup with 2x LDAP, 2x mailstore, and 2x MTA\Proxy. When I noticed high CPU usage, I looked around and found some forum threads and blogs. This was a couple weeks ago.

I installed the 8.6 patch and thought I had cleaned it, but apparently not. A couple days ago, I moved to 8.8.12 because I was seeing other weird issues.

Apparently, the servers weren't clean. Well, I only saw symptoms on one mailstore. The other has seemed clean the whole time. I've ran `dpkg -l zimbra* | grep ^ii | awk '{print $2}' | xargs debsums -c` to see what files may be different. There are several files different on both servers (known infected at one point and the other assumed never infected) that show changed. Given the second mailstore hasn't shown above a 1.0 load on a 4 CPU system during any of this, I'm assuming it's clean. The infected server often showed a load well into the 30s and 40s. Anyway, I only saw one file different between them, "/opt/zimbra/jetty_base/webapps/zimbra/public/login.jsp". Is that likely infected?

Code: Select all

Known infected server
root@Zimbra8-Mailstore1:/home/mhammett# dpkg -l zimbra* | grep ^ii | awk '{print $2}' | xargs debsums -c
/opt/zimbra/conf/localconfig.xml
/opt/zimbra/libexec/zmdiaglog  
/opt/zimbra/libexec/zmmailboxdmgr
/opt/zimbra/libexec/zmmailboxdmgr.unrestricted
/opt/zimbra/bin/zmthrdump
/opt/zimbra/bin/zmplayredo
/opt/zimbra/jetty_base/webapps/zimbraAdmin/WEB-INF/classes/messages/ZaMsg.properties
/opt/zimbra/jetty_base/webapps/zimbraAdmin/WEB-INF/jetty-env.xml
/opt/zimbra/jetty_base/webapps/zimbraAdmin/WEB-INF/web.xml
/opt/zimbra/jetty_base/webapps/service/WEB-INF/web.xml
/opt/zimbra/jetty_base/webapps/zimbra/WEB-INF/jetty-env.xml
/opt/zimbra/jetty_base/webapps/zimbra/WEB-INF/web.xml
/opt/zimbra/jetty_base/webapps/zimbra/public/login.jsp
/opt/zimbra/jetty_base/work/zimbra/jsp/org/apache/jsp/public_/Boot_jsp.class
/opt/zimbra/jetty_base/work/zimbra/jsp/org/apache/jsp/public_/Boot_jsp.java
/opt/zimbra/jetty_base/work/zimbra/jsp/org/apache/jsp/public_/Resources_jsp.class
/opt/zimbra/jetty_base/work/zimbra/jsp/org/apache/jsp/public_/Resources_jsp.java
/opt/zimbra/jetty_base/work/zimbra/jsp/org/apache/jsp/public_/error_jsp.class
/opt/zimbra/jetty_base/work/zimbra/jsp/org/apache/jsp/public_/error_jsp.java
/opt/zimbra/jetty_base/work/zimbra/jsp/org/apache/jsp/public_/launchZCS_jsp.class
/opt/zimbra/jetty_base/work/zimbra/jsp/org/apache/jsp/public_/launchZCS_jsp.java
/opt/zimbra/jetty_base/work/zimbra/jsp/org/apache/jsp/public_/login_jsp.class
/opt/zimbra/jetty_base/work/zimbra/jsp/org/apache/jsp/public_/login_jsp.java
/opt/zimbra/common/etc/java/cacerts
debsums: missing file /opt/zimbra/lib/patches/nginx-lookup.jar (from zimbra-patch package)
/opt/zimbra/common/lib/perl5/XML/SAX/ParserDetails.ini
/opt/zimbra/lib/ext/nginx-lookup/nginx-lookup.jar
root@Zimbra8-Mailstore1:/home/mhammett# ls -hal /opt/zimbra/jetty_base/webapps/zimbra/public/login.jsp
-rwxrwxrwx 1 zimbra zimbra 41K Jun 14 02:37 /opt/zimbra/jetty_base/webapps/zimbra/public/login.jsp
root@Zimbra8-Mailstore1:/home/mhammett# md5sum /opt/zimbra/jetty_base/webapps/zimbra/public/login.jsp
3c497b19d993c008f4211514a6bf21c0  /opt/zimbra/jetty_base/webapps/zimbra/public/login.jsp

Code: Select all

Assumed clean server
root@Zimbra8-Mailstore2:/home/mhammett# dpkg -l zimbra* | grep ^ii | awk '{print $2}' | xargs debsums -c
/opt/zimbra/conf/localconfig.xml
/opt/zimbra/libexec/zmdiaglog
/opt/zimbra/libexec/zmmailboxdmgr
/opt/zimbra/libexec/zmmailboxdmgr.unrestricted
/opt/zimbra/bin/zmthrdump
/opt/zimbra/bin/zmplayredo
/opt/zimbra/jetty_base/webapps/zimbraAdmin/WEB-INF/classes/messages/ZaMsg.properties
/opt/zimbra/jetty_base/webapps/zimbraAdmin/WEB-INF/jetty-env.xml
/opt/zimbra/jetty_base/webapps/zimbraAdmin/WEB-INF/web.xml
/opt/zimbra/jetty_base/webapps/service/WEB-INF/web.xml
/opt/zimbra/jetty_base/webapps/zimbra/WEB-INF/jetty-env.xml
/opt/zimbra/jetty_base/webapps/zimbra/WEB-INF/web.xml
/opt/zimbra/jetty_base/work/zimbra/jsp/org/apache/jsp/public_/Boot_jsp.class
/opt/zimbra/jetty_base/work/zimbra/jsp/org/apache/jsp/public_/Boot_jsp.java
/opt/zimbra/jetty_base/work/zimbra/jsp/org/apache/jsp/public_/Resources_jsp.class
/opt/zimbra/jetty_base/work/zimbra/jsp/org/apache/jsp/public_/Resources_jsp.java
/opt/zimbra/jetty_base/work/zimbra/jsp/org/apache/jsp/public_/error_jsp.class
/opt/zimbra/jetty_base/work/zimbra/jsp/org/apache/jsp/public_/error_jsp.java
/opt/zimbra/jetty_base/work/zimbra/jsp/org/apache/jsp/public_/launchZCS_jsp.class
/opt/zimbra/jetty_base/work/zimbra/jsp/org/apache/jsp/public_/launchZCS_jsp.java
/opt/zimbra/jetty_base/work/zimbra/jsp/org/apache/jsp/public_/login_jsp.class
/opt/zimbra/jetty_base/work/zimbra/jsp/org/apache/jsp/public_/login_jsp.java
/opt/zimbra/common/etc/java/cacerts
debsums: missing file /opt/zimbra/lib/patches/nginx-lookup.jar (from zimbra-patch package)
/opt/zimbra/common/lib/perl5/XML/SAX/ParserDetails.ini
/opt/zimbra/lib/ext/nginx-lookup/nginx-lookup.jar
root@Zimbra8-Mailstore2:/home/mhammett# ls -hal /opt/zimbra/jetty_base/webapps/zimbra/public/login.jsp
-rw-r--r-- 1 zimbra zimbra 40K Jun 14 02:37 /opt/zimbra/jetty_base/webapps/zimbra/public/login.jsp
root@Zimbra8-Mailstore2:/home/mhammett# md5sum /opt/zimbra/jetty_base/webapps/zimbra/public/login.jsp
e82ea1127ac694dc63d937a72a042977  /opt/zimbra/jetty_base/webapps/zimbra/public/login.jsp

mhammett
Advanced member
Advanced member
Posts: 133
Joined: Sat Jul 19, 2014 7:07 am
ZCS/ZD Version: Release 8.6.0.GA.1153.UBUNTU14.64 U

Re: Exploit Questions

Post by mhammett »

I was able to extract the original file from the .deb and ran an md5sum across all login.jsps. That one was indeed infected and I replaced it.

Is it common for all of those other files to be different?
mhammett
Advanced member
Advanced member
Posts: 133
Joined: Sat Jul 19, 2014 7:07 am
ZCS/ZD Version: Release 8.6.0.GA.1153.UBUNTU14.64 U

Re: Exploit Questions

Post by mhammett »

Here are some notes from earlier when I was working to clean up the infection.

At one point the load was into the 40s, possibly higher.

Code: Select all

root@Zimbra8-Mailstore1:/home/mhammett# top
top - 15:20:34 up 1 day,  5:32,  1 user,  load average: 28.93, 28.71, 27.21
Tasks: 268 total,   5 running, 263 sleeping,   0 stopped,   0 zombie
%Cpu(s): 60.3 us, 16.7 sy,  0.0 ni,  0.0 id, 22.9 wa,  0.0 hi,  0.2 si,  0.0 st
KiB Mem:   8174984 total,  7899044 used,   275940 free,   548572 buffers
KiB Swap:  8386556 total,    21284 used,  8365272 free.  2906484 cached Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
  652 zimbra    20   0  452056   6868      4 S 136.2  0.1  86:41.91 zmswatch
 1292 zimbra    20   0   13880   2412   2252 R  41.3  0.0   0:30.07 sed
 4285 zimbra    20   0 4730028  45044  18712 S  36.0  0.6   0:01.10 java
 4286 zimbra    20   0 4730028  42992  18724 S  28.1  0.5   0:00.86 java
32454 zimbra    20   0   10992   2020   1808 D  20.2  0.0   0:19.52 sed
 2952 zimbra    20   0   11000   1976   1776 R  11.2  0.0   0:04.16 sed
 1594 zimbra    20   0   10992   1988   1784 D  10.6  0.0   0:08.87 sed
31210 zimbra    20   0   11000   2004   1788 R   9.3  0.0   0:39.69 sed
 3782 zimbra    20   0   11000   2000   1808 R   4.3  0.0   0:01.11 sed
 2190 zimbra    20   0 5913932 2.183g  24608 S   3.3 28.0 141:21.90 java
17309 zimbra    20   0  194972  14444   5108 D   2.0  0.2  16:20.86 zmlogger
 1908 zimbra    20   0 3095644 1.065g  16832 S   1.0 13.7 107:12.12 mysqld
 4097 root      20   0   25104   3132   2476 R   0.7  0.0   0:00.16 top
32505 zimbra    20   0   13872   2396   2240 D   0.7  0.0   0:31.92 sed
 1035 root      20   0   19316   2020   1860 S   0.3  0.0   0:31.28 irqbalance
 1148 landsca+  20   0  233164  24936  11644 S   0.3  0.3   1:41.45 landscape-monit
 1932 zimbra    20   0   13872   2380   2220 D   0.3  0.0   0:22.98 sed
 2323 zimbra    20   0   10992   1960   1768 D   0.3  0.0   0:07.75 sed
 2618 zimbra    20   0   45784   9244   3692 S   0.3  0.1   0:59.50 zmstat-mysql
16903 zimbra    20   0 3867040 357868  20696 S   0.3  4.4   4:52.18 java
17308 zimbra    20   0   54584  21060   2636 S   0.3  0.3   2:10.59 /opt/zimbra/com
    1 root      20   0   33492   3276   2680 S   0.0  0.0   0:11.05 init
    2 root      20   0       0      0      0 S   0.0  0.0   0:00.04 kthreadd
    3 root      20   0       0      0      0 S   0.0  0.0   1:14.08 ksoftirqd/0
    5 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/0:0H
    7 root      20   0       0      0      0 S   0.0  0.0   1:28.18 rcu_sched
    8 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcu_bh
    9 root      rt   0       0      0      0 S   0.0  0.0   0:04.23 migration/0
   10 root      rt   0       0      0      0 S   0.0  0.0   0:00.61 watchdog/0
   11 root      rt   0       0      0      0 S   0.0  0.0   0:00.50 watchdog/1
   12 root      rt   0       0      0      0 S   0.0  0.0   0:04.21 migration/1
   13 root      20   0       0      0      0 S   0.0  0.0   1:26.67 ksoftirqd/1
   15 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/1:0H
   16 root      rt   0       0      0      0 S   0.0  0.0   0:01.40 watchdog/2
   17 root      rt   0       0      0      0 S   0.0  0.0   0:04.56 migration/2
   18 root      20   0       0      0      0 S   0.0  0.0   1:21.34 ksoftirqd/2
   20 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/2:0H
   21 root      rt   0       0      0      0 S   0.0  0.0   0:02.11 watchdog/3
   22 root      rt   0       0      0      0 S   0.0  0.0   0:04.62 migration/3
   23 root      20   0       0      0      0 S   0.0  0.0   1:43.09 ksoftirqd/3
   25 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/3:0H
   26 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kdevtmpfs
   27 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 netns
   28 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 perf
   29 root      20   0       0      0      0 S   0.0  0.0   0:00.16 khungtaskd
   30 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 writeback
   31 root      25   5       0      0      0 S   0.0  0.0   0:00.00 ksmd
   32 root      39  19       0      0      0 S   0.0  0.0   0:02.70 khugepaged
   33 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 crypto
   34 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kintegrityd
   35 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 bioset
   36 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kblockd
   37 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 ata_sff
   38 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 md
Here is what was running after I executed `/etc/init.d/zimbra stop` a couple of times.

Code: Select all

root@Zimbra8-Mailstore1:/home/mhammett# ps aux | grep zimbra | grep -v sed
zimbra    6316  0.2  0.0   3340   972 ?        S    22:17   0:00 /opt/zimbra/bin/zmqueuelog
zimbra    6386  0.0  0.0   4456   784 ?        S    22:17   0:00 sh -c ps aux | grep -v grep | grep '/tmp/.cache' | awk '{print $2}' | xargs kill -9 > /dev/null 2>&1;
zimbra    6391  0.0  0.0   7492   664 ?        S    22:17   0:00 xargs kill -9
root      6393  0.0  0.0  11764  2240 pts/1    S+   22:17   0:00 grep --color=auto zimbra
zimbra   15847  0.0  0.0   3340   972 ?        Ss   18:00   0:09 /opt/zimbra/libexec/zmmysqlstatus
zimbra   20502  0.0  0.0   3908  1648 ?        S    17:00   0:02 /opt/zimbra/bin/zmclientcertmgr
zimbra   20512  0.0  0.0   3900  1640 ?        S    17:00   0:02 /opt/zimbra/libexec/zmtrainsa
mhammett
Advanced member
Advanced member
Posts: 133
Joined: Sat Jul 19, 2014 7:07 am
ZCS/ZD Version: Release 8.6.0.GA.1153.UBUNTU14.64 U

Re: Exploit Questions

Post by mhammett »

I am having some issues, though I don't know if they're related to the infection, the upgrade, or something else.

Every now and then I get 502 errors on the webmail site and in Zimbra Desktop.

Here is what I found in nginx.log on MTA1 (where my desktop is currently connected). 10.1.8.63 is Zimbra8-Mailstore1 and 10.1.8.64 is Zimbra8-Mailstore2.

Code: Select all

2019/06/23 14:03:21 [info] 5035#0: *98512 epoll_wait() reported that client prematurely closed connection, so upstream connection is closed too while sending request to upstream, client: [redacted], server: zimbra8-mta1.ics-il.net, request: "POST /service/soap/WaitSetRequest HTTP/1.1", upstream: "https://10.1.8.63:443/service/soap/WaitSetRequest", host: "webmail.ics-il.net"
2019/06/23 14:03:29 [info] 5037#0: *98517 epoll_wait() reported that client prematurely closed connection, so upstream connection is closed too while reading response header from upstream, client: [redacted], server: zimbra8-mta1.[redacted], request: "POST /service/soap/WaitSetRequest HTTP/1.1", upstream: "https://10.1.8.64:443/service/soap/WaitSetRequest", host: "webmail.[redacted]"
2019/06/23 14:04:26 [info] 5037#0: *98523 epoll_wait() reported that client prematurely closed connection, so upstream connection is closed too while reading response header from upstream, client: [redacted], server: zimbra8-mta1.[redacted], request: "POST /service/soap/WaitSetRequest HTTP/1.1", upstream: "https://10.1.8.63:443/service/soap/WaitSetRequest", host: "webmail.[redacted]"
2019/06/23 14:04:34 [info] 5037#0: *98526 epoll_wait() reported that client prematurely closed connection, so upstream connection is closed too while sending request to upstream, client: [redacted], server: zimbra8-mta1.[redacted], request: "POST /service/soap/WaitSetRequest HTTP/1.1", upstream: "https://10.1.8.64:443/service/soap/WaitSetRequest", host: "webmail.[redacted]"
2019/06/23 14:05:31 [info] 5037#0: *98529 epoll_wait() reported that client prematurely closed connection, so upstream connection is closed too while sending request to upstream, client: [redacted], server: zimbra8-mta1.[redacted], request: "POST /service/soap/WaitSetRequest HTTP/1.1", upstream: "https://10.1.8.63:443/service/soap/WaitSetRequest", host: "webmail.[redacted]"
2019/06/23 14:05:39 [info] 5037#0: *98532 epoll_wait() reported that client prematurely closed connection, so upstream connection is closed too while sending request to upstream, client: [redacted], server: zimbra8-mta1.[redacted], request: "POST /service/soap/WaitSetRequest HTTP/1.1", upstream: "https://10.1.8.64:443/service/soap/WaitSetRequest", host: "webmail.[redacted]"
Here is what I get in Zimbra Desktop:

Code: Select all

Failure communicating with remote server. Please try again later.
Debug message: error while proxying request to target server: HTTP/1.1 502 Bad Gateway

Exception:

com.zimbra.common.service.ServiceException: error while proxying request to target server: HTTP/1.1 502 Bad Gateway
ExceptionId:sync-mbox-billing@ics-il.com:1561316399185:11f1b5476337f349
Code:service.PROXY_ERROR Arg:(url, STR, "https://webmail.ics-il.net/service/soap/SyncRequest")
	at com.zimbra.common.service.ServiceException.PROXY_ERROR(ServiceException.java:323)
	at com.zimbra.common.soap.SoapHttpTransport.invoke(SoapHttpTransport.java:247)
	at com.zimbra.common.soap.SoapHttpTransport.invoke(SoapHttpTransport.java:164)
	at com.zimbra.common.soap.SoapTransport.invoke(SoapTransport.java:407)
	at com.zimbra.common.soap.SoapTransport.invokeWithoutSession(SoapTransport.java:393)
	at com.zimbra.cs.mailbox.ZcsMailbox.sendRequest(ZcsMailbox.java:690)
	at com.zimbra.cs.mailbox.ZcsMailbox.sendRequest(ZcsMailbox.java:652)
	at com.zimbra.cs.mailbox.ZcsMailbox.sendRequest(ZcsMailbox.java:647)
	at com.zimbra.cs.mailbox.ZcsMailbox.sendRequest(ZcsMailbox.java:640)
	at com.zimbra.cs.mailbox.ZcsMailbox.sendRequest(ZcsMailbox.java:636)
	at com.zimbra.cs.mailbox.ZcsMailbox.sendRequest(ZcsMailbox.java:632)
	at com.zimbra.cs.mailbox.ZcsMailbox.sendRequest(ZcsMailbox.java:628)
	at com.zimbra.cs.mailbox.ZcsMailbox.sendRequestWithNotification(ZcsMailbox.java:624)
	at com.zimbra.cs.mailbox.DeltaSync.sync(DeltaSync.java:103)
	at com.zimbra.cs.mailbox.DeltaSync.sync(DeltaSync.java:89)
	at com.zimbra.cs.mailbox.MailboxSync.sync(MailboxSync.java:192)
	at com.zimbra.cs.mailbox.ZcsMailbox.sync(ZcsMailbox.java:125)
	at com.zimbra.cs.mailbox.ZcsMailbox.syncOnTimer(ZcsMailbox.java:106)
	at com.zimbra.cs.mailbox.SyncMailbox$1.run(SyncMailbox.java:279)
	at java.util.TimerThread.mainLoop(Unknown Source)
	at java.util.TimerThread.run(Unknown Source)
If issue persists please post description and debug information here.
Here is what I get in the web browser:

Code: Select all

HTTP ERROR 502
Problem accessing ZCS upstream server. Cannot connect to the ZCS upstream server. Connection is refused.
Possible reasons:

upstream server is unreachable
upstream server is currently being upgraded
upstream server is down
Please contact your ZCS administrator to fix the problem.

Powered by Nginx-Zimbra://








After some amount of time, it just starts working again. Some amount of time after that, it fails again.
Post Reply