Possible Zimbra exploit need validation

[dfwtx]

I have a zinbra 8.8.15_GA_4859.FOSS. I have some strange traffic I notice. I see my zimbra server calling out on:

tcp 0 112 my_ip:465 5.188.206.203:40319 FIN_WAIT1 -
tcp 0 0 my_ip:465 212.70.149.71:10836 SYN_RECV -
tcp 0 0 my_ip:465 212.70.149.71:44112 SYN_RECV -

Zimbra shouldn't be calling out thru the ports and certainly not with FIN_WAIT and SYNC_RECV. So dug around and found a hidden .udev directory. Working on going thru the kernel modules now. chkrootkit seems to have found this:

Checking `chkutmp'... => possibly 336 deletion(s) detected in /var/run/utmp !
> The tty of the following user process(es) were not found
> in /var/run/utmp !
> ! RUID PID TTY CMD
> ! 27983 se=/opt/zimbra/mailboxd ncoding=UTF-8 -server -Dhttps.protocols=TLSv1,TLSv1.1,TLSv1.2 -Djdk.tls.client.protocols=TLSv1,TLSv1.1,TLSv1.2 -Djava.awt.headless=true -Dsun.net.inetaddr.ttl= -Dorg.apache.jasper.compiler.disablejsr199=true -XX:+UseG1GC -XX:SoftRefLRUPolicyMSPerMB=1 -XX:-Omchkutmp: nothing deleted

The process shouldn't be hidden, so that has me wondering what is going on. I don't have any ports open on this box except what is needed by Zimbra. I am wondering if there is an unknown exploit out that we might not know about. For now I have rebuilt the server and have this blocked so I can study it and figure out what is going on. So far have not figure out where they got in. But it is clear they loaded a kernel module and the kernel module was deleting entries form utmp.

Anyone seen this?

[xorcz]

Yes, I see similar:

Checking `chkutmp'... The tty of the following user process(es) were not found
in /var/run/utmp !
! RUID PID TTY CMD
! 27983 se=/opt/zimbra/mailboxd ncoding=UTF-8 -server -Dhttps.protocols=TLSv1,TLSv1.1,TLSv1.2 -Djdk.tls.client.protocols=TLSv1,TLSv1.1,TLSv1.2 -Djava.awt.headless=true -Dsun.net.inetaddr.ttl= -Dorg.apache.jasper.compiler.disablejsr199=true -XX:+UseG1GC -XX:SoftRefLRUPolicyMSPerMB=1 -XX:-Omchkutmp: nothing deleted

[jholder]

I think you have it backward. Those are inbound connections to port 465 (smtp auth) and are ephemeral:

The allocation of an ephemeral port is temporary and only valid for the duration of the communication session.

https://en.wikipedia.org/wiki/Ephemeral_port

tcp 0 112 my_ip:465 5.188.206.203:eport FIN_WAIT1 -
tcp 0 0 my_ip:465 212.70.149.71:eport SYN_RECV -
tcp 0 0 my_ip:465 212.70.149.71:eport SYN_RECV -

Where my_ip is your server and eport is the ephemeral inbound connection port from the connecting side.

For instance, if I run netsstat:

Code: Select all

$ sudo netstat -np | grep 56375
tcp        0      0 192.168.1.1:56375      217.70.184.11:993       ESTABLISHED 3256/thunderbird

56375 is the ephemeral port on the incoming connection and 993 is on the server.

[JDunphy]

I was just looking at the source from chkrootkit.org to this chkutmp.c and they are looking to see if ut.ut_time == 0 and if it is they say it is deleted. The program written in 2004 will open the utmp file and read structures of utmp entries. The problem I see is that over the years, some OS's have modified the utmp structure including 64bit support and time is now represented by a structure of tv_sec, tv_usec see: utmp(5). If this wasn't properly ported to the OS you are running, then they are overlaying a structure onto the file which doesn't work if their utmp structure being read doesn't match the code that writes utmp structure by other apps. There are a few functions getutent(3) that normally is used to safeguard these type of problems and should be thread safe for the writers ... but they are reading a utmp structure at a time and hoping for the best; so it may be a false positive when they find 0's to be in those locations when they reference the field.

Not saying you were NOT hacked but I would be suspicious of that code and also look into that. How did the chkutmp binary come to be on your OS and what version of the chkutmp are you running?

Code: Select all

        while (read(f, &ut, sz_ut) > 0 && curp <= endp) {
#if !defined(__sun)
            if (ut.ut_time == 0)
                del_cnt++;      /* ut_time shouldn't be zero */
#endif
            if (strlen(ut.ut_user) > 0) {
                strncpy(curp->ut_tty, ut.ut_line, UT_LINESIZE);
                curp->ut_pid = ut.ut_pid;
                curp->ut_type = ut.ut_type;
                i++;
                curp++;
            }
        }
        close(f);
        if (del_cnt > 0)
            printf("=> possibly %d deletion(s) detected in %s !\n",

Jim

[xorcz]

I dove into this because wazuh notified a few times:

Code: Select all

Received From: (mail) any->rootcheck
Rule: 521 fired (level 11) -> "Possible kernel level rootkit"
Portion of the log(s):

Process '28627' hidden from /proc. Possible kernel level rootkit.
title: Process '28627' hidden from /proc.

I have not found anything related in the logs.

chkrootkit is from
ftp://ftp.pangeia.com.br/pub/seg/pac/chkrootkit.tar.gz
seems to be 0.55 packed June 11 2021.

It is strange the chkrootkit always reads the same PID 27983, so it looks it probably reads it wrong.

[jholder]

From the FAQ:

How accurate is chkproc?
If you run chkproc on a server that runs lots of short time processes it could report some false positives. chkproc compares the ps output with the /proc contents. If processes are created/killed during this operation chkproc could point out these PIDs as suspicious.

Zimbra definitely has many procs that are short-lived.

> ! 27983 se=/opt/zimbra/mailboxd ncoding=UTF-8 -server -Dhttps.protocols=TLSv1,TLSv1.1,TLSv1.2 -Djdk.tls.client.protocols=TLSv1,TLSv1.1,TLSv1.2 -Djava.awt.headless=true -Dsun.net.inetaddr.ttl= -Dorg.apache.jasper.compiler.disablejsr199=true -XX:+UseG1GC -XX:SoftRefLRUPolicyMSPerMB=1 -XX:-Omchkutmp: nothing deleted

This might be a typo but encoding is missing the "e".

Try grepping your mailbox.log/zimbra.log/kernel messages for the PID. Also look for any zombie procs.

[xorcz]

I installed clean OS and zimbra and transfered data this way viewtopic.php?f=13&t=70488&p=304180#p304180

I noticed two suspicious activities:
1. outgoing traffic to DST=23.185.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=29238 DF PROTO=TCP SPT=35450 DPT=443 WINDOW=29200 RES=0x00 SYN URGP=0
it is https://pantheon.io/ so I will have to inspect SSL

2. process logged in /var/log/audit/audit.log
type=USER_CMD msg=audit(1648048693.564:11340): pid=28045 uid=998 auid=0 ses=5 msg='cwd="/opt/zimbra" cmd=2F6F70742F7A696D6272612F6C6962657865632F7A6D6D61696C626F78646D677220737461747573 terminal=? res=success'

it is the same command as here:
viewtopic.php?t=24561


Sorry to hear John Holder passed away

[jeastman]

1. outgoing traffic to DST=23.185.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=29238 DF PROTO=TCP SPT=35450 DPT=443 WINDOW=29200 RES=0x00 SYN URGP=0
it is https://pantheon.io/ so I will have to inspect SSL

I believe that is the Zimbra website (and dig seems to confirm that).

Code: Select all

;; ANSWER SECTION:
www.zimbra.com.		1777	IN	A	23.185.0.2

Check your crontab file. You should see:

Code: Select all

# Check zimbraVersionCheckURL for new update. 
# Only runs if this server matches zimbraVersionCheckServer 
# Only executes on zimbraVersionCheckInterval. min 2h interval
#
18 */2 * * * /opt/zimbra/libexec/zmcheckversion -c >> /dev/null 2>&1

The zmcheckversion script checks for posted update notifications on the Zimbra website.

Sorry to hear John Holder passed away

Yes, this is indeed sad news. Thank you for the sentiment.