CVE-2019-9670 being actively exploited (Hacked Server)

[isra]

So I patched and restarted the server on Monday night... Seemed to work, and all was working on Tuesday.

Today I got a call asking if I knew why it was coming up with 403 (which it certainly wasn't on Tuesday). After much reading of logs and looking at whether ports were misconfigured, I decided to recheck the symptoms of this exploit.... And we've got 2 new .jsp files (Ajax.jsp and XZimbra.jsp) created today. These appear not to be present in our backup from last night.

Is there another exploit/bug?

i have the same 403 issue after clean mi server. How did you solve the misconfigured ports?

[tin]

I'm fairly sure there was nothing wrong with ports - we'd had a similar 403 issue last year that was caused by wrong ports.... This time it looks like the exploit has broken the web interface, and being the weekend I haven't looked into it yet. Monday job.

[halfgaar]

The malware is getting worse. Now if you delete if from /tmp it starts downloading in /var/tmp and there are no l.sh nor s.sh files around.
It's using wget to download the zmcat to the server if you delete it every 10-15 seconds.
Looked for jsp files and didn't find anything suspicious around.
Is there a way to prevent linux from creating the zmcat file for example? so that if deletes it immediately?
for the time being I removed the wget program to avoid the automatic download of the zmcat file to the server.

Why did you not do anything I suggested? Do you see suspecious entries with shell commands in the logs? Do you have suspicous keys in /opt/zimbra/.ssh/authorized_keys? Did you regenerate the keys?

I also think you're missing the point of the hack. You're asking to block the zmcat, but using your CPU for mining is one thing. Still having access is the real problem. Who knows what they will do next.

[maxxer]

Looked for jsp files and didn't find anything suspicious around.
Is there a way to prevent linux from creating the zmcat file for example? so that if deletes it immediately?

Did you try the dpkg/rpm commands to check for modified files?

To prevent zmcat creation create it yourself then chown root and chmod 600

[JDunphy]

I am working on a few tools to help with some proactive detection based on log analysis and came across greynoise today which can provide some information about the reputation of connecting ip's.

If anyone has any logs from the origin of these attacks, could you plug those ip's into greynoise and report back if they have been used previously. It will show you the other type of attacks coming from this ip address. Here is an example using their visualization tool from a nuisance ip hitting a zimbra server. https://viz.greynoise.io/ip/5.188.210.101. You can replace the ip or use their interface to add new ones. I have a curl command that does the same for tool writers. You have to grab the jq command to pretty print the JSON data and it will tell you the types of attacks they have participated in with this ip.

Code: Select all

% curl -s -XPOST -d 'ip=89.248.172.16' 'http://api.greynoise.io:8888/v1/query/ip' | jq '.'

Which will dump all the rows of attacks this ip has been previously involved in.

[halfgaar]

Maxxer, you may want to be more explicit in your blog post about changing the LDAP password. As I understand it, a local file inclusion hack was used to obtain the /opt/zimbra/conf/localconfig.xml, and the admin password in it was used to log in with account 'zimbra'. Meaning, all passwords were compromised.

Ultimately, I did:

Code: Select all

zmldappasswd <random>
zmldappasswd -r <random>
zmldappasswd -a <random>
zmldappasswd -n <random>
zmldappasswd -p <random>

zmmypasswd <random>
zmmypasswd --root <random>

# Probably a good idea:
zmcontrol restart

The LDAP ones respectively for: ldap_admin_password (the most important one), root password, amavis password, nginx password and postfix password.

[timido]

How can I replace the base from a more recent version (with a virus) into a healthy version. after launch through old snapshot

[halfgaar]

You mean restore an old snapshot but migrate data from your current version? I suppose it's different on the Network edition then it is on the open source edition.

How many accounts do you have? There is an account export function, per account, that you could use perhaps. It would be relatively easy to write a bash script to do that for all accounts. Like this in a loop:

Code: Select all

curl --insecure --silent --show-error --user admin:$admin_password "https://localhost:7071/home/$target_account/?fmt=tgz" > "$target_account.tgz"

You'll have to experiment with this, and subsequently restoring them.

With the Network edition you may have other options entirely.

[tin]

We're running 8.7.11. I will probably restore the jetty folder from a backup on Monday. Or is that a bad idea?

Well, I renamed the old "webapps" directory, made a new empty one, and restored just that folder... Got the web interface back to working... For us anyway.

I'm not convinced we're 100% clean and safe, but we had already been talking about moving to a simpler SMTP+IMAP server without all the fancy stuff, so that's how we're going to "fix" ours - just make a new server with different software

[maxxer]

How many accounts do you have? There is an account export function, per account, that you could use perhaps. It would be relatively easy to write a bash script to do that for all accounts. Like this in a loop:

this is unreliable. see bgo#101760. Nowadays better use ZeXtas migration tool, it's free for 30 days