Page 1 of 1

Seemingly untraceable login

Posted: Mon May 23, 2016 3:31 pm
by chauvetp
Hello,

We've been having some brute force login attempts (where the attackers appear to be trying 20 or so attempts for a user before moving on). Normally, these are easy to trace for me, but I cannot find the source of these. There is no oip in the log entries, just the ip of one of our Zimbra proxies. I don't know what I'm missing, or how these attempts are even coming through. Any advice on where to look here would be appreciated!


Here's an example below (I've left out the java stack trace for the failed logins). The IPs and usernames have been redacted.

mailbox.log on one of the Zimbra stores:
2016-05-22 13:00:02,046 WARN [qtp1937601231-4651783:https://ZSTORE_IP:7071/service/admin/soap/] [name=USER@OURDOMAIN;ip=PROXY_IP;] account - ldap auth for domain OURDOMAIN failed, fall back to zimbra default auth mechanism
com.zimbra.cs.account.AccountServiceException$AuthFailedServiceException: authentication failed for [USER@OURDOMAIN]
ExceptionId:qtp1937601231-4651783:https://ZSTORE_IP:7071/service/admin/soap/:1463936402046:531d9a48b8676902

2016-05-22 13:00:02,050 INFO [qtp1937601231-4651783:https://ZSTORE_IP:7071/service/admin/soap/] [name=USER@OURDOMAIN;ip=PROXY_IP;] SoapEngine - handler exception: authentication failed for [USER@OURDOMAIN], invalid password
2016-05-22 13:00:02,050 INFO [qtp1937601231-4651783:https://ZSTORE_IP:7071/service/admin/soap/] [name=USER@OURDOMAIN;ip=PROXY_IP;] soap - AuthRequest elapsed=148
2016-05-22 13:00:02,663 WARN [qtp1937601231-4651813:https://ZSTORE_IP:7071/service/admin/soap/] [name=USER@OURDOMAIN;ip=PROXY_IP;] account - ldap auth for domain OURDOMAIN failed, fall back to zimbra default auth mechanism
com.zimbra.cs.account.AccountServiceException$AuthFailedServiceException: authentication failed for [USER@OURDOMAIN]
ExceptionId:qtp1937601231-4651813:https://ZSTORE_IP:7071/service/admin/soap/:1463936402663:531d9a48b8676902

Code:account.AUTH_FAILED

zimbra.log on one of the proxies:
May 22 12:59:59 zproxy2 saslauthd[11500]: auth_zimbra: USER@OURDOMAIN auth failed: authentication failed for [USER@OURDOMAIN]
May 22 12:59:59 zproxy2 saslauthd[11500]: do_auth : auth failure: [user=USER@OURDOMAIN] [service=smtp] [realm=OURDOMAIN] [mech=zimbra] [reason=Unknown]
May 22 13:00:00 zproxy2 saslauthd[11501]: zmpost: url='https://ZSTORE_HOST:7071/service/admin/soap/' returned buffer->data='<soap:Envelope xmlns:soap="http://www.w3.org/2003/05/soap-envelope"><soap:Header><context xmlns="urn:zimbra"/></soap:Header><soap:Body><soap:Fault><soap:Code><soap:Value>soap:Sender</soap:Value></soap:Code><soap:Reason><soap:Text>authentication failed for [USER@OURDOMAIN]</soap:Text></soap:Reason><soap:Detail><Error xmlns="urn:zimbra"><Code>account.AUTH_FAILED</Code><Trace>qtp821866309-3692748:https://ZSTORE_IP:7071/service/admin/soap/:1463936400366:71b4000474b86f75</Trace></Error></soap:Detail></soap:Fault></soap:Body></soap:Envelope>', hti->error=''
May 22 13:00:00 zproxy2 saslauthd[11501]: auth_zimbra: USER@OURDOMAIN auth failed: authentication failed for [USER@OURDOMAIN]
May 22 13:00:00 zproxy2 saslauthd[11501]: do_auth : auth failure: [user=USER@OURDOMAIN] [service=smtp] [realm=OURDOMAIN] [mech=zimbra] [reason=Unknown]
May 22 13:00:00 zproxy2 saslauthd[11496]: zmpost: url='https://ZSTORE_HOST:7071/service/admin/soap/' returned buffer->data='<soap:Envelope xmlns:soap="http://www.w3.org/2003/05/soap-envelope"><soap:Header><context xmlns="urn:zimbra"/></soap:Header><soap:Body><soap:Fault><soap:Code><soap:Value>soap:Sender</soap:Value></soap:Code><soap:Reason><soap:Text>authentication failed for [USER@OURDOMAIN]</soap:Text></soap:Reason><soap:Detail><Error xmlns="urn:zimbra"><Code>account.AUTH_FAILED</Code><Trace>qtp777206150-5193478:https://ZSTORE_IP:7071/service/admin/soap/:1463936400867:525dc054ceb2ff22</Trace></Error></soap:Detail></soap:Fault></soap:Body></soap:Envelope>', hti->error=''


audit.log on one of the mailbox servers:
2016-05-22 13:00:02,049 WARN [qtp1937601231-4651783:https://ZSTORE_IP:7071/service/admin/soap/] [name=USER@OURDOMAIN;ip=PROXY_IP;] security - cmd=Auth; account=USER@OURDOMAIN; protocol=soap; error=authentication failed for [USER@OURDOMAIN], invalid password;
2016-05-22 13:00:02,665 WARN [qtp1937601231-4651813:https://ZSTORE_IP:7071/service/admin/soap/] [name=USER@OURDOMAIN;ip=PROXY_IP;] security - cmd=Auth; account=USER@OURDOMAIN; protocol=soap; error=authentication failed for [USER@OURDOMAIN], invalid password;



I don't see anything else of use in the nginx logs on the proxies or the maillog file on the MTAs.

Re: Seemingly untraceable login

Posted: Mon May 23, 2016 8:18 pm
by liverpoolfcfan
Check out this document. It should help you. https://wiki.zimbra.com/wiki/Log_Files#Logging_the_Originating_IP

Re: Seemingly untraceable login

Posted: Mon May 23, 2016 8:47 pm
by chauvetp
Hi liverpoolfcfan,

Thanks for the link. That's the odd thing. We almost always have the originating IP.

It's there for every session I can specifically identify as IMAP, ActiveSync, SMTP, and the Zimbra Web Client. Access to the admin port (7071) is only allowed from the other Zimbra servers and a very small number of internal IPs.

To confirm the settings in that link, I've double checked the following:
- the zimbra_http_originating_ip_header is set on all proxy servers.
- all our mail stores have the zimbraMailTrustedIP set with the proxies.

It just seems that these weird connections are somehow bypassing this (or accessing some service that doesn't process/pass along the header).

Re: Seemingly untraceable login

Posted: Mon May 23, 2016 9:14 pm
by liverpoolfcfan
In that case it is probably the smtpd submission port.

Look in the zimbra.log file for one of the failures, then look back a few lines and you should find connection messages similar to this

May 23 12:33:24 mail postfix/submission/smtpd[5884]: connect from mail-vk0-f49.google.com[209.85.213.49]
May 23 12:33:25 mail postfix/submission/smtpd[5884]: Anonymous TLS connection established from mail-vk0-f49.google.com[209.85.213.49]: TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)

Re: Seemingly untraceable login

Posted: Tue May 24, 2016 1:43 pm
by chauvetp
Thanks! That was what I was missing. It seems a bit difficult to track though. Is there a way to match the saslauthd message with the postfix/smtps/smtpd message? Or is it just a matter of proximity to the logs?

Alternatively, is there a way to log (in /var/log/maillog) the username that failed authentication instead of just the following?

May 22 13:00:00 zproxy1 postfix/smtps/smtpd[871]: warning: SASL authentication failure: Password verification failed
May 22 13:00:00 zproxy1 postfix/smtps/smtpd[871]: warning: unknown[46.161.40.200]: SASL PLAIN authentication failed: authentication failure



Either way, whatever is trying to login appears to be a botnet of some sort due to how much the IPs are changing.