Hello,
We've been having some brute force login attempts (where the attackers appear to be trying 20 or so attempts for a user before moving on). Normally, these are easy to trace for me, but I cannot find the source of these. There is no oip in the log entries, just the ip of one of our Zimbra proxies. I don't know what I'm missing, or how these attempts are even coming through. Any advice on where to look here would be appreciated!
Here's an example below (I've left out the java stack trace for the failed logins). The IPs and usernames have been redacted.
mailbox.log on one of the Zimbra stores:
2016-05-22 13:00:02,046 WARN [qtp1937601231-4651783:https://ZSTORE_IP:7071/service/admin/soap/] [name=USER@OURDOMAIN;ip=PROXY_IP;] account - ldap auth for domain OURDOMAIN failed, fall back to zimbra default auth mechanism
com.zimbra.cs.account.AccountServiceException$AuthFailedServiceException: authentication failed for [USER@OURDOMAIN]
ExceptionId:qtp1937601231-4651783:https://ZSTORE_IP:7071/service/admin/soap/:1463936402046:531d9a48b8676902
2016-05-22 13:00:02,050 INFO [qtp1937601231-4651783:https://ZSTORE_IP:7071/service/admin/soap/] [name=USER@OURDOMAIN;ip=PROXY_IP;] SoapEngine - handler exception: authentication failed for [USER@OURDOMAIN], invalid password
2016-05-22 13:00:02,050 INFO [qtp1937601231-4651783:https://ZSTORE_IP:7071/service/admin/soap/] [name=USER@OURDOMAIN;ip=PROXY_IP;] soap - AuthRequest elapsed=148
2016-05-22 13:00:02,663 WARN [qtp1937601231-4651813:https://ZSTORE_IP:7071/service/admin/soap/] [name=USER@OURDOMAIN;ip=PROXY_IP;] account - ldap auth for domain OURDOMAIN failed, fall back to zimbra default auth mechanism
com.zimbra.cs.account.AccountServiceException$AuthFailedServiceException: authentication failed for [USER@OURDOMAIN]
ExceptionId:qtp1937601231-4651813:https://ZSTORE_IP:7071/service/admin/soap/:1463936402663:531d9a48b8676902
Code:account.AUTH_FAILED
zimbra.log on one of the proxies:
May 22 12:59:59 zproxy2 saslauthd[11500]: auth_zimbra: USER@OURDOMAIN auth failed: authentication failed for [USER@OURDOMAIN]
May 22 12:59:59 zproxy2 saslauthd[11500]: do_auth : auth failure: [user=USER@OURDOMAIN] [service=smtp] [realm=OURDOMAIN] [mech=zimbra] [reason=Unknown]
May 22 13:00:00 zproxy2 saslauthd[11501]: zmpost: url='https://ZSTORE_HOST:7071/service/admin/soap/' returned buffer->data='<soap:Envelope xmlns:soap="http://www.w3.org/2003/05/soap-envelope ... r><context xmlns="urn:zimbra"/></soap:Header><soap:Body><soap:Fault><soap:Code><soap:Value>soap:Sender</soap:Value></soap:Code><soap:Reason><soap:Text>authentication failed for [USER@OURDOMAIN]</soap:Text></soap:Reason><soap:Detail><Error xmlns="urn:zimbra"><Code>account.AUTH_FAILED</Code><Trace>qtp821866309-3692748:https://ZSTORE_IP:7071/service/admin/soap/:1463936400366:71b4000474b86f75</Trace></Error></soap:Detail></soap:Fault></soap:Body></soap:Envelope>', hti->error=''
May 22 13:00:00 zproxy2 saslauthd[11501]: auth_zimbra: USER@OURDOMAIN auth failed: authentication failed for [USER@OURDOMAIN]
May 22 13:00:00 zproxy2 saslauthd[11501]: do_auth : auth failure: [user=USER@OURDOMAIN] [service=smtp] [realm=OURDOMAIN] [mech=zimbra] [reason=Unknown]
May 22 13:00:00 zproxy2 saslauthd[11496]: zmpost: url='https://ZSTORE_HOST:7071/service/admin/soap/' returned buffer->data='<soap:Envelope xmlns:soap="http://www.w3.org/2003/05/soap-envelope ... r><context xmlns="urn:zimbra"/></soap:Header><soap:Body><soap:Fault><soap:Code><soap:Value>soap:Sender</soap:Value></soap:Code><soap:Reason><soap:Text>authentication failed for [USER@OURDOMAIN]</soap:Text></soap:Reason><soap:Detail><Error xmlns="urn:zimbra"><Code>account.AUTH_FAILED</Code><Trace>qtp777206150-5193478:https://ZSTORE_IP:7071/service/admin/soap/:1463936400867:525dc054ceb2ff22</Trace></Error></soap:Detail></soap:Fault></soap:Body></soap:Envelope>', hti->error=''
audit.log on one of the mailbox servers:
2016-05-22 13:00:02,049 WARN [qtp1937601231-4651783:https://ZSTORE_IP:7071/service/admin/soap/] [name=USER@OURDOMAIN;ip=PROXY_IP;] security - cmd=Auth; account=USER@OURDOMAIN; protocol=soap; error=authentication failed for [USER@OURDOMAIN], invalid password;
2016-05-22 13:00:02,665 WARN [qtp1937601231-4651813:https://ZSTORE_IP:7071/service/admin/soap/] [name=USER@OURDOMAIN;ip=PROXY_IP;] security - cmd=Auth; account=USER@OURDOMAIN; protocol=soap; error=authentication failed for [USER@OURDOMAIN], invalid password;
I don't see anything else of use in the nginx logs on the proxies or the maillog file on the MTAs.
Seemingly untraceable login
-
- Elite member
- Posts: 1112
- Joined: Sat Sep 13, 2014 12:47 am
Re: Seemingly untraceable login
Check out this document. It should help you. https://wiki.zimbra.com/wiki/Log_Files# ... inating_IP
Re: Seemingly untraceable login
Hi liverpoolfcfan,
Thanks for the link. That's the odd thing. We almost always have the originating IP.
It's there for every session I can specifically identify as IMAP, ActiveSync, SMTP, and the Zimbra Web Client. Access to the admin port (7071) is only allowed from the other Zimbra servers and a very small number of internal IPs.
To confirm the settings in that link, I've double checked the following:
- the zimbra_http_originating_ip_header is set on all proxy servers.
- all our mail stores have the zimbraMailTrustedIP set with the proxies.
It just seems that these weird connections are somehow bypassing this (or accessing some service that doesn't process/pass along the header).
Thanks for the link. That's the odd thing. We almost always have the originating IP.
It's there for every session I can specifically identify as IMAP, ActiveSync, SMTP, and the Zimbra Web Client. Access to the admin port (7071) is only allowed from the other Zimbra servers and a very small number of internal IPs.
To confirm the settings in that link, I've double checked the following:
- the zimbra_http_originating_ip_header is set on all proxy servers.
- all our mail stores have the zimbraMailTrustedIP set with the proxies.
It just seems that these weird connections are somehow bypassing this (or accessing some service that doesn't process/pass along the header).
-
- Elite member
- Posts: 1112
- Joined: Sat Sep 13, 2014 12:47 am
Re: Seemingly untraceable login
In that case it is probably the smtpd submission port.
Look in the zimbra.log file for one of the failures, then look back a few lines and you should find connection messages similar to this
May 23 12:33:24 mail postfix/submission/smtpd[5884]: connect from mail-vk0-f49.google.com[209.85.213.49]
May 23 12:33:25 mail postfix/submission/smtpd[5884]: Anonymous TLS connection established from mail-vk0-f49.google.com[209.85.213.49]: TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)
Look in the zimbra.log file for one of the failures, then look back a few lines and you should find connection messages similar to this
May 23 12:33:24 mail postfix/submission/smtpd[5884]: connect from mail-vk0-f49.google.com[209.85.213.49]
May 23 12:33:25 mail postfix/submission/smtpd[5884]: Anonymous TLS connection established from mail-vk0-f49.google.com[209.85.213.49]: TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)
Re: Seemingly untraceable login
Thanks! That was what I was missing. It seems a bit difficult to track though. Is there a way to match the saslauthd message with the postfix/smtps/smtpd message? Or is it just a matter of proximity to the logs?
Alternatively, is there a way to log (in /var/log/maillog) the username that failed authentication instead of just the following?
Either way, whatever is trying to login appears to be a botnet of some sort due to how much the IPs are changing.
Alternatively, is there a way to log (in /var/log/maillog) the username that failed authentication instead of just the following?
May 22 13:00:00 zproxy1 postfix/smtps/smtpd[871]: warning: SASL authentication failure: Password verification failed
May 22 13:00:00 zproxy1 postfix/smtps/smtpd[871]: warning: unknown[46.161.40.200]: SASL PLAIN authentication failed: authentication failure
Either way, whatever is trying to login appears to be a botnet of some sort due to how much the IPs are changing.