Tracking down source of account lockouts

rsaeks · Post by **rsaeks** » Wed Feb 24, 2016 8:44 pm

Hi all,
We've recently been faced with an issue where we have people attempting to guess user account passwords and therefore locking out legitimate accounts. The lockout has been working and preventing access. To work around this I've spent some time consolidating our mailbox.log and audit.log files into a searchable syslog server (through vmware log insight) and have been able to pull together data from those log files about the number of invalid password attempts, number of lockouts, number of invalid account attempts and a breakdown of invalid password attempts group by user. (I've attached a screenshot of that in case anyone is interested) There are also notification rules setup from the log monitoring system where generally within 2 minutes of an account lockout we receive an E-Mail notification.
What we are now wondering is how we can track down the source of the attempts. We've found that when attempting to connect via IMAP or SMTP the source IP address appears, but most of the invalid password attempts that are causing issues are through SOAP. What we are able to see during the times are entries like the following:

2016-02-24 19:26:02,593 WARN [qtp1480581246-137387:https://192.168.40.8:7071/service/admin/soap/] [name=USERNAME@DOMAIN;ip=192.168.40.8;] security - cmd=Auth; account=USERNAME@DOMAIN; protocol=soap; error=authentication failed for [USERNAME@DOMAIN], invalid password;
2016-02-24 15:32:30,406 WARN [qtp1480581246-133399:https://192.168.40.8:7071/service/admin/soap/] [name=USERNAME@DOMAIN;ip=192.168.40.8;] security - cmd=Auth; account=USERNAME@DOMAIN; protocol=soap; error=authentication failed for [USERNAME@DOMAIN], account lockout;
The account lockouts / invalid passwords never seem to show a source IP address. Are there any other log files we might be able to use to try and locate that information? Is there some logging we might be able to increase to find out the IP address so we can possible look to block that in some manner?

Thanks for any insight!

jorgedlcruz · Post by **jorgedlcruz** » Thu Feb 25, 2016 5:08 am

Hi rsaeks,

This is a great Forum post, thank you. What you have been done with VMware LogInsight it's pretty cool, did you follow the Wiki to implement it? https://wiki.zimbra.com/wiki/Centralize ... og_Insight

Well, back to your post, you can always increase the level of the log to Debug, but be careful because the size and the amount of data you will have during DEBUG - https://wiki.zimbra.com/wiki/Using_log4 ... xd_Logging

Let us know if you are able to see more information in DEBUG

Best regards

howanitz · Post by **howanitz** » Thu Feb 25, 2016 9:47 am

This is something I have had to deal with, although the majority of cases here have /not/ been attempts through SOAP, so most the time I have been able to track down the ip address easily. I hope this is not a trend, and would like to hear a better answer also. Maybe we need to request a change in logging.

I have a simple script to monitor and alert me when an account goes into lockout mode, I have posted it here:

https://community.zimbra.com/collaborat ... /t/1141135

I don't think it will be of use to you, since your monitoring system is more sophisticated, but I mention it should it be useful to others. (It does alert almost immediately.) It monitors /opt/zimbra/log/audit.log as yours does.

rsaeks · Post by **rsaeks** » Thu Feb 25, 2016 5:53 pm

Thanks for the replies. I was looking at changing the debug levels a bit then them stumbled across this post:

http://community.zimbra.com/collaborati ... /t/1136676

Running: zmlocalconfig zimbra_http_originating_ip_header

my output states this is a null value key

Running: zmprov gcf zimbraMailTrustedIP returns nothing

It sounds like setting those values may add in the oip item into the requests. Would that be the private IP of the server, 127.0.0.1 and our pre-zimbra archiving solution private IP? I can post back my results.

howanitz · Post by **howanitz** » Tue Mar 01, 2016 7:56 am

Looks like this would give us the correct oip for soap logins from his report - have not tried it myself yet:

https://community.zimbra.com/collaborat ... 42#1589542

zmprov mcf +zimbraMailTrustedIP 192.168.5.3

zmmailboxdctl restart

rsaeks · Post by **rsaeks** » Tue Mar 01, 2016 1:12 pm

Keith - Thanks for that information! We went ahead and made the change and at that point began to see the oip field popping in for those accessing email through the web.

Since a majority of our users access Zimbra through the web interface we were able to layer in a tcpdump on the server looking at port 25 and 587 originating outside our private IP block and take the capture data into wireshark to get some analysis and visibility into the incoming connections that were causing issue and blocking those blocks on our firewall.

tcpdump -vv -X -x -s 1500 -i eth0 'src net !192.168.0.0/16 and (port 25 or 587)' -w /opt/zimbra/smtpcapture.pcap

liverpoolfcfan · Post by **liverpoolfcfan** » Thu Mar 03, 2016 3:32 pm

You can also use fail2ban or your own scripts to monitor the /var/log/zimbra.log file - smtpd authorization failures are logged there. The following regex is what I use in the configuration file.

failregex = .*[<HOST>]: SASL PLAIN authentication failed: authentication failure

jasggomes · Post by **jasggomes** » Thu Sep 28, 2017 7:56 am

Hi everyone,

Actually I got one user guessed password, and that make a mess...

SO, does anyone have a good script to monitor the locked accounts and send an email to admin to alert it? The one's mentioned above are no longer available.

I'm not using Vmware anymore, we moved to Hyper-v free one year ago ... maybe i switch it back ...

But for now I'm dealing with the aftermath of this mess.

Thanks in advance.

JG

zimico · Post by **zimico** » Mon Jan 08, 2018 2:59 am

Dear JG,
You can use zmauditswatch to email to if there is any account is locked out.
https://wiki.zimbra.com/wiki/Zmauditswatch
Just one of my issue is I can not activate it in boot event using systemd and init.d
Regards,
Minh.

Zimbra Forums