Cant shake bruteforce monkey

Discuss your pilot or production implementation with other Zimbra admins or our engineers.
copowpow
Posts: 20
Joined: Mon Mar 26, 2018 3:34 pm

Cant shake bruteforce monkey

Post by copowpow »

I am having issues combating brute-force attempts on my ZCS server.

I have fail2ban in place but it is not helping in this situation, they just come back from a different IP address. My accounts keep getting locked out.Ive blocked entire country's (brazil, indonesia, china) with ip lists from ip2location.com

The only work around is to rename the account and create a new dummy account with the old name, let it lock out and forward the email to the old re-named address. Thats the only way I can keep some of my users from getting locked out, and it seems to be getting worse as accounts that were not targets are starting to become targets.

Is there anything I can do to help with this? Re-captcha on the login screen? Change the URL of the login page (does that even help?)?

Any suggestions would help greatly as we cant seem to get a break from these botnets!
User avatar
vavai
Advanced member
Advanced member
Posts: 174
Joined: Thu Nov 14, 2013 2:41 pm
Location: Indonesia
ZCS/ZD Version: 0
Contact:

Re: Cant shake bruteforce monkey

Post by vavai »

Hi
copowpow wrote:I am having issues combating brute-force attempts on my ZCS server.

I have fail2ban in place but it is not helping in this situation, they just come back from a different IP address. My accounts keep getting locked out.Ive blocked entire country's (brazil, indonesia, china) with ip lists from ip2location.com

The only work around is to rename the account and create a new dummy account with the old name, let it lock out and forward the email to the old re-named address. Thats the only way I can keep some of my users from getting locked out, and it seems to be getting worse as accounts that were not targets are starting to become targets.

Is there anything I can do to help with this? Re-captcha on the login screen? Change the URL of the login page (does that even help?)?

Any suggestions would help greatly as we cant seem to get a break from these botnets!
What is your main problem? Do you have some compromised account due to brute-force attack or does it simply too many attempt on SMTP connection and worry it would lead to high CPU usage?

Do you activating "failed login lockout policy" and it randomly lock some account due to failed login attempt?

I've never activate "failed login lockout policy" as it can be used by wrong people (or robot) to brute-force attack and affecting innocent account. What I do :

1. Enforce strong password
2. Activate fail2ban
3. Limit trusted network
4. Activate 2FA
5. If possible, change default port, something like disable port 25 or 587, activate other port as incoming port and submission port and adjust my cloud anti spam destination port.
User avatar
L. Mark Stone
Ambassador
Ambassador
Posts: 2796
Joined: Wed Oct 09, 2013 11:35 am
Location: Portland, Maine, US
ZCS/ZD Version: 10.0.6 Network Edition
Contact:

Re: Cant shake bruteforce monkey

Post by L. Mark Stone »

If you use fail2ban and a password lockout policy, just set the fail2ban threshold lower than the lockout policy.

In this way, a brute force attacker will get their IP blocked before the user's mailbox is set to Locked Out status; the legitimate user can keep using their email.

You can do the same thing with Zimbra's DoSFilter, instead of fail2ban if you wish.

Hope that helps,
Mark
___________________________________
L. Mark Stone
Mission Critical Email - Zimbra VAR/BSP/Training Partner https://www.missioncriticalemail.com/
AWS Certified Solutions Architect-Associate
User avatar
zimico
Outstanding Member
Outstanding Member
Posts: 225
Joined: Mon Nov 14, 2016 8:03 am
Location: Vietnam
ZCS/ZD Version: 8.8.15 P3
Contact:

Re: Cant shake bruteforce monkey

Post by zimico »

Hi Mark,
I have tested zimbra dosfilter with lower threshold and fail login policy. Dosfilter recoginzed bad ip but the account was still locked out.
Could you show me your current configuration.
Regards,
Minh.
User avatar
L. Mark Stone
Ambassador
Ambassador
Posts: 2796
Joined: Wed Oct 09, 2013 11:35 am
Location: Portland, Maine, US
ZCS/ZD Version: 10.0.6 Network Edition
Contact:

Re: Cant shake bruteforce monkey

Post by L. Mark Stone »

zimico wrote:Hi Mark,
I have tested zimbra dosfilter with lower threshold and fail login policy. Dosfilter recoginzed bad ip but the account was still locked out.
Could you show me your current configuration.
Regards,
Minh.
Just did a blog post on this:
https://www.missioncriticalemail.com/20 ... -together/

Please take a look and let me know if that works for you?

All the best,
Mark
___________________________________
L. Mark Stone
Mission Critical Email - Zimbra VAR/BSP/Training Partner https://www.missioncriticalemail.com/
AWS Certified Solutions Architect-Associate
User avatar
JDunphy
Outstanding Member
Outstanding Member
Posts: 889
Joined: Fri Sep 12, 2014 11:18 pm
Location: Victoria, BC
ZCS/ZD Version: 9.0.0_P39 NETWORK Edition

Re: Cant shake bruteforce monkey

Post by JDunphy »

L. Mark Stone wrote:
Just did a blog post on this:
https://www.missioncriticalemail.com/20 ... -together/
Mark
That is really well done Mark. Got me thinking that perhaps an enhancement should someone wish to take this further at the fw/packet level.

1) add a single entry via iptables that matches an ipset that is time based for removal
2) add the ip address dynamically from information provided by dosfilter

We have a 4 hr ipset and a 12 hr ipset that our incoming relays will put any bad actors into time out. I have seen 1000's of ip's added in a few minutes without load to the server. Super efficient and the bad guy burns all their ip's and hopefully goes away.

where this is in your /etc/sysconfig/iptables or whatever method you use for your fw rules.

Code: Select all


-A Block -m set --match-set blacklist24hr src -j DROP
and the ipset's have been created via:

Code: Select all

% ipset create blacklist4hr hash:ip hashsize 4096 timeout 14400
% ipset create blacklist24hr hash:ip hashsize 4096 timeout 86400
then you have a script looking for triggers from logs or whatever and they are added like thus:

Code: Select all

ipset add blacklist4hr X.X.X.X
Not as elaborate as fail2ban but ipsets are super efficient and scale massively and the block is in real-time without any restart of your firewall. If this was a multi-server zimbra install, one could do it on the parameter via syslog and a script looking for those ip addresses.Note: Because the ip's decay automatically and are removed there is nothing further to do after addition... ie. a hacker that has used TOR being a usage case and why you want to might want to decay those ip's.

Note: there are variations one could do without writing any scripts and using the ipset just with FW rules. Example: control number of connections per ip over a time interval.

Code: Select all

-A Block -m set --match-set someblacklist src -j DROP
-A Block -p tcp --dport 25 -m state --state NEW -m recent --set --name SMTP
-A Block -p tcp --dport 25 -m state --state NEW -m recent --update --seconds 60 --hitcount 10 --rttl --name SMTP -j SET --add-set someblacklistr src
#allow incoming traffic if not more than 10 connections in 60 seconds per ip
-A Block -m state --state NEW -m tcp -p tcp --dport 25 -j ACCEPT
User avatar
vavai
Advanced member
Advanced member
Posts: 174
Joined: Thu Nov 14, 2013 2:41 pm
Location: Indonesia
ZCS/ZD Version: 0
Contact:

Re: Cant shake bruteforce monkey

Post by vavai »

Hi Mark & JDunphy,

Thank you for share valuable discussion. I'll be take a chance to deploy what you described on your blog as It seems it will help me a lot to deal with brute-force attack and improving our anti spam strategy.
copowpow
Posts: 20
Joined: Mon Mar 26, 2018 3:34 pm

Re: Cant shake bruteforce monkey

Post by copowpow »

L. Mark Stone wrote: Just did a blog post on this:
https://www.missioncriticalemail.com/20 ... -together/

Please take a look and let me know if that works for you?

All the best,
Mark

Thank you Mark, I am implementing this tonight! I will report back once I get a few days under these new settings. Very glad that Zimbra is adding these features to the DoS protection!
Newt0n
Posts: 1
Joined: Wed Oct 24, 2018 1:23 pm

Re: Cant shake bruteforce monkey

Post by Newt0n »

L. Mark Stone wrote:If you use fail2ban and a password lockout policy, just set the fail2ban threshold lower than the lockout policy.

In this way, a brute force attacker will get their IP blocked before the user's mailbox is set to Locked Out status; the legitimate user can keep using their email.

You can do the same thing with Zimbra's DoSFilter, instead of fail2ban if you wish.

Hope that helps,
Mark
It might not run in this case because the brute force attack is using different IP so you can't block by IP because you can have 15 failed logins from 15 diferent IP in one minute.
copowpow
Posts: 20
Joined: Mon Mar 26, 2018 3:34 pm

Re: Cant shake bruteforce monkey

Post by copowpow »

Well it looks like the DoS filter is not working (zcs 8.8.8) , accounts still getting locked out after following Marks blog post,

As per https://wiki.zimbra.com/wiki/DoSFilter, it says to check /opt/zimbra/log/zmmailboxd.out and /opt/zimbra/log/sync.log

/opt/zimbra/log/zmmailboxd.out is packed with new Broken Pipe errors (see pastebin link, too large to post here):

https://paste.ee/p/fbZMZ


/opt/zimbra/log/sync.log shows:

Code: Select all

2018-10-22 18:33:07,219 INFO  [Thread-12] [] sync - ZeXtras Mobile stopped
2018-10-22 18:33:43,495 INFO  [main] [] sync - ZeXtras Mobile started
2018-10-22 18:42:11,002 INFO  [Thread-12] [] sync - ZeXtras Mobile stopped
2018-10-22 18:42:44,183 INFO  [main] [] sync - ZeXtras Mobile started




Not sure what to do now, can i fix these broken pipe errors so the thing actually works? I would really like this to work. Is there a log somewhere that shows what IP's its trying to block or is it supposed to be in the zmmailboxd.out ?
Post Reply