Stopping attacks in real-time

Have a great idea for extending Zimbra? Share ideas, ask questions, contribute, and get feedback.
juneros
Posts: 2
Joined: Wed May 17, 2023 6:25 pm

Re: Stopping attacks in real-time

Post by juneros »

I recommend not using chat gpt for this.
phoenix
Ambassador
Ambassador
Posts: 27272
Joined: Fri Sep 12, 2014 9:56 pm
Location: Liverpool, England

Re: Stopping attacks in real-time

Post by phoenix »

juneros wrote: Wed May 17, 2023 6:26 pm I recommend not using chat gpt for this.
How about expanding on the reason you don't recommend using chat gpt.
Regards

Bill

Rspamd: A high performance spamassassin replacement

Per ardua ad astra
juneros
Posts: 2
Joined: Wed May 17, 2023 6:25 pm

Re: Stopping attacks in real-time

Post by juneros »

Chatbots are often limited in their capability to understand human language and accurately respond to customers. Additionally, they may lack the ability to recognize customer emotions or nuances in customer conversations, leading to a less-than-ideal customer interaction. Furthermore, chatbot customer service can be frustrating for customers, as they may find themselves repeating the same information or being asked the same questions multiple times without the help of a human operator.
User avatar
JDunphy
Outstanding Member
Outstanding Member
Posts: 889
Joined: Fri Sep 12, 2014 11:18 pm
Location: Victoria, BC
ZCS/ZD Version: 9.0.0_P39 NETWORK Edition

Re: Stopping attacks in real-time

Post by JDunphy »

juneros wrote: Thu May 18, 2023 7:51 am Chatbots are often limited in their capability to understand human language and accurately respond to customers. Additionally, they may lack the ability to recognize customer emotions or nuances in customer conversations, leading to a less-than-ideal customer interaction. Furthermore, chatbot customer service can be frustrating for customers, as they may find themselves repeating the same information or being asked the same questions multiple times without the help of a human operator.
Sorry I could not resist. ;-) ;-)
chat.openai.com (ChatGPT-4) wrote: "Thank you for contributing to the discussion. Indeed, chatbots, like all technologies, come with certain limitations. However, this particular thread is focused on exploring ModSecurity 3, Nginx, and Zimbra-specific rules to develop real-time threat mitigation strategies. Your insights into chatbot technologies are valuable and it would be great if you could initiate a separate thread for this topic. "
But seriously, I have no intention of integrating chatbots into this. :-)

So where are we on this? First, I pulled my 24hr ipset for an ipset without timeouts to see if it's possible to reduce their frequency and if there are any savings in load on a zimbra server as a result. This means I now have a growing and vetted list of attacking ip's that are accessing this zimbra host by its ip address only in contrast to its FQDN. Contacting this server makes no sense given it has no public dns entry. Some of these ip's escalated their queries and begin adding Host:zmhostname in the request header in addition to looking for previous exploits. I am capturing a great amount of information for these requests via nginx.access.log + modsec_audit.log + modsec_debug.log and have started to analyze how and what they were attempting.

I may post the ip's which is now approaching 200 to a github area with the obfuscated nginx.access.logs for those requests and how they ended up in this list. Maybe the start of a threat network for zimbra. Early days but I am interested how they are manipulating the request headers and bodies for potential future and specific zimbra rules.

Currently, I have 1 zimbra specific rule and the entire OWASP core rule set that is targeting these attacks/bots. I continue to see no issues with normal zimbra traffic or admin zimbra traffic with these rules in place. At some point, I need to change the paranoia level to see how far we could go before a FP. I should also see if Zimbra has any test suites they might want to share and have it run against the rules.

Another issue I am currently thinking about is after a user has authenticated and left themselves logged in. Those session and trust tokens.... It is not uncommon to find user's clicking on attack vectors that could allow a 3rd party to harvest their browser cookies and then login to the zimbra server as the user. Whitelisting users would allow that attack to continue and bypass the rules. For that reason, I am considering options how to handle this. One way is to allow the admin to choose and use built-in paranoia levels to dictate what/how this is handled.

Jim
User avatar
JDunphy
Outstanding Member
Outstanding Member
Posts: 889
Joined: Fri Sep 12, 2014 11:18 pm
Location: Victoria, BC
ZCS/ZD Version: 9.0.0_P39 NETWORK Edition

Re: Stopping attacks in real-time

Post by JDunphy »

I have a tangent idea. I have this perl program/daemon that uses a single UDP socket with a Rijndael cipher for encryption/decryption to pass information to peers of itself running with FW access where this daemon can add an ip address to an ipset.

It is fairly efficient since it opens one socket, changes the ip address, sends the datagram and then continues going through a group of peers (ip addresses of zimbra servers) until everyone has the datagram which contains an ip address and count which I will change to represent the modsecurity rule number. The receive side is handled with a select on this socket so it is not looping and can process the packet in an event driven fashion. The program is both a client/server and listens to this udp socket but I will update the code so that it also will listen on a unix domain socket which the nginx connector can feed based on zimbra modsecurity rules and paranoia levels. That means changing the lua code which is currently a fork/exec via that action statement to a write to this unix domain socket. That solves a performance issue as it eliminates extra system calls, the thread re-entrant issue for some external programs, and also can allow further input sanitization of input should nginx, or the nginx connector be tricked. Inside this program is where ipset, fail2ban or other programs can be executed from and it can work in a cooperative fashion with other peers based on each admin's trust relationship with these peers - ie) password for encrypt/decrypt, agreed upon port, fw incoming/outgoing access to peers for this port.

I was using this program to watch nginx.access.log and talking to peer zimbra instances that worked as a collective adding ip's to ipsets to temporarily block for some type of attacks. It came about 3-4 years ago when we had active attacks against the community's unpatched zimbra sites which is how check_attacks.pl came about from which morphed into a rewrite of that for testing the concept of a poor man's threat network.

To make this work, I will bring up another modsecurity 3 connector with Zimbra 9 which I have around to test patches against but is normally turned off and the two zimbra instances can start sharing ip addresses using that single ip address vs FQDN modsecurity rule that I have thus far. I have these zimbra instances with different providers and in different data centers so it will be interesting to see if that makes any difference in threat mitigation. The zimbra 9 instance will start with the ip's from my Zimbra 8 instance with it's ipset.

The goal continues to see if we can prevent active attacks and this allows me to tie up a few more loose ends of how this might work with cooperating sites.

I have not put a lot of time into this recently as Stanley Cup is on so as a Canadian, I am required to watch all games. ;-) :-) ... but this is where I am headed.

I am still grappling with how to handle a stolen credential like a session token, etc. Given the cookies have been stolen in the scenario I am targeting, it seems to me that there could be information in the request header or request bodies that we would be privy to that someone re-playing those tokens might not have. So in a sense, fingerprinting the authenticated connection.

Jim
User avatar
JDunphy
Outstanding Member
Outstanding Member
Posts: 889
Joined: Fri Sep 12, 2014 11:18 pm
Location: Victoria, BC
ZCS/ZD Version: 9.0.0_P39 NETWORK Edition

Re: Stopping attacks in real-time

Post by JDunphy »

Posting a few lua scripts today.... If you are already running fail2ban, this might be the easiest way to add ip's to your existing jails with custom rules.

Code: Select all

% cat tell2syslog.lua

-- Note: update /etc/rsyslog.conf so that local2.info goes to a file that you are watching
--       For example, fail2ban could be looking for patterns
--        verify location of logger (/usr/bin or /bin, etc)
function send_to_syslog(remote_addr, rule_id)
    local cmd = "/bin/logger -p local2.info zimbra says block this ip address " .. remote_addr .. " because of rule " .. rule_id
    os.execute(cmd)
end

function main()

    send_to_syslog(m.getvar("REMOTE_ADDR"), m.getvar("TX.ruleid"))

end
lua can write to syslog directly but would require an additional library to install and to keep the above example simple without installing more lua code; I am using os.execute included in the base install. You would then be able to parse this logfile with tools like fail2ban and block the ip address within a few seconds vs real-time which is probably good enough. For example, many bots attack our zimbra servers with ip addresses and no Host: header. This would stop and add their ip's to a log where one could grab the ip's and block them from future access.

Code: Select all

# missing Host: header
SecRule &REQUEST_HEADERS:Host "@eq 0" \
    "id:950401,\
    phase:1,\
    block,\
    log,\
    msg:'Missing Host header',\
    tag:'application-multi',\
    tag:'language-multi',\
    tag:'platform-multi',\
    tag:'attack-bot',\
    setvar:tx.ruleid=%{rule.id},t:none"

# if ruleid is set, we call lua script to alert other programs
SecRule TX:ruleid "!@eq 0" \
    "id:12346,phase:1,deny,nolog,t:none,\
     exec:/etc/nginx/modsec/tell2syslog.lua"
I continue to experiment with this:

Code: Select all

cat send2daemon.lua

-- Caveat:
--     permissions: /tmp/modsec.fd
--     luarocks install luasocket
--              /usr/lib64/lua/5.3/socket/unix.so 

local socket = require("socket.unix")

-- no state so no way to not open/close the socket each time
function add_to_blacklist(remote_addr, rule_id)
    -- create and connect to the Unix domain socket
    local unix_socket = assert(socket())
    assert(unix_socket:connect("/tmp/modsec.fd"))
    
    -- prepare the datagram
    local datagram = remote_addr .. ":" .. rule_id
    
    -- send the datagram
    unix_socket:send(datagram)
    
    -- close the Unix socket
    unix_socket:close()

    local cmd = "/bin/logger -p local2.info NETWORK zimbra is adding blacklist24hr " .. remote_addr .. " with rule " .. rule_id
    os.execute(cmd)
end

function main()

    add_to_blacklist(m.getvar("REMOTE_ADDR"), m.getvar("TX.ruleid"))

end
Where I have another program reading from /tmp/modsec.fd and adding the ip's into ipset's locally and sending to peers that do likewise. I'll post that program as it become simpler. What I don't like about this is that I had to install luasocket so yet more potential code on a production server. Worse, there is a class of attacks that I am not able to get in front of yet with modsecurity 3 rules which include contacting the server with http://mail.example.com:443/ vs https://mail.example.com:443/ ... nginx.log will tell you that:

Code: Select all

client sent plain HTTP request to HTTPS port while reading client request headers
There are few others like this which has me asking if I have to watch the logs anyway then why bother with lua that add more installation dependencies. We still get all the other rules in the CRS protecting against tons of different sort of attacks so still plugging along in this investigation. A lot of my struggle is not knowing how to write rules and finding documentation that works with the version I am running on my zimbra server.

One positive, our web servers which already use modsecurity are about to get an upgrade in I can pretty much now understand all the OWASP CRS rules which has been my teacher on how to write new rules. Baby steps but taking way too long.

Jim
User avatar
L. Mark Stone
Ambassador
Ambassador
Posts: 2796
Joined: Wed Oct 09, 2013 11:35 am
Location: Portland, Maine, US
ZCS/ZD Version: 10.0.6 Network Edition
Contact:

Re: Stopping attacks in real-time

Post by L. Mark Stone »

JDunphy wrote: Thu May 25, 2023 4:22 pm
<snip>

Worse, there is a class of attacks that I am not able to get in front of yet with modsecurity 3 rules which include contacting the server with http://mail.example.com:443/ vs https://mail.example.com:443/ ... nginx.log will tell you that:

Code: Select all

client sent plain HTTP request to HTTPS port while reading client request headers
<snip>

Jim
Hi Jim,

This really is fascinating and useful, thank you.

Wouldn't this particular probe however be easy enough to include in Fail2ban, provided we configured a jail for /opt/zimbra/log/nginx.log?

And can you help me to understand why it's malicious please? (Never mind... I see the myriad of GET requests...)

Thanks, and all the best,
Mark
___________________________________
L. Mark Stone
Mission Critical Email - Zimbra VAR/BSP/Training Partner https://www.missioncriticalemail.com/
AWS Certified Solutions Architect-Associate
User avatar
JDunphy
Outstanding Member
Outstanding Member
Posts: 889
Joined: Fri Sep 12, 2014 11:18 pm
Location: Victoria, BC
ZCS/ZD Version: 9.0.0_P39 NETWORK Edition

Re: Stopping attacks in real-time

Post by JDunphy »

L. Mark Stone wrote: Thu May 25, 2023 7:10 pm Wouldn't this particular probe however be easy enough to include in Fail2ban, provided we configured a jail for /opt/zimbra/log/nginx.log?
Hi Mark,

Yes. That is the conclusion I am coming to also given the simplicity of watching logs for blocking a lot more than we currently do. One doesn't even need lua to log if one still wanted to add modsecurity for the core rules and use fail2ban to mop up ip's... Something like this to any rule:

Code: Select all

    log,\
    msg:'Blocked IP address: %{REMOTE_ADDR}',\
Now because I can't give up ;-), my next plan of attack is to finger print these requests using the request bodies/headers to see if one could identify bots and tools that escape the normal core rule checks identified in the text files - (scanners-users-agent.data and crawlers-user-agents.data) with over 120. I am determined to somehow find a use for lua and extending modsecurity it would appear. ;-) If I am lucky, I can find a pattern where we finger print browsers with some level of confidence and everything else will be a bot. Goal isn't to be 100% but knock out some of the sophisticated tools that can drill down on a zimbra server looking for attack vectors.

I have walked back from doing this as a plugin as I want to be able to get in front of the core rules so I'll end up with a zimbra.conf containing a bunch of rules, some lua scripts necessary for more advanced rules that can be enabled, and a one shot shell script to build, install, and test everything. At some point, I need to package it up because I don't want to install all this development cruft on a production server. Just what is required to protect a zimbra servers should someone need to enable this to an existing zimbra server. That is where I am heading with all these detours in between. :-) ;-)

Jim
User avatar
JDunphy
Outstanding Member
Outstanding Member
Posts: 889
Joined: Fri Sep 12, 2014 11:18 pm
Location: Victoria, BC
ZCS/ZD Version: 9.0.0_P39 NETWORK Edition

Re: Stopping attacks in real-time

Post by JDunphy »

I decided to run fail2ban (never used it before) and have this as my action to simply add ip's to my ipset we are using with Modsecurity 3.

Code: Select all

% cat /etc/fail2ban/action.d/nginx.conf 
# Fail2Ban configuration file
#
# Author: J Dunphy 5/26/2023
#

[Definition]

# Option:  actionstart
# Notes.:  command executed once at the start of Fail2Ban.
# Values:  CMD
#
actionstart =

# Option:  actionstop
# Notes.:  command executed once at the end of Fail2Ban
# Values:  CMD
#
actionstop =

# Option:  actioncheck
# Notes.:  command executed once before each actionban command
# Values:  CMD
#
actioncheck =

# Option: actionban
# Notes: command executed when banning an IP. When the IP is banned, add it to the ipset blacklist.
# Tags:  <ip>  IP address
#        <failures>  number of failures
#        <time>  unix timestamp of the ban time
#
actionban = ipset add blacklist24hr <ip> -exist
            /bin/logger -p local2.info "NETWORK fail2ban zimbra is adding <ip> to blacklist24hr"

# Option: actionunban
# Notes: command executed when unbanning an IP. No action for unban is taken as blacklist removal is not required.
# Tags:  <ip>  IP address
#        <failures>  number of failures
#        <time>  unix timestamp of the ban time
#
actionunban =

[Init]
Nothing fancy other than setting ban time to 60 seconds because I have no actionunban and want ipset to handle everything so the bantime expires with fail2ban no longer managing it. I had tried 1 second but fail2ban optimized the ban completely out and it never happened.

Here is an example of a filter for nginx.log

Code: Select all

% cat /etc/fail2ban/filters.d/nginx.log
#
# Fail2Ban configuration file
#
#

[Definition]

# matching: client sent plain HTTP request to HTTPS port while reading client request headers, client: 64.227.164.183, server
#           client sent invalid method while reading client request line, client: 36.225.122.233, server:
#failregex = ^.*client: <HOST>, server:.*
failregex = client\s+sent\s+plain\s+HTTP\s+request\s+to\s+HTTPS\s+port\s+while\s+reading\s+client\s+request\s+headers,\s+client:\s+<HOST>,\s+server
            client\s+sent\s+invalid\s+method\s+while\s+reading\s+client\s+request\s+line,\s+client:\s+<HOST>,\s+server
ignoreregex =
and my snippet I append to the jail.conf file to call all this

Code: Select all

[nginx]
enabled  = true
port     = http,https
filter   = nginx
logpath  = /opt/zimbra/log/nginx.log
maxretry = 1
#bantime  = 86400
# ipset is managing the jail itself, no actionunban defined in action.d/nginx.conf
bantime  = 60
action   = nginx
I test it like this:

Code: Select all

% fail2ban-regex /opt/zimbra/log/nginx.log /etc/fail2ban/filter.d/nginx.conf --print-all-matched
The ipset exists and was created like this:

Code: Select all

# ipset create blacklist24hr hash:ip hashsize 4096 timeout 86400
My FW rule is:

Code: Select all

-A Input -m set --match-set blacklist24hr src -j DROP
Note: ipsets have the concept of save/restore if you want them to persist after a reboot or you can swap them if you want to change the parameters mid run from saying 24hr to shorter/longer intervals or no expiration.

We are blocking somethings with modsecurity 3 and lower level attacks on the nginx server with fail2ban that modsecurity 3 will not see. In both cases, they end up in an ipset called blacklist24hr. I'll watch my nginx.log to see if better regex's could be found as I learn about this tool.

Thanks to Mark Stone for the discussion on fail2ban.

Jim
Last edited by JDunphy on Sat May 27, 2023 10:39 pm, edited 1 time in total.
User avatar
L. Mark Stone
Ambassador
Ambassador
Posts: 2796
Joined: Wed Oct 09, 2013 11:35 am
Location: Portland, Maine, US
ZCS/ZD Version: 10.0.6 Network Edition
Contact:

Re: Stopping attacks in real-time

Post by L. Mark Stone »

Thanks Jim for the conversation and the feedback! I updated my Fail2Ban blog post (https://www.missioncriticalemail.com/20 ... practices/) for those of us that continue to use "route" as the ban action, instead of using ipsets like you are doing, and I gave you a shoutout there.

Best regrds to all,
Mark

P.S. You'll see in your fail2ban logs like this:

Code: Select all

root@mail:~# grep nginx /var/log/fail2ban.log
2023-05-27 21:03:44,839 fail2ban.jail           [6613]: INFO    Creating new jail 'zimbra-nginx'
2023-05-27 21:03:44,839 fail2ban.jail           [6613]: INFO    Jail 'zimbra-nginx' uses pyinotify {}
2023-05-27 21:03:44,843 fail2ban.filter         [6613]: INFO    Added logfile: '/opt/zimbra/log/nginx.log' (pos = 0, hash = 4d041668215a2f24daf20544843b64bddd6d164b)
2023-05-27 21:03:47,303 fail2ban.jail           [6613]: INFO    Jail 'zimbra-nginx' started
2023-05-27 21:03:47,306 fail2ban.filter         [6613]: WARNING Error decoding line from '/opt/zimbra/log/nginx.log' with 'UTF-8'. Consider setting logencoding=utf-8 (or another appropriate encoding) for this jail. Continuing to process line ignoring invalid characters: b'2023/05/27 20:56:20 [info] 1159#0: *26076473 client sent invalid method while reading client request line, client: 80.66.66.140, server: mail.missioncriticalemail.com, request: "\x03\x00\x00/*\xe0\x00\x00\x00\x00\x00Cookie: mstshash=Administr"\n'
2023-05-27 21:03:47,307 fail2ban.filter         [6613]: INFO    [zimbra-nginx] Found 80.66.66.140 - 2023-05-27 20:56:20
2023-05-27 21:03:47,504 fail2ban.actions        [6613]: NOTICE  [zimbra-nginx] Ban 80.66.66.140
...
___________________________________
L. Mark Stone
Mission Critical Email - Zimbra VAR/BSP/Training Partner https://www.missioncriticalemail.com/
AWS Certified Solutions Architect-Associate
Post Reply