Stopping attacks in real-time

Have a great idea for extending Zimbra? Share ideas, ask questions, contribute, and get feedback.
User avatar
JDunphy
Outstanding Member
Outstanding Member
Posts: 925
Joined: Fri Sep 12, 2014 11:18 pm
Location: Victoria, BC
ZCS/ZD Version: 9.0.0_P42 NETWORK Edition

Re: Stopping attacks in real-time

Post by JDunphy »

Just wanted to give an update on the status. I have been preoccupied with other issues last week but the project continues to run and gather data.

If I was to give it a grade, I would say it is working better than my initial expectations. I was under the impression that it was going to take some effort to make it work with the core rule set but other than a few changes, it works perfectly on my test server. In the past, if I ran check_attacks.pl on a zimbra host with 443 open; I would see loads of status codes and multiple attacks from the same ip address. Check_attacks.pl now reporting single line output per ip address with a 403 status code - 100's of those but that means it seems to be working and we are blocking those incursions. If they attack at a lower part of the stack where modsecurity never see's the connection then fail2ban is mopping up those ip's and adding to the same set.

What the limited data is showing me given I still don't have a DNS entry for the host is that there are 100's of commercial and opportunistic bots investigating what is at port 443 at just the ip address. At present, the list is about 500 ip's but it has been slowing to a trickle as they exhaust their ipv4 ranges. Given that fail2ban seems to be required, having it do all the adds to null routes or ipset might be easier on the software install for a production system without the lua stack. Currently, I have lua enabled so I can extend modsecurity 3. If I can't find a use case (session tracking might be one where it could help) then I would probably not recommend building that which can make the install easier and smaller. My instructions however in this thread contains the full build including lua.

Zimbra's web interfaces seem to be working under this. I did some extensive work the other day in both the admin interface and web interface for my user account and forgot that it was in front as I was testing some things with gmail and focused on postfix issues and email send/receive. There is still a lot to do and one area I am thinking of exploring is having fail2ban whitelist successful authorized zimbra accounts in audit.log to an ipset and correlate that on the cookies in the headers. That should allow one to track mobile users as their ip addresses change but we don't see the normal authorization. In theory, I could create an ipset that is checked before I check the blacklist so we don't lock out users using this method should a FP happen. A major concern continues to be hijacked user credentials that are played back and how to track that.

Note: eventually, I will move the ipset to the raw/prerouting for efficiency but it is at filter/input so that I can make sure I don't get blocked while I am testing it. I have started down the path of learning metasploit which is slowing me down but I want to write some modules so I can verify if modsecurity would have protected an unpatched zimbra P38 from the new security patches in P40 for example.

At some point, I need to polish the instructions for building the modsecurity connector so this doesn't become another cold fusion example where no one can reproduce the results. :-) I have been careful here because the 3rd party tree has some danger in the makefile for nginx where a production server could be wiped out if someone forgot and typed make. To be safe, do it on a development machine if someone wants to explore this thread. I add a rule that becomes the default that makes modSecurity as the first thing I do when using that tree is to modify its Makefile.

Code: Select all

modSecurity:
        @echo $(PKG_BUILD) $(specfile)
        @echo "installing /etc/nginx/zimbra-modsecurity/modules/ngx_http_modsecurity_module.so"
        cp ./build/RHEL8_64/zimbra-nginx/rpm/BUILD/nginx-1.20.0-zimbra/objs/ngx_http_modsecurity_module.so  /etc/nginx/zimbra-modsecurity/modules/

all: clean getsrc build pkgrm1
and change this also in the Makefile:

Code: Select all

#JAD#build: pkgadd setup build_$(PKG_EXT)
build: setup build_$(PKG_EXT)
and I modify configure in the zimbra-nginx/rpm/SPECS/nginx.spec:

Code: Select all

  --add-dynamic-module=/usr/src/public/www/zimbra-nginx/ModSecurity-nginx \
  
and add to this section in the spec file

Code: Select all

%files
%defattr(-,root,root)
OZC/sbin
/opt/zimbra/data/nginx
/opt/zimbra/common/modules/ngx_http_modsecurity_module.so
%exclude OZC/conf
I do not however use the RPM yet... as my process is generally:

Code: Select all

% make getsrc  # only once
% make build
% make modSecurity
Here is the reference how to build nginx without modsecurity: viewtopic.php?p=309246#p309246

Without doing it in the 3rd party tree as shown above, the process I used before I learned how to build nginx was:

Code: Select all

% cat step2-connector.sh
#
# nginx -v
#
git clone --depth 1 https://github.com/SpiderLabs/ModSecurity-nginx.git
wget http://nginx.org/download/nginx-1.20.0.tar.gz
tar zxvf nginx-1.20.0.tar.gz
cd nginx-1.20.0
#
# Pulled these options from: 
#   https://github.com/Zimbra/packages/blob/develop/thirdparty/nginx/zimbra-nginx/rpm/SPECS/nginx.spec
# Note: just need the binary format... we are not replacing nginx with the compiled version here.
#
# Replace these
# OZCL is /opt/zimbra/common/lib
# OZC is /opt/zimbra/common
# OZCI is /opt/zimbra/common/include
#
#LDFLAGS="-Wl,-rpath,OZCL"; export LDFLAGS; \
#CFLAGS="-g -O0"; export CFLAGS; \
#./configure --prefix=OZC \
#--with-cc-opt="-g -IOZCI" \
#--with-ld-opt="-Wl,-rpath,/opt/zimbra/common/lib -L/opt/zimbra/common/lib" \
LDFLAGS="-Wl,-rpath,/opt/zimbra/common/lib"; export LDFLAGS; \
CFLAGS="-g -O0"; export CFLAGS; \
./configure --prefix=/opt/zimbra/common \
  --with-cc-opt="-g -I/opt/zimbra/common/include" \
  --with-ld-opt="-Wl,-rpath,/opt/zimbra/common/lib -L/opt/zimbra/common/lib" \
  --add-dynamic-module=../ModSecurity-nginx \
  --with-debug \
  --with-ipv6 \
  --with-http_ssl_module \
  --with-http_stub_status_module \
  --with-http_v2_module \
  --with-pcre \
  --with-mail \
  --with-mail_ssl_module \
  --error-log-path=/opt/zimbra/log/nginx.log \
  --http-log-path=/opt/zimbra/log/nginx.access.log \
  --http-client-body-temp-path=/opt/zimbra/data/tmp/nginx/client \
  --http-proxy-temp-path=/opt/zimbra/data/tmp/nginx/proxy \
  --http-fastcgi-temp-path=/opt/zimbra/data/tmp/nginx/fastcgi \
  --without-http_scgi_module \
  --without-http_uwsgi_module 
make
echo "cp ./objs/ngx_http_modsecurity_module.so /etc/nginx/zimbra-modsecurity/modules/"
Eventually, I will post my wiki article that condenses these working notes and my single script that combines all the parts, builds, pulls rules, etc, etc.

Jim
User avatar
JDunphy
Outstanding Member
Outstanding Member
Posts: 925
Joined: Fri Sep 12, 2014 11:18 pm
Location: Victoria, BC
ZCS/ZD Version: 9.0.0_P42 NETWORK Edition

Re: Stopping attacks in real-time

Post by JDunphy »

A quick walk through of what logs might look like. This could be more interesting for those that use fail2ban of what you might be able to do.

Here was the test scenario... rules are in place but the action of putting it in an ipset and blocking it at the firewall was disabled. That generated a list of activities like this where the bot refused to leave and kept coming back:

Code: Select all

%  check_attacks.pl --file=nginx.access.log --srcip=216.218.206.67 | obfuscate.pl 
	[ 403] HEAD https://mail.example.com/owa/  Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36 Edg/109.0.1518.70
	[ 404] GET https://mail.example.com/owa/  Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36
	[ 404] GET https://mail.example.com/owa/auth/x.js  Mozilla/5.0 (Windows NT 10.0; rv:109.0) Gecko/20100101 Firefox/109.0
	[ 403] GET https://X.X.X.X/geoserver/web/  Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36 Edg/109.0.1518.70
	[ 403] GET https://X.X.X.X/.git/config  Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/110.0
        [ 403] GET https://X.X.X.X/  Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.1 Safari/605.1.15
 Attacker from  216.218.206.67                  6 Requests - Score 100% 
 ------------------------------------------------------------------------------------------------------------

Next I enabled the action of blocking to an ipset. That was the last activity for this bot and here is what that looked like from modsecurity's perspective:

The format is flexible and sections can be specified in the configurations. The interesting section for fail2ban users would be H where we see this line: msg "Blocked IP address: 216.218.206.67". That is a result of this rule where we issue that message:

Code: Select all

SecRule REQUEST_HEADERS:Host "@rx ^(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})" \
   "id:950000,log, pass, phase:1, t:none, capture, setvar:'tx.host_was_ip=1', chain"
    SecRule REMOTE_ADDR "^(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})" \
        "capture,nolog,setvar:tx.addr=%{tx.1},setvar:tx.ip_block=1,chain"
    SecRule TX:ip_block "@streq 1" \
       "id:950003, block, log, msg:'Blocked IP address: %{REMOTE_ADDR}', status:403, setenv:REMOTE_ADDR=tx.addr,\
        setvar:tx.ruleid=%{rule.id}"
#        setvar:tx.ruleid=%{rule.id},exec:/etc/nginx/modsec/send2daemon.lua"
        #exec:/etc/nginx/modsec/add_to_blacklist.lua"
We are allowing the rule to continue to further rule processing. I have commented out a few lines as I am testing it by sending the ip to a daemon that is adding ip's locally and passing it on to peers who reciprocate when they add ip's to their local ipsets. For simplicity, fail2ban users could use syslog remote ability and the command logger in your action statement to pass those ip's on to peers and not require this extra code for the same result.

Continuing... The rule to do send that ip address to be blocked is:

Code: Select all

SecRule TX:ruleid "!@eq 0" \
    "id:12346,phase:1,deny,nolog,t:none,\
    exec:/etc/nginx/modsec/send2daemon.lua"
It is simpler to just add the ip to an ipset in the initial rule but send2daemon.lua will write it to a unix domain socket and I am going to begin scoring based on rules so actions are blocking vs denying at this point. Not to mention, I have other rules that will ultimately hit this action of blocking the ip and only wanted to do it in one place.

For fail2ban users, this 12346 rule executing this lua script could be commented out and you could parse modsec_audit.log and look for Blocked IP address X.X.X.X. To reduce the log size, you could also specify the minimum sections in your report. See below what those sections mean.

Code: Select all

---N5ecBmfu---A--
[04/Jun/2023:05:10:40 -0700] 168588064098.120962 216.218.206.67 52718 X.X.X.X 443
---N5ecBmfu---B--
GET / HTTP/1.1
Host: X.X.X.X
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.1 Safari/605.1.15
Accept: */*
Accept-Encoding: gzip

---N5ecBmfu---D--

---N5ecBmfu---F--
HTTP/1.1 403

---N5ecBmfu---H--
ModSecurity: Warning. Matched "Operator `StrEq' with parameter `1' against variable `TX:ip_block' (Value: `1' ) [file "/etc/nginx/modsec/zimbra.conf"] [line "72"] [id "950000"] [rev ""] [msg "Blocked IP address: 216.218.206.67"] [data ""] [severity "0"] [ver ""] [maturity "0"] [accuracy "0"] [hostname "X.X.X.X"] [uri "/"] [unique_id "168588064098.120962"] [ref "o0,13o0,13v21,13o0,14o0,14v0,14"]

---N5ecBmfu---I--

---N5ecBmfu---J--

---N5ecBmfu---Z--
Your decoder ring for the sections and changing the output for your report logs:

Code: Select all

 Default ABCFHZ.
#
# Available audit log parts:
#    A – audit log header (mandatory)
#    B – request headers
#    C – request body (present only if the request body exists and ModSecurity is configured to intercept it)
#    D - RESERVED for intermediary response headers, not implemented yet.
#    E – intermediary response body (present only if ModSecurity is configured to intercept response bodies, and if the audit log
#        engine is configured to record it). Intermediary response body is the same as the actual response body unless
#        ModSecurity intercepts the intermediary response body, in which case the actual response body will contain the
#        error message (either the Apache default error message, or the ErrorDocument page).
#    F – final response headers (excluding the Date and Server headers, which are always added by Apache in the late stage of content delivery).
#    G – RESERVED for the actual response body, not implemented yet.
#    H - audit log trailer
#    I - This part is a replacement for part C. It will log the same data as C in all cases except when multipart/form-data
#        encoding in used. In this case it will log a fake application/x-www-form-urlencoded body that contains the information
#        about parameters but not about the files. This is handy if you don't want to have (often large) files stored in your audit logs.
#    J - RESERVED. This part, when implemented, will contain information about the files uploaded using multipart/form-data encoding.
#    Z – final boundary, signifies the end of the entry (mandatory)
For completeness, this is the lua code that is blocking it.

Code: Select all

cat /etc/nginx/modsec/send2daemon.lua

-- Caveat:
--     permissions: /tmp/modsec.fd
--     luarocks install luasocket
--              /usr/lib64/lua/5.3/socket/unix.so 

local socket = require("socket.unix")

-- no state so no way to not open/close the socket each time
function add_to_blacklist(remote_addr, rule_id)
    -- create and connect to the Unix domain socket
    local unix_socket = assert(socket())
    assert(unix_socket:connect("/tmp/modsec.fd"))
    
    -- prepare the datagram
    local datagram = remote_addr .. ":" .. rule_id
    
    -- send the datagram
    unix_socket:send(datagram)
    
    -- close the Unix socket
    unix_socket:close()

    local cmd = "/bin/logger -p local2.info NETWORK zimbra is adding blacklist24hr " .. remote_addr .. " with rule " .. rule_id
    os.execute(cmd)
end

function main()

    add_to_blacklist(m.getvar("REMOTE_ADDR"), m.getvar("TX.ruleid"))

end
For maximum efficiency, don't use the lua function call and have fail2ban do all the work. That was what I ment in a previous post that if we have to use fail2ban to protect nginx.log anyway, why not have it do the action of adding it to an ipset or null routing the ip to be blocked and save having to support lua and it's tree of support files - think perl and it's modules for a conceptual model.

Hope this makes sense and provides some examples of how this might fit together with zimbra and provide additional capabilities for admin's.

We are also currently protected against the following categories of threats. That is on top of the zimbra specific rules which I have firing first.

Code: Select all

% ls 
crawlers-user-agents.data                             REQUEST-900-EXCLUSION-RULES-BEFORE-CRS.conf.example   REQUEST-941-APPLICATION-ATTACK-XSS.conf               RESPONSE-999-EXCLUSION-RULES-AFTER-CRS.conf 
iis-errors.data                                       REQUEST-901-INITIALIZATION.conf                       REQUEST-942-APPLICATION-ATTACK-SQLI.conf              RESPONSE-999-EXCLUSION-RULES-AFTER-CRS.conf.example 
java-classes.data                                     REQUEST-905-COMMON-EXCEPTIONS.conf                    REQUEST-943-APPLICATION-ATTACK-SESSION-FIXATION.conf  restricted-files.data 
java-code-leakages.data                               REQUEST-911-METHOD-ENFORCEMENT.conf                   REQUEST-944-APPLICATION-ATTACK-JAVA.conf              restricted-upload.data 
java-errors.data                                      REQUEST-913-SCANNER-DETECTION.conf                    REQUEST-949-BLOCKING-EVALUATION.conf                  scanners-headers.data 
lfi-os-files.data                                     REQUEST-920-PROTOCOL-ENFORCEMENT.conf                 RESPONSE-950-DATA-LEAKAGES.conf                       scanners-urls.data 
php-config-directives.data                            REQUEST-921-PROTOCOL-ATTACK.conf                      RESPONSE-951-DATA-LEAKAGES-SQL.conf                   scanners-user-agents.data 
php-errors.data                                       REQUEST-922-MULTIPART-ATTACK.conf                     RESPONSE-952-DATA-LEAKAGES-JAVA.conf                  scripting-user-agents.data 
php-errors-pl2.data                                   REQUEST-930-APPLICATION-ATTACK-LFI.conf               RESPONSE-953-DATA-LEAKAGES-PHP.conf                   sql-errors.data 
php-function-names-933150.data                        REQUEST-931-APPLICATION-ATTACK-RFI.conf               RESPONSE-954-DATA-LEAKAGES-IIS.conf                   ssrf.data 
php-function-names-933151.data                        REQUEST-932-APPLICATION-ATTACK-RCE.conf               RESPONSE-955-WEB-SHELLS.conf                          unix-shell.data 
php-variables.data                                    REQUEST-933-APPLICATION-ATTACK-PHP.conf               RESPONSE-959-BLOCKING-EVALUATION.conf                 web-shells-php.data 
REQUEST-900-EXCLUSION-RULES-BEFORE-CRS.conf           REQUEST-934-APPLICATION-ATTACK-GENERIC.conf           RESPONSE-980-CORRELATION.conf                         windows-powershell-commands.data 
where the data files are text and you can add your own data to them... looking at one of the smaller files

Code: Select all

% cat /etc/nginx/modsec/rules/restricted-upload.data
# Apache webserver
.htaccess
.htdigest
.htpasswd
# WordPress configuration file
wp-config.php
# Symfony configuration files
config.yml
config_dev.yml
config_prod.yml
config_test.yml
parameters.yml
routing.yml
security.yml
services.yml
# Drupal configuration files
default.settings.php
settings.php
settings.local.php
# Magento configuration files
local.xml
# dotenv configuration file
.env
The rules are in the .conf files and the data for them have an .data extension. If you want to look at the what these rules might look at... here is a reference to explore with your browser.

Ref: https://github.com/coreruleset/corerule ... /dev/rules

Jim
User avatar
JDunphy
Outstanding Member
Outstanding Member
Posts: 925
Joined: Fri Sep 12, 2014 11:18 pm
Location: Victoria, BC
ZCS/ZD Version: 9.0.0_P42 NETWORK Edition

Re: Stopping attacks in real-time

Post by JDunphy »

I am not sure blocking by ip address is good enough anymore. Over time, one would end up with 10,000's if not millions of ip's to manage all being potential and future false positives as ipv4 addresses are reused by different entities. This morning, there was a lot of activity from 85.203.34.0/24. I pulled up a list of what they were trying. Nothing dangerous for Zimbra other than a waste of resources and a lot of log noise.

I am going to play with some rules to redirect first to a static blank page with no execute environment and then an action to put up a capture for the entire /24 when we see activity like this from similar address space. Those two rules could come in handy for future rules that watch for brute force password guessing for example.

Investigating 85.203.34.0/24 has me wondering if we just block everything it would be too simplistic.

Code: Select all

% check_attacks.pl --file=nginx.access.log -srcip '85.203.34' --display=date | obfuscate.pl 
	[ 403] POST https://X.X.X.X/admin/ckeditor/plugins/ajaxplorer/phpunit/src/Util/PHP/eval-stdin.php  06/Jun/2023:03:11:31
 Attacker from  85.203.34.100                   1 Requests - Score 100% 
------------------------------------------------------------------------------------------------------------
	[ 403] POST https://X.X.X.X/vendor/phpunit/phpunit/src/Util/PHP/eval-stdin.php  06/Jun/2023:03:11:10
 Attacker from  85.203.34.51                    1 Requests - Score 100% 
------------------------------------------------------------------------------------------------------------
	[ 403] GET https://X.X.X.X/debug/default/view.html  06/Jun/2023:03:15:49
 Attacker from  85.203.34.52                    1 Requests - Score 100% 
------------------------------------------------------------------------------------------------------------
	[ 403] GET https://X.X.X.X/smtp.json  06/Jun/2023:03:17:15
 Attacker from  85.203.34.53                    1 Requests - Score 100% 
------------------------------------------------------------------------------------------------------------
	[ 403] GET https://X.X.X.X/  06/Jun/2023:03:10:29
 Attacker from  85.203.34.54                    1 Requests - Score 100% 
------------------------------------------------------------------------------------------------------------
	[ 403] POST https://X.X.X.X/api/vendor/phpunit/phpunit/src/Util/PHP/Template/eval-stdin.php  06/Jun/2023:03:11:37
 Attacker from  85.203.34.55                    1 Requests - Score 100% 
------------------------------------------------------------------------------------------------------------
	[ 403] POST https://X.X.X.X/phpunit/phpunit/Util/PHP/eval-stdin.php  06/Jun/2023:03:11:17
 Attacker from  85.203.34.56                    1 Requests - Score 100% 
------------------------------------------------------------------------------------------------------------
	[ 403] POST https://X.X.X.X/.env.production  06/Jun/2023:03:12:27
 Attacker from  85.203.34.57                    1 Requests - Score 100% 
------------------------------------------------------------------------------------------------------------
	[ 403] GET https://X.X.X.X/.git/config  06/Jun/2023:03:10:58
 Attacker from  85.203.34.59                    1 Requests - Score 100% 
------------------------------------------------------------------------------------------------------------
	[ 403] POST https://X.X.X.X/lib/phpunit/phpunit/Util/PHP/eval-stdin.php  06/Jun/2023:03:11:48
 Attacker from  85.203.34.60                    1 Requests - Score 100% 
------------------------------------------------------------------------------------------------------------
	[ 403] POST https://X.X.X.X/lib/phpunit/phpunit/Util/PHP/eval-stdin.php  06/Jun/2023:03:11:26
 Attacker from  85.203.34.61                    1 Requests - Score 100% 
------------------------------------------------------------------------------------------------------------
	[ 403] GET https://X.X.X.X/cms/.env  06/Jun/2023:03:13:23
 Attacker from  85.203.34.62                    1 Requests - Score 100% 
------------------------------------------------------------------------------------------------------------
	[ 403] POST https://X.X.X.X/lib/phpunit/src/Util/PHP/eval-stdin.php  06/Jun/2023:03:11:27
 Attacker from  85.203.34.63                    1 Requests - Score 100% 
------------------------------------------------------------------------------------------------------------
	[ 403] GET https://X.X.X.X/sources/.env  06/Jun/2023:03:14:35
 Attacker from  85.203.34.64                    1 Requests - Score 100% 
------------------------------------------------------------------------------------------------------------
	[ 403] POST https://X.X.X.X/admin-app/.env  06/Jun/2023:03:12:42
 Attacker from  85.203.34.65                    1 Requests - Score 100% 
------------------------------------------------------------------------------------------------------------
	[ 403] GET https://X.X.X.X/  06/Jun/2023:03:14:57
 Attacker from  85.203.34.66                    1 Requests - Score 100% 
------------------------------------------------------------------------------------------------------------
	[ 403] POST https://X.X.X.X/phpunit/phpunit/src/Util/PHP/eval-stdin.php  06/Jun/2023:03:11:16
 Attacker from  85.203.34.67                    1 Requests - Score 100% 
------------------------------------------------------------------------------------------------------------
	[ 403] POST https://X.X.X.X/lib/phpunit/phpunit/src/Util/PHP/eval-stdin.php  06/Jun/2023:03:11:25
 Attacker from  85.203.34.68                    1 Requests - Score 100% 
------------------------------------------------------------------------------------------------------------
	[ 403] POST https://X.X.X.X/vendor/phpunit/phpunit/Util/PHP/eval-stdin.php  06/Jun/2023:03:11:11
 Attacker from  85.203.34.70                    1 Requests - Score 100% 
------------------------------------------------------------------------------------------------------------
	[ 403] GET https://X.X.X.X/.env.dist  06/Jun/2023:03:13:50
 Attacker from  85.203.34.71                    1 Requests - Score 100% 
------------------------------------------------------------------------------------------------------------
	[ 403] POST https://X.X.X.X/laravel/vendor/phpunit/phpunit/src/Util/PHP/eval-stdin.php  06/Jun/2023:03:11:43
 Attacker from  85.203.34.72                    1 Requests - Score 100% 
------------------------------------------------------------------------------------------------------------
	[ 403] GET https://X.X.X.X/core/.env  06/Jun/2023:03:14:01
 Attacker from  85.203.34.73                    1 Requests - Score 100% 
------------------------------------------------------------------------------------------------------------
	[ 403] GET https://X.X.X.X/system/.env  06/Jun/2023:03:13:11
 Attacker from  85.203.34.74                    1 Requests - Score 100% 
------------------------------------------------------------------------------------------------------------
	[ 403] POST https://X.X.X.X/lib/phpunit/phpunit/src/Util/PHP/eval-stdin.php  06/Jun/2023:03:11:47
 Attacker from  85.203.34.76                    1 Requests - Score 100% 
------------------------------------------------------------------------------------------------------------
	[ 403] GET https://X.X.X.X/  06/Jun/2023:03:10:30
 Attacker from  85.203.34.77                    1 Requests - Score 100% 
------------------------------------------------------------------------------------------------------------
	[ 403] GET https://X.X.X.X/admin-app/.env  06/Jun/2023:03:12:35
 Attacker from  85.203.34.78                    1 Requests - Score 100% 
------------------------------------------------------------------------------------------------------------
	[ 403] GET https://X.X.X.X/.env.save  06/Jun/2023:03:12:08
 Attacker from  85.203.34.80                    1 Requests - Score 100% 
------------------------------------------------------------------------------------------------------------
	[ 403] POST https://X.X.X.X/  06/Jun/2023:03:15:03
 Attacker from  85.203.34.81                    1 Requests - Score 100% 
------------------------------------------------------------------------------------------------------------
	[ 403] GET https://X.X.X.X/_info.php  06/Jun/2023:03:18:34
 Attacker from  85.203.34.83                    1 Requests - Score 100% 
------------------------------------------------------------------------------------------------------------
	[ 403] GET https://X.X.X.X/_phpinf.php  06/Jun/2023:03:18:35
 Attacker from  85.203.34.85                    1 Requests - Score 100% 
------------------------------------------------------------------------------------------------------------
	[ 403] POST https://X.X.X.X/vendor/phpunit/src/Util/PHP/eval-stdin.php  06/Jun/2023:03:11:12
 Attacker from  85.203.34.86                    1 Requests - Score 100% 
------------------------------------------------------------------------------------------------------------
	[ 403] POST https://X.X.X.X/api/vendor/phpunit/phpunit/src/Util/PHP/eval-stdin.php  06/Jun/2023:03:11:36
 Attacker from  85.203.34.87                    1 Requests - Score 100% 
------------------------------------------------------------------------------------------------------------
	[ 403] GET https://X.X.X.X/?phpinfo=-1  06/Jun/2023:03:18:23
 Attacker from  85.203.34.88                    1 Requests - Score 100% 
------------------------------------------------------------------------------------------------------------
	[ 403] GET https://X.X.X.X/  06/Jun/2023:03:10:57
 Attacker from  85.203.34.90                    1 Requests - Score 100% 
------------------------------------------------------------------------------------------------------------
	[ 403] POST https://X.X.X.X/laravel_web/vendor/phpunit/phpunit/src/Util/PHP/eval-stdin.php  06/Jun/2023:03:11:42
 Attacker from  85.203.34.91                    1 Requests - Score 100% 
------------------------------------------------------------------------------------------------------------
	[ 403] GET https://X.X.X.X/1_1_PhpInfo.php  06/Jun/2023:03:19:14
 Attacker from  85.203.34.92                    1 Requests - Score 100% 
------------------------------------------------------------------------------------------------------------
	[ 403] GET https://X.X.X.X/.env.development  06/Jun/2023:03:12:23
 Attacker from  85.203.34.94                    1 Requests - Score 100% 
------------------------------------------------------------------------------------------------------------
	[ 403] POST https://X.X.X.X/lib/vendor/phpunit/phpunit/src/Util/PHP/eval-stdin.php  06/Jun/2023:03:11:53
 Attacker from  85.203.34.95                    1 Requests - Score 100% 
------------------------------------------------------------------------------------------------------------
	[ 403] GET https://X.X.X.X/.env.production  06/Jun/2023:03:12:21
 Attacker from  85.203.34.96                    1 Requests - Score 100% 
------------------------------------------------------------------------------------------------------------
	[ 403] POST https://X.X.X.X/.env.save  06/Jun/2023:03:12:14
 Attacker from  85.203.34.97                    1 Requests - Score 100% 
------------------------------------------------------------------------------------------------------------
	[ 403] GET https://X.X.X.X/cp/.env  06/Jun/2023:03:12:59
 Attacker from  85.203.34.99                    1 Requests - Score 100% 
-------------------------------------------------------------------------------------------------------------
Continuing... I do a lookup to see where it is SWIP'd to.

Code: Select all

% whois -h whois.arin.net 85.203.34.0
...
...
Found a referral to whois.ripe.net.

% This is the RIPE Database query service.
% The objects are in RPSL format.
%
% The RIPE Database is subject to Terms and Conditions.
% See http://www.ripe.net/db/support/db-terms-conditions.pdf

% Note: this output has been filtered.
%       To receive output for a database update, use the "-B" flag.

% Information related to '85.203.34.0 - 85.203.34.255'

% Abuse contact for '85.203.34.0 - 85.203.34.255' is 'jeroen@falco-networks.com'

inetnum:        85.203.34.0 - 85.203.34.255
netname:        VPN-Consumer-Network
descr:          VPN Consumer Network
descr:          Slough Trading Estate
descr:          2 Buckingham Ave
descr:          2 Buckingham Ave
geoloc:         51.523504 -0.636006
country:        GB
admin-c:        JVV19-RIPE
tech-c:         JVV19-RIPE
status:         ASSIGNED PA
mnt-lower:      MNT-FALCO
mnt-routes:     MNT-FALCO
mnt-by:         MNT-FALCO
mnt-domains:    MNT-FALCO
created:        2015-12-09T15:56:54Z
last-modified:  2019-06-03T00:18:33Z
source:         RIPE

person:         Jeroen van veen
address:        Falco IPR B.V.
address:        De Hoefsmid 11-13
address:        1851 PZ Heiloo
address:        The Netherlands
phone:          +31 72 532 3744
nic-hdl:        JVV19-RIPE
created:        2002-09-16T13:46:49Z
last-modified:  2018-05-01T12:11:21Z
source:         RIPE # Filtered
mnt-by:         MNT-FALCO

% Information related to '85.203.34.0/24AS206092'

route:          85.203.34.0/24
origin:         AS206092
mnt-by:         MNT-FALCO
mnt-lower:      MNT-FALCO
mnt-routes:     MNT-FALCO
created:        2022-02-14T23:04:27Z
last-modified:  2022-02-14T23:04:27Z
source:         RIPE

% This query was served by the RIPE Database Query Service version 1.106.1 (DEXTER)

Best guess was someone/something is using a VPN service and then proceeded to use a lot of their ip space as they became blocked in real-time. After they are gone, we are left with ip's to manage and block which accomplishes nothing if they never use this space again. Maybe I am back to, identify our users and whitelist them. Everyone else gets redirected to a capture if caught in different types of attacks.

Perhaps the correct response for some types of rules that trigger might be to redirect their entire /24 or larger to a capture for short duration until they prove they are human and can authenticate. If the attacking activity looks to be persistent, we could escalate them into longer windows of redirects or where we do block them for short duration. Ultimately this needs to be left up to the admin but having options is probably the correct course here. I am leaning toward a setting where different values represent how the rules will respond.

I don't believe I have provided where and how I am patching nginx. Here are my initial patch scripts, I have been using for nginx.

Code: Select all

% cat  nginx-modsecurity-web.sh
#!/bin/sh
#
#  Add modSecurity 3 to nginx config file (nginx.conf.web.template)
#
nginx_web_template=/opt/zimbra/conf/nginx/templates/nginx.conf.web.template

grep -q JAD $nginx_web_template
if [ $? -eq 1 ]; then
   cp $nginx_web_template $nginx_web_template.bak

   echo "Appying patch to $nginx_web_template"
   sed -i -e 's|\(exact_version_check [^;]*;\)|\1\n    #JAD\n    include /etc/nginx/modsec/main.conf;|g' $nginx_web_template

else
   echo "nothing to patch"
fi

#
#  Add modSecurity 3 to nginx config files (nginx.conf.web)
#
nginx_web_includes=/opt/zimbra/conf/nginx/includes/nginx.conf.web
grep -q JAD $nginx_web_includes
if [ $? -eq 1 ]; then
   cp $nginx_web_includes $nginx_web_includes.bak

   echo "Appying patch to $nginx_web_includes"
   sed -i -e 's|\(exact_version_check [^;]*;\)|\1\n    #JAD\n    include /etc/nginx/modsec/main.conf;|g' $nginx_web_includes
else
   echo "nothing to patch"
fi

exit
and to enable the rules

Code: Select all

cat nginx-modsecurity-web.sh 
#!/bin/sh
#
#  Add modSecurity 3 to nginx config file (nginx.conf.web.template)
#
nginx_web_template=/opt/zimbra/conf/nginx/templates/nginx.conf.web.template

grep -q JAD $nginx_web_template
if [ $? -eq 1 ]; then
   cp $nginx_web_template $nginx_web_template.bak

   echo "Appying patch to $nginx_web_template"
   sed -i -e 's|\(exact_version_check [^;]*;\)|\1\n    #JAD\n    include /etc/nginx/modsec/main.conf;|g' $nginx_web_template

else
   echo "nothing to patch"
fi

#
#  Add modSecurity 3 to nginx config files (nginx.conf.web)
#
nginx_web_includes=/opt/zimbra/conf/nginx/includes/nginx.conf.web
grep -q JAD $nginx_web_includes
if [ $? -eq 1 ]; then
   cp $nginx_web_includes $nginx_web_includes.bak

   echo "Appying patch to $nginx_web_includes"
   sed -i -e 's|\(exact_version_check [^;]*;\)|\1\n    #JAD\n    include /etc/nginx/modsec/main.conf;|g' $nginx_web_includes
else
   echo "nothing to patch"
fi

exit
You can run them over and over again. They are storing something JAD as a comment that we can grep for but you can change that to something unique. On my servers, anything I modify I put my initials which is my convention for our servers to know something isn't default. After updates, one runs the scripts and they will only patch if they have to. To disable modsecurity, you can comment out the 2 lines or modify the include files they reference to disable modsecurity. I tried to find places where I thought would be simplest to see and work for most environments.

Jim
User avatar
JDunphy
Outstanding Member
Outstanding Member
Posts: 925
Joined: Fri Sep 12, 2014 11:18 pm
Location: Victoria, BC
ZCS/ZD Version: 9.0.0_P42 NETWORK Edition

Re: Stopping attacks in real-time

Post by JDunphy »

I've been revisiting the idea of adding dynamic protections for Zimbra servers, especially as recent discussions have highlighted related challenges. For instance, there's an ongoing request for a Web Application Firewall (WAF) enhancement, which you can vote on here: https://pm.zimbra.com/p/web-application ... for-zimbra. A WAF feature that could be toggled on/off within Nginx would be ideal for many admins.

Currently, I'm exploring the feasibility of using a CAPTCHA mechanism to slow down bots during large-scale attacks targeting Zimbra servers. This is still in the planning stage, but I've started gathering ideas, with some initial input generated using GPT-4. You can review the shared details here: https://chatgpt.com/share/6744ad03-32 ... 632774bd71 ChatGPT Generated Ideas

For reference, the IP blocking via ipset is already functional, so my focus has shifted. Most admins might prefer a setup where logged IPs are handled by fail2ban, as opposed to directly calling ipset through a script, which has concerns about fork/exec overhead. My expertise with mod_security and custom rule writing has improved since I began exploring this, but there's always more to refine.

I’ve also considered using Nginx's counters for logging to reduce dependency on a WAF. However, past chatbot-generated recommendations about Nginx syntax have been prone to inaccuracies, so thorough verification is needed. One of those generated ideas was using lua code so that could solve for both captcha and dynamic ip blocking.

Before proceeding further, has anyone implemented a CAPTCHA-based approach to dynamically mitigate bot activity and enhance DosFilter protection? I see significant potential in a WAF system that Zimbra could use to push out rules for 0-day threats, buying time for patch development and QA testing.

Jim
Post Reply