Real-time Security Blocking from logs and sharing with peers

Have a great idea for extending Zimbra? Share ideas, ask questions, contribute, and get feedback.
Post Reply
User avatar
JDunphy
Outstanding Member
Outstanding Member
Posts: 889
Joined: Fri Sep 12, 2014 11:18 pm
Location: Victoria, BC
ZCS/ZD Version: 9.0.0_P39 NETWORK Edition

Real-time Security Blocking from logs and sharing with peers

Post by JDunphy »

With the recent events with the RCE/SSRF, I am going to release some software (framework) I have been using here as I watched this unfold on 8.6, 8.7, and 8.8 releases. It is a single perl program and it has UDP peer to peer communications that is encrypted so you can broadcast discoveries to all your hosts as it discovers attacking signatures. It is fairly lightweight and is best described as a tail -f but is a simple perl program using File-Tail-Multi-0.1.tar.gz (patched by me for bugs) and using a select for the single socket necessary for its UDP peers. Its not very big but has a lot of functionality at present. That requires that you would need to install 3 perl modules which are documented as comments for those that are scared of perl module installation. I kept it to a minimum of 3 modules ... Example:

Code: Select all

# Need this
wget -4 https://cpan.metacpan.org/authors/id/L/LD/LDS/Crypt-CBC-2.33.tar.gz
tar zxvf Crypt-CBC-2.33.tar.gz
cd Crypt-CBC-2.33
perl Makefile.PL
make
make test
make install

# need this
wget -4 https://cpan.metacpan.org/authors/id/L/LE/LEONT/Crypt-Rijndael-1.13.tar.gz
tar zxvf Crypt-Rijndael-1.13.tar.gz
cd Crypt-Rijndael-1.13
perl Makefile.PL
make
make test
make install
It can operate in 2 modes - discovery and an operations mode. It can operate by itself or with the cooperation of its peers running the same single program. You choose how they operate... watching, sharing, or all of the above and blocking ip's. The function is to watch various logs simultaneously and to add ip addresses in real-time for short duration's to ride out attackers when Zimbra is under attack using an ipset. You control when the ip will expire when you initially create the ipset. That also means you only require 1 line in your fw to enable a rule that can block 1 or 1000's of ips so your firewall will not slow down with every incoming packet as the blocking ip count increases. It also can prime it's caches with previous logs (that is your history) and can share that with its peers so the program starts out smarter before the first log entry is processed.

The program can watch multiple logs formats simultaneously and understand them in detail - meaning sometimes you need to see all the requests processed previously and what they did before you can decide if the incoming packet is an attacker. It also keeps track by count of how many things this ip has done in the past and status code. That means if someone comes from that ip address with a user-agent via python-requests but then again as Mozilla we know it - that was useful in tracking ip addresses of the recent attacker as it was sharing this discovery across all my hosts. It has the concept of number of badlookups (set by you) before it would put this ip in timeout.

The goal was to give more knowledge to those logs... so some logs have different formats but in aggregate we can make more intelligent choices... Similarly, it would be possible to notify MX peers of an attacker should they go after email addresses "spam traps" you establish. Those that go after your secondary MX with spam traps will not be able to connect to your primary MX as a result. It would also be able to determine account guessing by hackers vs users at some point when I add in the audit logs or postfix logs. Ultimately it would score this with other information to allow more aggressive rules. This is all post log analysis (already happened if we see it in the log) but in the future it would be able to use mod_security and stop the initial attack. Currently it is only proactive for your peers since that was information was propagated in real-time on discovery. Its a single program but really is a framework to make intelligent decisions of when to block an ip address. I will also add a zimlet at some point to show what it is doing and provide some configuration... Lots of stuff to do - fun!

The script came about from learning about attacks on nginx/zimbra viewtopic.php?f=15&t=66092 and watching '400' status codes and trying to understand what is normal for our users.

I also discovered that 400 codes can happen for normal traffic (mobile) at times so extra knowledge is required than just say you got a 400 and you did it 4 times so you are blocked. It is smarter than that. Currently, it blocks 400 status codes but they have to come from hackers and uses requests, user_agents, number of tries from that ip, etc. It will however stop the case when an attacker changes the user-agent to get around checking should you have history on that ip. We treat 400 status codes as an indicator of wanting to do harm. It will also block an attacker by user-agent or by request... example: POST /attackingYou

Because it use udp to send ip's and ip status counts to its peers. That communication is encrypted with Rijndael which is the encryption chosen for AES and necessary since udp can be spoofed. You set the password in the script, you lock down the ip's of the peers in your FW if necessary and it will encrypt and make sure that an attacker can not spoof that packet.

It can also take log input from standard-in so you can do stuff like this (note: normally it works on nginx.access.log but this is close enough so I ran it through. Who knew. :-)

Code: Select all

% grep python-requests /opt/zimbra/log/access_log* | ./build_mail_ipset.pl -v --prime
attacker 5.188.210.101 and count is 3 \x05\x01\x00
ipset add blacklist24hr 5.188.210.101 -exists
attacker 5.188.210.101 and count is 3 \x04\x01\x00P\x05\xBC\xD2e\x00
ipset add blacklist24hr 5.188.210.101 -exists
attacker 5.188.210.101 and count is 4 GET http://5.188.210.101/echo.php HTTP/1.1
ipset add blacklist24hr 5.188.210.101 -exists
attacker 93.157.62.162 and count is 3 hello
ipset add blacklist24hr 93.157.62.162 -exists
attacker 93.157.62.162 and count is 3 POST /Autodiscover/Autodiscover.xml HTTP/1.1
ipset add blacklist24hr 93.157.62.162 -exists
attacker 93.157.62.162 and count is 3 POST /Autodiscover/Autodiscover.xml HTTP/1.1
ipset add blacklist24hr 93.157.62.162 -exists
...
attacker 61.219.11.153 and count is 3 \x01\x02\x03\x04\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00
ipset add blacklist24hr 61.219.11.153 -exists
Normally you would run it as a daemon and let it work like a tail -f watching for changes to your files. Options thus far:

Code: Select all

&GetOptions("tail|t" => sub { $useSTDIN=0 },
            "debug|d" => \$debug,
            "verbose|v" => \$verbose,
            "ipset|i" => \$ipset,
            "p2p" => \$p2p,
            "prime|p" => \$prime);
I wanted to start the dialog for ideas and comments before I release this ... I need more production time before I trust running it in background. New software needs to earn my trust and while it has worked, it hasn't had enough baking if that makes any sense. I got side tracked by some horrible bugs in the tail module but am ready to get back into this... Note: NHL Stanley Cup Finals are on so as a Canadian I am obligated to stop what I am doing and watch so that could delay this too! LOL

I hope to use this thread and share my experience with this tool as it evolves.

Big Picture Idea:
I could see this run as a cloud service eventually by Zimbra, etc so the community could peer and share knowledge of attacks and stop active attacks before immediately and give admins time to patch. You decide if you want to block or not or just report and use ip's yourself when you start up your peer, etc. I am already sharing some ip's with our web farms since its been really good at identifying and stopping botnets. I am not advocating it work that way initially... Each man/woman for themselves and customize it to your environment as we learn what is happening to our zimbra production sites.

PS... learned of a zero day nginx this morning ZDI-CAN-8296. Doesn't effect Zimbra from my understanding but we need more tools because who can keep up???
User avatar
zimico
Outstanding Member
Outstanding Member
Posts: 225
Joined: Mon Nov 14, 2016 8:03 am
Location: Vietnam
ZCS/ZD Version: 8.8.15 P3
Contact:

Re: Real-time Security Blocking from logs and sharing with peers

Post by zimico »

Dear JDunphy,
It sounds great! I am not a software developer and very appreciate your work. Making zimbra system more safer always my request when asked by Zimbra support team. I am eager to see this zimlet soon.
Best regards,
Minh.
Klug
Ambassador
Ambassador
Posts: 2747
Joined: Mon Dec 16, 2013 11:35 am
Location: France - Drôme
ZCS/ZD Version: All of them
Contact:

Re: Real-time Security Blocking from logs and sharing with peers

Post by Klug »

I'll have a deeper look during the week-end.
JDunphy wrote:PS... learned of a zero day nginx this morning ZDI-CAN-8296. Doesn't effect Zimbra from my understanding but we need more tools because who can keep up???
Yes, seen that yesterday.
1.x versions of nginx are impacted.

However, considering how nginx is compiled/embedded in ZCS I understood we should be ok.
Post Reply