Simple program to report successful/fail ip logins and sorted by count
Re: Simple program to report successful/fail ip logins and sorted by count
Hello -- my single zimbra server runs behind a reverse proxy server (nginx).
As today it reads as all successful connection are coming from the reverse proxy IP.
I noticed that my Zimbra server audit.log trace correctly the "oip=xxx.xxx.xxx.xxx" but someway the script reports the "$ip" as the reverse proxy server IP
Anyway to allow the script read the originating IP ?
Many thanks for sharing this little nice piece of code ...
As today it reads as all successful connection are coming from the reverse proxy IP.
I noticed that my Zimbra server audit.log trace correctly the "oip=xxx.xxx.xxx.xxx" but someway the script reports the "$ip" as the reverse proxy server IP
Anyway to allow the script read the originating IP ?
Many thanks for sharing this little nice piece of code ...
- JDunphy
- Outstanding Member
- Posts: 901
- Joined: Fri Sep 12, 2014 11:18 pm
- Location: Victoria, BC
- ZCS/ZD Version: 9.0.0_P39 NETWORK Edition
Re: Simple program to report successful/fail ip logins and sorted by count
Try this... where X.X.X.X is your single server ip.milauria wrote:Hello -- my single zimbra server runs behind a reverse proxy server (nginx).
Anyway to allow the script read the originating IP ?
As today it reads as all successful connection are coming from the reverse proxy IP.
Code: Select all
# allow /opt/zimbra/log/audit.log to show correct client ip address
# because we are now behind proxy.
% zmprov mcf +zimbraMailTrustedIP 127.0.0.1 +zimbraMailTrustedIP X.X.X.X
% zmmailboxdctl restart
- JDunphy
- Outstanding Member
- Posts: 901
- Joined: Fri Sep 12, 2014 11:18 pm
- Location: Victoria, BC
- ZCS/ZD Version: 9.0.0_P39 NETWORK Edition
Re: Simple program to report successful/fail ip logins and sorted by count
Hi Labsy,Labsy wrote: Maybe another one for you, since you seem to know what sort is about - would it be possible to sort users by failure count? So healthy ones would be listed first, followed by more and more failed ones.
BTW...Your base program was really nice to work on
I took your modifications as my new base and reworked it... This should work much better for large sites that require filtering. I found I liked your ip to domain name feature even for the non failure cases so there is switch for that now too! It allows me to understand what are normal providers for my users. To show only failed accounts which I believe you were asking about, we have some new switches. Note: that fail=ip, fail=all, fail=none are possible options now. Because it can be expensive for large sites to do reverse lookups on large logs, the default is only for login failures and it can be turned off completely via options. Note: -g none| --gethost=all | --gethost=fail
Code: Select all
% check_login.pl --fail=ip
Code: Select all
usage: % check_login.pl
[--color=<color name (i.e. RED)>]
[--srchuser=<username>]
[--fail=<user|ip|none>]
[--gethost=<all|fail|none>]
[--help]
where:
--color|c: color to be used for FAILED login message information
--srchuser|s: print ONLY the logins/failed logins for <username>
--fail|f: if 'user': print ONLY users who have failed logins. If 'ip': print ONLY the failed login attempts. 'none': print all records regardless if failure
--gethost|g: values of 'all', 'fail' or 'none'. Perform a GETHOSTBYADDR for all IPs, only on FAILED login attempts, or don't perform this action (none)
--help|h: this message
";
example: % check_login.pl -f=user #only the accounts with failed logins
% check_login.pl -f ip #only the accounts and the ip that failed
% check_login.pl -fail=ip # same as above
% check_login.pl --fail ip # same as above
% check_login.pl -f ip -h #list only ip's that failed for accounts resolve ip to domain name
% check_login.pl -g fail #list only accounts that had a failed login
% check_login.pl -g all #list all accounts and resolve ip to domain name
% check_login.pl -c RED -f ip #change color and list only failed ip's
Code: Select all
#!/usr/bin/perl
#
# NOTE: To jump to the various sections within this program search
# on the string 'SECTION'
#
# usage: check_login.pl [Options]
#
# ======================================================================
#========================================================================
# SECTION - Modules, Variables, etc.
#========================================================================
use Term::ANSIColor;
use Data::Dumper qw(Dumper);
use Getopt::Long;
use Socket qw( inet_aton AF_INET );
%ip_list = (); #ip list
%fip_list = (); #failed ip list
#========================================================================
# SECTION - FUNCTIONS
#========================================================================
# Displays program usage
sub usage {
print <<"END";
usage: % check_login.pl
[--color=<color name (i.e. RED)>]
[--srchuser=<username>]
[--fail=<user|ip|none>]
[--gethost=<all|fail|none>]
[--help]
where:
--color|c: color to be used for FAILED login message information
--srchuser|s: print ONLY the logins/failed logins for <username>
--fail|f: if 'user': print ONLY users who have failed logins. If 'ip': print ONLY the failed login attempts. 'none': print all records regardless if failure
--gethost|g: values of 'all', 'fail' or 'none'. Perform a GETHOSTBYADDR for all IPs, only on FAILED login attempts, or don't perform this action (none)
--help|h: this message\n";
example: % check_login.pl -f=user #only the accounts with failed logins
% check_login.pl -f ip #only the accounts and the ip that failed
% check_login.pl -fail=ip # same as above
% check_login.pl --fail ip # same as above
% check_login.pl -f ip -h #list only ip's that failed for accounts resolve ip to domain name
% check_login.pl -g fail #list only accounts that had a failed login
% check_login.pl -g all #list all accounts and resolve ip to domain name
% check_login.pl -c RED -f ip #change color and list only failed ip's
% check_login.pl -s user -f ip -g fail #list all failed ip addresses with ip to domain name
% check_login.pl -s user@example.com -f ip -g fail #list all failed ip addresses with ip to domain name
END
exit 0;
}
sub setlists {
my ($user, $ip, $typeval) = @_;
++$ip_list{$user}{$ip}; #we loop by this for report
++$fip_list{$user}{$ip}{'count'};
++$fip_list{$user}{$ip}{$typeval};
return;
}
# get the hostname for the iP address if requested
sub gethostname {
my ($gethost) = @_;
my $attacker = "";
if (($gethost =~ m/all/i)
|| (($gethost =~ /fail/i) && ((exists $fip_list{$user}{$ip}) && ($fip_list{$user}{$ip}{count})))) {
if((gethostbyaddr(inet_aton($ip), AF_INET)) =~ /([a-z0-9_\-]{1,5})?(:\/\/)?(([a-z0-9_\-]{1,})(:([a-z0-9_\-]{1,}))?\@)?((www\.)|([a-z0-9_\-]{1,}\.)+)?([a-z0-9_\-]{3,})(\.[a-z]{2,4})(\/([a-z0-9_\-]{1,}\/)+)?([a-z0-9_\-]{1,})?(\.[a-z]{2,})?(\?)?(((\&)?[a-z0-9_\-]{1,}(\=[a-z0-9_\-]{1,})?)+)?/) {
$attacker="$10$11$15";
}
}
return $attacker;
}
# Print results out
sub printresults {
my($ucolor, $uattr, $user, $msgcolor, $msg) = @_;
print color($ucolor, $uattr), " ";
printf "%-47s", $user;
print color('reset');
print color($msgcolor), $msg;
print color('reset');
return;
}
# Pretty clear - output a big line
sub drawline {
print "\n------------------------------------------------------------------------------------------------------------\n";
return;
}
#========================================================================
# SECTION - GET input parameters
#========================================================================
# Get the command line parameters for processing
my $fcolor = 'YELLOW';
my $srchuser = '@';
my $failtype = 'none';
my $gethost = 'fail';
my $help, $dbug = 0;
&GetOptions( "color=s" => \$fcolor,
"srchuser=s" => \$srchuser,
"fail=s" => \$failtype, # user, ip, none
"gethost=s" => \$gethost, # all, fail, none
"debug" => \$dbug, # turn on debugging
"help" => \$help);
# call Help if parameters do not meet expected values or help is requested
usage() if($help || ($failtype !~ m/^user|ip|none$/i) || ($gethost !~ m/^all|fail|none$/i));
#========================================================================
# SECTION - PARSE audit.log files & process accordingly
#========================================================================
chdir "/opt/zimbra/log";
for (glob 'audit.log*') {
$lines = 0;
$audit_log = $_ eq 'audit.log' ? 1 : 0;
#print "Opening file $_";
open (IN, sprintf("zcat -f %s |", $_))
or die("Can't open pipe from command 'zcat -f $filename' : $!\n");
# part the audit logs looking for access types
while (<IN>)
{
if (m#invalid#i)
{
#print $_;
if ((m#ImapS#i) && !(m#INFO#))
{
my($ip,$user) = m#.*\s+\[ip=.*;oip=(.*);via=.*;\]\s*.* failed for\s+\[(.*)\].*$#i;
$uagent = "imap";
#print " - ip is $ip, user is $user, agent is $uagent\n";
#print $_;
setlists($user, $ip, $uagent);
}
elsif ((m#Pop#i) && !(m#INFO#))
{
my($ip,$user) = m#oip=(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3});.* failed for\s+\[(.*)\].*$#i;
$uagent = "pop";
#print " - ip is $ip, user is $user, agent is $uagent\n";
#print $_;
setlists($user, $ip, $uagent);
}
elsif (m#http#i)
{
my($user,$ip,$uagent) = m#.*\s+\[name=(.*);oip=(.*);ua=(.*);\].*$#i;
$uagent = "web";
#print " - ip is $ip, user is $user, agent is $uagent\n";
#print $_;
setlists($user, $ip, $uagent);
}
elsif (m#qtp#i)
{
my($ip,$user) = m#.*\s+\[name=.*;oip=(.*);oport.*\]\s*.* failed for\s+\[(.*)\].*$#i;
$uagent = "smtp";
#print " - ip is $ip, user is $user, agent is $uagent\n";
#print $_;
setlists($user, $ip, $uagent);
}
}
elsif (m#AuthRequest#i && ($_ !~ m/zimbra/i))
{
my($user,$ip,$uagent) = m#.*\s+\[name=(.*);oip=(.*);ua=(.*);\].*$#i;
++$ip_list{$user}{$ip};
#if ($audit_log == 1) { print $_; }
#printf "%4d: - ip is %15s, user is %45s, agent is %s\n",$lines,$ip,$user,$uagent;
}
$lines++;
} # End While (<IN>) loop
close (IN);
}
#========================================================================
# SECTION - PRINT / MAIN
#========================================================================
#debug
#print Dumper \%ip_list;
#print Dumper \%fip_list;
drawline();
# Print out the arrays by username. Flag failures.
for $user (sort keys %ip_list )
{
# Skip this user, if -s parameter is given and user is not in search string
next if(index($user,$srchuser) == -1);
# Proceed only if we're only looking for users who have failed logins recorded
next if(($failtype =~ /user|ip/i) && !(exists $fip_list{$user}));
$total = 0;
$totalf = 0;
for $ip (sort {$ip_list{$user}{$b} <=> $ip_list{$user}{$a}} keys %{$ip_list{$user}} )
{
# See count of how many times
if(($failtype !~ /ip/i) || (($failtype =~ /ip/) && exists $fip_list{$user}{$ip})) {
printf ("[%4d] logins from IP %15s ", $ip_list{$user}{$ip},$ip);
}
$total = $total+$ip_list{$user}{$ip}; # Count all for this username
# lookup the domain if requested
my $attacker = "[" . gethostname($gethost) . "]";
if ((exists $fip_list{$user}{$ip}) && ($fip_list{$user}{$ip}{count}))
{
print color($fcolor);
printf "%-30s", $attacker if ($gethost !~ /none/i);
printf " Failed [%4d] : ", $fip_list{$user}{$ip}{count};
for $etypes (keys %{$fip_list{$user}{$ip}}) {
next if $etypes =~ /count/;
printf " using %s [%4d] ", $etypes, $fip_list{$user}{$ip}{$etypes};
}
print color('reset');
printf ("\n");
$totalf = $totalf+$fip_list{$user}{$ip}{count}; # Count all failed for this username
} elsif (($gethost =~ /all/i) && !($failtype =~ /ip/i))
{
printf "$attacker\n";
} elsif ($failtype !~ /ip/) {
printf ("\n");
}
}
# Print out user information & message totals
if ($totalf>0) {
my $msg = sprintf("%d failed of total %d login attempts!!!", $totalf, $total);
$msgcolor = $totalf==$total ? "RED" : "YELLOW";
printresults("WHITE", "BOLD", $user, $msgcolor, $msg);
} elsif ($failtype !~ /ip/i) {
my $msg = " No failed logins. Yeeee :)";
printresults("WHITE", "BOLD", $user, "GREEN", $msg);
}
drawline();
}
printf ("\n");
print color('reset'); # make sure we clean up
Re: Simple program to report successful/fail ip logins and sorted by count
Hi JDunphy,
that's one huge code rewrite, really love it how you incorporated all changes. And love it being organized
BUT there's still work to do - I noticed results are buggy/different, if I switch the order of log parsing elsif's for "http" and "qtp". Results are much different and it looks like WEB and SMTP parsers rule out each other:
QTP before HTTP:
HTTP before QTP:
So it makes me wonder if other checks also rule each other out?
If I get some free time I'll check those out, too.
that's one huge code rewrite, really love it how you incorporated all changes. And love it being organized
BUT there's still work to do - I noticed results are buggy/different, if I switch the order of log parsing elsif's for "http" and "qtp". Results are much different and it looks like WEB and SMTP parsers rule out each other:
QTP before HTTP:
Code: Select all
------------------------------------------------------------------------------------------------------------
[ 7] logins from IP 84.41.99.5
[ 6] logins from IP 89.212.81.25 [hostmachine.net] Failed [ 6] : using smtp [ 6]
[ 1] logins from IP 212.103.140.219
user1@domain.com 6 failed of total 14 login attempts!!!
------------------------------------------------------------------------------------------------------------
[ 2] logins from IP 89.212.238.52
[ 2] logins from IP 90.157.166.108
[ 1] logins from IP 46.123.249.67
[ 1] logins from IP 46.123.250.247
[ 1] logins from IP 46.164.7.43
user2@domain.com No failed logins. Yeeee :)
------------------------------------------------------------------------------------------------------------
Code: Select all
------------------------------------------------------------------------------------------------------------
[ 7] logins from IP 84.41.99.5
[ 1] logins from IP 212.103.140.219
user1@domain.com No failed logins. Yeeee :)
------------------------------------------------------------------------------------------------------------
[ 2] logins from IP 90.157.166.108
[ 2] logins from IP 46.123.249.67 [simobil.net] Failed [ 1] : using web [ 1]
[ 2] logins from IP 89.212.238.52
[ 1] logins from IP 46.164.7.43
[ 1] logins from IP 46.123.250.247
user2@domain.com 1 failed of total 8 login attempts!!!
------------------------------------------------------------------------------------------------------------
If I get some free time I'll check those out, too.
Re: Simple program to report successful/fail ip logins and sorted by count
The zimbraMailTrustedIP was already set correctlyJDunphy wrote:Try this... where X.X.X.X is your single server ip.milauria wrote:Hello -- my single zimbra server runs behind a reverse proxy server (nginx).
Anyway to allow the script read the originating IP ?
As today it reads as all successful connection are coming from the reverse proxy IP.Code: Select all
# allow /opt/zimbra/log/audit.log to show correct client ip address # because we are now behind proxy. % zmprov mcf +zimbraMailTrustedIP 127.0.0.1 +zimbraMailTrustedIP X.X.X.X % zmmailboxdctl restart
Instead in my nginx reverse proxy I was missing the 2 headers to be able to transfer correctly the real IP:
Code: Select all
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
- JDunphy
- Outstanding Member
- Posts: 901
- Joined: Fri Sep 12, 2014 11:18 pm
- Location: Victoria, BC
- ZCS/ZD Version: 9.0.0_P39 NETWORK Edition
Re: Simple program to report successful/fail ip logins and sorted by count
This is one area where I will need your help. I wasn't able to verify your qtp section. When I generated a MUA authentication error it didn't appear in my 8.7+ audit.logs. The only place I could find it was in /var/log/maillog. Could you provide me with the output of:Labsy wrote:Hi JDunphy,
that's one huge code rewrite, really love it how you incorporated all changes. And love it being organized
BUT there's still work to do - I noticed results are buggy/different, if I switch the order of log parsing elsif's for "http" and "qtp". Results are much different and it looks like WEB and SMTP parsers rule out each other:
Code: Select all
% zcat -f /opt/zimbra/log/audit.log* |grep -i invalid | grep qtp
I know in my logs, simply asking for qtp will not work because I have lines like this...
Code: Select all
2018-11-04 19:12:39,934 WARN [qtp2036958521-453494:http://localhost:8080 ...
((m#http#) && (#zclient#)) for the http section... or require 'soap' or 'failed', etc. Without knowing what the qtp line we are pattern matching against, I hesitate to recommend a fix. Certainly with my logs, it would yield incorrect results to place qtp in front of http in the parsing order.
Thanks
Jim
- JDunphy
- Outstanding Member
- Posts: 901
- Joined: Fri Sep 12, 2014 11:18 pm
- Location: Victoria, BC
- ZCS/ZD Version: 9.0.0_P39 NETWORK Edition
Re: Simple program to report successful/fail ip logins and sorted by count
Yep that is it. Glad you got it working. I am guessing for whatever reason this wasn't defined.milauria wrote: The zimbraMailTrustedIP was already set correctly
Instead in my nginx reverse proxy I was missing the 2 headers to be able to transfer correctly the real IP:Thanks anyway for the hintCode: Select all
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header Host $http_host;
Code: Select all
% zmlocalconfig zimbra_http_originating_ip_header
zimbra_http_originating_ip_header = X-Forwarded-For
Code: Select all
% zmproxyconfgen -v -n
Re: Simple program to report successful/fail ip logins and sorted by count
Hi Jim,JDunphy wrote: This is one area where I will need your help. I wasn't able to verify your qtp section. When I generated a MUA authentication error it didn't appear in my 8.7+ audit.logs. The only place I could find it was in /var/log/maillog. Could you provide me with the output of:JimCode: Select all
% zcat -f /opt/zimbra/log/audit.log* |grep -i invalid | grep qtp
here's the output you asked for.
Seems like there are two different types of logs under "qtp" type:
Code: Select all
04.11.2018 17:12:41 WARN [qtp1684106402-196362:http://localhost:8080/service/soap/AuthRequest] [name=sara@domain.com;oip=46.123.248.184;ua=zclient/8.8.7_GA_1964;soapId=1e47190;] security - cmd=Auth; account=sara@domain.com; protocol=soap; error=authentication failed for [sara@domain.com], invalid password;
04.11.2018 20:52:46 WARN [qtp1684106402-197556:https:https://zimbra.domain.com:7073/service/admin/soap/] [name=info@domain.com;oip=145.249.106.201;oport=41692;oproto=smtp;soapId=1e4782f;] security - cmd=Auth; account=info@domain.com; protocol=soap; error=authentication failed for [info@domain.com], invalid password;
I think we need to parse them BOTH, each under its own category.
- JDunphy
- Outstanding Member
- Posts: 901
- Joined: Fri Sep 12, 2014 11:18 pm
- Location: Victoria, BC
- ZCS/ZD Version: 9.0.0_P39 NETWORK Edition
Re: Simple program to report successful/fail ip logins and sorted by count
This is what I added to my version:Labsy wrote: Seems like there are two different types of logs under "qtp" type:Code: Select all
04.11.2018 17:12:41 WARN [qtp1684106402-196362:http://localhost:8080/service/soap/AuthRequest] [name=sara@domain.com;oip=46.123.248.184;ua=zclient/8.8.7_GA_1964;soapId=1e47190;] security - cmd=Auth; account=sara@domain.com; protocol=soap; error=authentication failed for [sara@domain.com], invalid password; 04.11.2018 20:52:46 WARN [qtp1684106402-197556:https:https://zimbra.domain.com:7073/service/admin/soap/] [name=info@domain.com;oip=145.249.106.201;oport=41692;oproto=smtp;soapId=1e4782f;] security - cmd=Auth; account=info@domain.com; protocol=soap; error=authentication failed for [info@domain.com], invalid password;
For http
Code: Select all
elsif ((m#http#i) && (m#zclient#))
Code: Select all
elsif ((m#oproto=smtp#) && (m#failed#))
Code: Select all
my($ip,$user) = m#oip=(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3});.* failed for\s+\[(.*)\].*$#i;
Re: Simple program to report successful/fail ip logins and sorted by count
I think it would be a good idea to add the date and time.