Very Slow Email Delivery

[combrolegit]

Hi,
You have to confirm that your server is not compromised before using inplace upgrade.
And if your server is ok, please see: https://wiki.zimbra.com/wiki/Spamming_troubleshooting
Regards,
Minh.

Sorry with this question, but how to identify that my server is compromised since the domain shows valid domain from our own server, please let me know how to identify that my server is compromised, thank you so much.

[combrolegit]

I just ran this command:

cat /var/log/zimbra.log | sed -n 's/.*sasl_username=//p' | sort | uniq -c | sort -nr, adn the result as follow:

Capture1.PNG (9 KiB) Viewed 693 times

Is that an ok value?

[zimico]

Dear,
In case you think your server may be compromised, Please investigate the output of:
#su – zimbra

$zmcontrol -v

$grep python-requests /opt/zimbra/log/access_log* $ grep downloads /opt/zimbra/log/access_log* | grep -i jsp

$ ls -lrth /var/tmp/*.sh
$ ls -lrth /opt/zimbra/log/*.sh

$ crontab -l | egrep -i ‘zmmailboxdwatch|zmstorewatch’
$ crontab -l | egrep -i ‘\.sh|\.py’

Best regards,
Minh.

[combrolegit]

Dear,
In case you think your server may be compromised, Please investigate the output of:
#su – zimbra

$zmcontrol -v

$grep python-requests /opt/zimbra/log/access_log* $ grep downloads /opt/zimbra/log/access_log* | grep -i jsp

$ ls -lrth /var/tmp/*.sh
$ ls -lrth /opt/zimbra/log/*.sh

$ crontab -l | egrep -i ‘zmmailboxdwatch|zmstorewatch’
$ crontab -l | egrep -i ‘\.sh|\.py’

Best regards,
Minh.

Hi Minh,

The result might not as expected, here is the result:

zimbra@mail03:/home/zmadmin$ zmcontrol -v
Release 8.8.15.GA.3869.UBUNTU18.64 UBUNTU18_64 FOSS edition, Patch 8.8.15_P6.
zimbra@mail03:/home/zmadmin$ grep python-requests /opt/zimbra/log/access_log* $ grep downloads /opt/zimbra/log/access_log* | grep -i jsp
grep: $: No such file or directory
grep: grep: No such file or directory
grep: downloads: No such file or directory
zimbra@mail03:/home/zmadmin$ ls -lrth /var/tmp/*.sh
ls: cannot access '/var/tmp/*.sh': No such file or directory
zimbra@mail03:/home/zmadmin$ ls -lrth /opt/zimbra/log/*.sh
ls: cannot access '/opt/zimbra/log/*.sh': No such file or directory
zimbra@mail03:/home/zmadmin$ crontab -l | egrep -i ‘zmmailboxdwatch|zmstorewatch’
zmstorewatch’: command not found
zimbra@mail03:/home/zmadmin$ crontab -l | egrep -i ‘\.sh|\.py’
.py’: command not found
zimbra@mail03:/home/zmadmin$

[combrolegit]

Well actually this is not an inplace upgrade, I move it to new server.

[zimico]

Hi,
Sorry for the cut & past cli. Here it is:

Code: Select all

grep python-requests /opt/zimbra/log/access_log*
grep downloads /opt/zimbra/log/access_log* | grep -i jsp
ls -lrth /var/tmp/*.sh
ls -lrth /opt/zimbra/log/*.sh
crontab -l | egrep -i 'zmmailboxdwatch|zmstorewatch'
crontab -l | egrep -i '\.sh|\.y'

If you don't have 100% CPU. I think you then can focus on spam troublshooting in the wiki which I listed in previous post.
Best regards,
Minh.

[combrolegit]

Hi,
Sorry for the cut & past cli. Here it is:

Code: Select all

grep python-requests /opt/zimbra/log/access_log*
grep downloads /opt/zimbra/log/access_log* | grep -i jsp
ls -lrth /var/tmp/*.sh
ls -lrth /opt/zimbra/log/*.sh
crontab -l | egrep -i 'zmmailboxdwatch|zmstorewatch'
crontab -l | egrep -i '\.sh|\.y'

If you don't have 100% CPU. I think you then can focus on spam troublshooting in the wiki which I listed in previous post.
Best regards,
Minh.

Hi Minh, here's the result, what do you think?

Code: Select all

zimbra@mail03:/home/zmadmin$ grep python-requests /opt/zimbra/log/access_log*
/opt/zimbra/log/access_log.2020-01-30:71.6.199.23 - - [30/Jan/2020:06:01:04 +0000] "GET /favicon.ico HTTP/1.1" 404 1477 "-" "python-requests/2.19.1" 12
/opt/zimbra/log/access_log.2020-01-31:188.165.216.213 - - [31/Jan/2020:05:05:28 +0000] "GET / HTTP/1.0" 200 4833 "-" "python-requests/2.22.0" 38
/opt/zimbra/log/access_log.2020-02-03:77.247.110.73 - - [03/Feb/2020:15:07:20 +0000] "GET //a2billing/customer/templates/default/footer.tpl HTTP/1.0" 404 1477 "-" "python-requests/2.6.0 CPython/2.7.5 Linux/3.10.0-1062.9.1.el7.x86_64" 25
/opt/zimbra/log/access_log.2020-02-03:77.247.110.73 - - [03/Feb/2020:19:42:09 +0000] "GET //a2billing/customer/templates/default/footer.tpl HTTP/1.0" 404 1477 "-" "python-requests/2.6.0 CPython/2.7.5 Linux/3.10.0-1062.9.1.el7.x86_64" 12
/opt/zimbra/log/access_log.2020-02-04:94.102.49.190 - - [04/Feb/2020:16:41:44 +0000] "GET /favicon.ico HTTP/1.0" 404 1477 "-" "python-requests/2.10.0" 10
/opt/zimbra/log/access_log.2020-02-06:176.31.110.135 - - [06/Feb/2020:20:52:59 +0000] "GET / HTTP/1.0" 200 4832 "-" "python-requests/2.22.0" 20
/opt/zimbra/log/access_log.2020-02-08:185.164.72.119 - - [08/Feb/2020:14:29:35 +0000] "POST /Autodiscover/Autodiscover.xml HTTP/1.1" 400 293 "-" "python-requests/2.22.0" 6
/opt/zimbra/log/access_log.2020-02-08:185.164.72.119 - - [08/Feb/2020:15:12:30 +0000] "POST /Autodiscover/Autodiscover.xml HTTP/1.0" 400 293 "-" "python-requests/2.22.0" 5
/opt/zimbra/log/access_log.2020-02-09:71.6.158.166 - - [09/Feb/2020:14:25:47 +0000] "GET /favicon.ico HTTP/1.0" 404 1477 "-" "python-requests/2.10.0" 20
/opt/zimbra/log/access_log.2020-02-09:185.164.72.119 - - [09/Feb/2020:15:07:42 +0000] "POST /Autodiscover/Autodiscover.xml HTTP/1.1" 400 293 "-" "python-requests/2.22.0" 11
/opt/zimbra/log/access_log.2020-02-09:185.164.72.119 - - [09/Feb/2020:23:55:31 +0000] "POST /Autodiscover/Autodiscover.xml HTTP/1.0" 400 293 "-" "python-requests/2.22.0" 53
/opt/zimbra/log/access_log.2020-02-11:80.82.77.139 - - [11/Feb/2020:07:10:29 +0000] "GET /favicon.ico HTTP/1.1" 404 1477 "-" "python-requests/2.13.0" 33
/opt/zimbra/log/access_log.2020-02-11:37.187.74.157 - - [11/Feb/2020:07:10:35 +0000] "GET / HTTP/1.1" 200 4831 "-" "python-requests/2.22.0" 18
/opt/zimbra/log/access_log.2020-02-12:185.164.72.119 - - [12/Feb/2020:09:46:20 +0000] "POST /Autodiscover/Autodiscover.xml HTTP/1.1" 400 293 "-" "python-requests/2.22.0" 6
zimbra@mail03:/home/zmadmin$ grep downloads /opt/zimbra/log/access_log* | grep -i jsp
zimbra@mail03:/home/zmadmin$ ls -lrth /var/tmp/*.sh
ls: cannot access '/var/tmp/*.sh': No such file or directory
zimbra@mail03:/home/zmadmin$ ls -lrth /opt/zimbra/log/*.sh
ls: cannot access '/opt/zimbra/log/*.sh': No such file or directory
zimbra@mail03:/home/zmadmin$ crontab -l | egrep -i 'zmmailboxdwatch|zmstorewatch'
zimbra@mail03:/home/zmadmin$ crontab -l | egrep -i '\.sh|\.y'
zimbra@mail03:/home/zmadmin$



[zimico]

Hi,
The output seems to be normal.
Did you check incoming spam issue and outgoing spam issue?
Please show the output of:
as root:

Code: Select all

# cat /var/log/zimbra.log | sed -n 's/.*sasl_username=//p' | sort | uniq -c | sort -nr
# /opt/zimbra/libexec/zmqstat
# cat /var/log/zimbra.log | grep -Eo '[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}'| sort | uniq -c | sort -nr
# netstat -np --protocol=inet | grep ESTABLISHED | grep :smtpd
# htop (or top)

Regards,
Minh.

[combrolegit]

Hi,
The output seems to be normal.
Did you check incoming spam issue and outgoing spam issue?
Please show the output of:
as root:

Code: Select all

# cat /var/log/zimbra.log | sed -n 's/.*sasl_username=//p' | sort | uniq -c | sort -nr
# /opt/zimbra/libexec/zmqstat
# cat /var/log/zimbra.log | grep -Eo '[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}'| sort | uniq -c | sort -nr
# netstat -np --protocol=inet | grep ESTABLISHED | grep :smtpd
# htop (or top)

Regards,
Minh.

Code: Select all

zimbra@mail03:/home/zmadmin$ cat /var/log/zimbra.log | sed -n 's/.*sasl_username=//p' | sort | uniq -c | sort -nr
     75 no-reply@te***up.com
     68 mthu1.11400057@t***m.co.id
     62 mthu1.11400281@t***m.co.id
     54 sogest02.21400046@t***m.co.id
     46 megar1.11400280@t***m.co.id
     43 seilam01.21400052@t***m.co.id
     43 metlam01.21400007@t***m.co.id
     42 mthr1.11400024@t***em.co.id
     39 sogmac01.21400035@t***m.co.id
     38 mthr1.11400281@t***m.co.id
     36 tntpackdev.adm@t***up.com
     36 sogcli01.21400004@t***m.co.id
     36 apjptt.mks@t***p.com
     33 mthu1.11400205@t***m.co.id



Currently mail queue already back to normal, but often happen later...

Code: Select all

root@mail03:/home/zmadmin# /opt/zimbra/libexec/zmqstat
incoming=0
corrupt=0
deferred=0
hold=0
active=2


Code: Select all

root@mail03:/home/zmadmin# cat /var/log/zimbra.log | grep -Eo '[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}'| sort | uniq -c | sort -nr
 486260 127.0.0.1
  75193 10.66.1.16
  21054 10.66.2.22
  18656 192.168.100.20
  16315 10.66.1.20
   7089 10.63.61.4
   6203 92.118.38.41
   5199 202.158.29.162
   4941 10.66.1.14
   4157 117.102.84.140
   1790 129.10.16.21
    735 141.0.0.118


Code: Select all

root@mail03:/home/zmadmin# netstat -np --protocol=inet | grep ESTABLISHED | grep :smtpd
root@mail03:/home/zmadmin#


[zimico]

Your system parameters seem to be normal. Is mail flow ok now?
Best regards,
Minh.