zimbra 8.7 ldap down for (exactly) one hour

[gherzig]

Hi all. We are facing a very strange problem with zimbra and ldap.

At random times, with no apparently reason, ldap stop working. The log just shows when it started to fail:

Code: Select all

Jul 23 15:46:13 azteca postfix/proxymap[6830]: error: dict_ldap_connect: Unable to set STARTTLS: -1: Can't contact LDAP server 
Jul 23 15:46:13 azteca postfix/trivial-rewrite[6829]: warning: proxy:ldap:/opt/zimbra/conf/ldap-transport.cf lookup error for "*" 
Jul 23 15:46:14 azteca postfix/proxymap[6830]: error: dict_ldap_connect: Unable to set STARTTLS: -1: Can't contact LDAP server 
Jul 23 15:46:14 azteca postfix/trivial-rewrite[6829]: warning: proxy:ldap:/opt/zimbra/conf/ldap-transport.cf lookup error for "*" 
Jul 23 15:46:15 azteca postfix/proxymap[6830]: error: dict_ldap_connect: Unable to set STARTTLS: -1: Can't contact LDAP server

At this point, the ldap process itself seems to be up:
zimbra@azteca:~$ ps aux|grep slap

Code: Select all

zimbra    9941  0.0  0.0   8876   868 pts/3    R+   16:20   0:00 grep slap 
zimbra   38892  0.0  0.2 84055784 39108 ?      Ssl  16:11   0:00 /opt/zimbra/common/libexec/slapd -l LOCAL0 -u zimbra -h ldap://azteca.our.site.org:389 ldapi:/// -F /opt
/zimbra/data/ldap/config

But a ldap search fails:
zimbra@azteca:~$ ldapsearch -x -h azteca.our.site.org -v -d 7                                           
ldap_initialize( ldap://azteca.our.site.org )
ldap_create
ldap_url_parse_ext(ldap://azteca.our.site.org)
ldap_sasl_bind
ldap_send_initial_request
ldap_new_connection 1 1 0
ldap_int_open_connection
ldap_connect_to_host: TCP azteca.our.site.org:389
ldap_new_socket: 3
ldap_prepare_socket: 3
ldap_connect_to_host: Trying 192.168.0.1:389
ldap_pvt_connect: fd: 3 tm: -1 async: 0
attempting to connect:
connect errno: 111
ldap_close_socket: 3
ldap_err2string
ldap_sasl_bind(SIMPLE): Can't contact LDAP server (-1)

* Restarting zimbra has no effect
* Restarting the host has no effect

Maybe the most strange thing: Afer exactly one hour of downtime, ldap start to work again, without admin intervention...
(This is the last error prior to ldap being up and running again)

Jul 23 16:46:00 azteca postfix/proxymap[32241]: error: dict_ldap_connect: Unable to set STARTTLS: -1: Can't contact LDAP server

See, one hour after the first error!!!
Im sure this has to be a clue, but...

This is Release 8.7.11.GA.1854.UBUNTU14.64 UBUNTU14_64 FOSS edition, Patch 8.7.11_P4, running on a Ubuntu 18.04 LTS

Any help would be appreciated (a lot!)

Thanks
Gerardo

[ccelis5215]

Hi all. We are facing a very strange problem with zimbra and ldap.

At random times, with no apparently reason, ldap stop working. The log just shows when it started to fail:

Code: Select all

Jul 23 15:46:13 azteca postfix/proxymap[6830]: error: dict_ldap_connect: Unable to set STARTTLS: -1: Can't contact LDAP server 
Jul 23 15:46:13 azteca postfix/trivial-rewrite[6829]: warning: proxy:ldap:/opt/zimbra/conf/ldap-transport.cf lookup error for "*" 
Jul 23 15:46:14 azteca postfix/proxymap[6830]: error: dict_ldap_connect: Unable to set STARTTLS: -1: Can't contact LDAP server 
Jul 23 15:46:14 azteca postfix/trivial-rewrite[6829]: warning: proxy:ldap:/opt/zimbra/conf/ldap-transport.cf lookup error for "*" 
Jul 23 15:46:15 azteca postfix/proxymap[6830]: error: dict_ldap_connect: Unable to set STARTTLS: -1: Can't contact LDAP server

At this point, the ldap process itself seems to be up:
zimbra@azteca:~$ ps aux|grep slap

Code: Select all

zimbra    9941  0.0  0.0   8876   868 pts/3    R+   16:20   0:00 grep slap 
zimbra   38892  0.0  0.2 84055784 39108 ?      Ssl  16:11   0:00 /opt/zimbra/common/libexec/slapd -l LOCAL0 -u zimbra -h ldap://azteca.our.site.org:389 ldapi:/// -F /opt
/zimbra/data/ldap/config

But a ldap search fails:
zimbra@azteca:~$ ldapsearch -x -h azteca.our.site.org -v -d 7                                           
ldap_initialize( ldap://azteca.our.site.org )
ldap_create
ldap_url_parse_ext(ldap://azteca.our.site.org)
ldap_sasl_bind
ldap_send_initial_request
ldap_new_connection 1 1 0
ldap_int_open_connection
ldap_connect_to_host: TCP azteca.our.site.org:389
ldap_new_socket: 3
ldap_prepare_socket: 3
ldap_connect_to_host: Trying 192.168.0.1:389
ldap_pvt_connect: fd: 3 tm: -1 async: 0
attempting to connect:
connect errno: 111
ldap_close_socket: 3
ldap_err2string
ldap_sasl_bind(SIMPLE): Can't contact LDAP server (-1)

* Restarting zimbra has no effect
* Restarting the host has no effect

Maybe the most strange thing: Afer exactly one hour of downtime, ldap start to work again, without admin intervention...
(This is the last error prior to ldap being up and running again)

Jul 23 16:46:00 azteca postfix/proxymap[32241]: error: dict_ldap_connect: Unable to set STARTTLS: -1: Can't contact LDAP server

See, one hour after the first error!!!
Im sure this has to be a clue, but...

This is Release 8.7.11.GA.1854.UBUNTU14.64 UBUNTU14_64 FOSS edition, Patch 8.7.11_P4, running on a Ubuntu 18.04 LTS

Any help would be appreciated (a lot!)

Thanks
Gerardo

Strange indeed.. do you have any firewall server?

ccelis

[gherzig]

Ccelis, thanks for your time.
There is a fail2ban server who generate iptables rules....oh boy...
(making some quick checks)

You hit it right in the center, men!!

Fail2ban was indeed banning their own server IP :/

Thank you so much!!!

Gerardo

[ccelis5215]

Ccelis, thanks for your time.
There is a fail2ban server who generate iptables rules....oh boy...
(making some quick checks)

You hit it right in the center, men!!

Fail2ban was indeed banning their own server IP :/

Thank you so much!!!

Gerardo

Glad to help you.

ccelis.