zimbra does not start after upgrade: ldap fail.

[gcalzada]

Hi all, I am having some issue with zcs after migration, specifically at ldap startup.

I have migrated zcs 8.0.4 to 8.7.1 successfully. Once the migration was finished zimbra started ok and either user accounts and administration console were accessible.

The problem appeared after the first "zmcontrol stop", after that, zcs did not start again and always finish with output:

[zimbra@######## ~]$ zmcontrol start
Host ########.###
Starting ldap...Done.
Failed.
Failed to start slapd. Attempting debug start to determine error.
TLS: error:02001002:system library:fopen:No such file or directory bss_file.c:175
TLS: error:2006D080:BIO routines:BIO_new_file:no such file bss_file.c:178
main: TLS init def ctx failed: -1



[zimbra@######## ~]$


I have read some forums, reimported commercial certificates, and checked "/opt/zimbra/data/ldap/config/cn=config.ldif"

I have found that "/opt/zimbra/data/ldap/config/cn=config.ldif" has much more parameters than before the upgrade. "cn=config.ldif" has "olcTLSDHParamFile: /opt/zimbra/conf/dhparam.pem" and that "dhparam.pem" does not exist at filesystem level.

Also, I have ran "/opt/zimbra/libexec/zmfixperms -extended" without success. (to try something)

So, can anyone help me? any ideas?

[kbish]

hi
in my opinion some system packages is missing: "System library:fopen:No such file or directory" . Try to check installed packages.

[NetCircle]

We have nearly the identical situation except we upgraded back in October 2016 from 8.0.7 to 8.7.0 and this host has been shutting down daily since then to rsync Zimbra to a backup directory without incident, however Saturday it shutdown without event, rsync'd to the backup directory but failed to restart:

Code: Select all

[root@### ####]# cat /tmp/zimbra-backup.log
20170121-0310 -- Stopping Zimbra to begin backup
Host ###.############.###
        Stopping zmconfigd...Done.
        Stopping zimlet webapp...Done.
        Stopping zimbraAdmin webapp...Done.
        Stopping zimbra webapp...Done.
        Stopping service webapp...Done.
        Stopping stats...Done.
        Stopping mta...Done.
        Stopping spell...Done.
        Stopping snmp...Done.
        Stopping cbpolicyd...Done.
        Stopping archiving...Done.
        Stopping opendkim...Done.
        Stopping amavis...Done.
        Stopping antivirus...Done.
        Stopping antispam...Done.
        Stopping proxy...Done.
        Stopping memcached...Done.
        Stopping mailbox...Done.
        Stopping logger...Done.
        Stopping dnscache...Done.
        Stopping ldap...Done.
Reloading crond: [  OK  ]
20170121-0321 -- Begin rsync from /opt/zimbra/ to /var/rsync-zimbra
`/opt/zimbra/data/ldap/mdb/db/data.mdb' -> `/var/rsync-zimbra/data/ldap/mdb/db/./data.mdb'
20170121-0332 -- rsync complete
Starting MySQL database
Starting mysqld...done.
Sleeping for 30 seconds while MySQL becomes available
20170121-0333 -- Dumping MySQL databases
Shutting down MySQL database
Stopping mysqld... done.
Host ###.############.###
        Starting ldap...Done.
Failed.
Failed to start slapd.  Attempting debug start to determine error.
TLS: error:02001002:system library:fopen:No such file or directory bss_file.c:175
TLS: error:2006D080:BIO routines:BIO_new_file:no such file bss_file.c:178
main: TLS init def ctx failed: -1

This slapd startup failure is identical to that posted by gcalzada.
Since the rsync was successful my backup copy is now identical to the working copy so there is no relief to be had by restoring the backup. It produces the same result.
On the chance there is a system file missing as kbish suggests, I ran the following check for missing files but nothing was returned:

Code: Select all

[root@### ####]#> for package in `rpm -qa`;do rpm -V $package;done|grep missing
[root@### ####]#>

This server is a one of a multi-server, master-master LDAP and host to several hundred mailboxes, some quite large. In the past whenever there has been suspected file corruption, we would reinstall the same Zimbra version using the upgrade option but the current installation packages require ldap to start which is the issue we're trying to fix.

Does anyone have an idea of how to diagnose this problem? We desperately need to recover this server and get it back online.

[NetCircle]

One of my colleagues was able to determine the problem which he corrected and Zimbra is back up and running.
Again, to document the problem:

Code: Select all

[zimbra@### ~]$ ldap start
Failed to start slapd.  Attempting debug start to determine error.
TLS: error:02001002:system library:fopen:No such file or directory bss_file.c:175
TLS: error:2006D080:BIO routines:BIO_new_file:no such file bss_file.c:178
main: TLS init def ctx failed: -1

After checking and searching, he found it was caused by missing /opt/zimbra/conf/dhparam.pem.

Code: Select all

[zimbra@### ~]$ openssl dhparam -out /opt/zimbra/conf/dhparam.pem 2048

LDAP was able to start after generating this file and Zimbra is working again now.

[gcalzada]

hi, thanks to everyone.

As NetCircle said, "dhparam" was no created by default and was the root cause for this issue.

Command:
[ZIMBRA PATH] openssl dhparam -out /opt/zimbra/conf/dhparam.pem 2048


### As a comment, other issue we faced after migration was that "zmfixperm" does not fix "zimbra/postfix" file permissions. If "mta" fail to start, you may need to use "/opt/zimbra/bin/postfix set-permissions; /opt/zimbra/bin/postfix check" commands ###

I apologize for my late confirmation as I was out on holydays.

zimbra 8.7.1 is up and running.

Migration successfull.

Thanks again.

[L. Mark Stone]

FWIW I created a bugzilla for this issue:

https://bugzilla.zimbra.com/show_bug.cgi?id=107576

All the best,
Mark