Here's my solution, that worked, and gave me an understanding of internal communications of ZCS. Credits to guys in IRC #zimbra, particularly JoBbZ and Jorge-zimb!
Environment
My environment is different from original posters, but in essence, I think, the base of problem route is the same. Please get your environment data correctly, to understand this solution. Here are my environment data (before upgrade, during modifications):
Zimbra Open Source 8.0.7 Ubuntu server 12.04 64-bit multi server install - each zimbra services item on own server node;
Planned intention - upgrade 8.0.7 > 8.6 ubuntu server 12.04 64-bit multi server, prior to Ubuntu OS upgrades to 14.04LTS;
Our server hostnames are 1:1 (FQDN), that are used in external (public facing) or internal Zimbra communications;
We use Split DNS, external and internal resolution works correctly and /etc/hosts files are in place;
Using mix of self signed SSLs (created by Zimbra) for internals, and commercial SSLs with intermediary and root server chaines in a single file, that are valid and not expired.
REMINDER: please do backups or snapshots, before modifications, and/or copy original files, you remove, to get you back on your feet, if something goes wrong
(it helped me...)
The problem source
The issue rise when a mix of commercially and self signed certificates are deployed in single server or multi server installations, which is "not supported by Zimbra" - read - it works, but you have to patch/solve it manually. It rise, because of of MITM flaw security improvement in ZCS 8.6, which was present in versions before. Zimbra all of the communications between ZCS instances run on self signed SSL certificates created and deployed by default. Introduced solution to strengthen zommunications is - mandatory CA check, which results in error in case of mixed SSLs, if not provided manually for slef signed ones.
Those, who use Zimbra prior 8.6, please secure your network communications between zimbra instances, especially, if you communicate over internet network!
In essence, when ZCS server instance, which had commercial SSLs deployed, could not successfully connect to Zimbra LDAP server, because it could not verify CA for self signed SSL. And it does not matter, whether you've configured ldap:// or ldaps:// paths for your communications.
Alternative solutions
There are a bunch of alternative solutions for this, but I didn't do it in my case, but one might consider it viable (I can not comment on solution with these below, especially in single server install). Purchase and deployment of commercial signed:
SSLs for each zimbra server;
named wildcard SSLs for group of named FQDNs (some providers offer lower cost ala wild card SSLs);
wildcard SSL certificate, that could be deployed to all ZCS servers.
Mind, that this solution puts some additional costs for SSL purchases, and some of SSL providers require additional payments for number of additional server instances, where particular SSL certificate is going to be installed (for example in HA server situations)!
Manual fix theory
To simplify understanding of this solution, the following separate Zimbra hosts are used:
TARGET: Zimbra LDAP server, which has working /opt/zimbra/conf/ca/ca.pem file, which is CA for selfsigned SSLs
CLIENT: Zimbra Proxy server, which acts as a client connecting to TARGET server, and needs correct CA file in the same location /opt/zimbra/conf/ca/ca.pem
REMINDER: I do not have an option, nor built a test lab for single server installation, so I can not confirm the issue described below, but you should be able to check and validate, that CLIENT server/service is using correct ca.pem file for internal communications!
As Zimbra installation and upgrade scripts rebuild self signed certificates (even if commercial ones are deployed), in multiserver environment, I've found out, that CLIENT and TARGET ca.pem files are different, as a result, CLIENT tries to connect to TARGET with wrong CA validation (remember, it's introduced as a security fix), resulting in error of connection to LDAP (TARGET).
Test for more information...
Quick way, to test, if particular ZCS server has errors in certificate configuration, use the following (remember to put it back after tests):
As root user:
Comment out a line, starting with TLS_REQCERT... in /opt/zimbra/.ldaprc file. If you do not do this, your communication will be correct, thus giving false results of this test. Still need to clear out things on this.
As zimbra user:
run ldapsearch -x -ZZ -H <hostname> -s base -b "" command, replacing <hostname> with your TARGET server path, for example ldap://ldap.example.com:389
If there are errors, you get notification, and some details, explaining the cause. More details could be get, if adding -d -1 switches to command above. In essence, if there are errors, your certificates are configured wrong, and you have to deal with this.
REMINDER: Remember to put modifications in first step back, after this test!
Manual fix runlist
Here are steps one by one, I did to solve this problem. Your milage may differ, and you may script these tasks too. I did it manually. As always, backup/snapshots are warmly welcome! May be I've done some steps which are not essential for solution, but I like to be clean, in case it's important in future.
After certificate correction, I run Zimbra restart, so plan for a downtime for this.
1. Verified, that TARGET and CLIENT server /opt/zimbra/conf/ca/ca.pem files differ. Just use ls -l on these folders, to compare hashes, which are symlinks to ca.pem file, located in indicated folder.
ca.pem file is Zimbra created CA for selfsigned SSL certificates. You will find other *.pem files there, which are commercial ones, including mix of intermediary and root certificates of Commercial ones, and ca.key file, related to ca.pem.
2. Archived CLIENT server ca.pem file, removed symlink (mind your hash), removed ca.pem from /opt/zimbra/conf/ca/ folder (I have multi server installation!).
3. Copied ca.pem file from TARGET (LDAP) server's /opt/zimbra/conf/ca/ to CLIENT server's /opt/zimbra/conf/ca/ folder. To be clean, please check ca.pem file ownership and permission on CLIENT server, which has to be zimbra:zimbra -rw-r-----. Permissions are important, or the other way, there will be error!
4. Re-hash ca.pem on CLIENT machine, to where ca.pem was copied from TARGET (LDAP) server
As root user on CLIENT server:
# cd /opt/zimbra/conf/ca
IMPORTANT! Please mind, that the following comand contains backtick (`), not apostrophe ('), as it executes shell command!
# ln -f -s ca.pem `openssl x509 -hash -noout -in ca.pem`.0
NOTE: I'm not particularly sure for index significance, behind hash, which in this case is .0 (dot zero), but as mentioned, I wanted to be as close, as it was before.
5. Check hash strings among TARGET and CLIENT servers - it has to be the same now. Hash strings, are these symlinks, you see linked to ca.pem files.
6. Restart Zimbra as zimbra user: $ zmcontrol restart
7. Test LDAP connection by means described above [Test for more information...]
If everything was done correct, your tests should be successfull, and you're DONE! If still you have errors in connection, first hand, check permissions on CLIENT server for ca.pem file - it has to be zimbra:zimbra -rw-r-----!
CLIENT machine may have ceveral .pem files, as mentioned before, but you have to get 1:1 TARGET and CLIENT machine hashes and ca.pem files - 100% match!
In addition, I'm not completely sure, but I think, that there's a way, just to symlink correct ca.pem file without runing -hash via openssl, but I did this way, as it runs a double check. if hashes are exact the same, after this command, I am sure, that this is correct ca.pem file
As well, there might be more advanced solutions to deploy these certs using zmcertmgr, as noted by JoBbZ on IRC, but I was ok with this manual method, so cannot commet on other options.
Run Zimbra upgrade
After corrections and tests, that ldap connection is successfull, run Zimbra upgrade as usual, without any additional switches. It should pass validation, and install upgrade without errors successfully.
Post upgrade tasks
As mentioned above, Zimbra upgrade script rebuilds certificates in last steps of installation/upgrade procedure, resulting again in wrong situation with ca.pem file. For this, I re-run correction steps mentioned above, once Zimbra upgrade completed.
NOTE: Those, who use this solution in single server install environment, please post your details, if they differ from this solution, especially ca.pem file locations and differences. This would help others to sleep a couple of nights more, in case they run into similar situation.
Please let me know, if any errors or unclear wording is present, to correct it.
