Error on deploying SSL certificates

Discuss your pilot or production implementation with other Zimbra admins or our engineers.
octet
Posts: 23
Joined: Thu Jan 06, 2022 2:35 pm

Error on deploying SSL certificates

Post by octet »

Hi guys,

Trying to install the LetsEncrypt SSL certificates, they're generated fine, but on deployment I got this error, any idea what could be please?

I'm using this script to generate them: https://wiki.zimbra.com/wiki/JDunphy-Letsencrypt

Code: Select all

[zimbra@mail tmp]$ /opt/zimbra/bin/zmcertmgr deploycrt comm mail.XXX.com.cer fullchain.cer
** Verifying 'mail.XXX.com.cer' against '/opt/zimbra/ssl/zimbra/commercial/commercial.key'
Certificate 'mail.XXX.com.cer' and private key '/opt/zimbra/ssl/zimbra/commercial/commercial.key' match.
** Verifying 'mail.XXX.com.cer' against 'fullchain.cer'
Valid certificate chain: mail.XXX.com.cer: OK
** Copying 'mail.XXX.com.cer' to '/opt/zimbra/ssl/zimbra/commercial/commercial.crt'
** Copying 'fullchain.cer' to '/opt/zimbra/ssl/zimbra/commercial/commercial_ca.crt'
** Appending ca chain 'fullchain.cer' to '/opt/zimbra/ssl/zimbra/commercial/commercial.crt'
** Importing cert '/opt/zimbra/ssl/zimbra/commercial/commercial_ca.crt' as 'zcs-user-commercial_ca' into cacerts '/opt/zimbra/common/lib/jvm/java/lib/security/cacerts'
** NOTE: restart mailboxd to use the imported certificate.
** Saving config key 'zimbraSSLCertificate' via zmprov modifyServer mail.XXX.com...failed (rc=1)
** Installing imapd certificate '/opt/zimbra/conf/imapd.crt' and key '/opt/zimbra/conf/imapd.key'
** Copying '/opt/zimbra/ssl/zimbra/commercial/commercial.crt' to '/opt/zimbra/conf/imapd.crt'
** Copying '/opt/zimbra/ssl/zimbra/commercial/commercial.key' to '/opt/zimbra/conf/imapd.key'
ERROR: imapd keytool(-delete -alias jetty) returned non-zero(1):
keytool error: java.security.cert.CertificateParsingException: signed overrun, bytes = 88
** Creating file '/opt/zimbra/ssl/zimbra/jetty.pkcs12'
** Creating keystore '/opt/zimbra/conf/imapd.keystore'
ERROR: com.zimbra.cert.MyPKCS12Import to '/opt/zimbra/ssl/zimbra/jetty.pkcs12' returned non-zero(1):
Exception in thread "main" java.security.cert.CertificateParsingException: signed overrun, bytes = 88
	at java.base/sun.security.x509.X509CertImpl.parse(X509CertImpl.java:1820)
	at java.base/sun.security.x509.X509CertImpl.<init>(X509CertImpl.java:188)
	at java.base/sun.security.provider.X509Factory.engineGenerateCertificate(X509Factory.java:105)
	at java.base/java.security.cert.CertificateFactory.generateCertificate(CertificateFactory.java:355)
	at java.base/sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:725)
	at java.base/sun.security.util.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:222)
	at java.base/java.security.KeyStore.load(KeyStore.java:1472)
	at com.zimbra.cert.MyPKCS12Import.main(MyPKCS12Import.java:104)
Tried everything in here: https://wiki.zimbra.com/wiki/Certificate_errors

Thanks a lot for your help.
User avatar
JDunphy
Outstanding Member
Outstanding Member
Posts: 889
Joined: Fri Sep 12, 2014 11:18 pm
Location: Victoria, BC
ZCS/ZD Version: 9.0.0_P39 NETWORK Edition

Re: Error on deploying SSL certificates

Post by JDunphy »

Certificate is fine but you have a permission problem. See if this helps.

viewtopic.php?f=15&t=70263

Note: zmcertmgr runs as zimbra so if you did something previously as root that might explain the problem.

Jim
octet
Posts: 23
Joined: Thu Jan 06, 2022 2:35 pm

Re: Error on deploying SSL certificates

Post by octet »

JDunphy wrote:Certificate is fine but you have a permission problem. See if this helps.

viewtopic.php?f=15&t=70263

Note: zmcertmgr runs as zimbra so if you did something previously as root that might explain the problem.

Jim
Hi Jim,

It was run as zimbra user, permissions seem fine:

Code: Select all

[zimbra@mail ~]$ ls -ld /opt/zimbra/ssl/zimbra/jetty.pkcs12 /opt/zimbra/ssl/zimbra
drwxr-x--- 5 zimbra zimbra   68 Jan  6 19:32 /opt/zimbra/ssl/zimbra
-rw------- 1 zimbra zimbra 2496 Jan  6 19:32 /opt/zimbra/ssl/zimbra/jetty.pkcs12
Noticed ldap doesn't start. I've probably messed up the host / confs as I've tried using a different host and now I can't change it back.

Code: Select all

[zimbra@mail ~]$ /opt/zimbra/libexec/zmsetservername -f -n mail.XXX.com
Getting local config zimbra_server_hostname=mail.XXX.com
Hostname is already mail.XXX.com.
Getting local config zimbra_ldap_userdn=uid=zimbra,cn=admins,cn=zimbra
Getting local config zimbra_ldap_password=XXXX
Getting local config ldap_is_master=true
Getting local config ldap_url=ldap://mail.XXXX.com:389
Getting local config ldap_master_url=ldap://mail.XXXX.com:389
Getting local config ldap_starttls_supported=0
Starting ldap...failed.
I've tried using web-mail.XXXX.com instead of mail.XXXX.com.

host file looks fine:

Code: Select all

[zimbra@mail ~]$ cat /etc/hosts
127.0.0.1       localhost.localdomain   localhost
x.x.x.x	mail.XXX.com	mail
x.x.x.x	web-mail.XXX.com	web-mail
User avatar
JDunphy
Outstanding Member
Outstanding Member
Posts: 889
Joined: Fri Sep 12, 2014 11:18 pm
Location: Victoria, BC
ZCS/ZD Version: 9.0.0_P39 NETWORK Edition

Re: Error on deploying SSL certificates

Post by JDunphy »

Interesting... can you provide some basic information.

1) version of zimbra you are running
2) which names you generated your certificate for (ie. -d mail.XXXX.com -d web-mail.XXXX.com etc
3) some history of this install... was this working previous and what/how did you generate your certificate... ie) certbot, commercial cert, admin interface, self-signed, etc. nothing fancy but general background vs... never had it working and this is my first time installing a certificate.

I have not seen this error myself but zmcertmgr isn't happy with deploying the certificate when copying and updating various locations with the cert it would appear. It was happy with verifying your certificate but depending on what happened prior to your successful generation of the certificate that might be a clue. zmcertmgr (perl script) has to install the certificate for nginx, mailboxd (java), ldap, and postfix. On that wiki page of mine you listed, I have my tripwire listing of all the files and location that zimbra touches when it installs the certificate. see section Notes to verify permissions/ownership and everything looks sane.

Here is the output when ldap isn't running and then again when it is when attempting to install a certificate. There are no other errors like what you show so that makes me think it might be ldap related thing if everything else checks out. Hopefully the output below can provide a clue of what you should expect.

Code: Select all

# su - zimbra
% zmcontrol stop
% cd .acme.sh
% ./acme.sh --deploy --deploy-hook zimbra -d mail.example.com
** Verifying '/opt/zimbra/.acme.sh/mail.example.com/mail.example.com.cer' against '/opt/zimbra/.acme.sh/mail.example.com/mail.example.com.key'
Certificate '/opt/zimbra/.acme.sh/mail.example.com/mail.example.com.cer' and private key '/opt/zimbra/.acme.sh/mail.example.com/mail.example.com.key' match.
** Verifying '/opt/zimbra/.acme.sh/mail.example.com/mail.example.com.cer' against '/opt/zimbra/.acme.sh/mail.example.com/ca.cer.real'
Valid certificate chain: /opt/zimbra/.acme.sh/mail.example.com/mail.example.com.cer: OK
** Verifying '/opt/zimbra/.acme.sh/mail.example.com/mail.example.com.cer' against '/opt/zimbra/ssl/zimbra/commercial/commercial.key'
Certificate '/opt/zimbra/.acme.sh/mail.example.com/mail.example.com.cer' and private key '/opt/zimbra/ssl/zimbra/commercial/commercial.key' match.
** Verifying '/opt/zimbra/.acme.sh/mail.example.com/mail.example.com.cer' against '/opt/zimbra/.acme.sh/mail.example.com/ca.cer.real'
Valid certificate chain: /opt/zimbra/.acme.sh/mail.example.com/mail.example.com.cer: OK
** Copying '/opt/zimbra/.acme.sh/mail.example.com/mail.example.com.cer' to '/opt/zimbra/ssl/zimbra/commercial/commercial.crt'
** Copying '/opt/zimbra/.acme.sh/mail.example.com/ca.cer.real' to '/opt/zimbra/ssl/zimbra/commercial/commercial_ca.crt'
...
...
...
** Copying '/opt/zimbra/ssl/zimbra/ca/ca.key' to '/opt/zimbra/conf/ca/ca.key'
** Copying '/opt/zimbra/ssl/zimbra/ca/ca.pem' to '/opt/zimbra/conf/ca/ca.pem'
** Creating CA hash symlink '4866e35b.0' -> 'ca.pem'
** Creating /opt/zimbra/conf/ca/commercial_ca_1.crt
** Creating CA hash symlink 'f85883ac.0' -> 'commercial_ca_1.crt'
** Creating /opt/zimbra/conf/ca/commercial_ca_2.crt
** Creating CA hash symlink '8d33f237.0' -> 'commercial_ca_2.crt'
** Creating /opt/zimbra/conf/ca/commercial_ca_3.crt
** Creating CA hash symlink '4042bcee.0' -> 'commercial_ca_3.crt'
[Thu Jan  6 13:59:24 PST 2022] Error deploy for domain:mail.example.com
[Thu Jan  6 13:59:24 PST 2022] Deploy error.

% ../bin/ldap start
Started slapd: pid 4112840
% ./acme.sh --deploy --deploy-hook zimbra -d mail.example.com
** Verifying '/opt/zimbra/.acme.sh/mail.example.com/mail.example.com.cer' against '/opt/zimbra/.acme.sh/mail.example.com/mail.example.com.key'
Certificate '/opt/zimbra/.acme.sh/mail.example.com/mail.example.com.cer' and private key '/opt/zimbra/.acme.sh/mail.example.com/mail.example.com.key' match.
** Verifying '/opt/zimbra/.acme.sh/mail.example.com/mail.example.com.cer' against '/opt/zimbra/.acme.sh/mail.example.com/ca.cer.real'
Valid certificate chain: /opt/zimbra/.acme.sh/mail.example.com/mail.example.com.cer: OK
** Verifying '/opt/zimbra/.acme.sh/mail.example.com/mail.example.com.cer' against '/opt/zimbra/ssl/zimbra/commercial/commercial.key'
Certificate '/opt/zimbra/.acme.sh/mail.example.com/mail.example.com.cer' and private key '/opt/zimbra/ssl/zimbra/commercial/commercial.key' match.
** Verifying '/opt/zimbra/.acme.sh/mail.example.com/mail.example.com.cer' against '/opt/zimbra/.acme.sh/mail.example.com/ca.cer.real'
Valid certificate chain: /opt/zimbra/.acme.sh/mail.example.com/mail.example.com.cer: OK
** Copying '/opt/zimbra/.acme.sh/mail.example.com/mail.example.com.cer' to '/opt/zimbra/ssl/zimbra/commercial/commercial.crt'
** Copying '/opt/zimbra/.acme.sh/mail.example.com/ca.cer.real' to '/opt/zimbra/ssl/zimbra/commercial/commercial_ca.crt'
** Appending ca chain '/opt/zimbra/.acme.sh/mail.example.com/ca.cer.real' to '/opt/zimbra/ssl/zimbra/commercial/commercial.crt'
** Importing cert '/opt/zimbra/ssl/zimbra/commercial/commercial_ca.crt' as 'zcs-user-commercial_ca' into cacerts '/opt/zimbra/common/lib/jvm/java/lib/security/cacerts'
** NOTE: restart mailboxd to use the imported certificate.
** Saving config key 'zimbraSSLCertificate' via zmprov modifyServer mail.example.com...ok
** Saving config key 'zimbraSSLPrivateKey' via zmprov modifyServer mail.example.com...ok
** Installing imapd certificate '/opt/zimbra/conf/imapd.crt' and key '/opt/zimbra/conf/imapd.key'
** Copying '/opt/zimbra/ssl/zimbra/commercial/commercial.crt' to '/opt/zimbra/conf/imapd.crt'
** Copying '/opt/zimbra/ssl/zimbra/commercial/commercial.key' to '/opt/zimbra/conf/imapd.key'
** Creating file '/opt/zimbra/ssl/zimbra/jetty.pkcs12'
** Creating keystore '/opt/zimbra/conf/imapd.keystore'
** Installing ldap certificate '/opt/zimbra/conf/slapd.crt' and key '/opt/zimbra/conf/slapd.key'
** Copying '/opt/zimbra/ssl/zimbra/commercial/commercial.crt' to '/opt/zimbra/conf/slapd.crt'
** Copying '/opt/zimbra/ssl/zimbra/commercial/commercial.key' to '/opt/zimbra/conf/slapd.key'
** Creating file '/opt/zimbra/ssl/zimbra/jetty.pkcs12'
** Creating keystore '/opt/zimbra/mailboxd/etc/keystore'
** Installing mta certificate '/opt/zimbra/conf/smtpd.crt' and key '/opt/zimbra/conf/smtpd.key'
** Copying '/opt/zimbra/ssl/zimbra/commercial/commercial.crt' to '/opt/zimbra/conf/smtpd.crt'
** Copying '/opt/zimbra/ssl/zimbra/commercial/commercial.key' to '/opt/zimbra/conf/smtpd.key'
** Installing proxy certificate '/opt/zimbra/conf/nginx.crt' and key '/opt/zimbra/conf/nginx.key'
** Copying '/opt/zimbra/ssl/zimbra/commercial/commercial.crt' to '/opt/zimbra/conf/nginx.crt'
** Copying '/opt/zimbra/ssl/zimbra/commercial/commercial.key' to '/opt/zimbra/conf/nginx.key'
** NOTE: restart services to use the new certificates.
** Cleaning up 9 files from '/opt/zimbra/conf/ca'
** Removing /opt/zimbra/conf/ca/commercial_ca_2.crt
** Removing /opt/zimbra/conf/ca/f85883ac.0
** Removing /opt/zimbra/conf/ca/ca.key
** Removing /opt/zimbra/conf/ca/8d33f237.0
** Removing /opt/zimbra/conf/ca/commercial_ca_3.crt
** Removing /opt/zimbra/conf/ca/4042bcee.0
** Removing /opt/zimbra/conf/ca/ca.pem
** Removing /opt/zimbra/conf/ca/commercial_ca_1.crt
** Removing /opt/zimbra/conf/ca/4866e35b.0
** Copying CA to /opt/zimbra/conf/ca
** Copying '/opt/zimbra/ssl/zimbra/ca/ca.key' to '/opt/zimbra/conf/ca/ca.key'
** Copying '/opt/zimbra/ssl/zimbra/ca/ca.pem' to '/opt/zimbra/conf/ca/ca.pem'
** Creating CA hash symlink '4866e35b.0' -> 'ca.pem'
** Creating /opt/zimbra/conf/ca/commercial_ca_1.crt
** Creating CA hash symlink 'f85883ac.0' -> 'commercial_ca_1.crt'
** Creating /opt/zimbra/conf/ca/commercial_ca_2.crt
** Creating CA hash symlink '8d33f237.0' -> 'commercial_ca_2.crt'
** Creating /opt/zimbra/conf/ca/commercial_ca_3.crt
** Creating CA hash symlink '4042bcee.0' -> 'commercial_ca_3.crt'
...
server stops and starts at this point. (zmcontrol restart)
Jim
octet
Posts: 23
Joined: Thu Jan 06, 2022 2:35 pm

Re: Error on deploying SSL certificates

Post by octet »

Hi Jim,

My answers below:

1. Zimbra version

Code: Select all

[zimbra@mail ~]$ zmcontrol -v    
Release 8.8.15_GA_3953.RHEL8_64_20200629025823 RHEL8_64 FOSS edition, Patch 8.8.15_P29.
2. The certificates were generated like this:

Code: Select all

./acme.sh --issue --force --dns dns_aws --log dns_cf -d mail.XXX.com
3. It's an old install, about 1-2 years old, was working fine until recently when the certificates generated with LE certbot expired and the renewal script didn't work anymore, we were getting:

Code: Select all

** Verifying 'cert.pem' against '/opt/zimbra/ssl/zimbra/commercial/commercial.key'
Certificate 'cert.pem' and private key '/opt/zimbra/ssl/zimbra/commercial/commercial.key' match.
** Verifying 'cert.pem' against 'chain.pem'
ERROR: Unable to validate certificate chain: cert.pem: O = Digital Signature Trust Co., CN = DST Root CA X3
error 10 at 3 depth lookup:certificate has expired
and eventually:

Code: Select all

An unexpected error occurred:
There were too many requests of a given type :: Error creating new order :: too many certificates (5) already issued for this exact set of domains in the last 168 hours: mail.XXX.com: see https://letsencrypt.org/docs/rate-limits/
Please see the logfiles in /var/log/letsencrypt for more details
Then my colleague tried using a new host, web-mail.XXX.com and he was getting the same errors.

Eventually I downloaded your script and it seems the certificates were generated fine, they're OK when verified, but they can't be deployed, ldap crashes too.

All permissions seems to be fine zimbra.zimbra and 640 with 750 on some folders. Also ran this several times:

Code: Select all

[root@mail ssl]# /opt/zimbra/libexec/zmfixperms -verbose
Fixing ownership and permissions on /opt/zimbra/conf
Fixing permissions on /opt/zimbra/conf/ca
Fixing permissions on /opt/zimbra/conf/ca/commercial_ca_1.crt
Fixing permissions on /opt/zimbra/conf/ca/commercial_ca_2.crt
Fixing permissions on /opt/zimbra/conf/ca/commercial_ca_3.crt
Fixing permissions on /opt/zimbra/conf/ca/ca.pem
Fixing ownership and permissions on /opt/zimbra/conf/ldap-canonical.cf
Fixing ownership and permissions on /opt/zimbra/conf/ldap-slm.cf
Fixing ownership and permissions on /opt/zimbra/conf/ldap-transport.cf
Fixing ownership and permissions on /opt/zimbra/conf/ldap-vad.cf
Fixing ownership and permissions on /opt/zimbra/conf/ldap-vam.cf
Fixing ownership and permissions on /opt/zimbra/conf/ldap-vmd.cf
Fixing ownership and permissions on /opt/zimbra/conf/ldap-vmm.cf
Fixing permissions and ownership on /opt/zimbra/conf/imapd.crt
Fixing permissions and ownership on /opt/zimbra/conf/nginx.crt
Fixing permissions and ownership on /opt/zimbra/conf/slapd.crt
Fixing permissions and ownership on /opt/zimbra/conf/smtpd.crt
Fixing permissions and ownership on /opt/zimbra/conf/imapd.key
Fixing permissions and ownership on /opt/zimbra/conf/nginx.key
Fixing permissions and ownership on /opt/zimbra/conf/slapd.key
Fixing permissions and ownership on /opt/zimbra/conf/smtpd.key
Fixing permissions and ownership on /opt/zimbra/conf/zmssl.cnf
Fixing ownership and permissions on /var/log/zimbra.log
Fixing ownership and permissions on /opt/zimbra/conf/crontabs
Fixing ownership and permissions on /opt/zimbra/common/lib/jylibs
Fixing ownership and permissions on /opt/zimbra/lib
Fixing ownership and permissions on /opt/zimbra/db
Fixing ownership and permissions on /opt/zimbra/data/sasl2/state
Fixing ownership and permissions on /opt/zimbra/data/amavisd
Fixing ownership and permissions on /opt/zimbra/jetty
Fixing ownership and permissions on /opt/zimbra/ssl
Fixing ownership and permissions on /opt/zimbra/data/ldap
Fixing ownership and permissions on /opt/zimbra/logger/db
Fixing ownership and permissions on /opt/zimbra/zmstat
Fixing postfix related permissions
Fixing ownership and permissions on /opt/zimbra/data/postfix
Ldap not starting:

Code: Select all

[zimbra@mail ~]$ ./bin/ldap start 
Failed to start slapd.  Attempting debug start to determine error.
61d80d68 daemon: bind(7) failed errno=99 (Cannot assign requested address)
61d80d68 slap_open_listener: failed on ldap://mail.XXX.com:389

[zimbra@mail ~]$ 
So annoying, considering reinstalling it from scratch, but I'm not that good with zimbra and have 0 knowledge on backing up the users, mailboxes and restoring them manually, considering current version on zimbra not starting.

Some more logs in case it helps:

Code: Select all

[zimbra@mail ~]$ zmcontrol restart
Host mail.XXX.com
	Stopping zmconfigd...Done.
	Stopping zimlet webapp...Failed.
Stopping mailboxd...mailboxd is not running.
Error: assertion '-d /opt/zimbra/log' failed


	Stopping zimbraAdmin webapp...Failed.
Stopping mailboxd...mailboxd is not running.
Error: assertion '-d /opt/zimbra/log' failed


	Stopping zimbra webapp...Failed.
Stopping mailboxd...mailboxd is not running.
Error: assertion '-d /opt/zimbra/log' failed


	Stopping service webapp...Failed.
Stopping mailboxd...mailboxd is not running.
Error: assertion '-d /opt/zimbra/log' failed


	Stopping stats...Done.
	Stopping mta...Done.
	Stopping spell...Done.
	Stopping snmp...Done.
	Stopping cbpolicyd...Done.
	Stopping archiving...Done.
	Stopping opendkim...Done.
	Stopping amavis...Done.
	Stopping antivirus...Done.
	Stopping antispam...Done.
	Stopping proxy...Done.
	Stopping memcached...Done.
	Stopping mailbox...Failed.
Stopping mailboxd...mailboxd is not running.
Error: assertion '-d /opt/zimbra/log' failed


	Stopping logger...Done.
	Stopping dnscache...Done.
	Stopping ldap...Done.
Host mail.mc3lab.com
	Starting ldap...Done.
Failed.
Failed to start slapd.  Attempting debug start to determine error.
61d8145c daemon: bind(7) failed errno=99 (Cannot assign requested address)
61d8145c slap_open_listener: failed on ldap://mail.XXX.com:389
and zimbra log:

Code: Select all

Jan  7 10:21:41 mail zimbramon[93012]: 93012:info: Stopping services initiated by zmcontrol
Jan  7 10:21:41 mail zimbramon[93012]: 93012:info: Stopping vmware-ha via zmcontrol
Jan  7 10:21:41 mail zimbramon[93012]: 93012:info: Stopping zmconfigd via zmcontrol
Jan  7 10:21:41 mail zimbramon[93012]: 93012:info: Stopping imapd via zmcontrol
Jan  7 10:21:41 mail zimbramon[93012]: 93012:info: Stopping zimlet via zmcontrol
Jan  7 10:21:42 mail zimbramon[93012]: 93012:info: Stopping zimbraAdmin via zmcontrol
Jan  7 10:21:42 mail zmmailboxdmgr[93081]: file /opt/zimbra/log/zmmailboxd_manager.pid does not exist
Jan  7 10:21:42 mail zmmailboxdmgr[93081]: assuming no other instance is running
Jan  7 10:21:42 mail zmmailboxdmgr[93081]: file /opt/zimbra/log/zmmailboxd.pid does not exist
Jan  7 10:21:42 mail zmmailboxdmgr[93081]: assuming no other instance is running
Jan  7 10:21:42 mail zmmailboxdmgr[93081]: no manager process is running
Jan  7 10:21:42 mail zimbramon[93012]: 93012:info: Stopping zimbra via zmcontrol
Jan  7 10:21:42 mail zimbramon[93012]: 93012:info: Stopping service via zmcontrol
Jan  7 10:21:42 mail zmmailboxdmgr[93108]: file /opt/zimbra/log/zmmailboxd_manager.pid does not exist
Jan  7 10:21:42 mail zmmailboxdmgr[93108]: assuming no other instance is running
Jan  7 10:21:42 mail zmmailboxdmgr[93108]: file /opt/zimbra/log/zmmailboxd.pid does not exist
Jan  7 10:21:42 mail zmmailboxdmgr[93108]: assuming no other instance is running
Jan  7 10:21:42 mail zmmailboxdmgr[93108]: no manager process is running
Jan  7 10:21:42 mail zmmailboxdmgr[93135]: file /opt/zimbra/log/zmmailboxd_manager.pid does not exist
Jan  7 10:21:42 mail zmmailboxdmgr[93135]: assuming no other instance is running
Jan  7 10:21:42 mail zmmailboxdmgr[93135]: file /opt/zimbra/log/zmmailboxd.pid does not exist
Jan  7 10:21:42 mail zmmailboxdmgr[93135]: assuming no other instance is running
Jan  7 10:21:42 mail zmmailboxdmgr[93135]: no manager process is running
Jan  7 10:21:42 mail zimbramon[93012]: 93012:info: Stopping stats via zmcontrol
Jan  7 10:21:42 mail zmmailboxdmgr[93162]: file /opt/zimbra/log/zmmailboxd_manager.pid does not exist
Jan  7 10:21:42 mail zmmailboxdmgr[93162]: assuming no other instance is running
Jan  7 10:21:42 mail zmmailboxdmgr[93162]: file /opt/zimbra/log/zmmailboxd.pid does not exist
Jan  7 10:21:42 mail zmmailboxdmgr[93162]: assuming no other instance is running
Jan  7 10:21:42 mail zmmailboxdmgr[93162]: no manager process is running
Jan  7 10:21:43 mail zimbramon[93012]: 93012:info: Stopping mta via zmcontrol
Jan  7 10:21:43 mail postfix/postalias[93225]: fatal: open database /etc/aliases.lmdb: MDB_PAGE_NOTFOUND: Requested page not found
Jan  7 10:21:44 mail postfix/postfix-script[93234]: fatal: the Postfix mail system is not running
Jan  7 10:21:45 mail zimbramon[93012]: 93012:info: Stopping spell via zmcontrol
Jan  7 10:21:45 mail zimbramon[93012]: 93012:info: Stopping snmp via zmcontrol
Jan  7 10:21:45 mail zimbramon[93012]: 93012:info: Stopping cbpolicyd via zmcontrol
Jan  7 10:21:45 mail zimbramon[93012]: 93012:info: Stopping archiving via zmcontrol
Jan  7 10:21:45 mail zimbramon[93012]: 93012:info: Stopping opendkim via zmcontrol
Jan  7 10:21:45 mail zimbramon[93012]: 93012:info: Stopping amavis via zmcontrol
Jan  7 10:21:45 mail zimbramon[93012]: 93012:info: Stopping antivirus via zmcontrol
Jan  7 10:21:45 mail zimbramon[93012]: 93012:info: Stopping antispam via zmcontrol
Jan  7 10:21:48 mail zimbramon[93012]: 93012:info: Stopping proxy via zmcontrol
Jan  7 10:21:48 mail zimbramon[93012]: 93012:info: Stopping memcached via zmcontrol
Jan  7 10:21:48 mail zimbramon[93012]: 93012:info: Stopping mailbox via zmcontrol
Jan  7 10:21:49 mail zimbramon[93012]: 93012:info: Stopping convertd via zmcontrol
Jan  7 10:21:49 mail zimbramon[93012]: 93012:info: Stopping logger via zmcontrol
Jan  7 10:21:49 mail zimbramon[93012]: 93012:info: Stopping dnscache via zmcontrol
Jan  7 10:21:49 mail zimbramon[93012]: 93012:info: Stopping ldap via zmcontrol
Jan  7 10:21:49 mail zimbramon[93012]: 93012:info: Starting services initiated by zmcontrol
Jan  7 10:21:49 mail zmmailboxdmgr[93512]: file /opt/zimbra/log/zmmailboxd_manager.pid does not exist
Jan  7 10:21:49 mail zmmailboxdmgr[93512]: assuming no other instance is running
Jan  7 10:21:49 mail zmmailboxdmgr[93512]: file /opt/zimbra/log/zmmailboxd.pid does not exist
Jan  7 10:21:49 mail zmmailboxdmgr[93512]: assuming no other instance is running
Jan  7 10:21:49 mail zmmailboxdmgr[93512]: no manager process is running
Jan  7 10:21:49 mail slapd[93543]: @(#) $OpenLDAP: slapd 2.4.59 (Sep 16 2021 16:28:51) $#012#011build@c887:/home/build/git/88/packages/thirdparty/openldap/build/RHEL8_64/zimbra-openldap/rpm/BUILD/openldap-2.4.59/servers/slapd
Jan  7 10:21:49 mail slapd[93543]: daemon: bind(7) failed errno=99 (Cannot assign requested address)
Jan  7 10:21:49 mail slapd[93543]: slapd stopped.
Jan  7 10:21:49 mail slapd[93543]: connections_destroy: nothing to destroy.
Jan  7 10:21:54 mail slapd[93550]: @(#) $OpenLDAP: slapd 2.4.59 (Sep 16 2021 16:28:51) $#012#011build@c887:/home/build/git/88/packages/thirdparty/openldap/build/RHEL8_64/zimbra-openldap/rpm/BUILD/openldap-2.4.59/servers/slapd
Jan  7 10:21:54 mail slapd[93550]: daemon: bind(7) failed errno=99 (Cannot assign requested address)
Jan  7 10:21:54 mail slapd[93550]: slapd stopped.
Jan  7 10:21:54 mail slapd[93550]: connections_destroy: nothing to destroy.
Jan  7 10:21:59 mail slapd[93557]: @(#) $OpenLDAP: slapd 2.4.59 (Sep 16 2021 16:28:51) $#012#011build@c887:/home/build/git/88/packages/thirdparty/openldap/build/RHEL8_64/zimbra-openldap/rpm/BUILD/openldap-2.4.59/servers/slapd
Jan  7 10:21:59 mail slapd[93557]: daemon: bind(7) failed errno=99 (Cannot assign requested address)
Jan  7 10:21:59 mail slapd[93557]: slapd stopped.
Jan  7 10:21:59 mail slapd[93557]: connections_destroy: nothing to destroy.
Jan  7 10:22:04 mail slapd[93631]: @(#) $OpenLDAP: slapd 2.4.59 (Sep 16 2021 16:28:51) $#012#011build@c887:/home/build/git/88/packages/thirdparty/openldap/build/RHEL8_64/zimbra-openldap/rpm/BUILD/openldap-2.4.59/servers/slapd
Jan  7 10:22:04 mail slapd[93631]: daemon: bind(7) failed errno=99 (Cannot assign requested address)
Jan  7 10:22:04 mail slapd[93631]: slapd stopped.
Jan  7 10:22:04 mail slapd[93631]: connections_destroy: nothing to destroy.
Jan  7 10:22:09 mail slapd[93638]: @(#) $OpenLDAP: slapd 2.4.59 (Sep 16 2021 16:28:51) $#012#011build@c887:/home/build/git/88/packages/thirdparty/openldap/build/RHEL8_64/zimbra-openldap/rpm/BUILD/openldap-2.4.59/servers/slapd
Jan  7 10:22:09 mail slapd[93638]: daemon: bind(7) failed errno=99 (Cannot assign requested address)
Jan  7 10:22:09 mail slapd[93638]: slapd stopped.
Jan  7 10:22:09 mail slapd[93638]: connections_destroy: nothing to destroy.
Jan  7 10:22:14 mail slapd[93654]: @(#) $OpenLDAP: slapd 2.4.59 (Sep 16 2021 16:28:51) $#012#011build@c887:/home/build/git/88/packages/thirdparty/openldap/build/RHEL8_64/zimbra-openldap/rpm/BUILD/openldap-2.4.59/servers/slapd
Jan  7 10:22:14 mail slapd[93654]: daemon: bind(7) failed errno=99 (Cannot assign requested address)
Jan  7 10:22:14 mail slapd[93654]: slapd stopped.
Jan  7 10:22:14 mail slapd[93654]: connections_destroy: nothing to destroy.
Jan  7 10:22:19 mail slapd[93662]: @(#) $OpenLDAP: slapd 2.4.59 (Sep 16 2021 16:28:51) $#012#011build@c887:/home/build/git/88/packages/thirdparty/openldap/build/RHEL8_64/zimbra-openldap/rpm/BUILD/openldap-2.4.59/servers/slapd
Jan  7 10:22:19 mail slapd[93662]: daemon: bind(7) failed errno=99 (Cannot assign requested address)
Jan  7 10:22:19 mail slapd[93662]: slapd stopped.
Jan  7 10:22:19 mail slapd[93662]: connections_destroy: nothing to destroy.
Jan  7 10:22:20 mail slapd[93668]: @(#) $OpenLDAP: slapd 2.4.59 (Sep 16 2021 16:28:51) $#012#011build@c887:/home/build/git/88/packages/thirdparty/openldap/build/RHEL8_64/zimbra-openldap/rpm/BUILD/openldap-2.4.59/servers/slapd
Jan  7 10:22:20 mail slapd[93668]: daemon: bind(7) failed errno=99 (Cannot assign requested address)
Jan  7 10:22:20 mail slapd[93668]: slapd stopped.
Jan  7 10:22:20 mail slapd[93668]: connections_destroy: nothing to destroy.
Jan  7 10:30:02 mail zimbramon[93942]: 93942:info: 2022-01-07 10:30:01, QUEUE: 0 0
Jan  7 10:30:02 mail postfix/postqueue[93958]: fatal: Queue report unavailable - mail system is down
Jan  7 10:30:02 mail postfix/postdrop[93991]: warning: unable to look up public/pickup: No such file or directory
I restored the /opt/zimbra/log directory and the new logs are now:

Code: Select all

[zimbra@mail ~]$ zmcontrol restart
Host mail.XXX.com
	Stopping zmconfigd...Done.
	Stopping zimlet webapp...Done.
	Stopping zimbraAdmin webapp...Done.
	Stopping zimbra webapp...Done.
	Stopping service webapp...Done.
	Stopping stats...Done.
	Stopping mta...Done.
	Stopping spell...Done.
	Stopping snmp...Done.
	Stopping cbpolicyd...Done.
	Stopping archiving...Done.
	Stopping opendkim...Done.
	Stopping amavis...Done.
	Stopping antivirus...Done.
	Stopping antispam...Done.
	Stopping proxy...Done.
	Stopping memcached...Done.
	Stopping mailbox...Done.
	Stopping logger...Done.
	Stopping dnscache...Done.
	Stopping ldap...Done.
Host mail.XXX.com
	Starting ldap...Done.
Failed.
Failed to start slapd.  Attempting debug start to determine error.
61d81cbf daemon: bind(7) failed errno=99 (Cannot assign requested address)
61d81cbf slap_open_listener: failed on ldap://mail.XXX.com:389
and

Code: Select all

Jan  7 10:57:28 mail zimbramon[95373]: 95373:info: Stopping services initiated by zmcontrol
Jan  7 10:57:28 mail zimbramon[95373]: 95373:info: Stopping vmware-ha via zmcontrol
Jan  7 10:57:28 mail zimbramon[95373]: 95373:info: Stopping zmconfigd via zmcontrol
Jan  7 10:57:28 mail zimbramon[95373]: 95373:info: Stopping imapd via zmcontrol
Jan  7 10:57:28 mail zimbramon[95373]: 95373:info: Stopping zimlet via zmcontrol
Jan  7 10:57:28 mail zimbramon[95373]: 95373:info: Stopping zimbraAdmin via zmcontrol
Jan  7 10:57:28 mail zmmailboxdmgr[95442]: stale pid 8484 found in /opt/zimbra/log/zmmailboxd_manager.pid: No such process
Jan  7 10:57:28 mail zmmailboxdmgr[95442]: assuming no other instance is running
Jan  7 10:57:28 mail zmmailboxdmgr[95442]: file /opt/zimbra/log/zmmailboxd.pid does not exist
Jan  7 10:57:28 mail zmmailboxdmgr[95442]: assuming no other instance is running
Jan  7 10:57:28 mail zmmailboxdmgr[95442]: no manager process is running
Jan  7 10:57:29 mail zimbramon[95373]: 95373:info: Stopping zimbra via zmcontrol
Jan  7 10:57:29 mail zimbramon[95373]: 95373:info: Stopping service via zmcontrol
Jan  7 10:57:29 mail zimbramon[95373]: 95373:info: Stopping stats via zmcontrol
Jan  7 10:57:28 mail zmmailboxdmgr[95469]: stale pid 8484 found in /opt/zimbra/log/zmmailboxd_manager.pid: No such process
Jan  7 10:57:28 mail zmmailboxdmgr[95469]: assuming no other instance is running
Jan  7 10:57:28 mail zmmailboxdmgr[95469]: file /opt/zimbra/log/zmmailboxd.pid does not exist
Jan  7 10:57:28 mail zmmailboxdmgr[95469]: assuming no other instance is running
Jan  7 10:57:28 mail zmmailboxdmgr[95469]: no manager process is running
Jan  7 10:57:29 mail zmmailboxdmgr[95496]: stale pid 8484 found in /opt/zimbra/log/zmmailboxd_manager.pid: No such process
Jan  7 10:57:29 mail zmmailboxdmgr[95496]: assuming no other instance is running
Jan  7 10:57:29 mail zmmailboxdmgr[95496]: file /opt/zimbra/log/zmmailboxd.pid does not exist
Jan  7 10:57:29 mail zmmailboxdmgr[95496]: assuming no other instance is running
Jan  7 10:57:29 mail zmmailboxdmgr[95496]: no manager process is running
Jan  7 10:57:29 mail zmmailboxdmgr[95523]: stale pid 8484 found in /opt/zimbra/log/zmmailboxd_manager.pid: No such process
Jan  7 10:57:29 mail zmmailboxdmgr[95523]: assuming no other instance is running
Jan  7 10:57:29 mail zmmailboxdmgr[95523]: file /opt/zimbra/log/zmmailboxd.pid does not exist
Jan  7 10:57:29 mail zmmailboxdmgr[95523]: assuming no other instance is running
Jan  7 10:57:29 mail zmmailboxdmgr[95523]: no manager process is running
Jan  7 10:57:30 mail zimbramon[95373]: 95373:info: Stopping mta via zmcontrol
Jan  7 10:57:30 mail postfix/postalias[95586]: fatal: open database /etc/aliases.lmdb: MDB_PAGE_NOTFOUND: Requested page not found
Jan  7 10:57:31 mail postfix/postfix-script[95595]: fatal: the Postfix mail system is not running
Jan  7 10:57:32 mail zimbramon[95373]: 95373:info: Stopping spell via zmcontrol
Jan  7 10:57:32 mail zimbramon[95373]: 95373:info: Stopping snmp via zmcontrol
Jan  7 10:57:32 mail zimbramon[95373]: 95373:info: Stopping cbpolicyd via zmcontrol
Jan  7 10:57:32 mail zimbramon[95373]: 95373:info: Stopping archiving via zmcontrol
Jan  7 10:57:32 mail zimbramon[95373]: 95373:info: Stopping opendkim via zmcontrol
Jan  7 10:57:32 mail zimbramon[95373]: 95373:info: Stopping amavis via zmcontrol
Jan  7 10:57:32 mail zimbramon[95373]: 95373:info: Stopping antivirus via zmcontrol
Jan  7 10:57:32 mail zimbramon[95373]: 95373:info: Stopping antispam via zmcontrol
Jan  7 10:57:35 mail zimbramon[95373]: 95373:info: Stopping proxy via zmcontrol
Jan  7 10:57:35 mail zimbramon[95373]: 95373:info: Stopping memcached via zmcontrol
Jan  7 10:57:35 mail zimbramon[95373]: 95373:info: Stopping mailbox via zmcontrol
Jan  7 10:57:35 mail zimbramon[95373]: 95373:info: Stopping convertd via zmcontrol
Jan  7 10:57:35 mail zimbramon[95373]: 95373:info: Stopping logger via zmcontrol
Jan  7 10:57:36 mail zimbramon[95373]: 95373:info: Stopping dnscache via zmcontrol
Jan  7 10:57:36 mail zimbramon[95373]: 95373:info: Stopping ldap via zmcontrol
Jan  7 10:57:36 mail zimbramon[95373]: 95373:info: Starting services initiated by zmcontrol
Jan  7 10:57:35 mail zmmailboxdmgr[95873]: stale pid 8484 found in /opt/zimbra/log/zmmailboxd_manager.pid: No such process
Jan  7 10:57:35 mail zmmailboxdmgr[95873]: assuming no other instance is running
Jan  7 10:57:35 mail zmmailboxdmgr[95873]: file /opt/zimbra/log/zmmailboxd.pid does not exist
Jan  7 10:57:35 mail zmmailboxdmgr[95873]: assuming no other instance is running
Jan  7 10:57:35 mail zmmailboxdmgr[95873]: no manager process is running
Jan  7 10:57:36 mail slapd[95904]: @(#) $OpenLDAP: slapd 2.4.59 (Sep 16 2021 16:28:51) $#012#011build@c887:/home/build/git/88/packages/thirdparty/openldap/build/RHEL8_64/zimbra-openldap/rpm/BUILD/openldap-2.4.59/servers/slapd
Jan  7 10:57:36 mail slapd[95904]: daemon: bind(7) failed errno=99 (Cannot assign requested address)
Jan  7 10:57:36 mail slapd[95904]: slapd stopped.
Jan  7 10:57:36 mail slapd[95904]: connections_destroy: nothing to destroy.
Jan  7 10:57:41 mail slapd[95911]: @(#) $OpenLDAP: slapd 2.4.59 (Sep 16 2021 16:28:51) $#012#011build@c887:/home/build/git/88/packages/thirdparty/openldap/build/RHEL8_64/zimbra-openldap/rpm/BUILD/openldap-2.4.59/servers/slapd
Jan  7 10:57:41 mail slapd[95911]: daemon: bind(7) failed errno=99 (Cannot assign requested address)
Jan  7 10:57:41 mail slapd[95911]: slapd stopped.
Jan  7 10:57:41 mail slapd[95911]: connections_destroy: nothing to destroy.
Jan  7 10:57:46 mail slapd[95918]: @(#) $OpenLDAP: slapd 2.4.59 (Sep 16 2021 16:28:51) $#012#011build@c887:/home/build/git/88/packages/thirdparty/openldap/build/RHEL8_64/zimbra-openldap/rpm/BUILD/openldap-2.4.59/servers/slapd
Jan  7 10:57:46 mail slapd[95918]: daemon: bind(7) failed errno=99 (Cannot assign requested address)
Jan  7 10:57:46 mail slapd[95918]: slapd stopped.
Jan  7 10:57:46 mail slapd[95918]: connections_destroy: nothing to destroy.
Jan  7 10:57:51 mail slapd[95925]: @(#) $OpenLDAP: slapd 2.4.59 (Sep 16 2021 16:28:51) $#012#011build@c887:/home/build/git/88/packages/thirdparty/openldap/build/RHEL8_64/zimbra-openldap/rpm/BUILD/openldap-2.4.59/servers/slapd
Jan  7 10:57:51 mail slapd[95925]: daemon: bind(7) failed errno=99 (Cannot assign requested address)
Jan  7 10:57:51 mail slapd[95925]: slapd stopped.
Jan  7 10:57:51 mail slapd[95925]: connections_destroy: nothing to destroy.
Jan  7 10:57:56 mail slapd[95932]: @(#) $OpenLDAP: slapd 2.4.59 (Sep 16 2021 16:28:51) $#012#011build@c887:/home/build/git/88/packages/thirdparty/openldap/build/RHEL8_64/zimbra-openldap/rpm/BUILD/openldap-2.4.59/servers/slapd
Jan  7 10:57:56 mail slapd[95932]: daemon: bind(7) failed errno=99 (Cannot assign requested address)
Jan  7 10:57:56 mail slapd[95932]: slapd stopped.
Jan  7 10:57:56 mail slapd[95932]: connections_destroy: nothing to destroy.
Jan  7 10:58:01 mail slapd[95939]: @(#) $OpenLDAP: slapd 2.4.59 (Sep 16 2021 16:28:51) $#012#011build@c887:/home/build/git/88/packages/thirdparty/openldap/build/RHEL8_64/zimbra-openldap/rpm/BUILD/openldap-2.4.59/servers/slapd
Jan  7 10:58:01 mail slapd[95939]: daemon: bind(7) failed errno=99 (Cannot assign requested address)
Jan  7 10:58:01 mail slapd[95939]: slapd stopped.
Jan  7 10:58:01 mail slapd[95939]: connections_destroy: nothing to destroy.
Jan  7 10:58:06 mail slapd[96013]: @(#) $OpenLDAP: slapd 2.4.59 (Sep 16 2021 16:28:51) $#012#011build@c887:/home/build/git/88/packages/thirdparty/openldap/build/RHEL8_64/zimbra-openldap/rpm/BUILD/openldap-2.4.59/servers/slapd
Jan  7 10:58:06 mail slapd[96013]: daemon: bind(7) failed errno=99 (Cannot assign requested address)
Jan  7 10:58:06 mail slapd[96013]: slapd stopped.
Jan  7 10:58:06 mail slapd[96013]: connections_destroy: nothing to destroy.
Jan  7 10:58:07 mail slapd[96019]: @(#) $OpenLDAP: slapd 2.4.59 (Sep 16 2021 16:28:51) $#012#011build@c887:/home/build/git/88/packages/thirdparty/openldap/build/RHEL8_64/zimbra-openldap/rpm/BUILD/openldap-2.4.59/servers/slapd
Jan  7 10:58:07 mail slapd[96019]: daemon: bind(7) failed errno=99 (Cannot assign requested address)
Jan  7 10:58:07 mail slapd[96019]: slapd stopped.
Jan  7 10:58:07 mail slapd[96019]: connections_destroy: nothing to destroy.
Jan  7 11:00:01 mail postfix/postqueue[96070]: fatal: Queue report unavailable - mail system is down
Jan  7 11:00:01 mail zimbramon[96052]: 96052:info: 2022-01-07 11:00:01, QUEUE: 0 0
Jan  7 11:00:01 mail postfix/postdrop[96094]: warning: unable to look up public/pickup: No such file or directory
Is there a manual way to get access to mysql DB embedded in zimbra and modify the tables with the host, I've manually changed all the configs /opt/zimbra/conf/ from the web-mail.XXX.com to mail.XXX.com, the host change script still doesn't work. I'm thinking that's the reason the ldap doesn't start, might still have the old hostname in the DB.

Code: Select all

[zimbra@mail ~]$ /opt/zimbra/libexec/zmsetservername -f -n mail.XXX.com
Getting local config zimbra_server_hostname= mail.XXX.com
Hostname is already mail.XXX.com.
Getting local config zimbra_ldap_userdn=uid=zimbra,cn=admins,cn=zimbra
Getting local config zimbra_ldap_password=XXX
Getting local config ldap_is_master=true
Getting local config ldap_url=ldap://mail.XXX.com:389
Getting local config ldap_master_url=ldap://mail.XXX.com:389
Getting local config ldap_starttls_supported=0
Starting ldap...failed.
octet
Posts: 23
Joined: Thu Jan 06, 2022 2:35 pm

Re: Error on deploying SSL certificates

Post by octet »

Some further information on mysql, trying to start it manually:

Code: Select all

$ /opt/zimbra/bin/mysql.server start
Starting mysqld...failed.
Log:

Code: Select all

220107 12:28:04 mysqld_safe Starting mysqld daemon with databases from /opt/zimbra/db/data
2022-01-07 12:28:05 140056623495040 [Note] /opt/zimbra/common/sbin/mysqld (mysqld 10.1.25-MariaDB) starting as process 103580 ...
2022-01-07 12:28:05 140056623495040 [Note] InnoDB: Using mutexes to ref count buffer pool pages
2022-01-07 12:28:05 140056623495040 [Note] InnoDB: The InnoDB memory heap is disabled
2022-01-07 12:28:05 140056623495040 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2022-01-07 12:28:05 140056623495040 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2022-01-07 12:28:05 140056623495040 [Note] InnoDB: Compressed tables use zlib 1.2.3
2022-01-07 12:28:05 140056623495040 [Note] InnoDB: Using Linux native AIO
2022-01-07 12:28:05 140056623495040 [Note] InnoDB: Using SSE crc32 instructions
2022-01-07 12:28:05 140056623495040 [Note] InnoDB: Initializing buffer pool, size = 2.3G
2022-01-07 12:28:05 140056623495040 [Note] InnoDB: Completed initialization of buffer pool
2022-01-07 12:28:05 140056623495040 [ERROR] InnoDB: Log file ./ib_logfile0 size 524314036 is not a multiple of innodb_page_size
2022-01-07 12:28:05 140056623495040 [ERROR] Plugin 'InnoDB' init function returned error.
2022-01-07 12:28:05 140056623495040 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2022-01-07 12:28:05 140056623495040 [Note] Plugin 'FEEDBACK' is disabled.
2022-01-07 12:28:05 140056623495040 [ERROR] Unknown/unsupported storage engine: InnoDB
2022-01-07 12:28:05 140056623495040 [ERROR] Aborting

220107 12:28:05 mysqld_safe mysqld from pid file /opt/zimbra/log/mysql.pid ended
my.conf file:

Code: Select all

[root@mail data]# cat /opt/zimbra/conf/my.cnf

[mysqld]

basedir        = /opt/zimbra/common
datadir        = /opt/zimbra/db/data
socket         = /opt/zimbra/data/tmp/mysql/mysql.sock
pid-file       = /opt/zimbra/log/mysql.pid
bind-address   = 127.0.0.1
port           = 7306
user           = zimbra
tmpdir         = /opt/zimbra/data/tmp

external-locking
slow_query_log = 1
slow_query_log_file = /opt/zimbra/log/myslow.log

general_log_file = /opt/zimbra/log/mysql-mailboxd.log

long_query_time  = 1
log_queries_not_using_indexes

thread_cache_size = 110
max_connections   = 110

# We do a lot of writes, query cache turns out to be not useful.
query_cache_type = 0

sort_buffer_size = 1048576
read_buffer_size = 1048576

# (Num mailbox groups * Num tables in each group) + padding
table_open_cache = 1200

innodb_data_file_path          = ibdata1:10M:autoextend
innodb_buffer_pool_size        = 2456449843
innodb_log_file_size           = 524314036
innodb_log_buffer_size         = 8388608
innodb_file_per_table

# Value is: 200 + max_connections + 2 * table_open_cache
innodb_open_files              = 2710

innodb_max_dirty_pages_pct     = 30
innodb_flush_method            = O_DIRECT
innodb_flush_log_at_trx_commit = 0
max_allowed_packet             = 16777216

[mysqld_safe]

log-error      = /opt/zimbra/log/mysqld.log
pid-file     = /opt/zimbra/log/mysql.pid

User avatar
JDunphy
Outstanding Member
Outstanding Member
Posts: 889
Joined: Fri Sep 12, 2014 11:18 pm
Location: Victoria, BC
ZCS/ZD Version: 9.0.0_P39 NETWORK Edition

Re: Error on deploying SSL certificates

Post by JDunphy »

octet wrote: There were too many requests of a given type :: Error creating new order :: too many certificates (5) already issued for this exact set of domains in the last 168 hours: mail.XXX.com: see https://letsencrypt.org/docs/rate-limits/
Please see the logfiles in /var/log/letsencrypt for more details[/code]
Ouch... just adding another name - anyname such as tmail.xxx.com would have allowed you to keep trying with the LE certs and not wait 168 hours. I keep a tmail around myself so that I can verify staging and test environments. nginx is handling this so provided you make the staging machine believe it is the actual zmhostname you can do this illusion. That is accomplished by entries in /etc/hosts and running a DNS server on the same box with the ip addresses of zimbra hostnames on the test server. Then just tell /etc/resolv.conf to use this DNS server that zimbra is running in your test environment. So you can have a mail.XXX.com and part of zimbra thinking that web-mail.XXX.com is the same exact machine when you provide this illusion. There is one problem and that is a bug with zimbra and that is that commented entries in /etc/resolv.conf can end up with nginx lookup entries (/opt/zimbra/conf/nginx/resolvers.conf)
octet wrote: Ldap not starting:

Code: Select all

[zimbra@mail ~]$ ./bin/ldap start 
Failed to start slapd.  Attempting debug start to determine error.
61d80d68 daemon: bind(7) failed errno=99 (Cannot assign requested address)
61d80d68 slap_open_listener: failed on ldap://mail.XXX.com:389
[/quote]
This does look like either there is something running on port 389 but most likely that mail.XXX.com doesn't resolve to the ip address of this running zimbra host. note: bind(2) assigns an ip address and port to a socket which you can see when you use netstat, etc.  This is where an extra /etc/hosts entry could help if you find that the name doesn't resolve to the ip address of this machine. 
Note: it use to be that we would just use /etc/hosts because /etc/nsswitch.conf would allow one to say use /etc/hosts before dns resolution etc. Over time, different libraries were used that no longer checked this file which is why we have to spoof in both ways now. In your case it would appear that the ip address of mail.XXX.com probably isn't this local machine that you are attempting to run ldap so the bind fails. Another method is to change this ldap host variable.

It kind of feels like things went off the rails here. Running an install.sh again will discover this is an update only and then SHOULD attempt to repair without loss of data. However, I would prefer you were testing that on a staging or clone of the broken machine.

Here is what I can recommend from my experience.

1) setup that split brain DNS I mentioned. I use bind myself and it's dead simple to do.
2) do the same with /etc/hosts
3) see if zimbra will come up

If that doesn't work, get that staging server setup. If this is a cloud instance then it can be fairly simple.
Or if you trust your backup/restore methodology - backup your server first.

Next attempt that zimbra install.sh to see if it can fix it

Perhaps someone has some better idea's but that is how I would approach the problem.

Jim
octet
Posts: 23
Joined: Thu Jan 06, 2022 2:35 pm

Re: Error on deploying SSL certificates

Post by octet »

JDunphy wrote: This does look like either there is something running on port 389 but most likely that mail.XXX.com doesn't resolve to the ip address of this running zimbra host. note: bind(2) assigns an ip address and port to a socket which you can see when you use netstat, etc. This is where an extra /etc/hosts entry could help if you find that the name doesn't resolve to the ip address of this machine.
Note: it use to be that we would just use /etc/hosts because /etc/nsswitch.conf would allow one to say use /etc/hosts before dns resolution etc. Over time, different libraries were used that no longer checked this file which is why we have to spoof in both ways now. In your case it would appear that the ip address of mail.XXX.com probably isn't this local machine that you are attempting to run ldap so the bind fails. Another method is to change this ldap host variable.

Jim

Code: Select all

[root@mail zimbra]# netstat -anp | grep :389 | grep LISTEN
[root@mail zimbra]# 
Nothing running on 389.

resolv.conf file:

Code: Select all

[root@mail zimbra]# cat /etc/resolv.conf 
nameserver 10.194.3.3
nameserver 1.1.1.1
nameserver 8.8.8.8
[root@mail zimbra]# 
hosts file:

Code: Select all

[root@mail zimbra]# cat /etc/hosts
127.0.0.1       localhost.localdomain   localhost
51.158.xx.xx	mail.XXXX.com	mail
51.158.xx.xx	web-mail.XXX.com	web-mail
host resolves to the correct IP:

Code: Select all

[root@mail zimbra]# host mail.XXX.com
mail.XXX.com has address 51.158.xx.xx
mail.XXX.com mail is handled by 10 feedback-smtp.eu-west-1.amazonses.com.

Code: Select all

[root@mail zimbra]# cat /opt/zimbra/conf/nginx/resolvers.conf
resolver 10.194.3.3;
[root@mail zimbra]# 
ldap confs all have the correct host:

Code: Select all

[root@mail conf]# ll ldap*
-rw-r----- 1 zimbra postfix 439 Jan  6 19:53 ldap-canonical.cf
-rw-r----- 1 zimbra postfix 599 Jan  6 19:58 ldap-slm.cf
-rw-r----- 1 zimbra zimbra  481 Jan  6 19:58 ldap-splitdomain.cf
-rw-r----- 1 zimbra postfix 367 Jan  6 19:53 ldap-transport.cf
-rw-r----- 1 zimbra postfix 355 Jan  6 19:54 ldap-vad.cf
-rw-r----- 1 zimbra postfix 557 Jan  6 19:57 ldap-vam.cf
-rw-r----- 1 zimbra postfix 355 Jan  6 19:57 ldap-vmd.cf
-rw-r----- 1 zimbra postfix 349 Jan  6 19:57 ldap-vmm.cf
[root@mail conf]# 

[root@mail conf]# cat ldap* | grep mail
server_host = ldap://mail.XXX.com:389
server_host = ldap://mail.XXX.com:389
server_host = ldap://mail.XXX.com:389
server_host = ldap://mail.XXX.com:389
server_host = ldap://mail.XXX.com:389
server_host = ldap://mail.XXX.com:389
server_host = ldap://mail.XXX.com:389
server_host = ldap://mail.XXX.com:389
[root@mail conf]# 
Tried installation with forced-upgrade:

Code: Select all

[root@mail zcs-8.8.15_GA_3953.RHEL8_64.20200629025823]# ./install.sh --force-upgrade 

Operations logged to /tmp/install.log.npsCg9RG
Checking for existing installation...
    zimbra-drive...FOUND zimbra-drive-1.0.13.1576152256-1
    zimbra-imapd...NOT FOUND
    zimbra-patch...FOUND zimbra-patch-8.8.15.1639579168
    zimbra-mta-patch...FOUND zimbra-mta-patch-8.8.15.1638533801
    zimbra-proxy-patch...FOUND zimbra-proxy-patch-8.8.15.1634196512
    zimbra-license-tools...NOT FOUND
    zimbra-license-extension...NOT FOUND
    zimbra-network-store...NOT FOUND
    zimbra-network-modules-ng...NOT FOUND
    zimbra-chat...FOUND zimbra-chat-3.0.1.1594306000-1
    zimbra-talk...NOT FOUND
    zimbra-ldap...FOUND zimbra-ldap-8.8.15_GA_3953
    zimbra-logger...FOUND zimbra-logger-8.8.15_GA_3953
    zimbra-mta...FOUND zimbra-mta-8.8.15_GA_3953
    zimbra-dnscache...NOT FOUND
    zimbra-snmp...FOUND zimbra-snmp-8.8.15_GA_3953
    zimbra-store...FOUND zimbra-store-8.8.15_GA_3953
    zimbra-apache...FOUND zimbra-apache-8.8.15_GA_3953
    zimbra-spell...FOUND zimbra-spell-8.8.15_GA_3953
    zimbra-convertd...NOT FOUND
    zimbra-memcached...FOUND zimbra-memcached-1.6.5-1zimbra8.7b1
    zimbra-proxy...FOUND zimbra-proxy-8.8.15_GA_3953
    zimbra-archiving...NOT FOUND
    zimbra-core...FOUND zimbra-core-8.8.15_GA_3953
ZCS upgrade from 8.8.15 to 8.8.15 will be performed.
Validating ldap configuration
Error: Unable to create a successful TLS connection to the ldap masters.
       Fix cert configuration prior to upgrading.
[root@mail zcs-8.8.15_GA_3953.RHEL8_64.20200629025823]# 
User avatar
JDunphy
Outstanding Member
Outstanding Member
Posts: 889
Joined: Fri Sep 12, 2014 11:18 pm
Location: Victoria, BC
ZCS/ZD Version: 9.0.0_P39 NETWORK Edition

Re: Error on deploying SSL certificates

Post by JDunphy »

Strange. ... how about we get ldap running manually. We will bind to all interfaces on your host for this test on port 389. ldapi is for unix domain sockets. The standard ldap for a single hosted machine is:

Code: Select all

/opt/zimbra/common/libexec/slapd -l LOCAL0 -u zimbra -h ldap://mail.example.com:389 ldapi:/// -F /opt/zimbra/data/ldap/config
To do this manually, we have to jump through some hoops because you need to be root to bind to tcp port 389 but it will run as the zimbra user

Code: Select all

# su - 
# /opt/zimbra/common/libexec/slapd -l LOCAL0 -u zimbra -h "ldap:/// ldapi:///" -F /opt/zimbra/data/ldap/config
# netstat -nalp |grep 389 | grep LISTEN
If that works find out why -h ldap://mail.example.com isn't your hosts ip address when you attempt to start it with ldap start. You can now try to see if you can re-install that certificate, etc or other commands that failed partially for you when switching names, etc.

Note: this is what /opt/zimbra/bin/ldap is doing (bash script around the start() function

Code: Select all

    # Our ldap url should be the first in the list in localconfig
    bind_url=$ldap_bind_url
    if [ x"$bind_url" = "x" ]; then
        bind_url=$(echo "${ldap_url}" | awk '{print $1}')
    fi
    for ((i =0; i <= 30; i++)); do
        checkrunning
        if [ $RUNNING = 0 ]; then
            if ((i % 5 == 0)); then
                sudo /opt/zimbra/libexec/zmslapd -l LOCAL0 \
                -u zimbra -h "${bind_url} ldapi:///" -F /opt/zimbra/data/ldap/config
            fi
        else
You can verify this yourself by doing this:

Code: Select all

# su - zimbra
% source /opt/zimbra/bin/zmshutil 
% zmsetvars
% env |egrep '(ldap_bind|ldap_url)'
You can also stop this if you got it started manually via:

Code: Select all

# su - zimbra
% ldap stop
Very confused why you can't bind to the ip address of your local machine with ldap given the previous error you showed.

During install.sh you also reported a certificate error most likely because it failed during the certificate installation. To get around this you could disable ldap tls to get it up and running until you get this sorted and a proper certificate installed. search for ldap_starttls_required in forums. This came up for me. Ref: viewtopic.php?f=15&t=68753&p=299147&hil ... ed#p299147

Code: Select all

# su - zimbra 
% zmlocalconfig -e ldap_starttls_required=false
% zmlocalconfig -e ldap_starttls_supported=0
% ldap start
Jim
octet
Posts: 23
Joined: Thu Jan 06, 2022 2:35 pm

Re: Error on deploying SSL certificates

Post by octet »

Got ldap started manually:

Code: Select all

[root@mail ~]# /opt/zimbra/common/libexec/slapd -l LOCAL0 -u zimbra -h “ldap:/// ldapi:///” -F /opt/zimbra/data/ldap/config
[root@mail ~]# netstat -nalp |grep 389 | grep LISTEN
tcp    0   0 0.0.0.0:389       0.0.0.0:*        LISTEN   176361/slapd     
tcp6    0   0 :::389         :::*          LISTEN   176361/slapd     
[root@mail ~]#
Tried deploying the certificates again:

Code: Select all

[zimbra@mail tmp]$ /opt/zimbra/bin/zmcertmgr deploycrt comm mail.XXX.com.cer fullchain.cer
** Verifying 'mail.XXX.com.cer' against '/opt/zimbra/ssl/zimbra/commercial/commercial.key'
Certificate 'mail.XXX.com.cer' and private key '/opt/zimbra/ssl/zimbra/commercial/commercial.key' match.
** Verifying 'mail.XXX.com.cer' against 'fullchain.cer'
Valid certificate chain: mail.mc3lab.com.cer: OK
** Copying 'mail.XXX.com.cer' to '/opt/zimbra/ssl/zimbra/commercial/commercial.crt'
** Copying 'fullchain.cer' to '/opt/zimbra/ssl/zimbra/commercial/commercial_ca.crt'
** Appending ca chain 'fullchain.cer' to '/opt/zimbra/ssl/zimbra/commercial/commercial.crt'
** Importing cert '/opt/zimbra/ssl/zimbra/commercial/commercial_ca.crt' as 'zcs-user-commercial_ca' into cacerts '/opt/zimbra/common/lib/jvm/java/lib/security/cacerts'
** NOTE: restart mailboxd to use the imported certificate.
** Saving config key 'zimbraSSLCertificate' via zmprov modifyServer mail.XXX.com...failed (rc=1)
** Installing imapd certificate '/opt/zimbra/conf/imapd.crt' and key '/opt/zimbra/conf/imapd.key'
** Copying '/opt/zimbra/ssl/zimbra/commercial/commercial.crt' to '/opt/zimbra/conf/imapd.crt'
** Copying '/opt/zimbra/ssl/zimbra/commercial/commercial.key' to '/opt/zimbra/conf/imapd.key'
ERROR: imapd keytool(-delete -alias jetty) returned non-zero(1):
keytool error: java.security.cert.CertificateParsingException: signed overrun, bytes = 88
** Creating file '/opt/zimbra/ssl/zimbra/jetty.pkcs12'
** Creating keystore '/opt/zimbra/conf/imapd.keystore'
ERROR: com.zimbra.cert.MyPKCS12Import to '/opt/zimbra/ssl/zimbra/jetty.pkcs12' returned non-zero(1):
Exception in thread "main" java.security.cert.CertificateParsingException: signed overrun, bytes = 88
	at java.base/sun.security.x509.X509CertImpl.parse(X509CertImpl.java:1820)
	at java.base/sun.security.x509.X509CertImpl.<init>(X509CertImpl.java:188)
	at java.base/sun.security.provider.X509Factory.engineGenerateCertificate(X509Factory.java:105)
	at java.base/java.security.cert.CertificateFactory.generateCertificate(CertificateFactory.java:355)
	at java.base/sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:725)
	at java.base/sun.security.util.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:222)
	at java.base/java.security.KeyStore.load(KeyStore.java:1472)
	at com.zimbra.cert.MyPKCS12Import.main(MyPKCS12Import.java:104)

Code: Select all

[zimbra@mail tmp]$ source /opt/zimbra/bin/zmshutil 
[zimbra@mail tmp]$ zmsetvars
[zimbra@mail tmp]$ env |egrep '(ldap_bind|ldap_url)'
ldap_url=ldap://mail.XXX.com:389
ldap_bind_url=
Tried disabling ldap SSL stuff and restart it:

Code: Select all

[zimbra@mail tmp]$ zmlocalconfig -e ldap_starttls_required=false
[zimbra@mail tmp]$ zmlocalconfig -e ldap_starttls_supported=0
[zimbra@mail tmp]$ ldap restart
Killing slapd with pid 176361. done.
Failed to start slapd.  Attempting debug start to determine error.
61d99216 daemon: bind(7) failed errno=99 (Cannot assign requested address)
61d99216 slap_open_listener: failed on ldap://mail.XXX.com:389
Could be some permission where it doesn't let it read the correct conf file when ran as zimbra I wonder? Where can I see the ldap log?

Also checked where the ldap pid files should be vs where it's trying to write them:

Code: Select all

[root@mail conf]# /opt/zimbra/common/libexec/slapd -l LOCAL0 -u zimbra -h "ldap:/// ldapi:///" -F /opt/zimbra/data/ldap/config
[root@mail conf]# locate slapd.pid
/opt/zimbra/data/ldap/state/run/slapd.pid
[root@mail conf]# locate slapd.args
/opt/zimbra/data/ldap/state/run/slapd.args
[root@mail conf]# 

Code: Select all

[root@mail lib]# cat /opt/zimbra/common/etc/openldap/slapd.conf | grep pidfile
pidfile		/opt/zimbra/data/ldap/state/run/slapd.pid
[root@mail lib]# cat /opt/zimbra/common/etc/openldap/slapd.conf | grep argsfile
argsfile	/opt/zimbra/data/ldap/state/run/slapd.args
This seems fine, too... I can't understand why it wouldn't let me bind to mail.XXX.com :roll:

Just checked the IP the server has, which is local IP, is this the reason it can't bind on the mail.XXX.com domain, as it is set in /etc/hosts?

Code: Select all

[root@mail conf]# ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.70.150.43  netmask 255.255.255.254  broadcast 0.0.0.0
        inet6 2001:bc8:634:2315::1  prefixlen 64  scopeid 0x0<global>
        inet6 fe80::dc1c:a4ff:fe70:a016  prefixlen 64  scopeid 0x20<link>
        ether de:1c:a4:70:a0:16  txqueuelen 1000  (Ethernet)
        RX packets 238528  bytes 71285916 (67.9 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 268041  bytes 365546797 (348.6 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

eth0:0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.168.168  netmask 255.255.255.0  broadcast 192.168.168.255
        ether de:1c:a4:70:a0:16  txqueuelen 1000  (Ethernet)

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 3144  bytes 207417 (202.5 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 3144  bytes 207417 (202.5 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
Post Reply