Weird SSL Certificate issue even though valid certs are installed

Discuss your pilot or production implementation with other Zimbra admins or our engineers.
Post Reply
realsparticle
Posts: 41
Joined: Sat Sep 13, 2014 3:29 am

Weird SSL Certificate issue even though valid certs are installed

Post by realsparticle »

HI I am looking for some help to understand and troubleshoot an issue with the zimbra web client timing out every few minutes due to ssl.

I have deployed certs into zimbra via certbot. All goes well and everything was working just fine until a number of clients started timing out.

Code: Select all


What would you like to do?
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1: Keep the existing certificate for now
2: Renew & replace the cert (limit ~5 per 7 days)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Select the appropriate number [1-2] then [enter] (press 'c' to cancel): 2
Renewing an existing certificate

IMPORTANT NOTES:
 - Congratulations! Your certificate and chain have been saved at:
   /etc/letsencrypt/live/mydomain.co.uk/fullchain.pem
   Your key file has been saved at:
   /etc/letsencrypt/live/mydomain.co.uk/privkey.pem
   Your cert will expire on 2020-04-12. To obtain a new or tweaked
   version of this certificate in the future, simply run certbot
   again. To non-interactively renew *all* of your certificates, run
   "certbot renew"
 - If you like Certbot, please consider supporting our work by:

   Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
   Donating to EFF:                    https://eff.org/donate-le

** Verifying '/opt/zimbra/ssl/letsencrypt/cert.pem' against '/opt/zimbra/ssl/letsencrypt/privkey.pem'
Certificate '/opt/zimbra/ssl/letsencrypt/cert.pem' and private key '/opt/zimbra/ssl/letsencrypt/privkey.pem' match.
** Verifying '/opt/zimbra/ssl/letsencrypt/cert.pem' against '/opt/zimbra/ssl/letsencrypt/zimbra_chain.pem'
Valid certificate chain: /opt/zimbra/ssl/letsencrypt/cert.pem: OK
** Verifying '/opt/zimbra/ssl/letsencrypt/cert.pem' against '/opt/zimbra/ssl/zimbra/commercial/commercial.key'
Certificate '/opt/zimbra/ssl/letsencrypt/cert.pem' and private key '/opt/zimbra/ssl/zimbra/commercial/commercial.key' match.
** Verifying '/opt/zimbra/ssl/letsencrypt/cert.pem' against '/opt/zimbra/ssl/letsencrypt/zimbra_chain.pem'
Valid certificate chain: /opt/zimbra/ssl/letsencrypt/cert.pem: OK
** Copying '/opt/zimbra/ssl/letsencrypt/cert.pem' to '/opt/zimbra/ssl/zimbra/commercial/commercial.crt'
** Copying '/opt/zimbra/ssl/letsencrypt/zimbra_chain.pem' to '/opt/zimbra/ssl/zimbra/commercial/commercial_ca.crt'
** Appending ca chain '/opt/zimbra/ssl/letsencrypt/zimbra_chain.pem' to '/opt/zimbra/ssl/zimbra/commercial/commercial.crt'
** Importing cert '/opt/zimbra/ssl/zimbra/commercial/commercial_ca.crt' as 'zcs-user-commercial_ca' into cacerts '/opt/zimbra/common/lib/jvm/java/jre/lib/security/cacerts'
** NOTE: restart mailboxd to use the imported certificate.
** Saving config key 'zimbraSSLCertificate' via zmprov modifyServer newzimbra.twhg.co.uk...ok
** Saving config key 'zimbraSSLPrivateKey' via zmprov modifyServer newzimbra.twhg.co.uk...ok
** Installing imapd certificate '/opt/zimbra/conf/imapd.crt' and key '/opt/zimbra/conf/imapd.key'
** Copying '/opt/zimbra/ssl/zimbra/commercial/commercial.crt' to '/opt/zimbra/conf/imapd.crt'
** Copying '/opt/zimbra/ssl/zimbra/commercial/commercial.key' to '/opt/zimbra/conf/imapd.key'
** Creating file '/opt/zimbra/ssl/zimbra/jetty.pkcs12'
** Creating keystore '/opt/zimbra/conf/imapd.keystore'
** Installing ldap certificate '/opt/zimbra/conf/slapd.crt' and key '/opt/zimbra/conf/slapd.key'
** Copying '/opt/zimbra/ssl/zimbra/commercial/commercial.crt' to '/opt/zimbra/conf/slapd.crt'
** Copying '/opt/zimbra/ssl/zimbra/commercial/commercial.key' to '/opt/zimbra/conf/slapd.key'
** Creating file '/opt/zimbra/ssl/zimbra/jetty.pkcs12'
** Creating keystore '/opt/zimbra/mailboxd/etc/keystore'
** Installing mta certificate '/opt/zimbra/conf/smtpd.crt' and key '/opt/zimbra/conf/smtpd.key'
** Copying '/opt/zimbra/ssl/zimbra/commercial/commercial.crt' to '/opt/zimbra/conf/smtpd.crt'
** Copying '/opt/zimbra/ssl/zimbra/commercial/commercial.key' to '/opt/zimbra/conf/smtpd.key'
** Installing proxy certificate '/opt/zimbra/conf/nginx.crt' and key '/opt/zimbra/conf/nginx.key'
** Copying '/opt/zimbra/ssl/zimbra/commercial/commercial.crt' to '/opt/zimbra/conf/nginx.crt'
** Copying '/opt/zimbra/ssl/zimbra/commercial/commercial.key' to '/opt/zimbra/conf/nginx.key'
** NOTE: restart services to use the new certificates.
** Cleaning up 7 files from '/opt/zimbra/conf/ca'
** Removing /opt/zimbra/conf/ca/2e5ac55d.0
** Removing /opt/zimbra/conf/ca/commercial_ca_1.crt
** Removing /opt/zimbra/conf/ca/ca.pem
** Removing /opt/zimbra/conf/ca/37d1c1aa.0
** Removing /opt/zimbra/conf/ca/ca.key
** Removing /opt/zimbra/conf/ca/4f06f81d.0
** Removing /opt/zimbra/conf/ca/commercial_ca_2.crt
** Copying CA to /opt/zimbra/conf/ca
** Copying '/opt/zimbra/ssl/zimbra/ca/ca.key' to '/opt/zimbra/conf/ca/ca.key'
** Copying '/opt/zimbra/ssl/zimbra/ca/ca.pem' to '/opt/zimbra/conf/ca/ca.pem'
** Creating CA hash symlink '37d1c1aa.0' -> 'ca.pem'
** Creating /opt/zimbra/conf/ca/commercial_ca_1.crt
** Creating CA hash symlink '4f06f81d.0' -> 'commercial_ca_1.crt'
** Creating /opt/zimbra/conf/ca/commercial_ca_2.crt
** Creating CA hash symlink '2e5ac55d.0' -> 'commercial_ca_2.crt'
I then restart zimbra and all seems well. The certs are valid and in date as shown below:

Code: Select all

/opt/zimbra/bin/zmcertmgr viewdeployedcrt
- imapd: /opt/zimbra/conf/imapd.crt
notBefore=Jan 13 18:56:11 2020 GMT
notAfter=Apr 12 18:56:11 2020 GMT
subject= /CN=mydomain.co.uk
issuer= /C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
SubjectAltName=mydomain.co.uk, myother.co.uk
- ldap: /opt/zimbra/conf/slapd.crt
notBefore=Jan 13 18:56:11 2020 GMT
notAfter=Apr 12 18:56:11 2020 GMT
subject= /CN=mydomain.co.uk
issuer= /C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
SubjectAltName=mydomain.co.uk, myother.co.uk
- mailboxd: /opt/zimbra/mailboxd/etc/mailboxd.pem
notBefore=Jan 13 18:56:11 2020 GMT
notAfter=Apr 12 18:56:11 2020 GMT
subject= /CN=mydomain.co.uk
issuer= /C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
SubjectAltName=mydomain.co.uk, myother.co.uk
- mta: /opt/zimbra/conf/smtpd.crt
notBefore=Jan 13 18:56:11 2020 GMT
notAfter=Apr 12 18:56:11 2020 GMT
subject= /CN=mydomain.co.uk
issuer= /C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
SubjectAltName=mydomain.co.uk, myother.co.uk
- proxy: /opt/zimbra/conf/nginx.crt
notBefore=Jan 13 18:56:11 2020 GMT
notAfter=Apr 12 18:56:11 2020 GMT
subject= /CN=mydomain.co.uk
issuer= /C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
SubjectAltName=mydomain.co.uk, myother.co.uk
However when connecting to zimbra using the webclient I see an outdated cert showing as Not Secure with the expiry date of the old certificate.

What am I missing? Why is the new certificate not being served to the client.

Any help appreciated.

Zimbra is running on Ubuntu 16.04.5 and is Release 8.8.9.GA.3019.UBUNTU16.64 UBUNTU16_64 FOSS edition, Patch 8.8.9_P10.

Also another issue that may be related is that Zimbra refused to start the other day. LDAP crashed out. IU had to issue the following commands after an urgent google foo session to get it up and running again.

Code: Select all

zmlocalconfig -e ldap_starttls_required=false
ldap_starttls_supported=0
Otherwise I could not get ldap to start. It may be connected to the same certficate issues.

Cheers
Spart
User avatar
gabrieles
Outstanding Member
Outstanding Member
Posts: 233
Joined: Tue Feb 14, 2017 9:40 am

Re: Weird SSL Certificate issue even though valid certs are installed

Post by gabrieles »

did you zmcontrol restart-ed after deploy?
realsparticle
Posts: 41
Joined: Sat Sep 13, 2014 3:29 am

Re: Weird SSL Certificate issue even though valid certs are installed

Post by realsparticle »

Yes multiple times!

If I look in the admin console all looks fine. Certs are there and in date. If I check using a checker site they show as the out of date ones so the webserver is serving teh right certs out.

Proxy, Ldap, Mailboxd and mta services all show valid in date certs.
Cheers
Spart
User avatar
JDunphy
Outstanding Member
Outstanding Member
Posts: 889
Joined: Fri Sep 12, 2014 11:18 pm
Location: Victoria, BC
ZCS/ZD Version: 9.0.0_P39 NETWORK Edition

Re: Weird SSL Certificate issue even though valid certs are installed

Post by JDunphy »

Weird... Run this against nginx and then work backwards from there depending on results.

Code: Select all

% openssl s_client -connect mail.example.com:443 | openssl x509 -noout -dates
depth=2 O = Digital Signature Trust Co., CN = DST Root CA X3
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
verify return:1
depth=0 CN = mail.example.com
verify return:1
notBefore=Dec  7 18:54:45 2019 GMT
notAfter=Mar  6 18:54:45 2020 GMT
^C
Example: If the above looks good, I would double check you renewed all the domains you were supposed to if you should have more than one domain (subject alternative names) present on that cert. Verify what zmhostname is and re-run that openssl command above to verify that.

or do something like this to list them to see if there are any surprises:

Code: Select all

% openssl s_client -connect mail.example.com:443 | openssl x509 -text | grep DNS
depth=2 O = Digital Signature Trust Co., CN = DST Root CA X3
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
verify return:1
depth=0 CN = mail.example.com
verify return:1
                DNS:mail.example.com, DNS:mail.example.net, DNS:tmail.example.com
^C
Generally, renewing letsencrypt certificates is fairly event free so you are correct that this is a "Weird SSL Certificate" issue.

Jim
realsparticle
Posts: 41
Joined: Sat Sep 13, 2014 3:29 am

Re: Weird SSL Certificate issue even though valid certs are installed

Post by realsparticle »

Weirdly again I get an error on the hostname as part of the output of the command you posted: The hostname is correct. DNS is correct.

verify error:num=19:self signed certificate in certificate chain

Then the command hangs and does not complete. But there are no self signed certs. Only letsencrypt certbot installed certs which are there in zimbra admin and the command line.

Cheers
Spart
User avatar
JDunphy
Outstanding Member
Outstanding Member
Posts: 889
Joined: Fri Sep 12, 2014 11:18 pm
Location: Victoria, BC
ZCS/ZD Version: 9.0.0_P39 NETWORK Edition

Re: Weird SSL Certificate issue even though valid certs are installed

Post by JDunphy »

realsparticle wrote:Weirdly again I get an error on the hostname as part of the output of the command you posted: The hostname is correct. DNS is correct.
You should not be getting a connection error on the hostname you provided... double check via netstat that you have something on port 443 for hostname:443. It should be zimbra's nginx.

Better find out why hostname:443 doesn't allow you to connect to it would be the start of determining for root cause. If you have done anything wonky like manually change permissions or ownership then perhaps start down that path also.

Code: Select all

# netstat -nap | grep 443 |grep LISTEN
tcp        0      0 0.0.0.0:443                 0.0.0.0:*                   LISTEN      7674/nginx          
# su - zimbra
% cd /opt/zimbra/ssl/zimbra
% ls -Rl
Everything under ssl/zimbra should be owned by zimbra and have correct read permission and you should see nginx in a listen state for port 443 for the ip address you are attempting connection. All's 0's in the above example means all the ip addresses on the host but it doesn't matter if you have it locked down to just a single interface (ie. ip address) provided that is the host you are attempting connection to.

If I was making a wild guess since there isn't a lot to go on here other than cert/zimbra/environment seems to be wonky, it would be the cert is fine and that mailboxd failed to restart as expected. :-) Perhaps something like this is in order:

Code: Select all

# su - zimbra
% ps |grep mailboxd
% /opt/zimbra/bin/zmmailboxdctl restart
% ps |grep mailboxd
You should be able to confirm that a new mailboxd started and is running with a different pid.

HTH,

Jim
realsparticle
Posts: 41
Joined: Sat Sep 13, 2014 3:29 am

Re: Weird SSL Certificate issue even though valid certs are installed

Post by realsparticle »

Jim,

Many thanks I see the below outputs:

Code: Select all

netstat -nap | grep 443 |grep LISTEN
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN      31118/nginx: worker
tcp        0      0 0.0.0.0:8443            0.0.0.0:*               LISTEN      -               

ls -lash /opt/zimbra/ssl/zimbra
total 28K
4.0K drwxr-x---  5 zimbra zimbra 4.0K Oct 10  2018 .
4.0K drwxr-xr-x 16 zimbra zimbra 4.0K Jan 13 19:56 ..
4.0K drwxr-x---  3 zimbra zimbra 4.0K Oct 10  2018 ca
4.0K drwxr-x---  2 zimbra zimbra 4.0K Oct 12  2018 commercial
8.0K -rw-r-----  1 zimbra zimbra 5.1K Jan 13 19:56 jetty.pkcs12
4.0K drwxr-x---  2 zimbra zimbra 4.0K Oct 10  2018 server

All is seemingly well. The zimbra server is running on a machine with webmin installed. I copied the certs to the webmin server and now the correct cert is being served externally. however I still do not see a green lock in chrome even thought he cert is recognised. I tsays Not Secure but the cert shows a valid. Confused. If I check the site via ssllabs it gets an A rating.

I could also now set the variables back:

Code: Select all

zmlocalconfig -e ldap_starttls_required=true
zmlocalconfig -e ldap_starttls_supported=1
To allow me to get ldap to start.

Anyway thank you for all your advice and assistance. I am not sure this is fully sorted but everything is now working again.

Cheers
Spart
Post Reply