CN = DST Root CA X3 error 10 at 3 depth lookup:certificate has expired

[realsparticle]

This morning our certificates were due for renewal.

Certbot did the necessary but failed when verifying as you can see below.

Never seen this type of message and could do with a little step by step help to resolve this issue which appears to be due to a CA Root certificate expiring.

How do we solve this problem?

Any help would be appreciated.

Cheers
Spart

Zimbra version Release 8.8.9.GA

Code: Select all

** Verifying '/opt/zimbra/ssl/letsencrypt/cert.pem' against '/opt/zimbra/ssl/letsencrypt/privkey.pem'
Certificate '/opt/zimbra/ssl/letsencrypt/cert.pem' and private key '/opt/zimbra/ssl/letsencrypt/privkey.pem' match.
** Verifying '/opt/zimbra/ssl/letsencrypt/cert.pem' against '/opt/zimbra/ssl/letsencrypt/zimbra_chain.pem'
ERROR: Unable to validate certificate chain: /opt/zimbra/ssl/letsencrypt/cert.pem: O = Digital Signature Trust Co., CN = DST Root CA X3
error 10 at 3 depth lookup:certificate has expired
OK
Unable to verify cert!

[phoenix]

Have you looked at any of the recent forum threads on the expiration of the Root certificate?

[realsparticle]

Have you looked at any of the recent forum threads on the expiration of the Root certificate?

We did quite a bit of google fu, but to be honest the many many stage solutions were like rocket science to us. We were hoping someone would provide a simple step by step solution to this problem. Without compromising our installation.

There were no posts we found that matched exactly what we were seeing.

The server has been so stable for so long, this was a shock.

Any help is appreciated.

Cheers
Spart

[realsparticle]

I followed this guide and I now have a fully working zimbra with up to date certificates.

Replace with your domain

Then:

cp /etc/letsencrypt/live//privkey.pem /opt/zimbra/ssl/zimbra/commercial/commercial.key
chown zimbra:zimbra /opt/zimbra/ssl/zimbra/commercial/commercial.key
wget -O /tmp/ISRG-X1.pem https://letsencrypt.org/certs/isrgrootx1.pem
wget -O /tmp/R3.pem https://letsencrypt.org/certs/lets-encrypt-r3.pem
The files in /etc/letsencrypt/live/ are symbolic links to files in /etc/letsencrypt/archive/.
Check which files they point at (cert.pem, chain.pem, ...)

Then perform this, but replace the part with correct one.

cat /tmp/R3.pem > /etc/letsencrypt/archive//chain.pem
cat /tmp/ISRG-X1.pem >> /etc/letsencrypt/archive//chain.pem
As zimbra (su - zimbra) user perform

/opt/zimbra/bin/zmcertmgr verifycrt comm /opt/zimbra/ssl/zimbra/commercial/commercial.key /etc/letsencrypt/live//cert.pem /etc/letsencrypt/live//chain.pem
If it runs successfully perform deploy. If it gives you file permissions error temporarily do this:

chmod o+rx /etc/letsencrypt/archive
chmod o+rx /etc/letsencrypt/live

And verify the cert again.

You can run certbot-zimbra deploy, or the below one:

I ran certbot_zimbra.sh and selected to use the cert that I already had from the failed previous attempt.

Afterwards remove the extra permissions (as root)

chmod o-rx /etc/letsencrypt/archive
chmod o-rx /etc/letsencrypt/live

Restart zimbra:

zmcontrol restart`

I had to restart the full server to stop issues when I was sending test emails. So something requires a reboot of the system to be fully recognised.

Cheers
Spart

[fiddels]

Works for me, thank you!

[svenna71]

Hey Thank you!
On one server ist worked just like that!

on an other server, I get the error:
ERROR: Unable to validate certificate chain: /etc/letsencrypt/live/..../cert.pem: C = US, O = Internet Security Research Group, CN = ISRG Root X1
error 2 at 2 depth lookup:unable to get issuer certificate

thanks in advance

Sven

[zimico]

Please show your certbot version.

Code: Select all

certbot --version

[JDunphy]

Letsencrypt has 2 chains. The default chain is signed like this:
<customer cert><LE R3 intermediate><ISRG Root X1 signed by DST Root X3 (expires 2024)><DST Root CA X3 (expired Sep 30 2021)>

If you specify the preferred chain ISRG you get:
<customer cert><LE R3 intermediate><ISRG Root X1>

Now the problem. Old android devices <= version 6 don't verify trust anchor (aka root cerificates) dates so they continue to work perfectly with the default chain as do most browsers.
... BUT openssl does verify trust anchor dates AND openssl <= 1.0.2 will have a problem by default with this default LE chain. zmcertmgr uses openssl to verify it's certs

Things to consider if you want to resolve the issue:
1) what version of openssl are you running
2) what LE chain you are attempting to validate and install

Most fullchains returned by acme clients I know of already provide LE R3 in their fullchain. You can guess if you do something like:

Code: Select all

grep -A 1 BEGIN fullchain.cer

If it comes back with 3 chains than you have the default. If it comes back with 2 chains than you have the alternative chain.

If you are running an openssl 1.0.2 and want the default chain than your option is to remove the DST Root CA X3 expired chain from your trust store or update openssl. The ISRG Root X1 certificate that is signed by DST Root CA X3 has the CA flag set to True, trust first bit set and is a root CA. Unfortunately, with openssl 1.0.2 and older it will not stop at that ISRG Root X1 certificate and will continue to validate up to the expired DST Root CA X3 certificate... but if you remove the DST Root CA X3 from the trust store, it will stop with the ISRG Root X1. Openssl 1.1 and above will stop at the ISRG Root X1 and not follow the chain to the DST Root X3.

For most of us, using the alternative chains is the simplest solution given we don't have client software connecting with really old trust stores.

Ref: https://www.youtube.com/watch?v=RIR-_V1fNrk

[raslin]

Dear Spart!
You are a lifesaver, thanks soo much for this.
All the best
Rasmus

I followed this guide and I now have a fully working zimbra with up to date certificates.

Replace with your domain

Then:

cp /etc/letsencrypt/live//privkey.pem /opt/zimbra/ssl/zimbra/commercial/commercial.key
chown zimbra:zimbra /opt/zimbra/ssl/zimbra/commercial/commercial.key
wget -O /tmp/ISRG-X1.pem https://letsencrypt.org/certs/isrgrootx1.pem
wget -O /tmp/R3.pem https://letsencrypt.org/certs/lets-encrypt-r3.pem
The files in /etc/letsencrypt/live/ are symbolic links to files in /etc/letsencrypt/archive/.
Check which files they point at (cert.pem, chain.pem, ...)

Then perform this, but replace the part with correct one.

cat /tmp/R3.pem > /etc/letsencrypt/archive//chain.pem
cat /tmp/ISRG-X1.pem >> /etc/letsencrypt/archive//chain.pem
As zimbra (su - zimbra) user perform

/opt/zimbra/bin/zmcertmgr verifycrt comm /opt/zimbra/ssl/zimbra/commercial/commercial.key /etc/letsencrypt/live//cert.pem /etc/letsencrypt/live//chain.pem
If it runs successfully perform deploy. If it gives you file permissions error temporarily do this:

chmod o+rx /etc/letsencrypt/archive
chmod o+rx /etc/letsencrypt/live

And verify the cert again.

You can run certbot-zimbra deploy, or the below one:

I ran certbot_zimbra.sh and selected to use the cert that I already had from the failed previous attempt.

Afterwards remove the extra permissions (as root)

chmod o-rx /etc/letsencrypt/archive
chmod o-rx /etc/letsencrypt/live

Restart zimbra:

zmcontrol restart`

I had to restart the full server to stop issues when I was sending test emails. So something requires a reboot of the system to be fully recognised.

Cheers
Spart

[davidkillingsworth]

I followed this guide and I now have a fully working zimbra with up to date certificates.

Replace with your domain

Then:

cp /etc/letsencrypt/live//privkey.pem /opt/zimbra/ssl/zimbra/commercial/commercial.key
chown zimbra:zimbra /opt/zimbra/ssl/zimbra/commercial/commercial.key
wget -O /tmp/ISRG-X1.pem https://letsencrypt.org/certs/isrgrootx1.pem
wget -O /tmp/R3.pem https://letsencrypt.org/certs/lets-encrypt-r3.pem
The files in /etc/letsencrypt/live/ are symbolic links to files in /etc/letsencrypt/archive/.
Check which files they point at (cert.pem, chain.pem, ...)

Then perform this, but replace the part with correct one.

cat /tmp/R3.pem > /etc/letsencrypt/archive//chain.pem
cat /tmp/ISRG-X1.pem >> /etc/letsencrypt/archive//chain.pem
As zimbra (su - zimbra) user perform

/opt/zimbra/bin/zmcertmgr verifycrt comm /opt/zimbra/ssl/zimbra/commercial/commercial.key /etc/letsencrypt/live//cert.pem /etc/letsencrypt/live//chain.pem
If it runs successfully perform deploy. If it gives you file permissions error temporarily do this:

chmod o+rx /etc/letsencrypt/archive
chmod o+rx /etc/letsencrypt/live

And verify the cert again.

You can run certbot-zimbra deploy, or the below one:

I ran certbot_zimbra.sh and selected to use the cert that I already had from the failed previous attempt.

Afterwards remove the extra permissions (as root)

chmod o-rx /etc/letsencrypt/archive
chmod o-rx /etc/letsencrypt/live

Restart zimbra:

zmcontrol restart`

I had to restart the full server to stop issues when I was sending test emails. So something requires a reboot of the system to be fully recognised.

Cheers
Spart

Legend! This fixed the issue for me.

Crucially, these parts allowed me to get past running /opt/zimbra/bin/zmcertmgr verifycrt comm without permissions errors.

Code: Select all

chmod o+rx /etc/letsencrypt/archive
chmod o+rx /etc/letsencrypt/live
chmod o-rx /etc/letsencrypt/archive
chmod o-rx /etc/letsencrypt/live