Installing Letsencrypt cert after root expiration

Discuss your pilot or production implementation with other Zimbra admins or our engineers.
GlooM
Advanced member
Advanced member
Posts: 127
Joined: Sat Sep 13, 2014 12:50 am

Installing Letsencrypt cert after root expiration

Post by GlooM »

Greetings!

Yesterday I received an email with a reminder about the expiration of the root certificate DST X3 from Letsencrypt.
https://letsencrypt.org/docs/dst-root-c ... mber-2021/

Previously, I installed the Letsencrypt certificates according to this instruction:
https://wiki.zimbra.com/wiki/Installing ... ertificate

According to the instructions required:
"Build the proper Intermediate CA plus Root CA
Let's Encrypt is almost perfect, but during the files the process built, they just add the chain.pem file without the root CA. You must to use the IdenTrust root Certificate and merge it after the chain.pem

https://letsencrypt.org/certs/trustid-x3-root.pem.txt"

So the full chain included that certificate that would be expired in September.

1) Can you tell me if there is an updated version of the certificate installation instructions? Will I need to add another root certificate to the full chain, or will I not need to add anything at all?

2) Can I now check whether Zimbra is ready to work with other root certificates without replacing the current certificate?

3) Do I understand correctly that Zimbra does not use the operating system certificate store, and it is necessary to check whether there is a new root in its own store? How can I check Zimbra root certs store?

Thank you!
User avatar
JDunphy
Outstanding Member
Outstanding Member
Posts: 889
Joined: Fri Sep 12, 2014 11:18 pm
Location: Victoria, BC
ZCS/ZD Version: 9.0.0_P39 NETWORK Edition

Re: Installing Letsencrypt cert after root expiration

Post by JDunphy »

Given the confusion about the 2 chains that letsencrypt plans on keeping, I pulled that X1 chain to test against. For acme.sh clients, our option is --preferred-chain before the default changes. For certbot, I listed the option above you would use.

Code: Select all

% acme.sh  --issue --dns dns_cf  --preferred-chain  "ISRG Root X1" -d mail.example.com -d mail.example.net
After I pulled the cert, I made sure that the IdentTrust cross sign wasn't included and the verify failed.

The solution was simple, I did this:

Code: Select all

wget -O /tmp/ISRG-X1.pem 'https://letsencrypt.org/certs/isrgrootx1.pem.txt'
And used /tmp/ISRG-X1.pem instead of IdentTrust and it verified and deployed as expected.

For other readers that are not comfortable with command line and wanted to try this with the acme.sh and are using the /opt/zimbra/acme.sh/deploy/zimbra.sh, you can do this:

Code: Select all

# su - zimbra
% cd .acme.sh
% wget -O IdentTrust.pem 'https://letsencrypt.org/certs/isrgrootx1.pem.txt'
Everything will work as before. The /opt/zimbra/.acme.sh/deploy/zimbra.sh generally does a wget and pulled that IdentTrust.pem so in the future, I will update that script to pull the ISRG-X1.pem instead and use the correct name.

Note: If you have some really really old android clients, you would probably want to use the other chain.

HTH,

Jim

Ref: https://github.com/acmesh-official/acme ... rred-Chain
Ref: https://letsencrypt.org/certificates/
User avatar
barrydegraaff
Zimbra Employee
Zimbra Employee
Posts: 242
Joined: Tue Jun 17, 2014 3:31 am
Contact:

Re: Installing Letsencrypt cert after root expiration

Post by barrydegraaff »

I can confirm that using --preferred-chain solves the issue.

Please note that you will need an up-to-date certbot, the ones packed in OS is too old. Consider using a snap. https://certbot.eff.org/lets-encrypt/ub ... pache.html

In a scripted way, this is how I use it now, I run all these commands on a dedicated Letsencrypt VM and mail4 is my Zimbra server.

Code: Select all

/usr/local/bin/certbot --manual --force-renewal --preferred-chain  "ISRG Root X1" --expand --manual-auth-hook /usr/local/sbin/hook.sh --manual-cleanup-hook /usr/local/sbin/cleanhook.sh --preferred-challenges dns -d "barrytest.tk" -d "*.barrytest.tk" -d "zimbratest.tk" -d "*.zimbratest.tk" certonly --manual-public-ip-logging-ok -n

/usr/bin/scp -r /etc/letsencrypt/live/barrytest.tk/* root@mail4:/etc/letsencrypt/live/barrytest.tk/
/usr/bin/ssh root@mail4 'cp /etc/letsencrypt/live/barrytest.tk/privkey.pem /opt/zimbra/ssl/zimbra/commercial/commercial.key'
/usr/bin/ssh root@mail4 'chown zimbra:zimbra /opt/zimbra/ssl/zimbra/commercial/commercial.key'
/usr/bin/ssh root@mail4 'wget -O /tmp/ISRG-X1.pem 'https://letsencrypt.org/certs/isrgrootx1.pem.txt'
/usr/bin/ssh root@mail4 'cat /tmp/ISRG-X1.pem >> /etc/letsencrypt/live/barrytest.tk/chain.pem'
/usr/bin/ssh zimbra@mail4 '/opt/zimbra/bin/zmcertmgr deploycrt comm /etc/letsencrypt/live/barrytest.tk/cert.pem /etc/letsencrypt/live/barrytest.tk/chain.pem'
see also: https://wiki.zimbra.com/wiki/Installing ... tificate?c
--
Barry de Graaff
Email: barry.degraaff [at] synacor [dot] com
Admin of Zimbra-Community Github: https://github.com/orgs/Zimbra-Community/ and the
Zimlet Gallery https://gallery.zetalliance.org/extend/
User avatar
barrydegraaff
Zimbra Employee
Zimbra Employee
Posts: 242
Joined: Tue Jun 17, 2014 3:31 am
Contact:

Re: Installing Letsencrypt cert after root expiration

Post by barrydegraaff »

--
Barry de Graaff
Email: barry.degraaff [at] synacor [dot] com
Admin of Zimbra-Community Github: https://github.com/orgs/Zimbra-Community/ and the
Zimlet Gallery https://gallery.zetalliance.org/extend/
rtkluttz
Posts: 27
Joined: Mon May 06, 2019 12:00 pm

Re: Installing Letsencrypt cert after root expiration

Post by rtkluttz »

I am really hoping for some more explanation. All of the help so far that I have found on these issues has a different starting point than me. I am in Zimbra 8.8.15 4059 and I started with Ubuntu 14.04 but upgraded over the years to 16.04. So my certbot install has been completely manual over the years and was brought to the machine with wget, not with ubuntu repos. I am Lost with a capital L and almost in a panic state. i have to figure this all out by tomorrow night.

Question 1: my certbot/acme client has been updating itself every time I ran it up until about 6 months ago. How can I ensure that my version of certbot supports the forced --preferred-chain "ISRG Root X1"? If not will I be able to use the snap install with Ubuntu 16.04?

The new version of Zimbra letsencrypt wiki shows only new install, nothing about renewal so it is unhelpful for me.

Barry's examples include lots of hook scripts that are completley foreign to me and do not even exist on my box.

Where can I find a simple walkthrough of RENEWAL of letsencrypt for Ubuntu 16.04 with Zimbra 8.8.15 that doesn't make the assumption that anything is automated for renewals?
GlooM
Advanced member
Advanced member
Posts: 127
Joined: Sat Sep 13, 2014 12:50 am

Re: Installing Letsencrypt cert after root expiration

Post by GlooM »

rtkluttz wrote:but upgraded over the years to 16.04.
Hi!

New certbot works fine with Ubuntu 16. See this instruction : https://certbot.eff.org/lets-encrypt/ubuntuxenial-other
You can generate certificates with the required chain without any problems. And then, having the correct certificates already install them using Barry's instructions.
However, I do not know how things are with the Ubuntu 16 version of Zimbra. I'm not sure if it will be able to work with such certificates. But you can try installing them.

By the way, Ubuntu 16 was removed from support in April 2021. Better upgrade to 18.
User avatar
kdmiller45
Advanced member
Advanced member
Posts: 59
Joined: Sun Jan 19, 2020 11:08 pm

Re: Installing Letsencrypt cert after root expiration

Post by kdmiller45 »

Appreciate the help with the instruction on using certbot with Zimbra however

the instructions in that link ( https://wiki.zimbra.com/wiki/Installing ... tificate?c)
Make sure to request a certificate with the --preferred-chain "ISRG Root X1" option. In case you already have a certificate but you have not used the option, you have to do a force renewal with the --force-renewal --preferred-chain "ISRG Root X1" options.

It does not give the syntax to perform the command to request the required certs can you expand on that and then you link can be used

ie: certbot blahhh blahhhh blahhhh
Keith
GlooM
Advanced member
Advanced member
Posts: 127
Joined: Sat Sep 13, 2014 12:50 am

Re: Installing Letsencrypt cert after root expiration

Post by GlooM »

kdmiller45 wrote:
It does not give the syntax to perform the command to request the required certs can you expand on that and then you link can be used

Keith
Change user to root and run:

certbot certonly --standalone --preferred-chain "ISRG Root X1"
User avatar
kdmiller45
Advanced member
Advanced member
Posts: 59
Joined: Sun Jan 19, 2020 11:08 pm

Re: Installing Letsencrypt cert after root expiration

Post by kdmiller45 »

This is the error I get when I run the command
Attachments
certbot error.gif
certbot error.gif (6.57 KiB) Viewed 12444 times
User avatar
barrydegraaff
Zimbra Employee
Zimbra Employee
Posts: 242
Joined: Tue Jun 17, 2014 3:31 am
Contact:

Re: Installing Letsencrypt cert after root expiration

Post by barrydegraaff »

Everyone uses Let's Encrypt in a different way... that is why on the wiki I only wrote the options you need to make it work on Zimbra. You can combine this information with whatever generic Letsencrypt guide you can find on the internet.

I will tell you how I do it, so you can look at it and hopefully it helps you.

I have a separate VM that has a cronjob that requests wildcard certificates via DNS validation, it is not the easiest solution, but the best if you need multiple sub domains, which I do need.

Code: Select all

/usr/local/bin/certbot --manual --expand --manual-auth-hook /usr/local/sbin/hook.sh --manual-cleanup-hook /usr/local/sbin/cleanhook.sh --preferred-challenges dns -d "barrytest.tk" -d "*.barrytest.tk" -d "zimbratest.tk" -d "*.zimbratest.tk" certonly --manual-public-ip-logging-ok -n
/usr/local/sbin/hook.sh this script is custom and it adds a TXT record to a bind DNS server and reloads bind

Code: Select all

#!/usr/bin/env bash

/usr/bin/ssh root@dnsserver "/usr/bin/echo '_acme-challenge   TXT    \"$CERTBOT_VALIDATION\"' >> /etc/bind/zones/$CERTBOT_DOMAIN.db"

#increment serials, sync and restart
/usr/bin/ssh root@dnsserver '/usr/bin/sed -i -e "s/[0-9]\{10\}/$(date +%10s)/" /etc/bind/zones/*.db'
/usr/bin/ssh root@dnsserver '/usr/bin/systemctl restart named'

#maybe a smaller sleep is fine, but yeah whatever
sleep 120
exit 0

/usr/local/sbin/cleanhook.sh removes challenge records

Code: Select all

#!/usr/bin/env bash

#remove challenge records
/usr/bin/ssh root@dnsserver '/usr/bin/sed -i -e "/acme-challenge/d" /etc/bind/zones/*.db'

#increment serials, sync and restart
/usr/bin/ssh root@dnsserver '/usr/bin/sed -i -e "s/[0-9]\{10\}/$(date +%10s)/" /etc/bind/zones/*.db'
/usr/bin/ssh root@dnsserver '/usr/bin/systemctl restart named'
exit 0
So if you do not use BIND, you have to alter the hook scripts, some DNS providers have native support in certbot, which you can google for and find the letsencrypt documentation.
--
Barry de Graaff
Email: barry.degraaff [at] synacor [dot] com
Admin of Zimbra-Community Github: https://github.com/orgs/Zimbra-Community/ and the
Zimlet Gallery https://gallery.zetalliance.org/extend/
Post Reply