Letsencrypt after October 1

Discuss your pilot or production implementation with other Zimbra admins or our engineers.
elderf
Advanced member
Advanced member
Posts: 65
Joined: Sat Sep 13, 2014 12:10 am

Letsencrypt after October 1

Post by elderf »

I have seen various methods in various forums, but so far none of them work for me.

Basically I have two doubts ..

What is the simplest method to install letsencrypt in zimbra on new servers after October 1st?

I have used cerbot + snap + this documentation and it does not work

https://wiki.zimbra.com/wiki/Installing ... tificate?c

Also this other method does not work for me in new installations
https://wiki.zimbra.com/wiki/JDunphy-Letsencrypt

Facilities with certificates issued before September 30 and still working, what is the new method to renew the certificate?

Thank you and I appreciate your help.
User avatar
JDunphy
Outstanding Member
Outstanding Member
Posts: 889
Joined: Fri Sep 12, 2014 11:18 pm
Location: Victoria, BC
ZCS/ZD Version: 9.0.0_P39 NETWORK Edition

Re: Letsencrypt after October 1

Post by JDunphy »

On any new installations and with newer versions of the acme.sh bash script, you must force it to letsencrypt as they changed the default to zerossl so using any other CA will fail the validation step when you give it to zmcertmgr. The description on these forums and wiki's is for letsencrypt. The default certificate authority for acme.sh changed to zerossl in the summer which I believe started with version 3.0+ of the bash script. The solution is simple enough. You run this one time after installing acme.sh and before you issue your first certificate.

Code: Select all

cd ~/.acme.sh
./acme.sh --set-default-ca --preferred-chain "ISRG" --server letsencrypt
If you can generate a valid certificate, then the zimbra step is trivial. You add the ISG_X1.pem to fullchain.cer before running it through zmcertmgr for validation. Previously it was IdenTrust.pem but they are now signing with ISG X1. That is all that changed from zimbra's perspective so any of the letsencrypt methods and acme clients will work if you add this cert since zimbra needs to validate the entire chain and is missing that ISG_X1.pem.

What isn't clear is what error you had. If you are saying validation doesn't work than you need to investigate why your fullchain.cer is missing ISG_X1.pem or if you have anything else present. I was helping someone recently that didn't realize that he was trying to validate an zeroSSL certificate with the letsencrypt procedures. Somewhere in the process he ended up with another cert in that fullchain file and removed it by hand solved his problem. acme.sh now support 4-5 different CA's but the wiki only describes how to work with letsencrypt with zimbra but any of the others would also work provided you substitute ISG_X1.pem with the corresponding cert from those CA's before having zmcertmgr validate the chain.

HTH,

Jim
User avatar
JDunphy
Outstanding Member
Outstanding Member
Posts: 889
Joined: Fri Sep 12, 2014 11:18 pm
Location: Victoria, BC
ZCS/ZD Version: 9.0.0_P39 NETWORK Edition

Re: Letsencrypt after October 1

Post by JDunphy »

Ignore what I said. You are correct... I just ran it on a new install with version 3.0.1 and it failed even after telling it no zeroSSL. Something new is going on with the latest version of acme.sh

The fix was removing that MIIFYDCCBEigA cert from fullchain.cer. I am going to investigate whether patching the code back to letsencrypt fixes this.

Jim
User avatar
JDunphy
Outstanding Member
Outstanding Member
Posts: 889
Joined: Fri Sep 12, 2014 11:18 pm
Location: Victoria, BC
ZCS/ZD Version: 9.0.0_P39 NETWORK Edition

Re: Letsencrypt after October 1

Post by JDunphy »

For new installations with acme.sh, there is going to be a little bit of a struggle. The reason is that you have not issued a certificate yet so the default is hard coded as zeroSSL in acme.sh. If you follow my instructions in the wiki then it will fail (at least until I change the wiki to reflect what I learned here). The solution they give is to:

Code: Select all

./acme.sh  --set-default-chain  --preferred-chain  ISRG  --server letsencrypt
which should find all the domains you attempted to create with letsencrypt and changed them to the ISG X1 signed chain. This isn't an ideal solution for someone just learning how to use this or install acme.sh

I think I have found the appropriate dance and workaround for new installations. The problem is that letsencrypt is returning the extra certificate in the fullchain.cer which is for compatibility unless you specify the exact chain which is the ISG X1 chain for zimbra; otherwise you can remove the last certificate and it will be the same as specifying the ISG X1 chain. Here is a complete solution assuming you know how to issue a certificate with acme.sh and modify account.conf should you use the DNS method with your account information so you don't have to understand the nuances.

Code: Select all

# su - 
# mkdir /opt/zimbra/.acme.sh
# chown zimbra:zimbra /opt/zimbra/.acme.sh
# su - zimbra
% cd /opt/zimbra/.acme.sh
% wget -O -  https://get.acme.sh | sh
% ./acme.sh --issue --dns dns_cf -d mail.example.com -d tmail.example.com --preferred-chain  ISRG --server letsencrypt
At this point it should be correct... One can tell because the older letsencrypt chain has 3 certificates and that will not validate with zimbra. There should be only 2 certs with the ISG X1 chain.

Code: Select all

% cd mail.example.com
% grep BEGIN fullchain.cer 
-----BEGIN CERTIFICATE----
-----BEGIN CERTIFICATE-----
Of course now you have to add the ISG_X1.pem certificate to fullchain.cer before attempting to validate it with zmcertmgr with your certificate. If you are using the deploy method and zimbra.sh with acme.sh then it does that for you; otherwise the wiki walks you through how to do this manually.

For older and existing acme.sh installations, one could simply do the following:

Code: Select all

% ./acme.sh  --set-default-chain  --preferred-chain  ISRG  --server letsencrypt
% ./acme.sh --renewAll     --force
and everything would work. That is because you already had a mail.example.com certificate directory using the example above which contains a file inside that directory called mail.example.com.conf that specified the default chain. With new installations that directory doesn't exist yet so the advice from the author of acme.sh doesn't help because any new installation would always try the zeroSSL as the default unless you specify it at the time of issue which is our workaround. I have a sed script on the wiki page to change the default in acme.sh back to letsencrypt but decided to take the authors word for it that using the --set-default-chain would be enough. It clearly isn't enough for the very first time you install acme.sh so specify that manually at the time of first certificate creation.

I hope this makes sense of what is happening. acme.sh and zimbra. Very frustrating for a lot of people including myself that thought it was working for everyone.

HTH,

Jim
stoffl
Posts: 3
Joined: Sat Mar 09, 2019 8:50 pm

Re: Letsencrypt after October 1

Post by stoffl »

Hi!

I'm just trying to renew my LetsEncrypt certificate on a centos 7 server.

I issued the command letsencrypt renew befor the certificate expired. But when I try to check the certificate all I get is

Code: Select all

ERROR: Unable to validate certificate chain: O = Digital Signature Trust Co., CN = DST Root CA X3
error 10 at 3 depth lookup: certificate has expired
The installed certificate expired at around7pm - but the renew command was run before.

What am I doing wrong?
User avatar
JDunphy
Outstanding Member
Outstanding Member
Posts: 889
Joined: Fri Sep 12, 2014 11:18 pm
Location: Victoria, BC
ZCS/ZD Version: 9.0.0_P39 NETWORK Edition

Re: Letsencrypt after October 1

Post by JDunphy »

stoffl wrote: I issued the command letsencrypt renew befor the certificate expired. But when I try to check the certificate all I get is

Code: Select all

ERROR: Unable to validate certificate chain: O = Digital Signature Trust Co., CN = DST Root CA X3
error 10 at 3 depth lookup: certificate has expired
The installed certificate expired at around7pm - but the renew command was run before.

What am I doing wrong?
I am not familiar with the acme client that you are using for renewal but there are 2 chains from letsencrypt currently. The older previous chain will not validate as the trust anchor has expired which will fail because zimbra validates the chain. You have 2 possible solutions as a workaround.

1) Find the fullchain or whatever it is called with the tool you are using to generate your certificates. There should be 2 certificates inside it but you have 3 in this case and most likely that last certificate is causing you problems since zimbra will not be able to validate it. Remove that certificate manually and then add the ISGR_X1 cert to the bottom so zimbra can validate and install it.

2) Re-issue the certificates but look for an option like --preferred-chain "ISRG" and use that with whatever acme client you are using to issue certificates. This will create a fullchain file that only has 2 certificates (basically doing what workaround 1 above did.

HINT: grep BEGIN fullchain to see the number of certificates present in your fullchain file (in the acme.sh letsencrypt client we call it: fullchain.cer)

The acme clients if they don't specify the ISRG X1 chain will get the older previous chain which is still the default from letsencrypt. Why would letsencrypt default to a chain with a trust anchor date that has expired? Because really old old old old android devices (did I say old) do not have the newer ISRG X1 cert with no upgrade option for those android devices. Android doesn't validate the date in the trust anchor in a certificate so browsers work fine with this older certificate chain as do the server software like nginx/apache/ldap/etc. Zimbra does validate the chain as part of the installation process so we don't use the older chain nor need to because letsencrypt has been out long enough that all recent browsers in the past 10 years no longer require the previous cross signed chain to validate the certificate. The solution is for us to specify the directly signed chain from letsencrypt and not use the older cross signed chain.

HTH,

Jim
stoffl
Posts: 3
Joined: Sat Mar 09, 2019 8:50 pm

Re: Letsencrypt after October 1

Post by stoffl »

ATM I'm updating Letsencrypt manually...

I'm totally new to acme.

What is the reccommended way?

Thanks.
User avatar
JDunphy
Outstanding Member
Outstanding Member
Posts: 889
Joined: Fri Sep 12, 2014 11:18 pm
Location: Victoria, BC
ZCS/ZD Version: 9.0.0_P39 NETWORK Edition

Re: Letsencrypt after October 1

Post by JDunphy »

zimbra likes this method and posted this in all the letsencrypt threads recently to help with the transition: https://wiki.zimbra.com/wiki/Installing ... ertificate

I prefer something simpler like acme.sh (single bash script) which I showed in this thread above your comment including the installation of the acme client and the issue of the certificate if one was using DNS for the validation method. I didn't show the installation of the certificate which is another command line

Code: Select all

# su - zimbra
% ./acme.sh --deploy --deploy-hook zimbra -d mail.example.com
The deploy hook calls a script called zimbra.sh that validates and installs the certificate including adding that ISRG X1 cert to your chain if it's missing. In the acme.sh bash script method, the installation of acme.sh also installs a cron entry that will renew and automate this renewal process every 60 days so you don't need to do anything in the future if you have added that zimbra.sh to the deploy directory in the acme.sh install directory.

It really doesn't matter what letsencrypt acme client as they all will require you to add that ISRG Root X1 cert to the fullchain. A simple concat to the fullchain is all that is required to make your chain validate. Pick a client that you are comfortable with.

So I wouldn't change the client if you have a previous working solution. It is trivial to install any letsencrypt certificate after you have a valid fullchain certificate file. If you want a simple explanation and can read bash code... here is the snippet from zimbra.sh that I reference in the deploy-hook above that does the install of the certificate:

Code: Select all

# Ref: zimbra.sh  which is installed in /opt/zimbra/.acme.sh/deploy/
# wget 'https://github.com/JimDunphy/acme.sh/blob/master/deploy/zimbra.sh' -O zimbra.sh

# Zimbra's still needs CA pem to verify on some versions
   ISG_X1="$(dirname "$_cca")/../ISG_X1.pem"
   _debug ISG_X1 "$ISG_X1"

   # grab root pem if we don't have it
   if [ ! -f "$ISG_X1" ]; then
      _debug No "$ISG_X1"
      wget -q "https://letsencrypt.org/certs/isrgrootx1.pem.txt" -O "$ISG_X1" || return 1
   fi

   # append root pem so verifycrt can walk the chain
   cat "$_cfullchain" "$(dirname "$_cca")/../ISG_X1.pem" > "${_cca}.real"
   /opt/zimbra/bin/zmcertmgr verifycrt comm "$_ckey" "$_ccert" "${_cca}.real" || return 1

   #if it verifies we can deploy it
   $(which logger) -p local2.info NETWORK "Certificate has been Renewed for $_cdomain"
   cp -f "$_ckey" /opt/zimbra/ssl/zimbra/commercial/commercial.key
   /opt/zimbra/bin/zmcertmgr deploycrt comm "$_ccert" "${_cca}.real" || return 1
   #/opt/zimbra/bin/ldap restart
   #/opt/zimbra/bin/zmmailboxdctl reload
   #/opt/zimbra/bin/zmproxyctl reload
   #/opt/zimbra/bin/zmmtactl reload
   /opt/zimbra/bin/zmcontrol restart
   return 0
Jim
User avatar
JDunphy
Outstanding Member
Outstanding Member
Posts: 889
Joined: Fri Sep 12, 2014 11:18 pm
Location: Victoria, BC
ZCS/ZD Version: 9.0.0_P39 NETWORK Edition

Re: Letsencrypt after October 1

Post by JDunphy »

To bring closure on this. The acme.sh letsencrypt client will work with NEW installs and OLD but you should do this one time.

Code: Select all

# su - zimbra
% cd /opt/zimbra/.acme.sh
% ./acme.sh --set-default-ca  --server letsencrypt
% ./acme.sh  --set-default-chain  --preferred-chain  ISRG  --server letsencrypt
The first command will modify account.conf file with a DEFAULT_ACME_SERVER entry so that letsencrypt will always be the default CA and not zeroSSL. Lastly it will make sure you only grab the letsencrypt ISRG X1 chain by adding an entry DEFAULT_PREFERRED_CHAIN to ca.conf in the ca directory. Everything else remains the same as it was before. Issue your certs, add the ISRG X1 certificate to the fullchain.cer, validate and install it with zmcertmgr.
elderf
Advanced member
Advanced member
Posts: 65
Joined: Sat Sep 13, 2014 12:10 am

Re: Letsencrypt after October 1

Post by elderf »

https://wiki.zimbra.com/wiki/JDunphy-Letsencrypt

So for those of us who are not so experts, this method continues to serve new installations with the installation you are commenting on, or is it necessary to add the additional information you comment on in the thread?

In the case of certificates issued with certbot and the old method, can they be renewed with this method?
Post Reply