Another Letsencrypt method

[JDunphy]

Thought I would share a method that I prefer over installing and allowing the letsencrypt software to take over my zimbra host. For those users that don't want to install loads of python code and other associated and required software, this might be preferable. It does not require root.
I had looked at this https://wiki.zimbra.com/wiki/Installing ... ertificate but wanted a less intrusive method.

UPDATE: 7/19/2018 This is condensed into a wiki article - https://wiki.zimbra.com/wiki/JDunphy-Letsencrypt
BEGIN: TL;DR All in one method using automatic dns validation. Can use any validation method in place of DNS.

Code: Select all

% su - zimbra
# This will append a cron entry to zimbra's crontab and create /opt/zimbra/.acme.sh with the acme.sh bash script.
% wget -O -  https://get.acme.sh | sh
% cd .acme.sh

#will create or renew certs in /opt/zimbra/.acme.sh/
% ./acme.sh --issue --dns dns_cf -d mail.example.com -d mail.example.net -d mail.example.org -d '*.wildcard.com'

# will deploy into zimbra
% ./acme.sh --issue --deploy --deploy-hook zimbra -d mail.example.com 

Future renewals will happen automatically every 60 days and be loaded going forward without further intervention.
END: TL;DR

This method requires very little software to be installed (bash script only) and can be done on an external machine if you want. It uses the acme protocol with the DNS method for verification. The first time you run it, it will provide some DNS entries you are required to add and then can verify against that. After this initial verification step, one can remove those dns entries. See this https://github.com/Neilpang/acme.sh for full documentation and other methods such as stand alone.

The process goes like this.

Code: Select all

git clone https://github.com/Neilpang/acme.sh
cd acme.sh
./acme.sh --install

Note: the above install command creates a directory ../.acme.sh so you don't need to be root. (Generally, that is $home/.acme.sh if you are running this from your own account which I do.

You can copy this .acme.sh directory to your production machines later if you are simply testing the methodology. I tend to test everything on a staging server first so that is my method.
Continuing...

Code: Select all

cd ..
source $home/.acme.sh/acme.sh.csh or simply login/logout first time if you are not sure. I just source .cshrc myself.
acme.sh --issue --dns -d mail.example.com -d mail.example.net -d mail.example.org -d tmail.example.com

Copy the bind entries that the above acme.sh script spits out and put them in your zone files and reload your zones. BTW, I tend to make the first -d entry my zmhostname.
Continuing...

UPDATE 3/29/2018: If you renew within 29 days, you will not go through the request/challenge. See my post on page 3 and 7 later explaining why and options. If you do that, you will need to add --force because acme.sh doesn't think it is time to renew. The Certificates are still good for 90 days. With the newer acme.sh software from 2018 or later it now requires you to add this switch --yes-I-know-dns-manual-mode-enough-go-ahead-please. The author wants to alert you to these changes by letsencrypt. Long term solution: use one of the 2 automatic DNS methods or the standalone server verification methods.

Code: Select all

acme.sh --renew -d mail.example.com -d mail.example.net -d mail.example.org -d tmail.example.com

Your certs are now verified and installed in $home/.acme.sh/mail.example.com

Here is what the directory structure looks like after you have executed the acme.sh script:

Code: Select all

testmail:~:43> cd .acme.sh
testmail:~/.acme.sh:44> ls
account.conf  acme.sh.csh  ca      dnsapi 
acme.sh       acme.sh.env  deploy  http.header  mail.example.com

testmail:~/.acme.sh:45> cd mail.example.com/

testmail:~/.acme.sh/mail.example.com:46> ls
ca.cer               mail.example.com.conf      mail.example.com.key
fullchain.cer        mail.example.com.csr
mail.example.com.cer  mail.example.com.csr.conf

Now its just a matter of testing them and/or installing them.
Continuing...

First time, we need to create a cross-signed IdentTrust CERT so that zimbra can verify our certificate chain properly. It is who gives LetEncrypt authority to sign these free certs.
I put the following CERT into a file named IdentTrust.pem so I can use this the next time I have to renew my certs. It comes from here: https://www.identrust.com/certificates/ ... ad-x3.html
Clarification of why this extra step:

Our intermediate is signed by ISRG Root X1. However, since we are a very new certificate authority, ISRG Root X1 is not yet trusted in most browsers. In order to be broadly trusted right away, our intermediate is also cross-signed by another certificate authority, IdenTrust, whose root is already trusted in all major browsers. Specifically, IdenTrust has cross-signed our intermediate using their DST Root CA X3.

Continuing...

Code: Select all

Create a file called $home/.acme.sh/IdentTrust.pem
-----BEGIN CERTIFICATE-----
MIIDSjCCAjKgAwIBAgIQRK+wgNajJ7qJMDmGLvhAazANBgkqhkiG9w0BAQUFADA/
MSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT
DkRTVCBSb290IENBIFgzMB4XDTAwMDkzMDIxMTIxOVoXDTIxMDkzMDE0MDExNVow
PzEkMCIGA1UEChMbRGlnaXRhbCBTaWduYXR1cmUgVHJ1c3QgQ28uMRcwFQYDVQQD
Ew5EU1QgUm9vdCBDQSBYMzCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB
AN+v6ZdQCINXtMxiZfaQguzH0yxrMMpb7NnDfcdAwRgUi+DoM3ZJKuM/IUmTrE4O
rz5Iy2Xu/NMhD2XSKtkyj4zl93ewEnu1lcCJo6m67XMuegwGMoOifooUMM0RoOEq
OLl5CjH9UL2AZd+3UWODyOKIYepLYYHsUmu5ouJLGiifSKOeDNoJjj4XLh7dIN9b
xiqKqy69cK3FCxolkHRyxXtqqzTWMIn/5WgTe1QLyNau7Fqckh49ZLOMxt+/yUFw
7BZy1SbsOFU5Q9D8/RhcQPGX69Wam40dutolucbY38EVAjqr2m7xPi71XAicPNaD
aeQQmxkqtilX4+U9m5/wAl0CAwEAAaNCMEAwDwYDVR0TAQH/BAUwAwEB/zAOBgNV
HQ8BAf8EBAMCAQYwHQYDVR0OBBYEFMSnsaR7LHH62+FLkHX/xBVghYkQMA0GCSqG
SIb3DQEBBQUAA4IBAQCjGiybFwBcqR7uKGY3Or+Dxz9LwwmglSBd49lZRNI+DT69
ikugdB/OEIKcdBodfpga3csTS7MgROSR6cz8faXbauX+5v3gTt23ADq1cEmv8uXr
AvHRAosZy5Q6XkjEGB5YGV8eAlrwDPGxrancWYaLbumR9YbK+rlmM6pZW87ipxZz
R8srzJmwN0jP41ZL9c8PDHIyh8bwRLtTcm1D9SZImlJnt1ir/md2cXjbDaJWFBM5
JDGFoqgCWjBH4d1QB7wCCZAA62RjYJsWvIjJEubSfZGL+T0yjWW06XyxV3bqxbYo
Ob8VZRzI9neWagqNdwvYkQsEjgfbKbYK7p2CNTUQ
-----END CERTIFICATE-----

We will use this file going forward to build a complete chain.
Continuing...

Code: Select all

cd $home/.acme.sh/mail.example.com
cat ../IdentTrust.pem >> fullchain.cer

At this point our directory structure looks like this:

Code: Select all

testmail:~/.acme.sh:47> ls
account.conf  acme.sh.csh  ca      dnsapi       IdentTrust.pem  
acme.sh       acme.sh.env  deploy  http.header  mail.example.com

So the process going forward every time you generate a NEW or renewed cert and want to verify it is:

Code: Select all

$home/.acme.sh/acme.sh --cron --home $home/.acme.sh    # this could generate a new cert if its time
cd $home/.acme.sh/mail.example.com
cat ../IdentTrust.pem >> fullchain.cer # only if you have new or renewed certs in mail.example.com directory
su zimbra -c "/opt/zimbra/bin/zmcertmgr verifycrt comm mail.example.com.key mail.example.com.cer fullchain.cer"

(NOTE: you don't need to append the IdentTrust.pem to the fullchain if you haven't renewed your key... So your fullchain always will have 3 PEM's in it only). Pay special attention to --force if you are doing manual DNS entries and not the automatic DNS method provided by acme.sh

If that looks like zimbra likes the cert than you can deploy it as in:

I like to backup first.

Code: Select all

# do this as the zimbra user
cd /opt/zimbra/ssl
tar cvf zimbra.tar.$(date "+%Y%m%d") zimbra

Here we go... no turning back unless we have some sort of backup tar image.
Note: zmcertmgr is running as zimbra for 8.7+ ... While I haven't tested with 8.6, you would run it as root for those earlier releases.
WARNING: Because zmcertmgr is running as zimbra, make sure that zimbra can read the $home/.acme.sh and its associated files.

Code: Select all

cd $home/.acme.sh
su zimbra -c "cp mail.example.key /opt/zimbra/ssl/zimbra/commercial/commercial.key"
su zimbra -c "/opt/zimbra/bin/zmcertmgr deploycrt comm mail.example.cer fullchain.cer"
zmcontrol restart

When it is time to renew, you need to regenerate your certificate by issuing the simple command below and repeat the steps from creating that fullchain.cer

Code: Select all

$home/.acme.sh/acme.sh --cron --home $home/.acme.sh 

You can run this as often as you like. The variable MAX_RENEW inside the acme.sh scripts determines when it will get a new cert. If you want to force it, use the --force option and/or modify that variable.

I know this is long but its fairly simple and you don't need to be root or be on the same production host to try it out. You can copy the .acme.sh to your production machine and see if zimbra likes your cert.
The days of self-signed certs are gone. I think it took me more time to write this than create a simple script to do all this.

BTW, You should also do this: https://wiki.zimbra.com/wiki/How_to_obt ... urity_Test so that sslabs gives you an A+ score for your cert.

Code: Select all

zmprov mcf +zimbraResponseHeader "Strict-Transport-Security: max-age=31536000"

% zmprov mcf zimbraReverseProxySSLCiphers 'ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128:AES256:HIGH:!aNULL:!eNULL:!EXPORT:!DES:!MD5:!PSK:!RC4'

% zmcontrol restart

Hope this helps as its going to getting harder and harder to use self-signed certs with many browsers and these letsencrypt free certs are really great!

UPDATE: 2/7/2017. See: https://github.com/JimDunphy/deploy-zim ... encrypt.sh for updated method that is simpler and allows other acme.sh methods including standalone that makes deploying these certificates easier to learn.

[phoenix]

I've been putting off getting a letsencrypt certificate simply because of the hoops you had to jump through getting it installed. Your write-up seems much more straightforward so I'll give this a go in the next few weeks, thanks for posting this.

[JDunphy]

I've been putt ing off getting a letsencrypt certificate simply because of the hoops you had to jump through getting it installed. Your writeup seems much more straightforward so I'll give this a go in the next few weeks, thanks for posting this.

That is what stopped me Too.

Before I knew it the letsencrypt installation had brought in a full development environment then asking me to take an outage because I needed to allow either port 80 or 443 access to their bot for verification. Hard to get started when that is going on behind the scenes and its your first time. This is a little dirtier but you control the process.

With the write-up above, here is my quick and dirty script I wrote the posting from. It has some checks but not a lot. It works for our environment and I will use it to renew the certs in 90 days when I forgot what I did.
It will require you to do 2 global replacements and comment out an exit before it will run. No one should run this without having your certs verified by the method above. I don't want anyone's zimbra server to stop working.

Perhaps with the write-up and the script, it makes the process clearer. You run it as root because it needs to su to zimbra since these instructions are for 8.7+ and with /opt/zimbra/bin/zmcertmgr deploycrt runs as zimbra and not root.

Hint: The debug function is there so I can verify each step of the way in another window. Super paranoid about replacing certs with zimbra. hahahaha

Code: Select all

#!/bin/bash

#
# Run as root
#
#   Assumes that acme.sh has been installed and has been run initially to 
#   create all your certs. They are verified.
#
#   https://github.com/Neilpang/acme.sh or see README.acme
#
#          and see write up
#   in forum.zimbra.com under "Another Letsencrypt Method"

#---------------------------------------------------------------------
# DO NOT RUN before doing this:
#
# global replace ... XXX to user that installed and ran $home/.acme.sh
# global replace ... mail.example.com to domains
#
# NOTE: if you are renewing... replace step1 with following syntax as outlined in the initial post or you will have to
#    add your DNS txt records again.
# In other words, you want acme.sh --renew -d XXX -d YYY -d ZZZ 
# 
# %%% Also, verify that the directory mail.example.com and its files
#        can be read by the zimbra user. If you are using 8.6 then zmcertmgr needs to be run as root so verify if you 
#       umask creates really secure files in your $home/.acme.sh/mail.example.com location, etc.
#  zmcertmgr will attempt to chdir to cwd during deployment but validation can still pass in some circumstances.  bug: 107454
#
echo "please read this script and then comment these next 2 lines out"
exit 0
#---------------------------------------------------------------------

certs=/home/XXX/.acme.sh/mail.example.com/
zimbra_certs=/opt/zimbra/ssl

# %%% --- we will kick this off by hand initially
#su zimbra -c "/opt/zimbra/bin/zmcertmgr checkcrtexpiration -days 30 > /dev/null"
#if [ $? == 0 ]; then
#       exit 0;
#fi

debug() {
	echo $1
	read var
} 

#
# Step 1
#
# Note: there is a min value of 80 so can reduce and issue the force.
# UPDATE 1/30/2017: Code has changed and this is no longer true with 2.6.6. Run this script more frequently than 59 days or automotically
#    so you don't go through the request/challenge again.
# %%%% If you are renewing... change syntax completely with --renew -d XXX -d yyy as outline in initial post in this thread
su XXX -c '/home/XXX/.acme.sh/acme.sh --force --cron --home /home/XXX/.acme.sh | grep "END CERTIFICATE"'
#su XXX -c '/home/XXX/.acme.sh/acme.sh --cron --home /home/XXX/.acme.sh | grep "END CERTIFICATE"'

# return 1 if didn't generate a new certificate
if [ $? == 1 ]; then
   echo Did not renew
   exit 1
fi

#debug "wait Please"

#
# Step 2 - backup old
#
#as zimbra
# Backup old location
cd $zimbra_certs
su zimbra -c "tar cvf zimbra.tar.$(date "+%Y%m%d") zimbra"

#debug "wait Please"

#
# Step 3 - verify cert
#
#as zimbra verify cert
cd "$certs"
# append IdentTrust ca to chain (we added the BEGIN/END statements)
su XXX <<-EOF
cat ../IdentTrust.pem >> fullchain.cer 
EOF
su zimbra -c "/opt/zimbra/bin/zmcertmgr verifycrt comm mail.example.com.key mail.example.com.cer fullchain.cer"
if [ $? == 1 ]; then
   echo "bad cert???"
   exit 1
fi

#debug "wait Please ... did you verify zimbra has rad permission"

# Step 4 - Deploy to Zimbra

# %%% NOTE: verify that $cert is readable before 
# copy private key 
cd $certs
su zimbra -c "cp mail.example.com.key /opt/zimbra/ssl/zimbra/commercial/commercial.key"
if [ $? == 1 ]; then
   echo "permission problem"
   exit 1
fi

#debug "wait Please"

# as zimbra
cd $certs
su zimbra -c "/opt/zimbra/bin/zmcertmgr deploycrt comm mail.example.com.cer fullchain.cer"
su - zimbra <<-'EOF'
   zmcontrol restart
EOF

[phoenix]

Thanks for the update and additional info. Certificates are not my favourite things, it always seems too easy to screw it up but I must finally get my head around this subject. Anything that makes my life easier and a complicated subject easier to manage and understand is highly appreciated by me.

I've also made this a sticky so others can profit from it and add further to the discussion.

[JDunphy]

I ran this script today with very little thought. It had worked perfectly on my test boxes and the few production box's I have. This morning, I completed my last 8.6+ upgrade to 8.7.1.
I commented out the last debug statement prior to zmcertmgr so I could abort should the cert not verify. Cert verified so I hit return.

That led to:

Code: Select all

zmcertmgr: ERROR deploycrt(comm mail.example.com.cer fullchain.cer) failed:
 chdir(/home/XXX/.acme.sh/mail.example.com) failed: Permission denied

followed by this scary message:

Code: Select all

Host mail.example.com
	Starting ldap...Done.
Unable to start TLS: SSL connect attempt failed error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed when connecting to ldap master.

My environment was different on this production box and the home directory had 700 permissions which I didn't realize until I saw the above message. I had verified my mail.example.com directory and its files had proper permissions but failed to look at the complete path. The fix:

Code: Select all

chmod 755 $home
and re-ran the above script.

Perfect and working. I point this out so that others don't do what I did. If you are going to run zmcertmgr as zimbra in 8.7+, one might want to su to zimbra first and to verify you have read access to your certs and prevent this ugly situation. I'll add some more checks so I don't do that to myself again. The script was full of warning about this and I still failed to pay attention. One of those mornings..

[centrex]

I am using https://github.com/VojtechMyslivec/letsencrypt-zimbra with great success on ZCS 8.7.1, Ubuntu 14.04 LTS

[JDunphy]

I am using https://github.com/VojtechMyslivec/letsencrypt-zimbra with great success on ZCS 8.7.1, Ubuntu 14.04 LTS

Yes! Thank you for that link.

I had studied that code first and it was going to be my method until the initial verification process. With letsencrypt for those that don't know, you have to prove that you control your domain before you can get your certs signed. The normal methods is to have them connect back to your domains (ie. those -d mail.example.com) at port 80 or port 443. That means with zimbra, you would need to shut those services down so they can prove you are who you say you are during that step. There after, you just renew as they expire every 90 days. They provide various methods to make that easy which is why you install quite a bit of code when you initially install letsencrypt.

Another method and the point of my original post was to show the method for the verification process using the acme protocol and the DNS method without installing letsencrypt. Once you get your domains initially verified then its fairly easy to renew the certs via scripts like the link you provided or the zimbra command 'zmcertmgr'. The key in all methods is to recognize running zmcertmgr as the correct user depending on zimbra version and copying the private key. For centos 6 or RHEL 6 zimbra admins, python versioning with a lot of warning about not supported for the letsencrypt certbot can give you some initial pause if you press through on those OS releases.

I hope the take away from this is that those letsencrypt certs are very good. I have studied their process and like the little things that many paid for certs do not do... ie. "they publishes revocation information into the normal revocation channels (i.e., CRLs, OCSP), so that relying parties such as browsers can know that they shouldn’t accept the revoked certificate".

It is extremely trivial once you see or 'get' the process. (Assumes you have verified your initial certs):

• renew cert
• verify cert with zimbra
• cp private key
• zmcertmgr to install it
• restart zimbra

It would be really great if the self-signed certs went away or they added another option to the zimbra console gui handling that initial letsencrypt verification using this acme protocol and its associated web challenge. That should provide minimal additional code and would only require a small snippet added to nginx. On restart, we could blow that line away. Out of the box, that would make zimbra so easy to setup and appealing to more admins one would think. I might try hacking at it to understand what we are up against. I know that the acme.sh script I mentioned will generate this token challenge without any additional code to be present so I need to look closer how it was done. For many, signed certs are a big hassle and the browsers are going to make self-signed really difficult for their users beginning in Jan 2017. Once one moves their zimbra certs to letsencrypt, it really is automatic the renewal and you don't need to go to that screen again. Then we begin to focus on CA's that have been compromised and looking for certs issued for your domains by others.

Code: Select all

# Allow access to the letsencrypt ACME Challenge ... replace with string from acme.sh script
location ~ /\.well-known\/acme-challenge {
    allow all;
}

[shockwavecs]

followed by this scary message:

Code: Select all

Host mail.example.com
	Starting ldap...Done.
Unable to start TLS: SSL connect attempt failed error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed when connecting to ldap master.

Can anyone verify that you can simply run the

Code: Select all

/opt/zimbra/libexec/zmfixperms -extended

after this comes up and it will fix the issue?

[JDunphy]

followed by this scary message:

Code: Select all

Host mail.example.com
	Starting ldap...Done.
Unable to start TLS: SSL connect attempt failed error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed when connecting to ldap master.

Can anyone verify that you can simply run the

Code: Select all

/opt/zimbra/libexec/zmfixperms -extended

after this comes up and it will fix the issue?

Guessing how/what you did but assuming you are following along with this thread, this may apply.

I am thinking zmfixperms will not help. If all you did was use zmcertgmr and you did it as the correct user depending on your version of zimbra then probably not...When I got that message from zmcertmgr, it failed because it wasn't able to read my cert to make the copy. My first guess would be to verify the path to the certs where zmcertmgr is running as the zimbra user and make sure your full chain has 3 certs. My second guess is to recheck with verify the certs you created from the command line.

Code: Select all

cd wherever_your_certs_are
grep BEGIN fullchain.cer 
-----BEGIN CERTIFICATE-----
-----BEGIN CERTIFICATE-----
-----BEGIN CERTIFICATE-----

If that looks good then you should be able to do this as the zimbra user if you are 8.7+ or root if you are 8.6

Code: Select all

/opt/zimbra/bin/zmcertmgr verifycrt comm mail.example.com.key mail.example.com.cer fullchain.cer

If and only if that works would you move to the next step and use zmcertmgr to perform the deployment which copies the certs. For me, zimbra did not have access to my certs so running that zmcertmgr verifycrt should alert you immediately if that will be the case. If it verifies, you should be able to do (zimbra or root depending on version you are running):

Code: Select all

cp mail.example.key /opt/zimbra/ssl/zimbra/commercial/commercial.key
ls -l /opt/zimbra/ssl/zimbra/commercial/commercial.key 
-rw-r--r-- 1 zimbra zimbra 1679 Nov  6 04:47 /opt/zimbra/ssl/zimbra/commercial/commercial.key

/opt/zimbra/bin/zmcertmgr deploycrt comm mail.example.cer fullchain.cer

And then restart zimbra.

If the verify fails, then look at how you created the IdentTrust.pem and make sure its at the end of the fullchain.cer

[shockwavecs]

awesome! thank you. it works