Another Letsencrypt method

Discuss your pilot or production implementation with other Zimbra admins or our engineers.
User avatar
JDunphy
Outstanding Member
Outstanding Member
Posts: 889
Joined: Fri Sep 12, 2014 11:18 pm
Location: Victoria, BC
ZCS/ZD Version: 9.0.0_P39 NETWORK Edition

Re: Another Letsencrypt method

Post by JDunphy »

I was able to test this a little more and while the script works perfectly for installation it fails to reload the ldap certificate and gives a false sense that everything worked perfectly. As a result - some point in the future that running ldap process will have an expired certificate. That causes a lot of side effects with stop/restarts/status etc. If you restart zimbra or reboot your hosts before the expiration then one might not notice this because the updated cert would have been reloaded. I have updated the wiki to reflect this code change. Too bad because restarting/reloading did shave a little time off the outage to update the certificate.

Note: Given how badly an expired ldap certificate behaves in this failure mode, I am going with the full restart vs finessing the addition of an ldap restart/reload to those other 3 restart/reloads myself.
User avatar
JDunphy
Outstanding Member
Outstanding Member
Posts: 889
Joined: Fri Sep 12, 2014 11:18 pm
Location: Victoria, BC
ZCS/ZD Version: 9.0.0_P39 NETWORK Edition

Re: Another Letsencrypt method

Post by JDunphy »

FYI - reminder if anyone is doing TLS-SNI-01 validation instead of the others methods such as DNS, HTTP, etc that it is deprecated. Here is the official wording:
Let’s Encrypt soon will disable support for the TLS-SNI-01 domain validation method in the ACME protocol. In January of last year, a vulnerability in TLS-SNI-01 was discovered by Frans Rosén from Detectify. The deprecation will likely cause problems for users of some stable Linux distributions.

TLS-SNI-01 requires a user to temporarily serve a certificate with a special, invalid domain name via the TLS SNI extension. However, under many cloud provider’s settings, it’s possible for users to exploit this scenario and get positive validation for domains hosted by other users at the same cloud provider. This affected Heroku and Amazon CloudFront, for example.

Let’s Encrypt decided that this inherent vulnerability of the TLS-SNI-01 method is too much of a risk and therefore to deprecate it fully. But until now, there was still an exception in place for some providers and for certificate renewals.

The final deadline for TLS-SNI-01 is February 13, 2019, after which all current setups using this method will stop working. Let’s Encrypt certificates have a relatively short lifetime of ninety days, and it heavily relies on automated renewal. Let’s Encrypt sent out warning emails in recent weeks to those who still use TLS-SNI-01, but not all users will get them because providing an email address isn’t mandatory to use Let’s Encrypt.
If you do anything odd with your firewall rules like blocking port 80, your automatic renewal could fail if you are expecting TLS/443 access with certbot.

Note: If you are using acme.sh with DNS validation it will continue to work. Another great use of this validation method - it works for servers even on RFC1918 address space such as home zimbra instances and plex media servers. So give it a try if you are tired of getting those untrusted cert errors. :-)
phoenix
Ambassador
Ambassador
Posts: 27272
Joined: Fri Sep 12, 2014 9:56 pm
Location: Liverpool, England

Re: Another Letsencrypt method

Post by phoenix »

I seem to be having a problem with the install of the acme script as the zimbra user, if I run the command as that user I get the following permission error (the curl method also fails with the same error)::

Code: Select all

[zimbra@mail01 ~]$ wget -O -  https://get.acme.sh | sh
--2019-03-23 21:20:28--  https://get.acme.sh/
Resolving get.acme.sh (get.acme.sh)... 2607:5300:201:3100::5663, 144.217.161.63
Connecting to get.acme.sh (get.acme.sh)|2607:5300:201:3100::5663|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 705 [text/plain]
Saving to: 'STDOUT'

100%[======================================>] 705         --.-K/s   in 0s      

2019-03-23 21:20:28 (147 MB/s) - written to stdout [705/705]

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  174k  100  174k    0     0   516k      0 --:--:-- --:--:-- --:--:--  627k
[Sat Mar 23 21:20:28 CET 2019] Installing from online archive.
[Sat Mar 23 21:20:28 CET 2019] Downloading https://github.com/Neilpang/acme.sh/archive/master.tar.gz
sh: line 5827: master.tar.gz: Permission denied
[Sat Mar 23 21:20:28 CET 2019] Download error.
Am I missing something blindingly obvious here?
Regards

Bill

Rspamd: A high performance spamassassin replacement

Per ardua ad astra
User avatar
JDunphy
Outstanding Member
Outstanding Member
Posts: 889
Joined: Fri Sep 12, 2014 11:18 pm
Location: Victoria, BC
ZCS/ZD Version: 9.0.0_P39 NETWORK Edition

Re: Another Letsencrypt method

Post by JDunphy »

Hi Bill,

My guess is that /opt/zimbra is owned by root. That automatic method they like for install is problematic sometimes.

Here is the output from my own directory. Notice that it installs it in the users home directory (~/.acme.sh) and not the directory you are in.

Code: Select all

tmail:~:42> mkdir k
tmail:~:43> cd k
tmail:~/k:44> wget -O -  https://get.acme.sh | sh
--2019-03-23 15:43:54--  https://get.acme.sh/
Resolving get.acme.sh... 2607:5300:201:3100::5663, 144.217.161.63
Connecting to get.acme.sh|2607:5300:201:3100::5663|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 705 [text/plain]
Saving to: “STDOUT”

100%[=====================================================================================================================================================>] 705         --.-K/s   in 0s      

2019-03-23 15:43:54 (110 MB/s) - written to stdout [705/705]

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  174k  100  174k    0     0   469k      0 --:--:-- --:--:-- --:--:-- 1077k
[Sat Mar 23 15:43:55 PDT 2019] Installing from online archive.
[Sat Mar 23 15:43:55 PDT 2019] Downloading https://github.com/Neilpang/acme.sh/archive/master.tar.gz
[Sat Mar 23 15:43:55 PDT 2019] Extracting master.tar.gz
[Sat Mar 23 15:43:55 PDT 2019] Installing to /home/jad/.acme.sh
[Sat Mar 23 15:43:55 PDT 2019] Installed to /home/jad/.acme.sh/acme.sh
[Sat Mar 23 15:43:55 PDT 2019] Installing alias to '/home/jad/.bashrc'
[Sat Mar 23 15:43:55 PDT 2019] OK, Close and reopen your terminal to start using acme.sh
[Sat Mar 23 15:43:55 PDT 2019] Installing alias to '/home/jad/.cshrc'
[Sat Mar 23 15:43:56 PDT 2019] Installing cron job
[Sat Mar 23 15:43:56 PDT 2019] Good, bash is found, so change the shebang to use bash as preferred.
[Sat Mar 23 15:43:56 PDT 2019] OK
[Sat Mar 23 15:43:56 PDT 2019] Install success!
The nice thing is after it is installed you just do this to upgrade it.

Code: Select all

% acme.sh --upgrade
https://github.com/Neilpang/acme.sh
v2.8.1
[Sat Mar 23 15:50:02 PDT 2019] Installing from online archive.
[Sat Mar 23 15:50:02 PDT 2019] Downloading https://github.com/Neilpang/acme.sh/archive/master.tar.gz
[Sat Mar 23 15:50:03 PDT 2019] Extracting master.tar.gz
[Sat Mar 23 15:50:03 PDT 2019] Installing to /home/jad/.acme.sh
[Sat Mar 23 15:50:03 PDT 2019] Installed to /home/jad/.acme.sh/acme.sh
[Sat Mar 23 15:50:03 PDT 2019] Good, bash is found, so change the shebang to use bash as preferred.
[Sat Mar 23 15:50:04 PDT 2019] OK
[Sat Mar 23 15:50:04 PDT 2019] Install success!
[Sat Mar 23 15:50:04 PDT 2019] Upgrade success!
% ./acme.sh --version
https://github.com/Neilpang/acme.sh
v2.8.1
He just sped it up so that is really nice when you have 3 or 4 dozen certs to renew... that 120 seconds was killer per cert :-)
User avatar
JDunphy
Outstanding Member
Outstanding Member
Posts: 889
Joined: Fri Sep 12, 2014 11:18 pm
Location: Victoria, BC
ZCS/ZD Version: 9.0.0_P39 NETWORK Edition

Re: Another Letsencrypt method

Post by JDunphy »

I forgot to mention that I did a grafana.sh deploy method since I use that for zimbra.

Code: Select all

#!/bin/bash

#Here is a script to deploy cert to grafana server.

#returns 0 means success, otherwise error.

########  Public functions #####################

#domain keyfile certfile cafile fullchain
grafana_deploy() {
  _cdomain="$1"
  _ckey="$2"
  _ccert="$3"
  _cca="$4"
  _cfullchain="$5"

  _debug _cdomain "$_cdomain"
  _debug _ckey "$_ckey"
  _debug _ccert "$_ccert"
  _debug _cca "$_cca"
  _debug _cfullchain "$_cfullchain"

  cp -f "$_ckey" /etc/grafana/certs/certkey.key
  cp -f "$_ccert" /etc/grafana/certs/fullchain.cer

  return 0

}
Not terribly sophisticated but it gets the job done. Note: This shows a certificate issue/install when example.com doesn't have a DNS API but you still want to use the automatic DNS method nonetheless. It works because example.com has CNAME's for graphana and relay11 pointing to _acme-challenge.domainInCloudflare.com ... Where domainInCloudflare.com is managed by Cloudflare. There are now over 70 api's for automatic DNS insertion. See: dnsapi directory.

Code: Select all

#!/bin/sh

PATH=/bin:/usr/bin:/usr/sbin:/home/jad/bin export PATH

cd /home/jad/.acme.sh

# renewal and install
./acme.sh --force --issue --dns dns_cf --challenge-alias domainInCloudflare.com -d grafana.example.com -d relay11.example.com
# install now
./acme.sh --issue --deploy --deploy-hook grafana --dns dns_cf --challenge-alias domainInCloudflare.com -d grafana.example.com -d relay11.example.com
if [ $? == 1 ]; then
   echo "cert did not verify"
   exit 1
fi

#
/etc/init.d/grafana-server restart

exit 0
Labsy
Outstanding Member
Outstanding Member
Posts: 411
Joined: Sat Sep 13, 2014 12:52 am

Re: Another Letsencrypt method

Post by Labsy »

JDunphy,
you did huge job on the subject, but please, please rewrite the instructions on WIKI to be more clear. I went through few times but still I am not sure what is the proper way, what some step does and when to run as root and when as zimbra.

For example, you start the article by giving instructions to install acme.sh and I guess it assumes to be root. But few steps later you write one more time to install acme.sh as Zimbra user.
Another not clear thing is you have same headers and I went step by step, Intro, Requirements...done, then Issue your certificate...OK?...ok, then Install certificate with Zimbra...and ups here...now it's Step 1...aaaaaaaa?!! What are then those steps before? Did I mess up my server already?? Are there steps even BEFORE "Step 1"???
And why install IdentTrust SSL chain? Is this your favourite, or necessity?

I mean, you obviously just did not know how to write down all that info from your head and you probably edited Wiki articel many times. I understand, so I suggest you edit once more and make it more fool-proof :)
But in any case, huge excellent job! Thank you!
User avatar
JDunphy
Outstanding Member
Outstanding Member
Posts: 889
Joined: Fri Sep 12, 2014 11:18 pm
Location: Victoria, BC
ZCS/ZD Version: 9.0.0_P39 NETWORK Edition

Re: Another Letsencrypt method

Post by JDunphy »

I am not following what changes you are suggesting since I didn't ask you to use root anywhere. The entire reason I wrote this thread was so I didn't have to use root.

Here is how I do it... Everything as zimbra.

Code: Select all

% su - zimbra
% wget -O -  https://get.acme.sh | sh
% cd .acme.sh

#will create/renew certs in /opt/zimbra/.acme.sh/mail.example.com but will not install... Repeat this step until you have all your certs validated.
% ./acme.sh --issue --dns dns_cf -d mail.example.com -d mail.example.net -d mail.example.org

# will deploy certs to zimbra
% ./acme.sh --issue --deploy --deploy-hook zimbra --dns dns_cf -d mail.example.com -d mail.example.net -d mail.example.org 
You need to have installed and configured acme.sh to use the method you plan on validating with. I chose automatic DNS, but it doesn't matter... See: https://github.com/Neilpang/acme.sh for all the methods. The acme.sh code is constantly changing so new stuff is happening all the time as the acme protocol changes. So get your certs.... repeat as often as it takes until they are all validated should you have errors. You can't and won't screw up zimbra. All it is doing is creating your certificate in the home directory of the user you are issuing the .acme.sh command as. If you do this as root, then it will be in the root directory... if you do it as zimbra then it will be in the home directory of the zimbra user. Same for your own user id. The problem is that if you do it as another user than zimbra, there is a nasty bug I mentioned in this thread that will allow the validation but fail the install... That is why I recommend you use the zimbra user in the wiki link or use something like my script in github that protects against this. [bug]107454[/bug]

Next: install it as the zimbra user. We have documented this in a lot of detail... I wrote line by line in the first page of this thread, a very verbose script to do it in my github and now I use the plugin architecture for acme.sh which is also in the wiki article and in my github acme.sh code base under its deploy directory... I use the --deploy --deploy-hook zimbra that I show above. It is really that simple. It handles the intermediate certificate so there is no manual process when using the deploy hook. If you don't use the deploy hook then you need to worry about the intermediate certificate that is documented in the wiki.

Note: even if you install acme.sh multiple times, it doesn't matter as it won't hurt anything. You can also scp -r /opt/zimbra/.acme.sh newmachine:/opt/zimbra/ ... so lots of ways of getting that .acme.sh installed.

What you can not do is use the zmcertmgr command as root for 8.7+ and above. You also need to be really careful of file permissions when you mix different users... ie) create certificates as one user and install as zimbra.

The reason for the intermediate certificate is documented here: viewtopic.php?f=15&t=60781&start=10#p273930.

Can you be specific where you ran into problems. It pretty hard to screw up your server with a cert installation IMO... See this thread... viewtopic.php?f=15&t=60781#p272438 Yep that was me messing up the install when I renewed for the first time with a script. Oops! :-) Get your certs validated then install. If you used my script from github, it creates a backup in /opt/zimbra/ssl as a tar file but I don't backup certs now since they are so easy to generate and are free.
User avatar
JDunphy
Outstanding Member
Outstanding Member
Posts: 889
Joined: Fri Sep 12, 2014 11:18 pm
Location: Victoria, BC
ZCS/ZD Version: 9.0.0_P39 NETWORK Edition

Re: Another Letsencrypt method

Post by JDunphy »

Labsy wrote: ... then Install certificate with Zimbra...and ups here...now it's Step 1...aaaaaaaa?!! What are then those steps before? Did I mess up my server already?? Are there steps even BEFORE "Step 1"???
And why install IdentTrust SSL chain? Is this your favourite, or necessity?
Ohhhhhhh, I see what you are talking about in the wiki article. Unfortunately, that contents section is created automatically from the wiki page and I don't know how to fix that. If you browse the page it makes sense and you don't get things out of order. I took another wiki as a template because I don't know the markup language and changed the content to explain the method in this thread. I will investigate fixing this but have never found any description of what this markup language they are using is. I think the problem is my use of the '= some text title=' as a section header which seems to generate the corresponding contents section.

Given your experience, I will add a recovery section so people don't panic should something go wrong with the certificate issue. How to fix the problem and reissue the certificate and if you are really broken and can't re-issue... self-signed so you can get up and reissue your real certs. I'll also add a reference to the acme protocol since it is now heading toward a standard with version 2 as of this month and the rational behind certificates that expire faster than what was common practice just a few years ago. They have a nice description of the problem space for CA's. BTW Gmail is now issuing every 3 weeks given the security threats against PKI.
phoenix
Ambassador
Ambassador
Posts: 27272
Joined: Fri Sep 12, 2014 9:56 pm
Location: Liverpool, England

Re: Another Letsencrypt method

Post by phoenix »

[quote="JDunphy"]Hi Bill,

My guess is that /opt/zimbra is owned by root. That automatic method they like for install is problematic sometimes.[/quote}Hi Jim

Thanks for your reply and sorry for the late post, I've been rather tied-up lately. For some reason I seem to have missed out the fact that I was trying the "all in one" method on my system, am I correct that method is an alternative to installing as a normal user?

I'm probably missing something here but my understanding is that the hook script is installed, you mention that first in the section but there's no such ".acme"directory created at that point - I obviously created the directory and saved the script.

Next, you have the "Complete example" and it's at this point I get lost. :( I run the commands you have posted in the wiki article and the first post of this thread:

Code: Select all

% su - zimbra
% wget -O -  https://get.acme.sh | sh
At this point I get the failure I had earlier:

Code: Select all

[zimbra@mail01 ~]$ wget -O -  https://get.acme.sh | sh
--2019-04-02 20:49:39--  https://get.acme.sh/
Resolving get.acme.sh (get.acme.sh)... 2607:5300:201:3100::5663, 144.217.161.63
Connecting to get.acme.sh (get.acme.sh)|2607:5300:201:3100::5663|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 705 [text/plain]
Saving to: 'STDOUT'

100%[======================================>] 705         --.-K/s   in 0s      

2019-04-02 20:49:39 (82.9 MB/s) - written to stdout [705/705]

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  174k  100  174k    0     0   658k      0 --:--:-- --:--:-- --:--:--  724k
[Tue Apr  2 20:49:40 CEST 2019] Installing from online archive.
[Tue Apr  2 20:49:40 CEST 2019] Downloading https://github.com/Neilpang/acme.sh/archive/master.tar.gz
sh: line 5827: master.tar.gz: Permission denied
[Tue Apr  2 20:49:40 CEST 2019] Download error.
[zimbra@mail01 ~]$
Yes, the directories are owned by root. Do I need to change any permissions to get this working? This is currently my live server on which I'm trying to get this operational. Does installing as the zimbra user on your server still work with the current ZCS 8.8.11?
Regards

Bill

Rspamd: A high performance spamassassin replacement

Per ardua ad astra
User avatar
JDunphy
Outstanding Member
Outstanding Member
Posts: 889
Joined: Fri Sep 12, 2014 11:18 pm
Location: Victoria, BC
ZCS/ZD Version: 9.0.0_P39 NETWORK Edition

Re: Another Letsencrypt method

Post by JDunphy »

Hi Bill,

I am making it worse the more I add to that wiki article. Yikes! ;-)

The first part of the wiki explains installing as a "non root" user and to give the administrator some knowledge of the steps involved and how to use one of the verification methods such as automated DNS with CloudFlare... At the end is the continuation of the learned concepts but now using the zimbra user and a deploy hook. So the simple example you mentioned with only 2 command lines is done as the zimbra user and not "any user" as was the use case in the first part. The reason is that .acme.sh is invoked as the zimbra user and calls the hook which does the install of the certificate and restarts zimbra. The other reason is that doing anything with the zimbra user should cause an admin to pause and it felt safer to teach install of acme.sh and issue/renew of certs as a normal user at the beginning to build some trust with the process and follow a more manual process.

The trade off with the hook install method is that everything needs to be setup for .acme.sh for the zimbra user but once it is done, you can renew or issue and install to zimbra with only 2 commands going forward. I am open to suggestions how to update that wiki to make more sense because it just gets longer everything I change it and not clearer.

I run 8.7.11 and my /opt/zimbra is owned by root but I thought I did that so it appears to be standard given your comments.

A few options... I like /opt/zimbra being owned by root myself so a few options to keep it that way should others wonder about this.
I would do this as root.

Code: Select all

% su -
# cd /opt/zimbra/
# mkdir .acme.sh
# chown zimbra:zimbra .acme.sh
# su - zimbra
% wget -O -  https://get.acme.sh | sh
That wget command is going to install in the users home directory that is running that command. Should that directory already exist, it will upgrade the contents of that directory but it will not remove any files that were there previously.

The other way is to move or copy .acme.sh directory from where it is currently installed to /opt/zimbra/.acm e.sh... for example, say I have it installed in /home/jad/.acme.sh because I was using another user previously.

Code: Select all

% su -
# mv /home/jad/.acme.sh /opt/zimbra/.acme.sh
# chown -R zimbra:zimbra /opt/zimbra/.acme.sh
# su - zimbra
% cd /opt/zimbra/.acme.sh
% ./acme.sh --upgrade
Doing the upgrade isn't really necessary but I show it for completeness. Similarly, backing up your cert creation environment is nothing more than this since we are contained in a single directory. You can take it from machine to machine.

Code: Select all

# su - zimbra
% cd /opt/zimbra
% tar cvf /tmp/acme.sh.tar ./acme.sh
Given it is so simple to install acme.sh and create certs, I no longer backup the .acme.sh directory and just contain some documentation with the commands I need for that particular instance with the correct domains since remembering all the domains is easy to forget. Perhaps backing up account.conf or documenting that if you are using the DNS method since it has your account information for the providers API.
Post Reply