Another Letsencrypt method

Discuss your pilot or production implementation with other Zimbra admins or our engineers.
User avatar
JDunphy
Outstanding Member
Outstanding Member
Posts: 889
Joined: Fri Sep 12, 2014 11:18 pm
Location: Victoria, BC
ZCS/ZD Version: 9.0.0_P39 NETWORK Edition

Re: Another Letsencrypt method

Post by JDunphy »

I have been through 2 renewal cycles now and thought I would share my experiences.

It's not exactly what I had expected because I am having to add the dns challenge back to my zones during the verification. What is odd is that if you remove the challenge from your dns as you SHOULD after getting your certs and rerun the the acme.sh script against it, you can get new certs... but at some point it seems to self expire and that verification stops working. I thought that my syntax was wrong so I added the --renew option when running my renewal process in my script. It still required that I add the new challenge which surprised me this morning. Next time, I am just going to uncomment the current challenge in my dns zones and push it out before running the renewal to see if that would work??? The other thing I need to look into is how zmcertmgr is cleaning up. It does its own backup so my extra step of doing a backup is redundant, etc.

Anyone else have any experiences with this DNS method on renewal? I still like it for it simplicity versus the certbot but need to change to a more automatic method now that I trust these letsencrypt certs and the verification process. Initially, I wanted this to be manual until I built some operational experience so I might be looking at the web version of the letscrypt acme protocol if I can't get around this DNS verification step. In my speed to deploy, I took a fairly quick read of the acme.sh code but need to take a deeper look before my next renewal in case I am missing something obvious here.

UPDATED Jan 30, 2017: I asked the author of acme.sh and he said that currently, if you renew within 59 days, there would not be a re-authorization but expect that to go lower from LetsEncrypt. That explains why it seemed to work and then not later on and force me through another challenge. It is in reference to this: https://community.letsencrypt.org/t/upc ... nges/17947. Since my original post, the code has begun to change to reflect this also. ... Given this, you have 2 options:
  • 1. renew more frequently
    2. use something like the webroot method with nginx and acme.sh
I am working on the acme/web method so might have something to post soon that is a little simpler. Once that is working, I will probably just add a few hooks to zmcertmgr and be done with it.
Last edited by JDunphy on Mon Jan 30, 2017 3:13 pm, edited 1 time in total.
User avatar
JDunphy
Outstanding Member
Outstanding Member
Posts: 889
Joined: Fri Sep 12, 2014 11:18 pm
Location: Victoria, BC
ZCS/ZD Version: 9.0.0_P39 NETWORK Edition

Re: Another Letsencrypt method

Post by JDunphy »

lytledd wrote: The bit you just posted would be perfect!
Doug
Perfect. I changed the wording about that IdentTrust CERT and added that clarification. At some point, I'll incorporate this thread into a wiki article so this was really great feedback.
User avatar
myriad
Advanced member
Advanced member
Posts: 90
Joined: Fri Sep 12, 2014 11:51 pm
ZCS/ZD Version: Zimbra 9.0.0_ZEXTRAS_20211118.FOSS

Re: Another Letsencrypt method

Post by myriad »

Big thank you to JD for an excellent write up! Once I got the CAT mistake out of the way (it didn't copy the dashes properly) it worked perfectly. Can't wait for the web how to.
netmax
Posts: 1
Joined: Fri Feb 03, 2017 1:52 pm

Re: Another Letsencrypt method

Post by netmax »

Thanks a ton, that was extremly helpful and worked right out of the box!

I ran into a nightmare yesterday when my Zimbra server which I've installed end of October, presented me a CERT error in Firefox. I am using StartCOM SSL certificates on most of my devices/sites, and I was shocked when I read that Mozilla decided to just block them from version 51 on ... without any option to add an exception. :evil: Strange decisions over there. Finally, I've got also a warning in Chrome this morning, so I'm in really trouble. They killed StartCOM over night and millions of mostly private and non-profit websites which can't afford expensive certificates.

Maybe a way to place LetsEncrypt as the only free CA on the market ...

However, my Mailserver is back running thanks to JD. :D

Question to the round: Did anyone figure out a complete automated renewal via cronjob? Or do I need to fill up my calendar with a reoccurring event every 60 days to renew my CERT? A bit uncomfortable with several sites ...

Marco
User avatar
JDunphy
Outstanding Member
Outstanding Member
Posts: 889
Joined: Fri Sep 12, 2014 11:18 pm
Location: Victoria, BC
ZCS/ZD Version: 9.0.0_P39 NETWORK Edition

Re: Another Letsencrypt method

Post by JDunphy »

Interesting Question.

acme.sh --install does add via cron an entry to automate this and update the cert every 60 days unless you force it. I commented out the cron entry because I copy the .acme.sh directory to the production machines and then ran my script against it. But when I was testing my process I was forcing it. Uncomment FORCE=1 in account.conf or run --force. If you do this before 60 days then you DO NOT go through the challenge. That is why the testing worked without challenge and had me confused later when I was running it with 60+ days renewal. The problem is that at some point, that 60 days will become less to further reduce the opportunity for forgeries so I want to learn more about my options. The dns method could also be automated via the hooks in acme.sh but that opens up other issues I want to avoid at present. He has hooks for most of the big DNS services for example so the challenge/response would work unattended.

Because I wanted to understand the zimbra components more, I chose not to automate this via cron. For some reason, I thought cert updates was very difficult and prone to failure. I think that had more to do with all the zimbra parts than the reality of updating a CERT. My experience has shown that it is very straightforward to update a certs with zmcertmgr. I still run my script by hand that does all the steps but I have run it earlier than 60 days and it just works for me.

My beef is that my process is more complicated than it needs to be for the cert challenge/response phase. For example, one method is to use acme.sh standalone so that it listens on port 80/443. Another is the webroot with a location directory from nginx pointing to the validation response. You would need to bring down any zimbra services that conflict so that it can listen on 80/443 for validation/challenge... the nginx method (aka webroot) could work while zimbra is running... at least that is how I think it would work. For now, I just keep renewing more often and hope to get some time to try a few of these different methods. I need to make a decision soon however since it is getting old doing this by hand based on a email reminder even if its just running a single command to handle all the parts... Maybe I am just scared to pull the trigger. :-)

PS. My biggest gripe is I hate taking ANY zimbra outage even if its less than 60 seconds to replace a new CERT. zmcontrol restart is way overkill!

I also would be interested in what others are doing.

Jim
rwebb616
Posts: 25
Joined: Mon Apr 11, 2016 1:52 pm

Re: Another Letsencrypt method

Post by rwebb616 »

I too would like to see an automated method. Mine expires in 90 days but really they should give people a minimum of a year if there is no automated method available for renewal.

I'm not sure how the process works but if the renewal just downloads new certs couldn't one just stop the proxy and mailbox services and copy the new cert into it's appropriate place and fire the proxy and mailbox services back up?

Rich
User avatar
JDunphy
Outstanding Member
Outstanding Member
Posts: 889
Joined: Fri Sep 12, 2014 11:18 pm
Location: Victoria, BC
ZCS/ZD Version: 9.0.0_P39 NETWORK Edition

Re: Another Letsencrypt method

Post by JDunphy »

Would this do?

I was documenting and reworking the script I use yesterday... It's here: https://github.com/JimDunphy/deploy-zim ... encrypt.sh ... I will double check that renew-hook option to acme.sh tomorrow. I decoupled the acme.sh stuff from my script. That way I can verify different configurations with acme.sh methods... and not just DNS. In theory you could recombine them like below into local cron but that means you would have to install acme.sh as zimbra or make the script deploy-zimbra-letsencrypt.sh run as zimbra for 8.7+ versions.

Code: Select all

acme.sh --issue --renew-hook /opt/letsencrypt/deploy-zimbra-letsencrypt.sh
You would have to modify min variable in the script because it is checking the min days before expiration so should match acme.sh for renewal interval when you use --force which should be 59 or less days for the DNS method. I have anther script I was using but saw this --renew-hook flag and thought it looked interesting. By decoupling the acme.sh from the the zimbra script you also can use any of the other methods like standalone, webroot, etc that acme.sh supports.

It's time to make this automatic. I trust the process now so lets get some options for people... multi-host, SNI, etc. I am currently scp'ing the .acme.sh directory after renewal to different zimbra configurations which is another reason the deploy-zimbra-letsencrypt.sh just sits in cron and updates it when its time on the production machines. That deploy script knows where to look and copies the .acme.sh directory as the zimbra user and loads them. In summary. I create my certs from a mgmt machine and then push them to production or staging servers and this was the simplest way I came up with. You also don't need acme.sh software installed on production servers for renewal in that configuration... but this is an edge case for sure. :-)

Also. It looks like ldap, postfix, mailboxd, and nginx all use the certs at least that is what zmcertmgr checkcrtexpiration returns. I am wondering if we can do reloads vs stop/start of the entire system. Need to verify.

PS. the new script just hardwires the IdentTrust to the fullchain so there is no need to cat or creating the file manually. The script handles that.
User avatar
JDunphy
Outstanding Member
Outstanding Member
Posts: 889
Joined: Fri Sep 12, 2014 11:18 pm
Location: Victoria, BC
ZCS/ZD Version: 9.0.0_P39 NETWORK Edition

Re: Another Letsencrypt method

Post by JDunphy »

Here is another gotcha waiting for people. It isn't a letsencrypt problem as zmcontrol restart issue which you will see because you are probably restarting more frequently that others because of CERT renewals.

I have a machine that has gone through about 25 of these automatic cycles until last night.

Code: Select all

Host mail.example.com
        amavis                  Running
        ...
        proxy                   Running
        service webapp          Running
        snmp                    Running
        spell                   Running
        stats                   Stopped   ------ problem
        zimbra webapp           Running
        zimbraAdmin webapp      Running
        zimlet webapp           Running
        zmconfigd               Running
The fix is easy enough... as root:

Code: Select all

ps auxw |grep zmstat-fd
then kill any OLD ones left around
su - zimbra
zmstatctl restart
zmstatctl status - no more warnings about PID file
zmcontrol status ... should be fine
Looking at zmstatctl, it just isn't very robust to catch old copies running. Making matters worse, the repair/fix needs to be run as root but adding the cert is as zimbra in 8.7+ so my simple deployment script can't just fix the problem unless it runs that part as root which increases the complexity I was striving for in the solution. sudo is an option to address this but yet another file to fix during upgrades/installs. I wanted a single hook via cron.

Instead of trying to fix this problem because a quick review of the forums and bugzilla shows this 'stat's problem in various forms, I am going to warn myself for this edge case and focus on doing reloads to get that CERT installed instead of zmcontrol restart and hopefully bypass these type of issues. Not sure why the old zmstat-fd's were still around but they are fairly obvious with very old date stamps when you list them.
User avatar
zimico
Outstanding Member
Outstanding Member
Posts: 225
Joined: Mon Nov 14, 2016 8:03 am
Location: Vietnam
ZCS/ZD Version: 8.8.15 P3
Contact:

Re: Another Letsencrypt method

Post by zimico »

Thank JDunphy a lot for your great work!
User avatar
JDunphy
Outstanding Member
Outstanding Member
Posts: 889
Joined: Fri Sep 12, 2014 11:18 pm
Location: Victoria, BC
ZCS/ZD Version: 9.0.0_P39 NETWORK Edition

Re: Another Letsencrypt method

Post by JDunphy »

Thought I would mention the CAA record for DNS. Because I run centos 6, I am using the type257 method because my bind version doesn't support the CAA type.

Code: Select all

; CAA record for letsencrypt
example.net.                    IN      TYPE257 \# 22 000569737375656C657473656E63727970742E6F7267
So in theory, browsers won't accept a cert that was not issued by letsencrypt and some CA's will only issue certs that are listed to help with the unauthorized/forged certs problem.

I used https://sslmate.com/labs/caa/ to generate my entry.

If I understood this correctly, mail.example.net, https://www.example.net would also need to be issued by letsencrypt because I applied it to the entire domain. If you use different CA's, then you would need to specify them with a more specific entry.

Reference: https://en.wikipedia.org/wiki/DNS_Certi ... horization
Post Reply