Another note on this... It's been a few years without problems but last night's renewal showed me something new. zmcontrol restart had a few problems.
Nothing cert related but related to how zimbra determines when a process is running to know when to restart it. I had already patched and reported the MTA bug
viewtopic.php?f=15&t=65332&hilit=potential+bug+mta
so postfix will always restart properly but it appears there are a lot more places. The pattern we are looking for is 'kill -0' in their startup scripts.
Given the recent security threats, I thought I would share what changes when you replace a certificate... Courtesy of a morning tripwire report:
Code: Select all
added: /opt/zimbra/.acme.sh/ca/acme-v02.api.letsencrypt.org
added: /opt/zimbra/.acme.sh/ca/acme-v02.api.letsencrypt.org/ca.conf
added: /opt/zimbra/.acme.sh/ca/acme-v02.api.letsencrypt.org/account.json
added: /opt/zimbra/.acme.sh/ca/acme-v02.api.letsencrypt.org/account.key
changed: /opt/zimbra/ssl/.rnd
changed: /opt/zimbra/ssl/zimbra/jetty.pkcs12
changed: /opt/zimbra/ssl/zimbra/commercial/commercial.crt
changed: /opt/zimbra/ssl/zimbra/commercial/commercial_ca.crt
changed: /opt/zimbra/conf/slapd.crt
changed: /opt/zimbra/conf/smtpd.crt
changed: /opt/zimbra/conf/ca/commercial_ca_1.crt
changed: /opt/zimbra/conf/nginx.crt
changed: /opt/zimbra/common/etc/java/cacerts
changed: /opt/zimbra/.acme.sh/http.header
changed: /opt/zimbra/.acme.sh/mail.example.com/mail.example.com.cer
changed: /opt/zimbra/.acme.sh/mail.example.com/ca.cer.real
changed: /opt/zimbra/.acme.sh/mail.example.com/fullchain.cer
changed: /opt/zimbra/.acme.sh/mail.example.com/mail.example.com.conf
changed: /opt/zimbra/.acme.sh/mail.example.com/ca.cer
changed: /opt/zimbra/.acme.sh/ca
Note: the changing to the latest acme protocol version 2 is not normal but you can expect it for acme.sh v2.8.2
Now what didn't start?
Code: Select all
$ zmcontrol status
Host mail.example.com
amavis Running
antispam Running
antivirus Running
convertd Running
ldap Running
logger Stopped
zmlogswatchctl is not running
mailbox Running
memcached Running
mta Running
opendkim Running
proxy Running
service webapp Running
snmp Stopped
zmswatch is not running.
spell Running
stats Running
zimbra webapp Running
zimbraAdmin webapp Running
zimlet webapp Running
zmconfigd Running
The solution was simple enough:
Code: Select all
# su - zimbra
% zmlogswatchctl start
% zmswatchctl start
This is hardly enterprise software.
Code: Select all
grep 'kill -0' zmlogswatchctl zmswatchctl
zmlogswatchctl: kill -0 $pid 2> /dev/null
zmlogswatchctl: kill -0 $zmrrdfetchpid 2> /dev/null
zmlogswatchctl: kill -0 $zmrrdfetchpid 2> /dev/null
zmlogswatchctl: kill -0 $pid 2> /dev/null
zmswatchctl: kill -0 $pid 2> /dev/null
zmswatchctl: kill -0 $pid 2> /dev/null
Looks like I have more patching to do around here. BTW, I don't see the point of reporting and showing them bug fixes if I can't get zmmtastatus patched on 8.7.11. Those that rely on zmcontrol restart to work reliably are on borrowed time if you do unattended automatic restarts for things like certificate renewal or backups. I guess I might go back to doing only ldap,postfix,nginx reloads and mailboxd restarting given that reality. The deploy hook I am using with acme.sh has them present so its easy enough to make that change and comment out the zmcontrol restart.
Hint: My biggest issues is that certificates get replaced without me noticing so I have a swatch monitor which runs on our central syslog machine notify me after the acme.sh Zimbra hook issues a logger command that it was renewed... I also have a script that does acme.sh --list and parses the renewal date to send me an email 1 day before I can expect to see a new certificate which is currently every 60 days. That keeps me in the loop. I can post this if there is interest. With a single zimbra instance this isn't a problem but if you do it for everything that isn't zimbra related that can be a lot of certs to keep track off. Fortunately, I have never had a problem with web farms, grafana, plex ... just the commercial stuff we pay for it would seem. The irony eh?