Caution With Unattended Upgrades

Ask questions about your setup or get help installing ZCS server (ZD section below).
Post Reply
User avatar
L. Mark Stone
Ambassador
Ambassador
Posts: 2796
Joined: Wed Oct 09, 2013 11:35 am
Location: Portland, Maine, US
ZCS/ZD Version: 10.0.6 Network Edition
Contact:

Caution With Unattended Upgrades

Post by L. Mark Stone »

Generally it is a good practice to install the unattended-upgrades package (in Ubuntu 16.04; there's a similar feature for CentOS/RHEL) to get the benefit of security and other bug fixes quickly. I know some disagree (and the conversation about that is for another forum please...), but I wanted to point out a problem I have seen now with two successive Patch releases.

Bottom Line is if you do run unattended upgrades, be sure to exclude Zimbra packages.

So here we have a Zimbra 8.8.8 Network Edition system on Ubuntu 16.04. Zimbra is at Patch 9. Check this:

Code: Select all

root@zimbra:~# apt-get update; apt list --upgradable
Hit:1 http://archive.ubuntu.com/ubuntu xenial InRelease
Hit:2 http://security.ubuntu.com/ubuntu xenial-security InRelease       
Hit:3 http://archive.ubuntu.com/ubuntu xenial-updates InRelease         
Hit:4 http://archive.ubuntu.com/ubuntu xenial-backports InRelease
Hit:5 https://repo.zimbra.com/apt/87 xenial InRelease
Hit:6 https://repo.zimbra.com/apt/zv1 xenial InRelease
Hit:7 https://repo.zimbra.com/apt/888patch xenial InRelease
Hit:8 https://repo.zimbra.com/apt/888patch-nw xenial InRelease
Reading package lists... Done
Listing... Done
zimbra-common-core-jar/unknown 1.0.0.1536227922-1.u16 amd64 [upgradable from: 1.0.0.1533613456-1.u16]
zimbra-network-modules-ng/unknown 1.0.24.1535704239-1.u16 amd64 [upgradable from: 1.0.23.1534260702-1.u16]
zimbra-nginx/unknown 1.7.1-1zimbra8.7b9.16.04 amd64 [upgradable from: 1.7.1-1zimbra8.7b7.16.04]
zimbra-patch/unknown 8.8.8.10.1536232008-2.u16 amd64 [upgradable from: 8.8.8.9.1535106934-2.u16]
zimbra-proxy-components/unknown 1.0.2-1zimbra8.7b1.16.04 all [upgradable from: 1.0.1-1zimbra8.7b1.16.04]
root@zimbra:~# 
Now, as of this writing, we expect that Patch 10 is to be released soon. And if you manually edit the URL for the Patch 9 Release Notes, you get the Patch 10 release Notes document as it now stands: https://wiki.zimbra.com/wiki/Zimbra_Releases/8.8.8/P10

Helpfully, that Release Notes document lists the updated packages included in the patch:
8.8.8 Patch10 Packages
Below are the latest available packages:

Package Name Version
FOSS:
zimbra-patch              ->  8.8.8.10.1536232008-1
zimbra-common-core-jar    ->  1.0.0.1536227922-1
zimbra-mbox-webclient-war ->  1.0.0.1527079283-1
zimbra-ldap-components    ->  1.0.1-1zimbra8.7b1
zimbra-openldap-client    ->  2.4.46-1zimbra8.7b2
zimbra-openldap-lib ->  2.4.46-1zimbra8.7b2
zimbra-openldap-server    ->  2.4.46-1zimbra8.7b2
zimbra-lmdb               ->  2.4.46-1zimbra8.7b2
zimbra-mta-components -> 1.0.5-1zimbra8.7b1
zimbra-openjdk -> 1.8.0u172b01-1zimbra8.7b5
zimbra-chat               ->  1.0.20.1532350417-2
zimbra-nginx -> 1.7.1-1zimbra8.7b9
zimbra-proxy-components -> 1.0.2-1zimbra8.7b1

NETWORK:
zimbra-patch              ->  8.8.8.10.1536232008-2
zimbra-network-modules-ng ->  1.0.24.1535704239-1
zimbra-talk               ->  1.0.11.1532349058-1
Comparing the two, basically what I see is that ZImbra is populating the repositories a few packages at a time, instead of waiting until right before the Patch is released publicly. And what my apt list --upgradable command shows is that not all of the Patch 10 packages are yet in the repos. Are there inter-package dependencies in all/some of the packages to be distributed as part of the patch? I don't know.

But I do know that if I just ran "apt-get update; apt-get upgrade" today, I'd have a partially installed Zimbra Patch 10, and I don't how that would impact the functioning of my Zimbra system.

So, best to exclude all Zimbra packages from unattended-upgrades.

To exclude all Zimbra packages from unattended-upgrades: as root run:

Code: Select all

nano /etc/apt/apt.conf.d/50unattended-upgrades
Then, add the "zimbra-"; line to this section:

Code: Select all

// List of packages to not update (regexp are supported)
Unattended-Upgrade::Package-Blacklist {
        "zimbra-";
//      "vim";
//      "libc6";
//      "libc6-dev";
//      "libc6-i686";
};
Restart the unattended-upgrades service, then run as root (to be sure):

Code: Select all

root@zimbra:/etc# unattended-upgrades --dry-run --debug
Initial blacklisted packages: zimbra-
Initial whitelisted packages: 
Starting unattended upgrades script
Allowed origins are: ['o=Ubuntu,a=xenial', 'o=Ubuntu,a=xenial-security', 'o=UbuntuESM,a=xenial', 'o=Ubuntu,a=xenial-updates']
Checking: zimbra-common-core-jar ([<Origin component:'zimbra' archive:'' origin:'Repository for UBUNTU16' label:'Repository for UBUNTU16' site:'repo.zimbra.com' isTrusted:True>])
skipping blacklisted package 'zimbra-common-core-jar'
Checking: zimbra-network-modules-ng ([<Origin component:'zimbra' archive:'' origin:'Repository for UBUNTU16' label:'Repository for UBUNTU16' site:'repo.zimbra.com' isTrusted:True>])
skipping blacklisted package 'zimbra-network-modules-ng'
Checking: zimbra-nginx ([<Origin component:'zimbra' archive:'' origin:'Zimbra Collaboration Suite 8.7 Ubuntu16' label:'Zimbra Collaboration Suite 8.7 Ubuntu16' site:'repo.zimbra.com' isTrusted:True>])
skipping blacklisted package 'zimbra-nginx'
Checking: zimbra-patch ([<Origin component:'zimbra' archive:'' origin:'Repository for UBUNTU16' label:'Repository for UBUNTU16' site:'repo.zimbra.com' isTrusted:True>])
skipping blacklisted package 'zimbra-patch'
Checking: zimbra-proxy-components ([<Origin component:'zimbra' archive:'' origin:'Zimbra Collaboration Suite 8.7 Ubuntu16' label:'Zimbra Collaboration Suite 8.7 Ubuntu16' site:'repo.zimbra.com' isTrusted:True>])
skipping blacklisted package 'zimbra-proxy-components'
pkgs that look like they should be upgraded: 
Fetched 0 B in 0s (0 B/s)                                                                                                            
fetch.run() result: 0
blacklist: ['zimbra-']
whitelist: []
No packages found that can be upgraded unattended and no pending auto-removals
root@zimbra:/etc# 
The above shows the partial inventory of Patch 10 packages already copied to the repos, but that none of them would be installed during an unattended-upgrade run on account of the exclusion.

I've asked Zimbra not to populate the repositories at all, until right before a Patch is formally released, but in the grand scheme of things this is not a big deal -- if you are not upgrading Zimbra packages automatically.

If someone wants to post the same steps to do this on RHEL/CentOS, that would be helpful as well.

Hope that helps,
Mark
___________________________________
L. Mark Stone
Mission Critical Email - Zimbra VAR/BSP/Training Partner https://www.missioncriticalemail.com/
AWS Certified Solutions Architect-Associate
User avatar
howanitz
Advanced member
Advanced member
Posts: 65
Joined: Mon Feb 01, 2016 9:27 am

Re: Caution With Unattended Upgrades

Post by howanitz »

Thank you so much for these notes.

I did a manual update without paying much attention, and included Zimbra upgrades that ran into problems. In addition to the issues you outlined above, I don't think their Ubuntu packages restart the services nor request a computer restart the way packages in the Ubuntu repositories do.

I am REALLY thankful they are making the move to distribute updates this way, but it seems to me that the process still needs some work and maturing. I guess my current feedback to Zimbra is: Thank you for this work - please make it mature soon!
phoenix
Ambassador
Ambassador
Posts: 27272
Joined: Fri Sep 12, 2014 9:56 pm
Location: Liverpool, England

Re: Caution With Unattended Upgrades

Post by phoenix »

I'd suggest you disable the Zimbra repository and oply enable it when you're doing an upgrade to the Zimbra products, you won't run into this problem if you do that.
Regards

Bill

Rspamd: A high performance spamassassin replacement

Per ardua ad astra
User avatar
howanitz
Advanced member
Advanced member
Posts: 65
Joined: Mon Feb 01, 2016 9:27 am

Re: Caution With Unattended Upgrades

Post by howanitz »

I also think that whenever they put an updated package into the apt repositories, there should be a corresponding note added to this page to tell us about it, and let us know if we will need to restart the server or restart services or apply some other patch manually first.

https://www.zimbra.com/downloads/zimbra-collaboration/

Maybe there is some other page where they are putting this documentation and I don't know about it???
phoenix
Ambassador
Ambassador
Posts: 27272
Joined: Fri Sep 12, 2014 9:56 pm
Location: Liverpool, England

Re: Caution With Unattended Upgrades

Post by phoenix »

There is usually an announcement and a link to the patch notes on the wiki which details what's being upgraded e.g.: viewforum.php?f=8 and https://wiki.zimbra.com/wiki/Zimbra_Releases/8.8.8/P5
Regards

Bill

Rspamd: A high performance spamassassin replacement

Per ardua ad astra
User avatar
L. Mark Stone
Ambassador
Ambassador
Posts: 2796
Joined: Wed Oct 09, 2013 11:35 am
Location: Portland, Maine, US
ZCS/ZD Version: 10.0.6 Network Edition
Contact:

Re: Caution With Unattended Upgrades

Post by L. Mark Stone »

Bill,

Yes, disabling the Zimbra repo is another option, so maybe just a personal preference. I can run for example:

Code: Select all

apt-get update; apt list --upgradable
and then just not upgrade any zimbra packages if I need to do a manual upgrade.

Zimbra's updated packages do not yet restart their own dependent services it is true, but if you update the zimbra packages from the command line, each package will output to the screen which (if any) zimbra services need to be restarted, and whether any other post-install commands need to be run, like doing a cache flush for example.

I agree that this information will be useful to include in the Release Notes, if only to know ahead of time whether a Patch install will be service-impacting or not, and I opened a Support Case with Zimbra to file an RFE for improving the Patch Release Notes documentation. FWIW, I specifically mentioned Citrix's HotFix release notes as a good template worthy of consideration, for example: https://support.citrix.com/article/CTX237086.

All the best,
Mark
___________________________________
L. Mark Stone
Mission Critical Email - Zimbra VAR/BSP/Training Partner https://www.missioncriticalemail.com/
AWS Certified Solutions Architect-Associate
Post Reply