Post-June 17 Patch 25/32 Success Stories Roll Call

Ask questions about your setup or get help installing ZCS server (ZD section below).
BradC
Outstanding Member
Outstanding Member
Posts: 265
Joined: Tue May 03, 2016 1:39 am

Re: Post-June 17 Patch 25/32 Success Stories Roll Call

Post by BradC »

JDunphy wrote:Single server on RHEL 8.6. Restored last nights backup of production server to new vm and applied update against that instance. 8.8.15.P31.1 to 8.8.15P32
This is how I do it also. Usually replicate the prod to a staging server then apply the patch. Unless there's a severe security issue that I can't externally mitigate I then tend to watch the forum for a couple of weeks to make sure there's no clangers.

Using a duplicate, I can re-run the clone/patch process to make sure it's all clean before rolling it out.

Of course I can really only afford this luxury because we have a small system on a single VM.
User avatar
JDunphy
Outstanding Member
Outstanding Member
Posts: 889
Joined: Fri Sep 12, 2014 11:18 pm
Location: Victoria, BC
ZCS/ZD Version: 9.0.0_P39 NETWORK Edition

Re: Post-June 17 Patch 25/32 Success Stories Roll Call

Post by JDunphy »

BradC wrote:
Using a duplicate, I can re-run the clone/patch process to make sure it's all clean before rolling it out.

Of course I can really only afford this luxury because we have a small system on a single VM.
It's all about rehearsal so the actual downtime is minimized for the users. Not to mention, most of the time you can not roll back if updates have happened to db schema, etc. It's also nice to have the old system and do a test side by side to see any differences which often leads to more understanding of breakage, new changes, and potential resolutions. IMHO, practicing various restore/backup strategies to prepare for disaster recover is a good idea and if I can accomplish that at the same time as an update it's a bonus. For a part time admin like myself, that practice is especially important because I don't work on Zimbra enough to feel like I have a deep understanding. That java monstrosity scares me.

The one time I failed to follow this process ended up with disaster and I have been at zimbra for a long time so you would think I should have known better. ;-) :-) ... Nope! my failed machine with P20 was like P32.

Ref: viewtopic.php?f=13&t=69414

Jim
BradC
Outstanding Member
Outstanding Member
Posts: 265
Joined: Tue May 03, 2016 1:39 am

Re: Post-June 17 Patch 25/32 Success Stories Roll Call

Post by BradC »

JDunphy wrote:It's all about rehearsal so the actual downtime is minimized for the users.
I don't think the value of this can be emphasized enough.

When I upgraded from 8.8.12 to 9 I used the ZeXtras backup/restore technique onto a clean server (VM), but I reckon I wiped, installed and brought up the server > 30 times over a very badly interrupted year while testing and developing a script to make sure I mimimised the potential for finger trouble. That gave me the opportunity to tweak the config on the prod box to minimise the amount of changes necessary when restoring the backup and plenty of time to poke and prod to figure out which customisations needed tweaking.

The older I get, the worse my memory gets and the more I rely on my documentation. Naturally that has required an improvement in the documentation process.

The other thing I've done since Zimbra started using it is maintained a local copy of the repo. A cron job syncs it every night, and I know when there's new stuff to look at when I get the cron E-mail with the list of files synced. This isn't the first time there have been post-patch corrections.
ghen
Outstanding Member
Outstanding Member
Posts: 258
Joined: Thu May 12, 2016 1:56 pm
Location: Belgium
ZCS/ZD Version: 9.0.0

Re: Post-June 17 Patch 25/32 Success Stories Roll Call

Post by ghen »

We have our lab succesfully upgraded to 8.8.15 P32, except we found an issue with the log4j config which broke all logging (mailbox.log, audit.log etc).

ERROR: service.FAILURE (system failure: org.apache.logging.log4j.core.config.ConfigurationException: No name attribute provided for Logger slogger)

There seems to be an error in /opt/zimbra/conf/log4j.properties.in, on line 211 (near the bottom of the file)
%%comment VAR:!zimbraLogHostname%%logger.slogger.name = zimbra.slogger
%%comment VAR:!zimbraLogHostname%%logger.slogger.level=INFO
%%uncomment VAR:!zimbraLogHostname%%logger.slogger.level=ERROR
%%comment VAR:!zimbraLogHostname%%logger.slogger.additivity = false
%%comment VAR:!zimbraLogHostname%%logger.slogger.appenderRef.SLOGGER.ref = sloggerAppender
That red part should be "%%comment" as well, or it refers to an undefined logger "slogger", if you're not using syslog.
shanmarsh28
Posts: 14
Joined: Tue Dec 12, 2017 8:14 pm

Re: Post-June 17 Patch 25/32 Success Stories Roll Call

Post by shanmarsh28 »

We have upgraded from patch 31 to patch 32 (8.8.15.GA.3869.UBUNTU18.64 UBUNTU18_64 FOSS edition, Patch 8.8.15_P32.). The upgrade itself went cleanly however on OS restart the server lost the ability to resolve DNS. systemd-resolve appeared to have lost its settings and its daemon service was disabled. Obviously, this affected everything from dig all the way through to SpamCop, email delivery (due to reject_unkown_helo_hostname & rekect_unknown_sender_domain), both being active and Zimbras ability to connect to our AWS SMTP relay.

For now, we had to edit /etc/systemd/resolved.conf and run commands like: sudo systemd-resolve --set-dns=<ip_addresss> --set-domain=eu-west-1.compute.internal --set-llmnr=yes --set-mdns=no --set-dnssec=no --interface=ens5

Within Zimbra I ran:
$ zmprov ms <servername> +zimbraDNSMasterIP 8.8.8.8

Currently, the resolution is not perfect but everything is running without errors. Our network interface via networkctl is still reporting routable/unmanaged - that might cause issues later on. Not quite sure how to resolve that, or even if I need to, to be honest!

No idea if this issue was related/created by the patch upgrade but it happened exactly the same time we installed the patch and restarted the server.

Shane
Last edited by shanmarsh28 on Tue Jun 21, 2022 3:34 pm, edited 1 time in total.
saket.patel
Zimbra Employee
Zimbra Employee
Posts: 137
Joined: Mon Apr 11, 2022 8:39 pm

Re: Post-June 17 Patch 25/32 Success Stories Roll Call

Post by saket.patel »

If you are using apt upgrade for patch upgrade then it might be possible that you have pulled some other packages which broke dns resolution
I don't think zimbra would be able to change dns related configurations as part of upgrade step
shanmarsh28
Posts: 14
Joined: Tue Dec 12, 2017 8:14 pm

Re: Post-June 17 Patch 25/32 Success Stories Roll Call

Post by shanmarsh28 »

saket.patel wrote:If you are using apt upgrade for patch upgrade then it might be possible that you have pulled some other packages which broke dns resolution
I don't think zimbra would be able to change dns related configurations as part of upgrade step
It's very possibly not related to Zimbra but I don't know enough to say one way or the other. I've upgraded in this way (using apt), for a very long time without any issues at all and this time around we had issues. I hope no one else runs into this.
ghen
Outstanding Member
Outstanding Member
Posts: 258
Joined: Thu May 12, 2016 1:56 pm
Location: Belgium
ZCS/ZD Version: 9.0.0

Re: Post-June 17 Patch 25/32 Success Stories Roll Call

Post by ghen »

Found another glitch with Patch 32; it redeploys all (standard) Zimlets, thereby overwriting all of their config stored in LDAP back to defaults, ie. losing any previous zimlet config changes.
For example we had some config changes with com_zimbra_url to disable youtubePreview (a feature that has been broken for a long time), this was reverted again by deploying the patch.
Last edited by ghen on Thu Jun 23, 2022 9:36 pm, edited 1 time in total.
Edx2Eu7
Posts: 11
Joined: Tue Jun 21, 2022 7:57 pm

Re: Post-June 17 Patch 25/32 Success Stories Roll Call

Post by Edx2Eu7 »

Sorry to cross post here - running into viewtopic.php?f=15&t=70897&p=305560#p305560 since upgrading to 8.8.15_P32.

Following the conversations here I currently assume that there is no other reliable way to get the system back to a working state, but restoring backup taken before applying patch 8.8.15_P32? Once restored to an older version re-apply 8.8.15_P32? I'm still hoping not to have to do a full restore.

Best regards
Ed
User avatar
L. Mark Stone
Ambassador
Ambassador
Posts: 2796
Joined: Wed Oct 09, 2013 11:35 am
Location: Portland, Maine, US
ZCS/ZD Version: 10.0.6 Network Edition
Contact:

Re: Post-June 17 Patch 25/32 Success Stories Roll Call

Post by L. Mark Stone »

(Eventual) Success Story...

Zimbra 9 Patch 30 single server on Ubuntu 20.

Installed the first three packages OK, but then there was an issue with the repos when we went to do "apt-get upgrade". The error morphed over the next hour into a different error, but basically if we had proceeded not all the new packages would have been installed.

We opened a Support Case, they felt there was an issue with at least one of the Ubuntu 20 repos not syncing properly. Fixed, they asked me to try again; same issue.

In the event, the net process was:

Code: Select all

apt clean
apt update
apt-get install zimbra-common-core-jar zimbra-common-core-libs zimbra-mbox-store-libs
apt-get upgrade
su - zimbra
zmcontrol restart
exit
apt-get dist-upgrade
su - zimbra
zmcontrol stop && exit
reboot now
Zimbra came up fine. Hope that helps.
___________________________________
L. Mark Stone
Mission Critical Email - Zimbra VAR/BSP/Training Partner https://www.missioncriticalemail.com/
AWS Certified Solutions Architect-Associate
Post Reply