Hello Zimbra Community,
Here is a summary of this week’s conference call. A few brief reminders:
June 15, 2021
Administrator Tip: Changing Disk IOPS In AWS With Zimbra
Matthew F. shared a tip and said that he recently changed the disk IOPS setting for his Zimbra servers hosted at AWS from 3,000 to 6,000 on a Monday morning to increase performance. The AWS documentation says that when doing so, you should receive IOPS performance between the old and new IOPS values while the disk re-configuration is taking place, but in practice Matthew found it dropped drastically to about 30-50 IOPS during the change, leading to an outage of his Zimbra servers for about 2 hours. The disk volume this change was applied to was 750 GB in size. Matthew said he found that since the Zimbra mailboxd service was unusable by users at such a low IOPS performance level, that stopping the mailboxd service slightly decreased the time that AWS required to re-configure the IOPS setting.
2-Factor Authentication and SAML Support in Zimbra
Mark S. said that he noticed in the recent Zextras 3.2.0 release notes (
https://docs.zextras.com/zextras-suite- ... gelog.html ), there was mention of improvements around 2-factor authentication (2FA). He asked how these changes relate to Synacor’s road map plans recently shared with Zimbra Partners, such as if Zimbra will introduce the capability to choose the 2FA engine that can be used within a Zimbra installation. John E. said that no information about this topic was publicly available at this time.
Mark S. also mentioned that the Zextras release notes discuss some changes relating to SAML (
https://en.wikipedia.org/wiki/Security_ ... p_Language ) extension in Zimbra. He said that one of his customers is trying to get Zimbra to work with a user identity management product from Okta (
https://www.okta.com/ ) and they have been encountering some challenges. John E. asked if Mark’s customer is using the built-in SAML extension in Zimbra (
https://zimbra.github.io/zimbra-9/admin ... plesamlphp )? Mark confirmed that the customer is using this feature. John said that the SAML implementation has been designed to be very simple and that Synacor has done many customer implementations with it. He suggested that it is straightforward to modify the SAML extension for the desired integration, but noted that it is impossible for Synacor to distribute a single implementation that works for all use cases. He added that one perk of Zimbra is that its authentication chain is very pluggable, so if you can dream it, it can likely be built. John said the SAML extension was significantly revamped between Zimbra 8.8.15 and 9.
When Zimbra 9 was first released, it was highly similar to 8.8.15, but has diverged greatly since, and John E. encouraged everyone to upgrade to Zimbra 9, if they have the means. As far as he is aware, the improvements to the SAML extension in 9 were not backported to 8.8.15. Additionally, the 2FA and self-service user password reset feature (reset forgotten passwords) were fixed a long time ago in Zimbra 9, but those fixes have not yet been backported to 8.8.15. John said he has been pushing internally at Synacor to backport these fixes to 8.8.15, but the developers have pushed back explaining that there would be significant changes required in 8.8.15 to make these backports possible. He said he is going to keep pushing for these backports, but it is not clear if this will ever occur, as the challenge is in deciding where to invest developer resources – either in backporting fixes, or focusing on newer Zimbra versions. John said that if coming from a Zimbra version earlier than 8.8.15, it is important to do a stepped upgrade, by upgrading to 8.8.15 first, then to Zimbra 9 or later, rather than attempting to directly upgrade from a version prior to 8.8.15 to 9.
Upcoming Changes In Zimbra Version Designation and Packaging
John E. shared that Synacor is likely to move away in the near future from identifying Zimbra versions in its product marketing with numbers like Zimbra 9 and instead change to a continuous release model, where the product simply becomes known as Zimbra. Packaging systems (Yum and Apt) will become the norm for distributing all updates, as compared to the current practice of downloading and running an installer script for major version changes. Mark S. asked if it will still be possible to upgrade to specific Zimbra versions by using a packaging system. John E. said he believes it will be possible to do a targeted update using only the packaging systems. Noah P. pointed out that for other software companies using the continuous release model, there are certain breakpoints where compatibility is broken with older versions, and asked how Synacor will handle this scenario, as he would not want to see Synacor in a position of needing to balloon their quality assurance process to support esoteric operating systems and very old Zimbra releases. John E. said that updates will still have package version numbers, while Zimbra will not be marketed with a specific release number, but could not comment further. Randy L. asked if Synacor plans to continue to integrate security fixes with new Zimbra builds in the packaging systems as they do now, or if a separate repo will be introduced containing security fixes only. John E. said he anticipates security patches will continue to be released with the same approach, as they are currently. Randy also asked if Zimbra will continue to maintain separate repos for each major release train, allowing Zimbra administrators a choice of the major release train they wish to subscribe to, or if only a single release repo will be used. John E. said he was not sure how this will be structured in the future. Mark S. asked how this future versioning and packaging change might affect anyone wanting to build Zimbra from source code. John E. said that this change should not impose any problems for those wanting to build from source, and that if problems do arise, it should be raised with Zimbra Support as a defect issue. He added that the build system for Zimbra can be complicated and finicky, but if you follow the documented instructions carefully, it always works. He also commented that Synacor continues to be very committed to releasing Zimbra source code, and if confidence in this statement is needed, consider the large number of on-premise customers that rely on Zimbra’s open source commitment, especially those in Asia and the EU, where they have mandates for using open source products. These are some of Synacor’s larger customers in those regions. Noah P. added that, even for Zimbra Partners who do not rely on the open source commitment, due to the availability of commercial support, he still uses the open source commitment as a marketing benefit to draw customers away from proprietary, closed platforms, like Microsoft or Google’s products. John E. said that the only reason that features like Exchange ActiveSync and Exchange Web Services, that are integrated in Zimbra, are not open sourced as well, is that those features have associated licensing by the patent holder (Microsoft) that needs to be paid as royalties.
Update For Zimbra Support Of Ubuntu 20.04 LTS
Randy L. recalled that it was discussed in an earlier Zeta Alliance call that Ubuntu 20.04 LTS support in Zimbra was projected for June 2021, and asked if any status updates were available, as he was waiting to do some Zimbra 9 upgrades pending its availability. John H. said that this was the original projection, but with COVID-19 significantly impacting the region where the Zimbra development team is located, Ubuntu 20.04 support has been pushed back to later this Summer.
Simultaneously Updating Zimbra Mailbox Servers
Matthew F. asked if anyone on the call patches multiple Zimbra mailbox servers simultaneously, or if they patch mailbox servers sequentially, one-at-a-time. Randy L. said that he routinely does simultaneous patching of his mailbox servers in a cluster, as he takes a snapshot of each before upgrading, allowing for quick and easy roll backs if anything goes wrong. He also added that this is frequently necessary to hit his maintenance window targets with customers. Mark S. said that he used to patch mailbox servers simultaneously, but no longer does so as he has had problems with this approach in the past. He added that multiple Zimbra versions should be able to co-exist within the same cluster as a rolling upgrade, so sequential roll out should not present a problem. Matthew F. said that his only concern in doing simultaneous updates is that he sees the patches undeploying and redeploying Zimlets, so he is unsure if he needs the other mailbox servers online while running an update. Randy L. said that this has been of concern to him too, but he has yet to see this present a problem post-update after installing countless patches in the past, perhaps because each mailbox server update process may be redundantly undeploying and redeploying Zimlets on each local mailbox server. Matthew F. said he will open a support case with Zimbra to try and obtain a more official answer on this topic.
Recommended Zimbra 8.8.15 to 9 Upgrade Procedure
Gary C. said that he will soon be doing a Zimbra 8.8.15 to 9 multi-server upgrade. He asked if things are pretty smooth now with the 8.8.15 to 9 upgrade process. John H. said that Synacor has a large number of customers that have made the upgrade without issue. Gary said he is planning to upgrade his LDAP servers first, then the proxies, followed by his MTA servers, and finally the mailbox servers. He asked, in terms of timing, if he does not upgrade all of his mailbox servers within the same short period of time, how might this affect the operation of the cluster and his Zimbra users? Mark S. said the official Zimbra rolling upgrade documentation says that you should have at least one mailbox server running version 9, even if the other mailbox servers have not yet been upgraded, so that the Zimbra Administration Console can be run from that Zimbra 9 mailbox server to manage the other 8.8.15 mailbox servers. Gary said in that case, he may roll out a new mailbox server with version 9 for this purpose and upgrade his existing 8.8.15 mailbox servers to 9 later. Noah P. said that, given the scenario Gary describes, he believes that all of Gary’s Zimbra users should see the new Zimbra Web Client login screen (Modern UI), but that users may see a mix of Classic and Modern UI experiences post-login. Mark S. said that Gary can also take 8.8.15 mailbox servers out of his reverse proxy list, with the appropriate zmprov command, so that those are not available for clients to use. Matthew F. added that Gary may need to do this for the Zimbra Administration Console as well. Noah P. asked if anyone knew of issues encountered by having a Zimbra 9 login page logging in to an 8.8.15 mailbox server. Mark S. said that he had a customer do an extended rolling Zimbra upgrade over a number of months, and there did not appear to be any problems with this approach.
Gary asked for the thoughts of everyone on the call about the wisdom of doing an in-place upgrade for Ubuntu with Zimbra installed. Mark S. said that if it is a single-server install of Zimbra, he finds that it is much less risky to build a new Ubuntu server with the desired version, then doing an incremental migration upgrade of Zimbra (
https://zimbra.github.io/zimbra-9/admin ... ith-backup ) from the old to the new server. Or, for multi-server Zimbra installations, he suggested first building new Ubuntu servers with the desired version, installing Zimbra LDAP, and promoting those new LDAP servers to leaders, then demoting the old Ubuntu LDAP servers to followers/replicas, and finally deleting those old LDAP servers from the cluster.
Gary C. asked, if he has HSM enabled on all of his mailbox servers with the Centralized Storage feature in Zimbra, and he performs mailbox moves (
https://docs.zextras.com/zextras-suite- ... mailstores ) from his 8.8.15 to 9 servers, would any issues be expected? Mark S. said Gary could encounter sporadic time out issues with moving exceptionally large mailboxes, including a user’s mailbox that Gary said is close to 1 TB in size. Gary asked if moving mailboxes in this manner would cause a mailbox in Zimbra’s Centralized Storage to revert to using Zimbra’s primary storage volume instead? Mark S. and Randy L. commented that this should not be an issue if Gary uses the proper options for the “zxsuite hsm doMailboxMove” command to respect the current storage policy. Randy added that he recalled a detailed description provided by Cine on an earlier Zeta Alliance call of the process the “zxsuite hsm doMailboxMove” command uses to check the capabilities of the source and destination mailbox servers and how it auto-negotiates which features of of the command are available based on the supported Zimbra and Zextras versions of each server.
Randy Leiker
Skyway Networks, LLC