June 2021 Zeta Alliance Weekly Call Summaries

Industry info, happenings near you, and new product integrations. Hosting an event? Invite people here.
Post Reply
User avatar
rleiker
Advanced member
Advanced member
Posts: 149
Joined: Tue Jan 07, 2020 8:23 pm
Location: Kansas City
Contact:

June 2021 Zeta Alliance Weekly Call Summaries

Post by rleiker »

Hello Zimbra Community,

Here is a summary of this week’s conference call. A few brief reminders: June 1, 2021

Upcoming 8.8.15 Patch 22 and 9.0 Patch 15
Mark S. asked if the next patches were coming out the week of May 31st. John H. said Zimbra is currently projecting either late in the week of May 31st or the following week. Mark said that the preview release notes for Zimbra partners suggest the patches will change the Zimbra Video Server status from beta to generally available and said he is looking forward to trying it out to see if it makes video calls with Zimbra Connect more scalable than using a standalone TURN server.

ARC Support In Zimbra
Noah P. said he has been recently working on implementing DKIM ( https://en.wikipedia.org/wiki/DomainKey ... ified_Mail ) in Zimbra and asked if anyone has implemented ARC support ( https://en.wikipedia.org/wiki/Authentic ... ived_Chain ) in Zimbra for re-signing messages. He said he recalled a Postfix Milter being discussed on a Zeta Alliance call last February 2020 ( http://forums.zimbra.org/viewtopic.php? ... 26#p295876 ) that he thought may make ARC support available. Mark S. said that he has not found that forwarding of DKIM signed messages has resulted in false positives for his Zimbra users and thinks that most email providers have found this to be a typical use case, so they are allowing it, even if it violates a DMARC policy (https://en.wikipedia.org/wiki/DMARC ). Randy L. said that he thinks John W. at Xmission may have been working on implementing ARC support in Zimbra and suggested that Noah may want to send him a note to see if he had any additional information to share.

Unexpectedly Large Zimbra Log Files
Jorge C. said that he recently migrated from a single Zimbra server install to a multi-server install as described in the Zimbra Administration guide. Post-migration, he said he encountered a problem where his zimbra.log file grew exponentially from 500 MB to 3 GB in about 3 hours. Similarly, his Zimbra stats log file grew to 27 GB in the same time period. He opened Zimbra support case # 01173606 and was assigned Zimbra bug ID: ZBUG-2234. John H. said this bug is an open issue and is caused by an incorrect virtual host name entry in Jorge’s /etc/rsyslog.d/60-zimbra.conf file. Jorge said he now has the following entries in his 60-zimbra.conf file:

local0.* -/var/log/zimbra.log
local1.* -/var/log/zimbra-stats.log
auth.* -/var/log/zimbra.log
mail.* -/var/log/zimbra.log

But previously had virtual host names listed like:

local0.* @hostname.domain
local1.* @hostname.domain
auth.* @hostname.domain
local0.* -/var/log/zimbra.log
local1.* -/var/log/zimbra-stats.log
auth.* -/var/log/zimbra.log
mail.* @hostname.domain
mail.* -/var/log/zimbra.log

Jorge explained that the virtual host lines were present when he had a single server Zimbra install and it did not have the rapid log file growth issue at that time, so he was not sure why this likewise did not also work in a multi-server install. Matthew F. asked which of the Zimbra servers has the Zimbra logger service running, as he wondered if Jorge had inadvertently setup two servers running the logger service, when there should only be a single server running the logger service. Jorge said he did not know the answer to Matthew’s question. Mark S. said that he has found that Zimbra patches will occasionally force syslogging in to a non-functional state, and in those instances that running the /opt/zimbra/libexec/zmsyslogsetup utility ( https://wiki.zimbra.com/wiki/Configuring-Logger-Host ) followed by a “service syslog restart” command helps bring things back in to an operational state. Mark added that he finds syslogging over UDP does not seem reliable, where some network switches do not handle it well, and suggested using TCP logging by changing:

local0.* @@hostname.domain

Then running: “service rsyslog restart”. Next, in /etc/rsyslog.conf set:

# provides UDP syslog reception
module(load="imudp")a
input(type="imudp" port="514")

# provides TCP syslog reception
#module(load="imtcp")
#input(type="imtcp" port="514")

Uncomment the TCP related lines above then do another restart of the rsyslog service.


Randy Leiker
Skyway Networks, LLC
User avatar
rleiker
Advanced member
Advanced member
Posts: 149
Joined: Tue Jan 07, 2020 8:23 pm
Location: Kansas City
Contact:

Re: June 2021 Zeta Alliance Weekly Call Summaries

Post by rleiker »

Hello Zimbra Community,

Here is a summary of this week’s conference call. A few brief reminders: June 8, 2021

Proposal To Change The Zeta Alliance Conferencing Platform
Barry D. proposed that the Zeta Alliance consider moving away from freeconferencecall.com as the conferencing platform for the weekly Tuesday calls and instead use the Jitsi integration in either Zimbra 9 or Zimbra Cloud. He said one advantage of using freeconferencecall.com over the years has been a dial-in number for those not using the freeconferencecall.com desktop app, and he was not sure if this would be possible to replicate within Jitsi. He suggested sending a note to the Zeta Alliance mailing list to hold a vote on this platform change.

Setting Up Centralized Logging For Zimbra With Elastic Stack
Barry D. said he has published a new guide ( https://github.com/Zimbra/elastic-stack ) describing how to setup centralized logging in Zimbra with Elastic Stack and Rsyslog for visualization of log events and gathering new insights. This also makes it easy to search historic logs, in addition to monitoring service uptime and SSL certificate status. He explained that his guide shows that you do not need to have Elastic Stack components installed on all of the Zimbra servers, but rather just on a dedicated server instead. He has also published a guide introducing how to use a Zimlet to capture events: ( https://github.com/Zimbra/zimlet-cli/wi ... e-a-Zimlet ). Barry shared the following links to other helpful development guides:
Integrating Prometheus, Fluentd, Elastic Stack, and Logstash With Zimbra
Marc G. asked if Synacor has plans to create an integration for Prometheus ( https://prometheus.io/ ) for Zimbra log data, as he is currently trying to feed log data to the logz.io platform ( https://logz.io/ ) and they require logs to be formatted with Prometheus. Barry D. said he is not aware of any current plans to do so, but said that if enough people ask for this feature, it would increase the probability of this integration in the future. John E. said that this should be possible, as Synacor is using an integration with Prometheus internally and much depends on how Marc opts to get logs out of Zimbra. He explained that he has found Fluentd ( https://www.fluentd.org/ ) and Fluent Bit ( https://fluentbit.io/ ) helpful. Fluent Bit can be deployed on individual Zimbra servers to avoid deploying the complete Fluentd stack on each Zimbra server. Fluent Bit then communicates with a dedicated server running Fluentd and it can push log data to just about any destination, since many log data transformers are available. John E. added that using Logstash ( https://www.elastic.co/logstash ) is also useful for parsing the logs, thereby taking the load of parsing off of the Zimbra servers.

Marc G. said that he is seeking to export both Zimbra logs and deep monitoring of metrics to detect known patterns of system usage known to create problems. John E. said that both log and metric data can co-exist when using Elastic Stack with Zimbra. Barry D. asked what is the benefit of using Prometheus instead of Elastic Stack? John E. said that Prometheus is good for collecting metrics and monitoring tasks, while Elastic Stack excels at collecting log data. He also said that Prometheus’ alert manager feature works well for monitoring metrics.

Marc G. said that he thinks auditing Zimbra logs for SIEM-like ( https://en.wikipedia.org/wiki/Security_ ... management ) things, such as security events, in addition to understanding mail flow would be very helpful. Another example includes showing the lifecycle of an email as it travels from entry to final destination, or to unexpected places too, like a user’s misconfigured Zimbra filter. John E. recommended using Elastic Stack with Logstash for this purpose. He said that Marc’s initial focus should be on getting his Zimbra log data in to Elastic Stack where he will discover many different ways to use this data. Barry D. said that this is the goal of the Elastic Stack guide that he recently published, to help get everyone started, and suggested that Marc begin with getting RSyslog configured first, as it is helpful for setting up security logging. Then, he can proceed to working on getting Elastic Stack configured. Marc G. asked how RSyslog handles re-sending log data if there is a communication fault? Barry D. said that RSyslog does not handle this scenario by default, but thinks it can be configured to re-try sending logs.

Barry D. asked how much does Prometheus costs? Marc G. said that it is free open source software. John E. said that a Prometheus integration is the kind of thing you may not want delivered as part of Zimbra, especially with the additional external infrastructure required to handle the log data, requiring a non-standardized setup for each Zimbra installation. Marc G. suggested that this would be a great open source project to make it easier for the Zimbra community to implement log visualization and metric monitoring features. He said that many log analysis vendors are charging eye watering amounts of money for storing log data like this, so he often stores his log data in S3 storage at AWS on limited redundancy disks. John E. said Marc might also consider storing his logs in AWS’ Glacier service, or somewhere similar, as cheaply as possible, then use Logstash to filter out what he is looking for, and import the remaining data in to Elastic Stack. John E. said that starting with log data for low volume email customers first, for a limited time frame is best, just to play around with it, and refine the filters to get the most interesting bits of log data out. He said you will then start to discover things out of the logs that you did not know you wanted. Barry D. added that it is very easy to break Elastic Stack, and can often be easier to restore a backup than trying to fix it, so using a test environment is important. He said that Elastic Stack auto updates too, and the same version can probably be run reasonably safely for about a year before needing to upgrade. Noah P. said that he has been looking in to this same need too, but there does not seem to be a cheap way around it with the log analysis vendors, so taking the free open source software route makes the most sense. Marc G. said that log analysis can feel like an endless rabbit hole, so it is important to identify the specific goals that you want to accomplish first from the Zimbra log data. Noah P. said he would love to see something for this purpose more turnkey ready from Synacor, as he finds he does not have the time in the day to get all of the components tuned and working smoothly. John E. said that Barry D’s recently published Elastic Stack guide comes the closest to Noah’s request, but this topic gets so far outside of what Synacor offers, he does not see it likely that this would be packaged and distributed within Zimbra. Barry D. said that he found in his work with Elastic Stack that storage is consumed quickly, where he had 100 GB initially allocated to his setup, that was fully consumed in just 2.5 months.


Randy Leiker
Skyway Networks, LLC
User avatar
rleiker
Advanced member
Advanced member
Posts: 149
Joined: Tue Jan 07, 2020 8:23 pm
Location: Kansas City
Contact:

Re: June 2021 Zeta Alliance Weekly Call Summaries

Post by rleiker »

Hello Zimbra Community,

Here is a summary of this week’s conference call. A few brief reminders: June 15, 2021

Administrator Tip: Changing Disk IOPS In AWS With Zimbra
Matthew F. shared a tip and said that he recently changed the disk IOPS setting for his Zimbra servers hosted at AWS from 3,000 to 6,000 on a Monday morning to increase performance. The AWS documentation says that when doing so, you should receive IOPS performance between the old and new IOPS values while the disk re-configuration is taking place, but in practice Matthew found it dropped drastically to about 30-50 IOPS during the change, leading to an outage of his Zimbra servers for about 2 hours. The disk volume this change was applied to was 750 GB in size. Matthew said he found that since the Zimbra mailboxd service was unusable by users at such a low IOPS performance level, that stopping the mailboxd service slightly decreased the time that AWS required to re-configure the IOPS setting.

2-Factor Authentication and SAML Support in Zimbra
Mark S. said that he noticed in the recent Zextras 3.2.0 release notes ( https://docs.zextras.com/zextras-suite- ... gelog.html ), there was mention of improvements around 2-factor authentication (2FA). He asked how these changes relate to Synacor’s road map plans recently shared with Zimbra Partners, such as if Zimbra will introduce the capability to choose the 2FA engine that can be used within a Zimbra installation. John E. said that no information about this topic was publicly available at this time.

Mark S. also mentioned that the Zextras release notes discuss some changes relating to SAML ( https://en.wikipedia.org/wiki/Security_ ... p_Language ) extension in Zimbra. He said that one of his customers is trying to get Zimbra to work with a user identity management product from Okta ( https://www.okta.com/ ) and they have been encountering some challenges. John E. asked if Mark’s customer is using the built-in SAML extension in Zimbra ( https://zimbra.github.io/zimbra-9/admin ... plesamlphp )? Mark confirmed that the customer is using this feature. John said that the SAML implementation has been designed to be very simple and that Synacor has done many customer implementations with it. He suggested that it is straightforward to modify the SAML extension for the desired integration, but noted that it is impossible for Synacor to distribute a single implementation that works for all use cases. He added that one perk of Zimbra is that its authentication chain is very pluggable, so if you can dream it, it can likely be built. John said the SAML extension was significantly revamped between Zimbra 8.8.15 and 9.

When Zimbra 9 was first released, it was highly similar to 8.8.15, but has diverged greatly since, and John E. encouraged everyone to upgrade to Zimbra 9, if they have the means. As far as he is aware, the improvements to the SAML extension in 9 were not backported to 8.8.15. Additionally, the 2FA and self-service user password reset feature (reset forgotten passwords) were fixed a long time ago in Zimbra 9, but those fixes have not yet been backported to 8.8.15. John said he has been pushing internally at Synacor to backport these fixes to 8.8.15, but the developers have pushed back explaining that there would be significant changes required in 8.8.15 to make these backports possible. He said he is going to keep pushing for these backports, but it is not clear if this will ever occur, as the challenge is in deciding where to invest developer resources – either in backporting fixes, or focusing on newer Zimbra versions. John said that if coming from a Zimbra version earlier than 8.8.15, it is important to do a stepped upgrade, by upgrading to 8.8.15 first, then to Zimbra 9 or later, rather than attempting to directly upgrade from a version prior to 8.8.15 to 9.

Upcoming Changes In Zimbra Version Designation and Packaging
John E. shared that Synacor is likely to move away in the near future from identifying Zimbra versions in its product marketing with numbers like Zimbra 9 and instead change to a continuous release model, where the product simply becomes known as Zimbra. Packaging systems (Yum and Apt) will become the norm for distributing all updates, as compared to the current practice of downloading and running an installer script for major version changes. Mark S. asked if it will still be possible to upgrade to specific Zimbra versions by using a packaging system. John E. said he believes it will be possible to do a targeted update using only the packaging systems. Noah P. pointed out that for other software companies using the continuous release model, there are certain breakpoints where compatibility is broken with older versions, and asked how Synacor will handle this scenario, as he would not want to see Synacor in a position of needing to balloon their quality assurance process to support esoteric operating systems and very old Zimbra releases. John E. said that updates will still have package version numbers, while Zimbra will not be marketed with a specific release number, but could not comment further. Randy L. asked if Synacor plans to continue to integrate security fixes with new Zimbra builds in the packaging systems as they do now, or if a separate repo will be introduced containing security fixes only. John E. said he anticipates security patches will continue to be released with the same approach, as they are currently. Randy also asked if Zimbra will continue to maintain separate repos for each major release train, allowing Zimbra administrators a choice of the major release train they wish to subscribe to, or if only a single release repo will be used. John E. said he was not sure how this will be structured in the future. Mark S. asked how this future versioning and packaging change might affect anyone wanting to build Zimbra from source code. John E. said that this change should not impose any problems for those wanting to build from source, and that if problems do arise, it should be raised with Zimbra Support as a defect issue. He added that the build system for Zimbra can be complicated and finicky, but if you follow the documented instructions carefully, it always works. He also commented that Synacor continues to be very committed to releasing Zimbra source code, and if confidence in this statement is needed, consider the large number of on-premise customers that rely on Zimbra’s open source commitment, especially those in Asia and the EU, where they have mandates for using open source products. These are some of Synacor’s larger customers in those regions. Noah P. added that, even for Zimbra Partners who do not rely on the open source commitment, due to the availability of commercial support, he still uses the open source commitment as a marketing benefit to draw customers away from proprietary, closed platforms, like Microsoft or Google’s products. John E. said that the only reason that features like Exchange ActiveSync and Exchange Web Services, that are integrated in Zimbra, are not open sourced as well, is that those features have associated licensing by the patent holder (Microsoft) that needs to be paid as royalties.

Update For Zimbra Support Of Ubuntu 20.04 LTS
Randy L. recalled that it was discussed in an earlier Zeta Alliance call that Ubuntu 20.04 LTS support in Zimbra was projected for June 2021, and asked if any status updates were available, as he was waiting to do some Zimbra 9 upgrades pending its availability. John H. said that this was the original projection, but with COVID-19 significantly impacting the region where the Zimbra development team is located, Ubuntu 20.04 support has been pushed back to later this Summer.

Simultaneously Updating Zimbra Mailbox Servers
Matthew F. asked if anyone on the call patches multiple Zimbra mailbox servers simultaneously, or if they patch mailbox servers sequentially, one-at-a-time. Randy L. said that he routinely does simultaneous patching of his mailbox servers in a cluster, as he takes a snapshot of each before upgrading, allowing for quick and easy roll backs if anything goes wrong. He also added that this is frequently necessary to hit his maintenance window targets with customers. Mark S. said that he used to patch mailbox servers simultaneously, but no longer does so as he has had problems with this approach in the past. He added that multiple Zimbra versions should be able to co-exist within the same cluster as a rolling upgrade, so sequential roll out should not present a problem. Matthew F. said that his only concern in doing simultaneous updates is that he sees the patches undeploying and redeploying Zimlets, so he is unsure if he needs the other mailbox servers online while running an update. Randy L. said that this has been of concern to him too, but he has yet to see this present a problem post-update after installing countless patches in the past, perhaps because each mailbox server update process may be redundantly undeploying and redeploying Zimlets on each local mailbox server. Matthew F. said he will open a support case with Zimbra to try and obtain a more official answer on this topic.

Recommended Zimbra 8.8.15 to 9 Upgrade Procedure
Gary C. said that he will soon be doing a Zimbra 8.8.15 to 9 multi-server upgrade. He asked if things are pretty smooth now with the 8.8.15 to 9 upgrade process. John H. said that Synacor has a large number of customers that have made the upgrade without issue. Gary said he is planning to upgrade his LDAP servers first, then the proxies, followed by his MTA servers, and finally the mailbox servers. He asked, in terms of timing, if he does not upgrade all of his mailbox servers within the same short period of time, how might this affect the operation of the cluster and his Zimbra users? Mark S. said the official Zimbra rolling upgrade documentation says that you should have at least one mailbox server running version 9, even if the other mailbox servers have not yet been upgraded, so that the Zimbra Administration Console can be run from that Zimbra 9 mailbox server to manage the other 8.8.15 mailbox servers. Gary said in that case, he may roll out a new mailbox server with version 9 for this purpose and upgrade his existing 8.8.15 mailbox servers to 9 later. Noah P. said that, given the scenario Gary describes, he believes that all of Gary’s Zimbra users should see the new Zimbra Web Client login screen (Modern UI), but that users may see a mix of Classic and Modern UI experiences post-login. Mark S. said that Gary can also take 8.8.15 mailbox servers out of his reverse proxy list, with the appropriate zmprov command, so that those are not available for clients to use. Matthew F. added that Gary may need to do this for the Zimbra Administration Console as well. Noah P. asked if anyone knew of issues encountered by having a Zimbra 9 login page logging in to an 8.8.15 mailbox server. Mark S. said that he had a customer do an extended rolling Zimbra upgrade over a number of months, and there did not appear to be any problems with this approach.

Gary asked for the thoughts of everyone on the call about the wisdom of doing an in-place upgrade for Ubuntu with Zimbra installed. Mark S. said that if it is a single-server install of Zimbra, he finds that it is much less risky to build a new Ubuntu server with the desired version, then doing an incremental migration upgrade of Zimbra ( https://zimbra.github.io/zimbra-9/admin ... ith-backup ) from the old to the new server. Or, for multi-server Zimbra installations, he suggested first building new Ubuntu servers with the desired version, installing Zimbra LDAP, and promoting those new LDAP servers to leaders, then demoting the old Ubuntu LDAP servers to followers/replicas, and finally deleting those old LDAP servers from the cluster.

Gary C. asked, if he has HSM enabled on all of his mailbox servers with the Centralized Storage feature in Zimbra, and he performs mailbox moves ( https://docs.zextras.com/zextras-suite- ... mailstores ) from his 8.8.15 to 9 servers, would any issues be expected? Mark S. said Gary could encounter sporadic time out issues with moving exceptionally large mailboxes, including a user’s mailbox that Gary said is close to 1 TB in size. Gary asked if moving mailboxes in this manner would cause a mailbox in Zimbra’s Centralized Storage to revert to using Zimbra’s primary storage volume instead? Mark S. and Randy L. commented that this should not be an issue if Gary uses the proper options for the “zxsuite hsm doMailboxMove” command to respect the current storage policy. Randy added that he recalled a detailed description provided by Cine on an earlier Zeta Alliance call of the process the “zxsuite hsm doMailboxMove” command uses to check the capabilities of the source and destination mailbox servers and how it auto-negotiates which features of of the command are available based on the supported Zimbra and Zextras versions of each server.


Randy Leiker
Skyway Networks, LLC
Post Reply