March 2021 Zeta Alliance Weekly Call Summaries

Industry info, happenings near you, and new product integrations. Hosting an event? Invite people here.
Post Reply
User avatar
rleiker
Advanced member
Advanced member
Posts: 149
Joined: Tue Jan 07, 2020 8:23 pm
Location: Kansas City
Contact:

March 2021 Zeta Alliance Weekly Call Summaries

Post by rleiker »

Hello Zimbra Community,

Here is a summary of this week’s conference call. A few brief reminders: March 2, 2021

Purging Expired Mailbox Data From S3 Backups
Marc G. and Matthew F. shared they have been trying to get Zimbra backups to S3 storage working, but have encountered a bug. They explained that they cannot purge files from the S3 backups if there is an empty mailbox account. The bug seems to present itself as a lot of missing meta data for a corrupted mailbox.

Matthew F. suspects this bug is two separate issues, as it is showing a corrupt mailbox, but then the purge goes further after the corrupted mailbox is fixed, and again gets stuck on another empty mailbox. He has observed that the S3 backup purges appear to get stuck on the same empty mailbox everyday. He explained that this is a new Zimbra mailbox server that originally had real-time NG backups periodically enabled, but has always used S3 for backups. If he runs a repair on the empty mailbox, it will fix the corruption, but as soon as the next purge runs, it will report corruption on the same mailbox again. Mark S. said he has a related issue with performing S3 backup purges where when he tries to run a purge on his mailbox server configured for S3 backups, the purge attempts to run on his local backup storage, rather than his backups in S3.

John H. said that the bug Matthew described was just reported to Zimbra and is anticipated to be fixed in 8.8.15 Patch 21 and 9.0 Patch 14. Cine said that it could be related to a bug in Zextras 3.1.8 Backup Core triggered by empty mailboxes, and he seemed to recall a similar resolved issue encountered when doing mailbox moves. He suggested that Matthew try and create a new Class Of Service (COS) with backups disabled, and then place the problematic empty mailboxes in the new COS.

Feedback Requested for New ZCO Partial Sync Feature
John H. asked if anyone on the call has started using the new partial synchronization feature ( https://zimbra.github.io/zm-windows-com ... mized_sync ) recently introduced in the Zimbra Connector for Outlook (ZCO). Noah P. said not yet, but that he is looking forward to trying it out for some of his customers that have very large mailboxes that currently exceed Outlook’s size limits.

Upcoming Zimbra Conferences
It was mentioned that the North America Zimbra Partners conference is scheduled for March 18th. For Spanish/Portuguese partners, there is a LatAm Zimbra conference scheduled for March 3rd. The conferences will include a partner update, product road map, and new announcements.

Handling GDPR Data Removal Requests In Zimbra
Mark S. asked, if a Zimbra server is using third-party integrations, such as the Zimlets integrated in to the Modern UI in Zimbra 9, and a GDPR ( https://en.wikipedia.org/wiki/General_D ... Regulation ) data removal request is received, is the data in the third-party services also deleted when a Zimbra mailbox is deleted. John E. said he did not think the third-party services would delete the data since Zimbra does not have direct control of the third-party services. It would be up to the user to contact those third-party services individually to request a GDPR data removal, as it relates to the service agreements the user has with each service and their EU citizenship status. He said the only exception is that in the Zimbra 9.0 Web Client, analytics data may need to be deleted separately by the Zimbra administrator, in addition to a mailbox deletion, if the Zimbra server administrator has opted to enable this feature.

Noah P. asked if it is possible to purge backups for a given mailbox prior to the predefined automatic purge date, such as if a GDPR data removal request is received. Mark S. said he did not believe this is possible and explained that he has added language to his service agreements with his Zimbra users stating that for any deleted mailboxes, the user’s mailbox data will be deleted during the regular backup expiration purge. Marc G. said that his company does the same as Mark S. described and has not encountered any issue to-date.


Randy Leiker
Skyway Networks, LLC
User avatar
rleiker
Advanced member
Advanced member
Posts: 149
Joined: Tue Jan 07, 2020 8:23 pm
Location: Kansas City
Contact:

Re: March 2021 Zeta Alliance Weekly Call Summaries

Post by rleiker »

Hello Zimbra Community,

Here is a summary of this week’s conference call. A few brief reminders: March 9, 2021

Zimbra 8.8.15 Patch 19 Experiences
Mark S. said he installed 8.8.15 Patch 19, and has had no problems so far. He did however encounter one possible issue in Zimbra’s mailbox.log: Code:mail.TOO_MUCH_METADATA Arg:(limit, NUM, "32768"). He has opened a Zimbra Support case for further investigation.

Jorge C. said he updated to 8.8.15 P19. A week after installing the patch, he began having an issue opening Zimbra Docs Server documents in both the Chrome and Edge web browsers, but this does not affect Firefox or Safari. The error he sees when opening a document is: “Document load failed, it is possible the document is damaged or more resources are needed. Please contact support.” He does not think the issue is occurring with the Zimbra Docs Server, since if it were, it should affect all web browsers, rather than just Chrome and Edge. Cine suggested activating developer mode in the Zimbra Web Client like: https://mail.example.com/?dev=1 . This will clear all of the caches and make it easier for Zimbra Support to diagnose. He also suggested that Edge and Chrome may share some of their browser cache too since they are both using Chromium. Jorge said he tried developer mode, but it did not appear to provide any insights. Cine said in developer mode, Jorge can check the browser’s JavaScript console for errors, which will be more readable than the minified version. Jorge said he also opened support cases with both Zimbra and Zextras to troubleshoot further.

Experiences With the Zimbra Connector Partial Synchronization Feature and Feedback
Noah P. said he began testing the new partial synchronization feature for Zimbra Connector for Outlook ( https://zimbra.github.io/zm-windows-com ... mized_sync ) and wanted to know what the experience would be like for mailbox users. He asked how the purge process worked. John H. said that if you set the partial sync feature in the Zimbra Connector for 60 days, anything older gets moved to the “Zimbra Trash” folder, that is a sub-folder of the regular Trash folder. Noah P. said that he was concerned having two folders called Trash would be confusing for users. John H. said he agrees and thinks that having older messages purge immediately would be best, rather than moving them to the trash sub-folder on the client-side in Outlook. Noah P. suggested perhaps renaming the folder to something similar to “Sync Purges” that might be less confusing. John H. said he wrote a new Wiki page ( https://wiki.zimbra.com/wiki/Zco_partial_sync ) that provides additional details on this feature. He has plans to expand this Wiki page with a better explanation of how the message purge process works. Noah asked, if you set the Zimbra Connector Partial Sync feature for 60 days, and items older than this date are moved to the trash folder, does that means that the retention for deleted items is set to 60 days, or does it mean that the deleted items hang around for 120 days? John H. confirmed that items are deleted after 60 days. John H. added that if you set the Partial Sync feature for 60 days, then later change it to 90 days, it will not bring back messages older than 60 days. The only way to get those older items back in Outlook is to delete and re-create the Outlook profile, thereby causing Outlook to perform a full re-synchronization of the complete mailbox from the Zimbra server.

Zimbra Docs Server
Chad said that he recently rolled out Zimbra Docs Server and has seen some issues where manual document saves fail, especially when working on collaborative document editing with team members. Cine suggested running the latest version of everything (Zimbra, Docs Server, etc.) as compared to a mix of old and new versions of the components, since this will often remedy these types of issues. Chad said he did not have the latest Zimbra patch installed, so he will try installing it to see if that resolves the issue.

Mark S. asked, when upgrading an existing Zimbra Docs server, is it still unsupported to copy the configuration file from the existing Docs server, deploy a new Docs server, then paste in the configuration file to the new Docs server for an upgrade? Cine said he did not know if this is officially supported. Mark S. also asked how does he know if he is running the latest version of the a Zimbra Docs Server? He wanted to know if Zextras sends a new Docs Server binary to Zimbra monthly or only when it is updated. John H. said that normally a new Docs Server version will be announced in the release notes of new Zimbra patches.

Follow-Up: Purging Expired Mailbox Data From S3 Backups
Following up on the March 2nd Zeta Alliance Call discussion related to an issue purging expired mailbox data backed up to S3 storage, and mailbox corruption that seems to affect empty mailboxes that are backed up, Mark S. said that in his testing of Zimbra’s Centralized Storage feature, he has only encountered the bug related to the purging of expired data from his S3 storage, but has not seen mailbox corruption due to empty mailboxes. Matthew F. said he currently suspects the mailbox corruption issue might be due to a busy mailbox where items are frequently added/removed that he is currently investigating.

Improving S3 Throughput for Zimbra Backups and HSM
Mark S. asked when using S3 for Zimbra backups, or as an HSM secondary volume, what are the recommended ways for improving the throughput to S3 in terms of items per second. Cine said he did not think there was any means to control this, but suggested to try using multiple S3 buckets, rather than a single bucket for everything. Cine also suggested considering using an SSD drive for the cache directory. Mark S. suggested an additional option of using a Ram disk, for the caching directory’s individual S3 buckets. Matthew F. said he will also give the Ram disk suggestion a shot to see if that helps provide a performance boost on a test server.


Randy Leiker
Skyway Networks, LLC
User avatar
rleiker
Advanced member
Advanced member
Posts: 149
Joined: Tue Jan 07, 2020 8:23 pm
Location: Kansas City
Contact:

Re: March 2021 Zeta Alliance Weekly Call Summaries

Post by rleiker »

Hello Zimbra Community,

Here is a summary of this week’s conference call. A few brief reminders: March 23, 2021

Zimbra vs. Office 365 Hosted Mailboxes
There was a discussion comparing the merits of Zimbra hosted mailboxes compared to Microsoft Office 365 hosted mailboxes. Some on the call shared misunderstandings by their customers who incorrectly compare the fully managed Zimbra hosted mailboxes to the un-managed, shared responsibility model used by Office 365. Specifically, the lack of robust backup and restore capabilities in Office 365 was mentioned, where 365 backups are often limited to a simple recycle bin folder, leaving it up to the Office 365 tenant (customer) to design and operate their own backup process using third-party software under the shared responsibility model that Microsoft uses. Office 365 tenants only first become aware of this lack of a proper backup and recover safety net until after a disaster or security incident, at which point it becomes too late to reverse the damage. Randy L. commented that most Office 365 tenants he has spoken with are often unaware that Microsoft also expects each 365 tenant to assume management of their email security, aside from the basic spam and malware filtering included in the 365 services. Marc G. added that he has observed the Office 365 spam filters seem to struggle considerably with false positives, as compared to the spam filtering capabilities built in to Zimbra which seem to better tuned. It was also commented by some on the call that Office 365 tenants seem to be willing to pay a price premium for Microsoft hosted email that is a multiple higher than comparable licensing with Zimbra, often based on the Microsoft brand name alone, when in fact they are receiving a significant decrease in overall service and features.

Apple Mail Appending (null) To All Sent Email Messages
Randy L. asked about an issue reported by one of his customers who is using Apple Mail with an IMAP Zimbra account, where it appends “(null)” at the end of messages sent. The customer indicated that just before the problem appeared, they had performed a migration of their data from an older Mac to a new Mac computer using an Apple migration tool. John E. suggested checking if an email signature in Apple Mail or a domain disclaimer in Zimbra is enabled, but empty, which may be responsible for the “(null)” text.

Suggestions For Setting the Maximum Outbound Email Size
Gary C. said that one of his customers requested an increase to the maximum outbound email size limit for sending large messages, and asked for everyone’s opinion of how the size limit should be set. John E. said the largest allowed size he has heard of is 50 MB per email. Randy L. said that he sometimes explains to customers, requesting very large outbound email size limits, that the standards for sending email on the Internet were first invented in the 1970s, and were never originally designed to carry attachments. The ability to include email attachments was bolted-on as an after thought, so consequently it is very inefficient (due to MIME encoding overhead) and often causes any attachment to inflate in size by 20-30%, making it difficult for senders to determine in advance if a large attachment will fit within an outbound email size limit, aside from if the recipient’s mail server will also accept it. Randy suggested it would be best for the customer to use a file sharing service for sending large files, so as to avoid the uncertainty of trying to send large attachments via email.


Randy Leiker
Skyway Networks, LLC
User avatar
rleiker
Advanced member
Advanced member
Posts: 149
Joined: Tue Jan 07, 2020 8:23 pm
Location: Kansas City
Contact:

Re: March 2021 Zeta Alliance Weekly Call Summaries

Post by rleiker »

Hello Zimbra Community,

Here is a summary of this week’s conference call. A few brief reminders: March 30, 2021

SpamAssassin Vulnerability Prior To Zimbra 8.8.15 P20 and Prior To 9.0 P13
Randy L. shared news of recently disclosed vulnerability in SpamAssassin 3.4.4 and prior that affects Zimbra 8.8.15 Patch 20 and 9.0 Patch 13 and prior. He said that the vulnerability has a CVSS score of 9.8 out of 10 and is known as CVE-2020-1946 ( https://nvd.nist.gov/vuln/detail/CVE-2020-1946 ). The vulnerability is fixed in SpamAssassin 3.4.5 and is a form of a supply chain attack, that can be exploited in a similar manner as the victims of the Solar Winds attack widely reported by the news media. It relies on an attacker who inserts malicious code within a SpamAssassin rules (.cf) file on a repo, that when downloaded by victims during routine rule updates, can silently exploit each victim’s server with no indication an attack has taken place. John E. suggested sending a message to security@zimbra.com. John H. said he will also bring this up in an internal meeting at Synacor to discuss the response plan. Randy L. said that given the high CVSS score of 9.8, he felt this would justify an out-of-band patch from Zimbra. [Editor Note: revised builds of 8.8.15 Patch 20 and 9.0 Patch 13 were released on April 8, 2021 that fixes this vulnerability by upgrading SpamAssassin to 3.4.5. Refer to the release notes of each respective patch for further details.]

Zimbra Support For Ubuntu 16.04 LTS and 20.04 LTS
Mark S. asked when Zimbra support for Ubuntu 16.04 LTS is expected to end. He explained that he has purchased extended support from Canonical for Ubuntu 16.04 LTS and was expecting to have a 1 year window to migrate to a newer Ubuntu version, which he felt now instead looked like he would be required to migrate by the end of Summer 2021, based on the observed Zimbra deprecation timeline for Ubuntu 14.04 LTS. He explained that he wants to avoid needing to upgrade to Ubuntu 18.04 LTS within the next few months, then to 20.04 LTS immediately thereafter. He asked when Zimbra is anticipating officially announcing support for Ubuntu 20.04 LTS so as to best schedule his Ubuntu version upgrade. John H. said that Zimbra support for Ubuntu 20.04 LTS is on the road map for the first half of 2021, sometime before July, and added that he was unsure of the time frame for the end of Zimbra support for Ubuntu 16.04 LTS. John explained that Zimbra support for a Ubuntu LTS version is normally aligned with when standard support for an LTS version ends from Canonical, irrespective of any extended maintenance contracts that customers may separately purchase from Canonical. John added that it has been Zimbra’s standing policy to support an operating system while it has support from the OS vendor, up until support is only available through an optional extended support maintenance contract from the OS vendor.

Splitting A Domain Between Zimbra and Office 365
Gary C. said one of his business customers migrated their organization from Zimbra to Office 365, and there has been a vocal group of individuals within the organization hating it, and they are seeking to move back to Zimbra. He said he is looking for options to split email delivery for the customer’s domain name, so that a portion of the domain’s mailboxes can remain on Office 365, while the other portion returns back to Zimbra. He said he has an external email gateway that allows for routing mail for individuals to Zimbra and others to Office 365, that he can setup in Zimbra to be used as an SMTP smart host, however he has concerns that inter-domain mail delivery between Zimbra and Office 365 will fail.

Mark S. suggested changing the domain from authoritative to relay mode in Office 365 and leaving the DNS MX record for the domain pointing to Zimbra. He then recommended changing the zimbraMailTransport attribute setting for individual users in Zimbra, and deleting the mailboxes from Office 365 that are moving back to Zimbra. He also advised caution when using external email gateways (Barracuda, MailScanner, etc.) to avoid creating mail loops.

Noah suggested this Wiki article ( https://wiki.zimbra.com/wiki/Split_Domain ). He said he has done something similar when splitting a domain between Zimbra and G Suite, by using the methods discussed in the Wiki article. He said the trick is that administration of both services (Zimbra and G Suite) needs to be managed very carefully to avoid mail loops.

Matthew F. suggested that Gary should setup any distribution lists that his customer has in Office 365, since he has found that Zimbra ignores the zimbraMailTransport attribute for distribution lists, that can lead to mail routing issues. Marc G. asked, in terms of collaboration like sharing and free/busy, does splitting a domain make things rather messy? Mark S. confirmed that it does and did not know if the free/busy connector in Zimbra works with Office 365. John H. said he thinks there is a feature enhancement request open for the Zimbra free/busy connector to support Office 365.

All Day Calendar Events Appearing On Wrong Day In the Zimbra Calendar
Matthew F. asked if anyone has had a problem with all day calendar events being moved by 24 hours to the wrong day when using ActiveSync with a Zimbra mailbox. John H. said there was an old issue like this several patches ago. Matthew said he has 8.8.15 Patch 18 installed and did not see any mention in the release notes of a fix in P19. John H. said he thinks this issue was originally fixed back in P10. Matthew said that the issue does not seem to appear for all day calendar appointments created from a PC when Outlook is configured to use Exchange ActiveSync (EAS), but the issue does appear when a calendar appointment is created from an Android device using ActiveSync. John E. said this is known issue, since Outlook is doing more than just ActiveSync, including some out-of-band stuff. He suggested using the Zimbra Connector for Outlook, and avoiding using EAS. John E. asked if Matthew was testing with a Samsung device, and if so, said that Samsung has created their own ActiveSync version which sometimes causes unexpected things to happen. Mark S. said there is a Microsoft KB article that discusses the differences between Exchange ActiveSync (EAS) and Exchange Web Services (EWS). He added that Microsoft considers EAS deprecated and that EWS is solely for use on mobile devices.


Randy Leiker
Skyway Networks, LLC
Post Reply