May 2021 Zeta Alliance Weekly Call Summaries

[rleiker]

Hello Zimbra Community,

Here is a summary of this week’s conference call. A few brief reminders:

• Conference calls are every Tuesday and open to all using either the FreeConferenceCall.com VoIP app or via a dial-in number: https://www.freeconferencecall.com/wall/zetalliance
• Each week’s call agenda can be found at: https://drive.google.com/drive/folders/ ... jMuGGtIl7J
• Constructive feedback on these call summaries is always welcome.

May 4, 2021

Experiences with 8.8.15 Patch 21
Mark S. said that he recently installed 8.8.15 P21 and encountered no issues post-install. He also enabled TLS 1.3 support (originally introduced in P20), and removed some weak encryption ciphers from the default configuration. David M. said he also installed 8.8.15 P21 without any issues. John H. said the only known issue related to P21 occurs in very large Zimbra installations with 25+ mailbox servers when both ABQ ( https://zimbra.github.io/adminguide/lat ... bq_service ) and Zextras Cluster Management are enabled. Under these circumstances, a significant negative performance impact is incurred when a synchronization occurs between the Zimbra servers in a cluster, and is being investigated by Zimbra and Zextras. Randy L. said he has also seen a few posts in the Zimbra Forums (example: viewtopic.php?f=13&t=69530#p301307 ) where some are discussing problems with SpamAssassin not performing rules updates after an upgrade to either 8.8.15 P20/P21 or 9.0 P13/P14. According to the reports, this appears to be related to a regression bug introduced in SpamAssassin 3.4.5, that was initially included in 8.8.15 P20 and 9.0 P13 to address a high severity security vulnerability. The issue is reportedly fixed in SpamAssassin in 3.4.6, but has not yet been upgraded in Zimbra. P21 and P14 both ship with SpamAssassin 3.4.5.

Follow-Up: Purging Expired Mailbox Data From S3 Backups
Following up on the March 2nd ( viewtopic.php?f=9&t=69488#p301132 ) and April 16th Zeta Alliance calls, Matthew F. asked if Mark S. had tried running a mailbox purge on his backups in his Zimbra Centralized Storage ( https://zimbra.github.io/adminguide/lat ... ed-storage ) after installing 8.8.15 Patch 21, since it includes a fix for NG Backup for this issue. Mark S. said he did, and it unfortunately resulted in a Java heap dump, so he has a Zimbra Support case open to investigate.

Using NFS As A Target For Zimbra Backups
Noah P. said that he has some extra spinning hard drives that he is interested in using to setup NFS as a target for his Zimbra backups. He plans to keep SSDs in place for mailbox metadata, but wondered if NFS can cause any issues with NG backups, for example, if an NFS mount point unexpectedly becomes unavailable causing backups or the associated Zimbra services to hang. Randy L. said he commonly uses NFS with his NG backups in Zimbra and has not seen any issues. He said that in the case of an NFS mount point going offline unexpectedly, he thinks it will cause NG Backup to send some warning messages to the Zimbra administrator about slow or failing backups, but otherwise does not seem to crash any Zimbra services. He explained that one of the keys to this resiliency is to use the “setBackupVolume Local” command line switch in NG Backup ( https://docs.zextras.com/zextras-suite- ... on_nfsfuse ). Randy said he thinks the worst that might happen is that, in the event of an extended NFS unmount, Noah might need to use the “zxsuite backup doCoherencyCheck” command to clean-up. Matthew F. added that in the past he has used a trick where he uses a loopback device as a backup target on the mailbox server to speed up backups by 10-20x for metadata, but this only works if Noah does not use the real-time SmartScan feature.

Birthdates Offset By 24 Hours
Marc G. said he has noticed that dates entered in the birthday field for Zimbra contacts get offset by 24 hours when ActiveSync is used with a mailbox. To workaround the issue, he has resorted to adding birth dates to the Notes field for contacts instead. John H. said he recalled a similar issue back in 8.8.15 Patch 5, and said the solution at that time was to do a full re-synchronization of an affected ActiveSync device. John E. said he thinks the original issue had something to do with a time zone offset, but was not sure. Adriano asked if Marc is using only US based dates for birthdates, as he thinks it could also be related to differences in the US date format, as compared to the European date format. Marc said he is using the European date format in the birth date field and agreed that might be a contributing factor.


Randy Leiker
Skyway Networks, LLC

[rleiker]

Hello Zimbra Community,

Here is a summary of this week’s conference call. A few brief reminders:

• Conference calls are every Tuesday and open to all using either the FreeConferenceCall.com VoIP app or via a dial-in number: https://www.freeconferencecall.com/wall/zetalliance
• Each week’s call agenda can be found at: https://drive.google.com/drive/folders/ ... jMuGGtIl7J
• Constructive feedback on these call summaries is always welcome.

May 11, 2021

Using doMailboxMove To Pre-Stage Mailbox Blobs
David M. said that he is currently working on moving mailboxes between two Zimbra mailbox servers that are in geographically dispersed data centers. While testing with the “zxsuite hsm doMailboxMove” command ( https://docs.zextras.com/zextras-suite- ... mailstores ) he noticed the “stages” parameter that can be used with the doMailboxMove command for moving only select portions of a mailbox, such as the blobs, backups, data (blobs + backups), or account (metadata). He asked if anyone on the call had experience using the “stages” parameter to pre-stage a mailbox’s blobs on the target mailbox server, from the source server, for the purpose of making the mailbox move occur very quickly from the mailbox owner’s perspective, particularly since the two mailbox servers are in geographically distant data centers? Cine said that using either the “stages blobs” or “stages data” parameters should accomplish what David is seeking to achieve. David asked if he uses the command “zxsuite hsm doMailboxMove stages blobs”, will this place the mailbox in Maintenance Mode, or will it continue to be available in Active Mode? Mark S. said that when you want to have greater control over the time frame of when a mailbox move happens, the “stages blobs” parameter will do an initial move by pre-staging the blobs on the target server, then catch the blobs up-to-date when the “stages account” parameter is used to move the mailbox metadata to the target server. He explained that Maintenance Mode is not triggered until the mailbox metadata is moved to the new mailbox server. He added that new email received for the mailbox while the metadata is being moved will begin queueing at the Zimbra MTA until the mailbox exits Maintenance Mode.

Matthew F. suggested that David may want to consider skipping moving backups, by omitting it from the “stages” parameter for greater mailbox moving speed. Noah P. asked if it is difficult to perform backup restores from the source server, if the backups are still only on the source server, post-mailbox move. Cine confirmed that you must move the backups to the target server, as part of the mailbox move, in order to perform a mailbox restore. Noah P. said he thinks the decision of whether or not to move backups for a mailbox will mainly depend on if your organization’s policy requires being able to restore earlier backups, prior to the mailbox move. Mark S. said that if David opts not to move the backups with the mailboxes, then the mailbox move command should provide an additional command in its console output that David can optionally use to manually purge the mailbox backup data from the source server. Cine added that if David does not manually purge the backups that are not moved, to free up space on the source server quickly, then he thinks the backups are purged automatically from the source server after about 7 days. He also said that if David wishes to move the mailbox back again from the target to the source server, then the doMailboxMove process is intelligent enough to move only the new items since the first mailbox move.

Customer Adoption Update For Zimbra Cloud
Noah P. asked how customer adoption is going with Zimbra Cloud? John E. said that there were customers previously hosted by Synacor, that have been moved over to Zimbra Cloud, but he could not comment further as no one from XMission was available on the call to share an update.

NG Backup Inconsistencies
Randy L. said he recently received an email notification from one of his Zimbra mailbox server’s daily SmartScans indicating an inconsistency had been found in the backups. The notification recommended running the command “zxsuite backup doSmartScan start deep true”. He said that he instead started investigating the issue by running the command “zxsuite backup doCoherencyCheck /path/to/backups fixBackup true” for the purpose of performing an in-depth integrity check of the backups. The doCoherencyCheck log contained a few mailboxes with consistency issues that were marked as fixed with log entries like: “user@example.com WARN Removing state 0 in <mailbox-id>/13463”. The log entries for the affected mailboxes were similar except that the “state 0” portion varied, for example “state 5”, or “state 14”. He asked if anyone on the call knew what type of backup integrity issue this referred to, as he could not find any documentation on the Zimbra or Zextras site explaining these log entries. Mark S. said that he has occasionally seen similar notifications from his NG Backups indicating an inconsistency was found, but upon investigation, has found them to be false positives. Cine said that “state 0” is the initial backup for a mailbox, and every subsequent state number is an incremental backup of the mailbox. Cine suggested that running a combination of a “zxsuite backup doSmartScan start deep true” command and a “zxsuite backup doCoherencyCheck /path/to/backups fixBackup true“ command should be enough to ensure a good backup of any affected mailboxes. He explained that each SmartScan backup performed keeps a counter of when the last backup took place. A normal SmartScan (without the deep parameter) will only look at new or changed mailbox items from the counter’s current position. However a SmartScan with the deep parameter will look at every mailbox item for a mailbox, effectively ignoring the counter for a mailbox.


Randy Leiker
Skyway Networks, LLC

[rleiker]

Hello Zimbra Community,

Here is a summary of this week’s conference call. A few brief reminders:

• Conference calls are every Tuesday and open to all using either the FreeConferenceCall.com VoIP app or via a dial-in number: https://www.freeconferencecall.com/wall/zetalliance
• Each week’s call agenda can be found at: https://drive.google.com/drive/folders/ ... jMuGGtIl7J
• Constructive feedback on these call summaries is always welcome.

May 18, 2021

“UID fetch failed” IMAP Error in the Zimbra Web Client
Randy L. said that he has a customer with two Zimbra mailboxes setup with IMAP in Thunderbird on Windows. Both mailboxes have an extensive folder hierarchy, with many levels of sub-folders. The mailboxes are 14 and 53 GB in size. The customer is encountering Thunderbird’s upper limit for the maximum number of folders, where Thunderbird will no longer allow the customer to create new folders. He suggested that the customer consider switching to the Zimbra Web Client (ZWC) to avoid the Thunderbird limit on folders. The customer asked if there was a means for him to login to just one his mailboxes in the ZWC from a single web browser instance, but see email from both mailboxes simultaneously, similar to what he can do in Thunderbird. Randy suggested that the customer could use the Preferences > Accounts > Add External Account feature in the ZWC by setting up one of his mailboxes as an IMAP account. This configuration appeared to initially work until the external account had synchronized several months of email messages after which an error message said: “UID FETCH failed: Error in response, Exception in data handler, Unexpected end of stream”. Randy said that one of his initial troubleshooting steps was to perform a “zxsuite hsm doCheckBlobs start mailbox_ids <mailbox-id>” command to check the customer’s mailbox for missing blobs or unexpected blobs, but no problems were found. He asked if anyone on the call had suggestions of what might cause the UID fetch error encountered. John E. said that the error suggests that an IMAP time out may be occurring, which is likely given the extensive folder hierarchy in the customer’s mailbox. He said he has encountered a similar error when synchronizing one of his very large mailboxes using IMAP, where attempts to save an item to the Drafts folder results in an error “no UID stored failed” and he has not yet found a solution to this issue. John H. suggested that Randy create a share of the mailbox root for one of his customer’s mailboxes, using the zmmailbox command line utility. Then, from the customer’s other mailbox, add the share. This will help avoid any IMAP sync or time out issues. Mark S. suggested using the zimbra_imap_folder_pagination_enabled and zimbra_imap_folder_pagination_size settings introduced in 8.8.15 Patch 15 ( https://wiki.zimbra.com/wiki/Zimbra_Releases/8.8.15/P15 ) that are designed to allow IMAP clients to more easily work with very large mailboxes or mailboxes with complex folder hierarchies. Randy said that he already has these two settings enabled, but they did not appear to avoid the UID fetch error in the ZWC.

Encrypting Data At Rest With NG Backups
Randy L. asked if anyone on the call knew of a means to configure Zimbra NG Backups to natively encrypt backup data? He gave a use case example where an off-site copy of the NG Backup data is being created, and the off-site backup needs to be encrypted to mitigate the risk of a data breach incident associated with lost or stolen backup media. No one on the call knew of how to implement such a configuration within NG Backups. Mark S. said that he relies on the auto encrypting disk feature available on his AWS servers. Noah P. said that he relies on self-encrypting disks for off-site backups. Randy said that if the capability to create encrypted backups natively with NG Backups is not possible, he is looking for a media-independent means to create encrypted backups. Mark S. suggested trying Linux Unified Key Setup aka LUKS ( https://en.wikipedia.org/wiki/Linux_Unified_Key_Setup ).

Upcoming 8.8.15 Patch 22 and 9.0 Patch 15
Mark S. asked if 8.8.15 Patch 22 and 9.0 Patch 15 is anticipated for release on or around June 1st. John H. said he has not heard otherwise, so this seems to indicate this is a likely release date.


Randy Leiker
Skyway Networks, LLC

[rleiker]

Hello Zimbra Community,

Here is a summary of this week’s conference call. A few brief reminders:

• Conference calls are every Tuesday and open to all using either the FreeConferenceCall.com VoIP app or via a dial-in number: https://www.freeconferencecall.com/wall/zetalliance
• Each week’s call agenda can be found at: https://drive.google.com/drive/folders/ ... jMuGGtIl7J
• Constructive feedback on these call summaries is always welcome.

May 25, 2021

Upcoming 8.8.15 Patch 22 and 9.0 Patch 15
Mark S. asked if the upcoming 8.8.15 Patch 22 and 9.0 Patch 15 will include the Zextras 3.1.11 version ( https://docs.zextras.com/zextras-suite- ... /home.html ). John H. said this is unlikely as the 3.1.11 version just started regression testing at Zimbra, so if all of the tests pass, it will likely instead be included within 8.8.15 P23 and 9.0 P16.

Suggestions For Frequency of Blob Checks and Backup Coherency Checks
Randy L. said that he is re-visiting his policies for how frequently blob checks (zxsuite hsm doCheckBlobs) and backup coherency checks (zxsuite backup doCoherencyCheck) are performed to proactively address blob or backup integrity issues on his Zimbra servers. He asked everyone on the call for their thoughts as to if they proactively run these checks as well, or reactively perform these checks when a problem is reported by a SmartScan or by a mailbox owner. Mark S. said that when he was first getting started with the Centralized Storage feature ( https://zimbra.github.io/zimbra-9/admin ... ed-storage ), there was a bug, that has since been fixed in a Zimbra patch, that would routinely indicate a problem with his mailbox backups, so he was frequently running backup coherency checks at that time. Since then, he normally does not run coherency checks unless alerted to a problem by either the daily or real-time SmartScans, as he feels the SmartScans perform an adequate incremental coherency check of backups. Mark added that when he moves mailboxes, he does occasionally experience issues with orphaned blobs that are not deleted for some reason, so those do need occasional clean-up with a blob check. Noah P. said that he does not normally run recurring backup coherency scans, but a proactive scan has recently uncovered quite a few inconsistencies in the backups. He suspects these might be due to bugs in earlier versions of Zimbra/Zextras mailbox tools, as he does a large number of mailbox moves in his Zimbra environment. He has not yet decided if he will continue running these checks regularly once all of the issues are fixed. He added that he finds the Zextras tools do a good job of consistently cleaning up problems, as compared to earlier deprecated utilities like zmblobchk. Mark S. said that in an instance where the current Zimbra backups are suspect, a strategy that can be used to get known good backups by the current Zimbra/Zextras version is to unmount the disk containing the current backups, then mount a new blank backup disk, and run a full backup. Both disks can then be maintained for a period of time until the unmounted disk, containing the earlier backups, ages out and can be deleted. Randy L. agreed that this is a good strategy when the backups are untrusted, but said that he was looking at it more from the perspective of ongoing proactive maintenance of blobs and backups, as compared to a one-time fix for backups with known issues.

Zimbra Desktop Beta
Following up on the April 13th Zeta Alliance call, John E. said the Zimbra Desktop beta is ongoing and offered that if anyone would like to join in on the beta testing, to reach out to him. Mark S. asked if the beta version self-updates? John E. said he does not think so, and was surprised to learn that it does not self-deactivate after the beta period ends either. Mark S. asked if the Zimbra Desktop beta is available somewhere for download? John E. said that it is by invitation only, so it is not on a public web site for download, and reiterated that it is currently beta-quality software at present.

Follow-Up: Using doMailboxMove To Pre-Stage Mailbox Blobs
Following-up on the May 11th Zeta Alliance call, Randy L. shared an update on David M’s experiences with using the “stages” parameter for mailbox moves (zxsuite hsm doMailboxMove stages) between geographically dispersed data centers. Randy said that he was discussing this topic in more depth with David in a separate conversation recently and David said that he has been able to successfully move mailboxes in two steps: by using the “stages blobs” parameter to move mailbox blobs first, followed by moving mailbox metadata (“stages account”). Previously, David had been attempting to move each mailbox in a single step, but found that moved mailboxes were experiencing issues where the Zimbra Web Client would fail to render all page objects, or moved mailboxes configured to use Outlook were reporting client-side time-out related errors. Matthew F. asked if David had opted to move his backups (“stages backup”) with the mailbox moves or not. Randy said he did not know, since David was unavailable on today’s Zeta Alliance call to comment. Matthew said that he has a Zimbra cluster spread across 4 geographically dispersed data centers, and has not observed client-side issues when performing mailbox moves, but added that he does not normally move backups with the mailboxes. Matthew said that he did make one tweak to Nginx in Zimbra to disable its default time out checks for mailbox servers (zimbraMailProxyReconnectTimeout and zimbraMailProxyMaxFails), as this has caused issues in the past where Nginx would incorrectly flag a mailbox server as being unavailable. Randy said that he thinks David said something earlier about disabling the Nginx mailbox server time out check too, but was not sure. In David’s case, he found success with changing the address that Memcached binds with on the proxy servers at both of his data center locations ("zmprov ms $(zmhostname) zimbraMemcachedBindAddress 127.0.0.1" and "zmprov ms $(zmhostname) zimbraMemcachedClientServerList 127.0.0.1" and "zmmemcachedctl restart").


Randy Leiker
Skyway Networks, LLC