November 2020 Zeta Alliance Weekly Call Summaries

Industry info, happenings near you, and new product integrations. Hosting an event? Invite people here.
User avatar
rleiker
Advanced member
Advanced member
Posts: 149
Joined: Tue Jan 07, 2020 8:23 pm
Location: Kansas City
Contact:

November 2020 Zeta Alliance Weekly Call Summaries

Post by rleiker »

Hello Zimbra Community,

Here is a summary of this week’s conference call. A few brief reminders: November 3, 2020

New Zextras Release
Cine reported that Zextras released version 3.1.4, on November 2nd and highlighted two new features:

1. Zimbra data can now be backed up to external volumes such as an Amazon Web Services (AWS) S3 Bucket or a Network File System (NFS).

2. Mobile Password management has been moved to the Zimbra Web Client, allowing end-users to manage the mobile password, rather than only Zimbra administrators. Additionally, an unlimited number of mobile passwords can now be set.

Marc G. asked if Zextras 3.1.4 will be released as part of the next Zimbra patch. Cine said he was not sure, but expected it may be part of 8.8.15 Patch 16 and 9.0 Patch 9. All of the changes included in the Zextras 3.1.4 release can be found at: https://docs.zextras.com/zextras-suite- ... /home.html

Saving Zimbra Backups To AWS S3 Buckets Or NFS
Cine said that this new feature in Zextras 3.1.4 relies on a local caching feature provided by Zextras HSM (Hierarchical Storage Management) since an external volume (S3 or NFS) may not be able to immediately write data. He explained that this will be especially helpful in cases where NFS is used as an external backup volume, since NFS does not always tell Zimbra when a write fails, leading to the possibility of an incomplete backup. This allows for S3 to be used as a local storage device on a Zimbra server, since Zimbra sees it as a local mount point. Cine said that the HSM cache ensures consistent writes to an external NFS volume, by checking up to several times with the NFS volume to confirm that a write succeeded. Marc G. asked how this new cache feature confirms that a write succeeds when using S3 storage? Cine said that it is tougher to verify with S3, but usually is not required since the writes to S3 tend to be more reliable, as compared to NFS. Matthew F. commented that S3 provides a hash value for each write, which should provide a means for Zimbra to confirm a write.

Mark S. asked how much data might be recoverable from the HSM cache, should a Zimbra server unexpectedly fail before the cache can be fully flushed to an S3 bucket. Cine said that the RPO (recovery point objective) should be near zero, since the Zimbra SmartScan feature can check what data was written to an S3 bucket, once the Zimbra server is recovered.

Matthew F. asked if the new Zextras 3.1.4 version will reduce the number of files that are written to S3, since earlier implementations required a very large number of files to be written within a short period of time. Cine said that the new version will watch for contention errors from S3, and if observed, the HSM cache will slow down the writes sent to an S3 bucket, allowing it additional time to catch up. This in affect throttles the speed at which data transfer takes place increasing the likelihood it can succeed writing all data for a Zimbra backup over a longer period of time.

Cine suggested that if using an S3 bucket as an external volume for Zimbra backups, it is a best practice to have a separate bucket for each Zimbra mailbox server. He said that after initially setting up external backups to S3, by default, Zimbra will use the same bucket for all mailbox servers in a cluster. However, he described a command line process whereby each mailbox server can be configured to use a different S3 bucket.

Cine commented that revisions to the Zimbra Administrator’s guide, in the Backup/Recovery section will be forthcoming that detail using external volumes such as the S3 and NFS options.

Managing Mobile Passwords
Cine said that in addition to this feature being moved from the Zimbra Administration Console to the Zimbra Web Client in Zextra 3.1.4, where end-users can self-manage their own mobile passwords, if the Zextras mobile app is installed on a user’s phone/tablet, they can optionally scan a QR code generated by the Zimbra Web Client, instead of entering a conventional user name and password, to login to their email account. The login session on the Zextras app then remains active for as long as the QR code is valid in the Zimbra Web Client.

Managing IMAP Mailboxes With Large Numbers Of Folders
Randy L. asked if anyone on the call had suggestions about the threshold at which it makes sense to turn on the new “zimbra_imap_folder_pagination_enabled” local configuration setting, introduced in the 8.8.15 Patch 15 and 9.0 Patch 8 patches, since the release notes ( https://wiki.zimbra.com/wiki/Zimbra_Releases/9.0.0/P8 ) indicate this setting is disabled by default. Mark S. said he enabled this feature earlier to try and lower the CPU load on his Zimbra servers, and has not observed any apparent differences in the operation of email apps on end-user devices, nor any performance related issues for the end-users. He added that on the Zimbra server-side, he has observed the Java garbage collector running less often. Enabling this local configuration setting requires a restart of the mailboxd service.

John E. said that when a user has about 1,500 or more folders they are synchronizing with IMAP, that this seems to be the threshold at which a performance bug appears, based on internal test cases run by Synacor. Mark S. asked why is the default value for “zimbra_imap_folder_pagination_size”, discussed in the release notes, set to 2,000 rather than 1,500? John E. looked-up additional metrics from the internal Synacor testing on the issue. He said that performance timing was done from a Mac running Thunderbird with 42,000 email folders, with each containing 4 sub-folders with the “zimbra_imap_folder_pagination_enabled” setting set to false. This resulted in 388% CPU usage with a load average of 6-7 per CPU core. No sub-folders in the email folder tree synchronized in Thunderbird until two hours passed. Next, the “zimbra_imap_folder_pagination_enabled” setting was changed to true. The CPU usage dropped to less than 100% with load averages of 1-2. Sub-folders synchronized in Thunderbird after 20 minutes. These testing results have been documented internally at Synacor as ZBUG-1694, and resolved in the most recent patches for 8.8.15 and 9.0. John E. said that the internal testing at Synacor showed that a default setting of 2000 for “zimbra_imap_folder_pagination_size” was selected as it was found to help with optimizing IMAP usage from Outlook clients.

Randy L. said he did not understand why the “zimbra_imap_folder_pagination_enabled” setting is disabled by default in the latest patches, but felt that it should be enabled by default, at least as a defensive measure to avoid a performance issue with end-user mailboxes containing large numbers of folders. Mark S. commented that he noticed in Office 365 they are limiting customers to a total of 1,000 email folders and felt that Zimbra’s support of very large numbers of folders is a competitive advantage since there are many use cases where a large folder tree may be needed. Marc G. commented that this is an example of where it is probably costing Zimbra BSPs (Business Service Providers) more money in the form of added server resources to provide service for IMAP-only users at a lower per mailbox licensing cost, than it would be if Synacor were to restructure the BSP program so that the ActiveSync feature could be used at some of the lower Zimbra mailbox licensing tiers, since ActiveSync provides a more efficient folder syncing process, as compared to IMAP.

Industry-Related Meetup Discussing Searching Encrypted Data
Mark S. shared an upcoming free meet-up on November 17th: https://www.meetup.com/Cloud-Security-A ... 274055654/ with a focus on how to perform searches on encrypted data, such as encrypted mailboxes. He said that this meet-up is a discussion on what is going on in the world of encrypted mailbox searching for those interested in the privacy implications of encrypting mailbox data at rest, while maintaining traditional mailbox searching features that normally require access to unencrypted mailbox data.

Disappearing Zimbra Logs In CentOS 8
Mark S. said that there is a previously reported issue with Zimbra on CentOS 8 where the default log file rotation in CentOS 8 causes Zimbra logs to disappear after 24 hours. A possible fix is documented at: viewtopic.php?f=15&t=68919 . He also said he has an open support case with Zimbra on this issue too.


Randy Leiker
Skyway Networks, LLC
Klug
Ambassador
Ambassador
Posts: 2747
Joined: Mon Dec 16, 2013 11:35 am
Location: France - Drôme
ZCS/ZD Version: All of them
Contact:

Re: November 2020 Zeta Alliance Weekly Call Summaries

Post by Klug »

There's this in ZeXtras current (3.1.4) documentation:
How to backup on NFS/Fuse

While at a first glance it might seem that due to the need of a local mountpoint specifically setting up the backup for NFS or FUSE has little utility, the backend differences in metadata handling ensure a greater degree of data safety.
Splitting the high-access metadata from the BLOBs ensures that disk failures, such as when the share becomes briefly available, are better handled thanks to the local cache granting a higher backup resilience.
To backup on "Local" shares such as NFS or Fuse, first mount the share and then use the appropriate command based on your need:
No pre-existing backup: zxsuite backup setBackupVolume Local
Running backup: zxsuite backup migrateBackupVolume Local
Both commands only require a single argument, which is the path to the local mountpoint of the NFS/FUSE share.
What does happen when the command is run on a existing/running backup?
Does it need a mailboxd restart?
User avatar
rleiker
Advanced member
Advanced member
Posts: 149
Joined: Tue Jan 07, 2020 8:23 pm
Location: Kansas City
Contact:

Re: November 2020 Zeta Alliance Weekly Call Summaries

Post by rleiker »

Klug wrote:There's this in ZeXtras current (3.1.4) documentation:
What does happen when the command is run on a existing/running backup?
Does it need a mailboxd restart?
Hi Klug,

On the most recent Zeta Alliance call on Nov. 10th, someone on the call asked a similar question too. Cine (from Zextras) explained that once the necessary command is run to make the conversion, it will use noticeably more Zimbra mailbox server resources while the conversion process runs, but is otherwise able to do so in the background. He said that it will simply run over a period of time until complete, but he did not specifically mention a need to restart mailboxd. I think there was mention too about it temporarily disabling SmartScans in the Network Modules NG feature while the conversion process takes place, but I'm not certain of this. If you have a Zimbra support contract, you could also open a support case to run this scenario by Zimbra/Zextras so they can offer you an official answer too.

I will have the Nov. 10th call summary posted as soon as I have it ready, which will have more helpful details on this topic too.
Klug
Ambassador
Ambassador
Posts: 2747
Joined: Mon Dec 16, 2013 11:35 am
Location: France - Drôme
ZCS/ZD Version: All of them
Contact:

Re: November 2020 Zeta Alliance Weekly Call Summaries

Post by Klug »

Thank you Randy.

Considering 8.8.15-P15 release notes, it doesn't seem to be integrated in the NG module yet.
It's too soon to ask Zimbra's support about this.
User avatar
rleiker
Advanced member
Advanced member
Posts: 149
Joined: Tue Jan 07, 2020 8:23 pm
Location: Kansas City
Contact:

Re: November 2020 Zeta Alliance Weekly Call Summaries

Post by rleiker »

Klug wrote:Thank you Randy.

Considering 8.8.15-P15 release notes, it doesn't seem to be integrated in the NG module yet.
It's too soon to ask Zimbra's support about this.
You are correct. It will be included in the next patch for 8.8.15 and 9.0 which is due out very soon. Synacor does have the information you are seeking available internally, so if needed, I think they could still answer your question prior to the release of 8.8.15 patch 16.
User avatar
rleiker
Advanced member
Advanced member
Posts: 149
Joined: Tue Jan 07, 2020 8:23 pm
Location: Kansas City
Contact:

Re: November 2020 Zeta Alliance Weekly Call Summaries

Post by rleiker »

Hello Zimbra Community,

Here is a summary of this week’s conference call. A few brief reminders: November 10, 2020

Upcoming Zimbra Patches
Mark S. asked if there were any updates from Zimbra available for 8.8.15 Patch 16 and 9.0 Patch 9. He noticed the Zimbra Partner Portal had not been updated yet with a preview of the release notes for those upcoming patches. Gayle B. said she will work on getting that updated shortly and that the Partner Portal is usually updated about 1 week before each patch release. She is going to see if this information can be posted to the Partner Portal sooner. Mark S. asked if the new Zextras 3.1.4 release discussed in the November 3rd Zeta Alliance call ( viewtopic.php?f=9&t=68942#p299660 ) will be incorporated into these upcoming patches. John E. said that 6.0.17 of NG modules will be included within 8.8.15 P16 and 9.0 P9, and Cine confirmed this is equivalent to the Zextras 3.1.4 release. Cine said there have been some stability improvements included too which will benefit AWS S3 connection handling that keeps HTTPS connections to a minimum, as some providers (Digital Ocean) have very low connection time outs that were causing issues for Zimbra administrators.

New ZURT Release
Mark S. said he noticed that there is a ZURT (Zimbra Usage Reporting Tool; used by Zimbra Service Providers for license usage reporting) version in the Zimbra repos. He asked if the new ZURT has any dependencies on the latest or upcoming patches, or if the ZURT can be installed independently. John H. said that ZURT can be installed independently and will restart the ZURT service automatically after it has been updated. He explained that the new ZURT has better support for older OSes like Red Hat 12 and also fixes a bug within the mailbox usage reporting, where there can be a difference between the license usage data in the CVS file generated by ZURT and the usage information reported automatically to Zimbra. Mark S. asked if Zimbra Connect usage is being reported automatically by ZURT yet. John H. said Connect usage still needs to be reported manually.

Controlling SMTP Mail Flow In Zimbra
Noah P. said he is experimenting with directing certain customer domains in Zimbra to specific Zimbra MTA servers. For example, customers that send mostly personal email can be directed to one or more MTAs, while other customers that send bulk or questionable mail can be sent to another MTA where throttling or more restrictive outbound SMTP policies can be applied. He said he has been using the “zimbraSmtpHostname” setting at the domain level in Zimbra to see if he can control which MTA is used for each customer domain. While this seems to work fine for all outgoing email sent from a given customer’s domain, it however does not seem to have any impact on outbound email sent from Zimbra distribution lists. Matthew F. said that he is also managing which customer domains may use which Zimbra MTA, and that he has found success by using SMTP relays, external to Zimbra, to manage the mail flow, but he too has the same issue of not being able to control which MTAs Zimbra uses internally for outbound distribution list email. John H. explained that when a message is delivered to a distribution list in Zimbra, the delivery address is first expanded at the MTA level, which will have the effect of ignoring the domain level setting for the MTA (zimbraSmtpHostname). Noah said he also looked at the “zimbraMailTransport” setting too, but did not know if it was supported for use with distribution lists. Noah opened Zimbra support case 01097971 on this issue.

Follow-Up: Restoring Zimbra Backups From AWS S3 Buckets
As a follow-up to the November 3rd Zeta Alliance call discussion ( viewtopic.php?f=9&t=68942#p299660 ) about backing up and restoring from AWS S3 buckets after 8.8.15 Patch 16 and 9.0 Patch 9 are released, Cine said that disaster recovery or data migration can be performed in Zimbra by restoring directly from S3, or by restoring from S3 + a local copy of the backup metadata on a new Zimbra server.

Cine explained that if Zimbra data is being restored from S3 only, and there is no local copy of the backup metadata, then the backup metadata will need to be downloaded to the new Zimbra server first, along with configuring the S3 bucket in Zimbra, before a restore operation can begin. However, the S3 bucket on the new Zimbra server should not be configured as a backup target, to avoid the risk of inadvertently overwriting data on S3, from the new Zimbra server. This will result in the old backup path being restored on the new Zimbra server, so that an external restore process can be started, along with specifying the backup archive option within the restore command, so that Zimbra knows to restore the mailbox blobs from S3.

Mark S. asked if it is wise to run more frequent SmartScans to lower the RPO (recovery point objective) for a disaster recovery, by ensuring that S3 receives more frequent data updates from Zimbra. He said his concern is what happens if a Zimbra mailbox server should become corrupted, including the backup metadata. Cine said that running more frequent SmartScans should not be necessary, but does recommend keeping an additional local backup copy of the Zimbra backup metadata if possible, as it will be much faster to begin the disaster recovery restore, than needing to first download the backup metadata from S3 first. To get a shorter RPO, Cine thinks that syncing the metadata to S3 more frequently would help and suggested this could be accomplished using Cron jobs multiple times per day, which will take more server resources, but not as many resources as frequent SmartScan operations. Matthew F. asked if there is any means to convert an existing Zimbra mail store’s backups to use the new S3 backup target capabilities in the upcoming 8.8.15 Patch 16 and 9.0 Patch 9. Cine said there is a command line option to do this, but it will require additional server resources and affect performance temporarily on the Zimbra mailbox servers while the conversation takes place over a period of time.

New Location For Zeta Alliance Call Agendas
Cine said that he is setting up a new Google Drive container for all of the Zeta Alliance call agendas at: https://drive.google.com/drive/folders/ ... sp=sharing and anticipated having this completed within about a week.


Randy Leiker
Skyway Networks, LLC
ghen
Outstanding Member
Outstanding Member
Posts: 258
Joined: Thu May 12, 2016 1:56 pm
Location: Belgium
ZCS/ZD Version: 9.0.0

Re: November 2020 Zeta Alliance Weekly Call Summaries

Post by ghen »

Have you also noticed that, if you set zimbra_imap_folder_pagination_enabled=true on 8.8.15 patch 15, it always logs, for every user:

"Total folder count - X is greater than folder pagination size - 2000"

regardless of the number of folders X actually exceeding the max or not...
User avatar
L. Mark Stone
Ambassador
Ambassador
Posts: 2796
Joined: Wed Oct 09, 2013 11:35 am
Location: Portland, Maine, US
ZCS/ZD Version: 10.0.6 Network Edition
Contact:

Re: November 2020 Zeta Alliance Weekly Call Summaries

Post by L. Mark Stone »

ghen wrote:Have you also noticed that, if you set zimbra_imap_folder_pagination_enabled=true on 8.8.15 patch 15, it always logs, for every user:

"Total folder count - X is greater than folder pagination size - 2000"

regardless of the number of folders X actually exceeding the max or not...
Yes, we noticed this. It logs the same result regardless of whether there are more or less than 2,000 folders. Seems like a logic bug in the code; Zimbra is now aware of this.
___________________________________
L. Mark Stone
Mission Critical Email - Zimbra VAR/BSP/Training Partner https://www.missioncriticalemail.com/
AWS Certified Solutions Architect-Associate
User avatar
L. Mark Stone
Ambassador
Ambassador
Posts: 2796
Joined: Wed Oct 09, 2013 11:35 am
Location: Portland, Maine, US
ZCS/ZD Version: 10.0.6 Network Edition
Contact:

Re: November 2020 Zeta Alliance Weekly Call Summaries

Post by L. Mark Stone »

ghen wrote:Have you also noticed that, if you set zimbra_imap_folder_pagination_enabled=true on 8.8.15 patch 15, it always logs, for every user:

"Total folder count - X is greater than folder pagination size - 2000"

regardless of the number of folders X actually exceeding the max or not...
Yes, we noticed this. It logs the same result regardless of whether there are more or less than 2,000 folders. Seems like a logic bug in the code; Zimbra is now aware of this.
___________________________________
L. Mark Stone
Mission Critical Email - Zimbra VAR/BSP/Training Partner https://www.missioncriticalemail.com/
AWS Certified Solutions Architect-Associate
maio
Posts: 1
Joined: Wed Nov 18, 2020 5:02 pm

Re: November 2020 Zeta Alliance Weekly Call Summaries

Post by maio »

Hello everyone,
Will we have the privilege of having functionalities similar to Zimbra Cloud soon, for example I am talking about the integration of Onlyoffce with really useful editing functions (see the video) : https://www.youtube.com/watch?v=J7dqGKjE2IU&t=2s
I know there is owncloud-zimlet with a straightforward approach to Onlyoffice using its APIs, but it is unstable, and this iframe access is not comfortable. :(
Post Reply