February 2021 Zeta Alliance Weekly Call Summaries

Industry info, happenings near you, and new product integrations. Hosting an event? Invite people here.
Post Reply
User avatar
rleiker
Advanced member
Advanced member
Posts: 149
Joined: Tue Jan 07, 2020 8:23 pm
Location: Kansas City
Contact:

February 2021 Zeta Alliance Weekly Call Summaries

Post by rleiker »

Hello Zimbra Community,

Here is a summary of this week’s conference call. A few brief reminders: February 2, 2021

Using Centralized Storage In Zimbra
Mark S. asked if anyone has implemented the Centralized Storage ( https://zimbra.github.io/adminguide/lat ... ed-storage ) feature in Zimbra? This allows for storing mailboxes from multiple Zimbra mail stores within the same directory structure on an S3 storage volume (AWS S3, Ceph, etc.). Cine commented that Centralized Storage speeds up mailbox moves between Zimbra mailbox servers considerably. John E. added that when using object storage (Centralized Storage), it uses a single name space for blob storage, that during a mailbox move, will then move only the mailbox meta data, so few mailbox blobs move at all. Mark S. asked if he has a single AWS S3 bucket with two name spaces, where all of his mailboxes are stored, and he performs a mailbox move, are mailbox blobs still moving? Cine said in that instance, yes, all mailbox blobs will still need to be moved between name spaces, but if he switches to using Centralized Storage under a single name space in his AWS bucket, then most mailbox blobs will not need to be moved – only the meta data. Noah P. asked if he has a primary Zimbra mailbox server volume on-site, and a secondary volume on an AWS S3 bucket, does Centralized Storage still work? Cine confirmed it does. Mark S. asked if using a Zimbra HSM policy that moves email older than 4 weeks to a secondary volume, when using Centralized Storage, would this mean that only the most recent 4 weeks of mailbox blobs move? Cine confirmed this is correct and that customers he has worked with who use an aggressive HSM policy of keeping only 3-7 days of email in their primary volume on-site with all older items moved to a secondary volume, when combined with Centralized Storage, mailbox moves are very fast.

Zimbra Disaster Recovery (DR) Restores
Mark S. asked, when restoring a Zimbra mailbox server that has suddenly failed in a DR situation, what is the recommended way to do a restore? Cine suggested using the Zextras Raw Restore feature ( https://zimbra.github.io/adminguide/latest/#raw-restore ), which is designed for DR use only. Mark S. asked if he has two mailbox servers, Server 1 and Server 2, and Server 1 fails, should he build Server 3 as the replacement using the raw restore feature, and if so, does the Raw Restore feature also update the Zimbra mailbox transport setting for each mailbox from the failed server, so the Zimbra MTAs (Postfix) knows the new location of each mailbox? Cine said that it is not necessary to create a new server name, as the failed server name can be re-used. Cine suggested referring to the Raw Restore documentation section that discusses “Running A Raw Restore” and “Usage Scenarios”. Mark S. commented that he is trying to save money on storage at AWS by putting as much on S3 storage as he can, but this has the consequence of also shortening his Recovery Point Objective during a DR incident.

Migrating Mailboxes From Exchange To Zimbra
Marc G. said he has a customer doing a migration from Microsoft Exchange to Zimbra and asked for suggestions on the best mailbox migration tool to use. Mark S. suggested taking a look at BitTitan ( https://www.bittitan.com/ ). He added that BitTitan supports bi-directional transfers that can migrate email, contacts, calendars, etc. It also works well for migrating Office 365 tenants between accounts, since it is aware of things like Microsoft Teams. This helps in scenarios where a parent company is spinning off a subsidiary company in to their own Office 365 account.

Zimbra and SELinux
Matthew F. said he is building new Zimbra servers and wondered if there has been any changes to earlier recommendations to avoid running Zimbra with SELinux in enforcing mode. Mark S. said he disables SELinux on his Zimbra servers and Randy L. said he runs SELinux in permissive mode on his Zimbra servers.

Obtaining Status Updates For a Bugzilla Pull Request
Cine asked for suggestions on the best option to request a status update of a Bugzilla pull request for the Zimbra Open Source Edition. John H. said that bug updates are only available through the Zimbra Support Portal at present. Mark S. commented that open source Zimbra users can buy support, so they can then gain access to the Support Portal. He also commented that when he sees Zimbra Forum users post issues that he knows affect Zimbra Network Edition, he has opened support cases in the past referencing those Forum posts. Cine said he has a friend that has found a memory leak bug in the Nginx version included in Zimbra, and has submitted a pull request to fix it. John H. suggested that Cine’s friend take a look at the beta version of Nginx which jumps from Nginx 1.18 to 1.9. John E. suggested that if Cine’s friend posts comments in the pull request, this may also help draw more attention to it. John H. added that for anyone willing to install the beta version of Nginx and OpenSSL, Zimbra is willing to provide support. If installing the beta version, he suggested opening a support case to give Zimbra Support a heads up and mention John Hurley’s name. Nginx has only two bugs that need to be resolved before it comes out of beta: ZBUG-2098 and ZBUG-2099, related to an issue with an HTTP/2 configuration file, and a second issue related to some buggy code that causes Nginx to crash.

Zimbra Suite Plus Road Map
Mark S. asked if anyone had heard about updates for the Zimbra Suite Plus road map. He said he has a prospective customer interested in Zimbra Suite Plus since they want basic mailboxes with mobile sync support. The customer is also interested in Zimbra Connect, but there does not seem to be a way to add it to Zimbra Suite Plus. No one had any updates to share and Cine said that it is correct that Zimbra Connect is not currently available with Zimbra Suite Plus.


Randy Leiker
Skyway Networks, LLC
Last edited by rleiker on Tue Apr 13, 2021 3:54 pm, edited 1 time in total.
User avatar
rleiker
Advanced member
Advanced member
Posts: 149
Joined: Tue Jan 07, 2020 8:23 pm
Location: Kansas City
Contact:

Re: February 2021 Zeta Alliance Weekly Call Summaries

Post by rleiker »

Hello Zimbra Community,

Here is a summary of this week’s conference call. A few brief reminders: February 9, 2021

Experiences With Zimbra Centralized Storage On AWS
To follow-up on the February 2, 2021 Zeta Alliance call, Mark S. shared that he has been continuing to explore using Zimbra’s Centralized Storage feature for his Zimbra servers hosted at AWS for the purposes of speeding up mailbox moves and improving recovery time objectives in the event of a disaster recovery scenario. He said that it seems to be working well, but that he found the Zextras documentation for Centralized Storage ( https://docs.zextras.com/zextras-suite- ... ed_storage ) was missing a discussion of options for setting up different storage classes for use with an AWS S3 volume. Cine commented that it can be a challenge for the documentation to remain vendor neutral and avoid becoming focused on providing AWS S3 support. Mark S. suggested perhaps the documentation could be updated to note that each vendor has varying options available for tiered storage so that Zimbra administrators are aware of the potential to save money on their storage hosting costs.

Mark S. said he wanted to use intelligent storage tiering, but found that his storage objects initially went into the standard storage class in AWS S3 instead. To correct this, he manually changed the volume settings on each Zimbra mailbox server so that future blobs will be correctly placed in to the intelligent storage tiering. He said he plans to test the Zimbra Raw Restore feature in Zimbra with this new setup in a lab environment and will share his findings in a future Zeta Alliance call. Mark added that he is not using Centralized Storage in Zimbra for his primary mailbox server volumes yet, but instead continues to use HSM, as he is unsure of how Centralized Storage could affect mailbox performance, particularly those mailboxes that have intensive IMAP users. He plans to ask his customers how they are using their mailbox data to set the most aggressive HSM policy possible, so as to keep the minimal amount of recent mailbox data outside of S3 storage. Marc G. asked if Mark S. is using blobless backups. Mark S. said he has not reached this point yet in his testing. Cine commented that blobless backups are designed to be used in conjunction with Centralized Storage.

New Lifesize Zimlet
Barry D. shared that a new Zimlet integration is now available for Lifesize ( https://github.com/Zimbra-Community/zim ... t-lifesize ), for creating virtual meeting rooms for virtual conferences. Marc G. asked if Barry has had experience with using Jitsi as he finds Lifesize tends to be very expensive. Barry said that there is already a beta Zimlet integration available for Jitsi in the Zimbra 9 Modern UI ( https://zimbra.github.io/zimbra-9/admin ... rly_access ). John E. added that a packaged Jitsi Zimlet is anticipated to be included in a future Zimbra version release.

Performance Testing With Zimbra Centralized Storage
Matthew F. shared that he is exploring options for using Centralized Storage in Zimbra, combined with HSM, and small AWS EBS storage volumes for his frequently accessed data on his Zimbra servers. He currently runs his Zimbra mailbox servers in AWS. He said that he has been conducting performance testing and has observed big performance hits when mailbox data is stored within AWS S3 volumes, when using Zimbra HSM. He shared his preliminary testing results so far in the section titled “S3 Testing Data” ( https://docs.google.com/document/d/1zuj ... oQkc8/edit ). He said his biggest concerns are how these performance tests will reflect real-world usage should a complete mailbox server restore be needed in the event of a disaster recovery scenario, in addition to mailbox users conducting full Zimbra Connector for Outlook synchronizations. His calculation of these performance test results suggest it could require between 1-2 weeks to fully restore a mailbox server from S3 storage, in the event of the loss of a complete mailbox server.

Cine said that performance testing of blob checks on S3 storage is the worst case scenario since it requires interacting with all components of a mailbox. He suggested using 4 parallel threads for a mailbox move or a disaster recovery, as past experience suggests this tends to be the right amount of threads, but said that it may require some trial and error to boost performance. Cine said that Matthew may want to take a look at this command where the number of threads can be specified: “zxsuite backup doExternalRestore /path/to/data/ domains domain1.com domain2.com concurrent_accounts 5”. Mark S. suggested going from an AWS T3.small server to a T5.large server and re-running the performance testing.

Mark S. commented that S3 has throttling limits that are supposedly account-wide, and Matthew might be reaching those upper limits while running the performance tests and suggested opening a support case with AWS to determine if this is the cause. He added that S3 will always be slower than block storage in AWS. Marc G. said that delays are expected with S3 storage, but the current challenge is that he and Matthew do not feel they can count on it for disaster recovery if restoring a whole mailbox server, given the calculated 1-2 weeks of estimated time to restore a complete mailbox server from S3 storage. Mark S. said that if the HSM policy is set appropriately, a disaster recovery scenario would not involve restoring all mailbox blobs, but rather just the most recent few days worth of mailbox data to the mailbox servers’ primary storage volumes, while leaving all older blobs in object (S3) storage.

Marc G. asked if anyone on the call had experience using Scality? John E. said that when using the S3 Connector or the EMC Connector for Zimbra, they utilize blobless backups, and the only potential risk is orphaning of blobs, which are defined as blobs (mailbox items) on disk, but no longer referenced within Zimbra’s MariaDB database. He explained the Zimbra NG modules provide compatibility with both the NG backups and HSM while the Scality S3 & EMC Connectors do not. John said that the user case for the Scality and EMC Connector is usually very large scale mailbox storage for some of the largest Zimbra service providers. The EMC Connector is available in the Zimbra GitHub repo if searching for “EMC”.

Different Zimbra Patch Versions When Moving Mailboxes
Matthew F. said while doing a mailbox move between two Zimbra mailbox servers with the same major and minor version number, but different Zimbra patch levels installed (Patch 14 versus Patch 18), he encountered a problem where the moved mailbox reported one unexpected blob, along with the Zimbra Connector for Outlook producing an error about a sync token being too old, thereby preventing Outlook syncing from operating. Matthew tried deleting the Outlook profile and re-creating it, but this did not resolve the issue. Mark S. said that he encountered a similar problem with mailbox moves, and ended up doing a restore of the mailbox to the destination server to fix the issue. Cine said he thinks mailbox moves between servers with different patch levels should work as there are some cross-safety checks that take place during a mailbox move operation. He added that the mailbox servers should negotiate the API level they can both support.


Randy Leiker
Skyway Networks, LLC
User avatar
rleiker
Advanced member
Advanced member
Posts: 149
Joined: Tue Jan 07, 2020 8:23 pm
Location: Kansas City
Contact:

Re: February 2021 Zeta Alliance Weekly Call Summaries

Post by rleiker »

Hello Zimbra Community,

Here is a summary of this week’s conference call. A few brief reminders: February 23, 2021

Demonstration of VNCmail
Andrea W. with VNClagoon demonstrated her company’s product, VNCmail ( https://vnclagoon.com/products/vncmail/ ). The product uses Zimbra at its core, and she said that it has 3 new front-end clients: a web client, Android, and iOS. They have also created their own search index based on Apache Solr in place of Lucene normally used within Zimbra. She explained that the Solr index includes searching of email, chat messages, tickets, tasks, and uploaded files for the integrated VNCsafe (file sharing) product. The VMCmail product can integrate either with OpenLDAP or Zimbra’s embedded OpenLDAP server. She said that these products can be hosted either on-premises or in a hosted environment.

Andrea demonstrated:
  • VMCcalendar: that can open a Zimbra calendar as a standalone web page outside of the Zimbra Web Client.
  • VNCtalk: that includes a chat feature, video/audio calls, and screen sharing, which is a replacement for Microsoft Teams, Zoom, and WhatsApp.
  • VNCproject: that includes an AI-based virtual assistant for agile management of projects.
Mark S. asked if these products can run on top of Zimbra Open Source Edition and Zimbra Network Edition. Andrea said that it supports both, but does not yet support Zimbra 9.0, where support is available only on 8.8.x. Marc G. asked if using Zimbra Open Source, does Andrea’s company have an integration with Outlook, similar to the Zimbra Connector for Outlook (ZCO). Andrea said that there is currently no integration available for Outlook from VNC, so customers can instead use ZCO, if they are running the Zimbra Network Edition. Marc G. asked if VNC products seamlessly integrate with NextCloud/ownCloud so that files are easily accessible on both desktop and mobile clients. Andrea said files can either be retrieved from the VNC product on the desktop, or the ownCloud app on mobile devices. M. Garbin asked if the VNC product supports shared folders for file sharing. Andrea said that the VNC products do support this capability.

Open Source Vulnerabilities Tracking Service
Randy L. shared news about the OSV project ( https://security.googleblog.com/2021/02 ... ility.html ) that may be a useful resource for both the Zimbra development team, in addition to Zimbra administrators, to keep track of security vulnerabilities in the many open source projects that comprise Zimbra. This Google project is initially focusing on populating the vulnerability database with the results of fuzzing tests on various open source projects, which is the process of injecting unusual input in to an app and checking for unexpected behavior or app crashes that could result in an exploit. They plan to expand beyond fuzzing to incorporate CVEs (Common Vulnerabilities and Exposures; https://cve.mitre.org/ ) and security researcher’s findings too. They are hoping to create a centralized resource for this information, which often needs to be tracked by administrators and developers manually from multiple sources.


Randy Leiker
Skyway Networks, LLC
Post Reply