Zimbra OSS - what are you using to install/upgrade?

[siliconalchemy]

Zimbra really did a number on opensource. As well as stopping publishing builds for OSS, the build process seems to be a complicated mess that only a select few understand with enough confidence to build reliable, secure packages, and even then there's confusion about which tags mean what, when.

So who do you use to install/upgrade Zimbra OSS? I've added a poll here out of interest. The problem I see is that Techfiles isn't updated nearly enough to be useful (and I'm not in any way denigrating what he's contributed to the community) - they're simply not released frequently enough to respond to security problems. Intalio scares me because it's repo based and there's no transparency to what it actually is or does. Maldua seems to be the most frequently updated, but I'm completely confused about what of the many versions on the download page to use, and what actually covers whatever serious security issue has been released today about Zimbra.

I've used Zimbra since it first came out in beta, I've deployed it to companies, I've run it personally, I've bought NE, I've bought Zextras, but I've really lost all confidence in Zimbra recently - even in the paid versions. It just all seems such as mess. And there are serious security vulns coming out quicker than I can process them.

Anyone else feel this way, or just me?

[JDunphy]

I used to feel exactly like what you have articulated about the build process when 9.0 FOSS binaries were no longer provided. It appeared complicated and coupled with bad documentation that made building a source release feel more like trial and error than reproducible at times. That changed when Adrian came up with some helper scripts to generate the Maldua builds. Previous to that, we had Ian's build script that would generate a build from the current branch in github. Both of them then produced binaries for those that didn't want to build their own releases from source code. A major concern I had and I believe they also had was that we have had companies jump in and claim they will support the FOSS builds and then it was silence and they never seem to share the build process they used.

Having said that, Zimbra/Synacor since version 10 has attempted to tag the repositories in a sane fashion in contrast to version 8.8.15 and 9.0 for build releases and also updates the github README.md file with the syntax and tags needed to build by tags.

A quick note on the tags... there are approx 54+ repositories and the build.pl script that runs everything will attempt the highest numbered tag (--git-default-tag) and if that doesn't exist, use the next highest, etc. Here is an example:

Code: Select all

% ./build_zimbra.sh --dry-run --version 10.1
#!/bin/sh
git clone --depth 1 --branch "10.1.1" "git@github.com:Zimbra/zm-build.git"
cd zm-build
ENV_CACHE_CLEAR_FLAG=true ./build.pl --ant-options -DskipTests=true --git-default-tag="10.1.2,10.1.1,10.1.0" --build-release-no="10.1.2" --build-type=FOSS --build-release="DAFFODIL_T100102C100101FOSS" --build-thirdparty-server=files.zimbra.com --no-interactive --build-release-candidate=GA

That script above will go through all the repositories and determine the best zm-build.git repository (tools to build a release) to use AND generate a list of the tags required for the latest version 10.1

Here is another example building a specific version 10.1.0

Code: Select all

% /build_zimbra.sh --dry-run --version 10.1.0
#!/bin/sh
git clone --depth 1 --branch "10.1.0" "git@github.com:Zimbra/zm-build.git"
cd zm-build
ENV_CACHE_CLEAR_FLAG=true ./build.pl --ant-options -DskipTests=true --git-default-tag="10.1.0" --build-release-no="10.1.0" --build-type=FOSS --build-release="DAFFODIL_T100100C100100FOSS" --build-thirdparty-server=files.zimbra.com --no-interactive --build-release-candidate=GA

Notice that the list of tags has changed but also the zm-build.git branch to checkout has changed. Tools like this can validate what the zm-build.git documentation says for building releases. If you leave off the --dry-run then it will build a tarball.

In other words, we have tools that can generate reproducible FOSS builds. We have at least 2 builders that are providing full binaries and have documented how they build their binaries. This uncertainty of building FOSS was a major reason I wanted to understand the build process and ended up with this build_zimbra.sh script to verify that FOSS wasn't dead. I also documented this in the wiki page below. One other piece of the puzzle is that Zimbra still provides the repositories for the 3rd party open source binaries (nginx,ldap, etc) meaning that most if not all of the FOSS binaries and builds will use these when you execute install.sh to install or update your FOSS zimbra.

Having said that, I still and have always used the network version in production so the above script while it works and generates FOSS tarballs are not production builds I use. They are on test machines and is my canary in the coal mine to know if something drastically changes within Zimbra.

Your point of verification of the security patches for FOSS is valid. It would seem like any tool we come up with could be abused by bad actors.

Ref: https://wiki.zimbra.com/wiki/JDunphy-Co ... mbraScript

[Martinwiertz]

Feel the same.

I had to switch to another FOSS-builder because Zextras discontinued. I used them for builds and Zextras suite. After considerations on available builders I went for Intalio. Works great.

I am waiting on the 10.1 way of working but so far all updates are good. I appreciate all opensource providers for their effort. Thank you.
I am not experienced enough to build on my own.

Currently a way of working with a docker image seems a way to go. I will try in test.

[halfgaar]

Maldua seems to be the most frequently updated, but I'm completely confused about what of the many versions on the download page to use, and what actually covers whatever serious security issue has been released today about Zimbra.

He waits with calling a release 'stable' for two weeks. His 'recent' means 'testing', sort of. I installed a 'recent' 10.1.1 when the security vulnerability CVE-2024-45519 in postjournal came out. But yes, there was nothing specially indicated about it containing a security patch. That would be hard to do, because Zimbra is also not very clear about it.

The state of Ubuntu 22.04 is also unclear. It's marked as BETA on one, but not the other. I wonder if that's a mistake. (for reference, on this page).

I asked that in his feedback forum thread, and I also reached out to him by private message when the recent vulnerability came out (because as a packager he'd be interested, likely), but he didn't reply. The PM is still in my outbox here on the forum, actually. I don't know if he's on vacation, or ... ? It also doesn't help that the forum's e-mail notifications seem broken. I haven't had one for a long time.

[H_Rocky]

Maldua seems to be the most frequently updated, but I'm completely confused about what of the many versions on the download page to use, and what actually covers whatever serious security issue has been released today about Zimbra.

He waits with calling a release 'stable' for two weeks. His 'recent' means 'testing', sort of. I installed a 'recent' 10.1.1 when the security vulnerability CVE-2024-45519 in postjournal came out. But yes, there was nothing specially indicated about it containing a security patch. That would be hard to do, because Zimbra is also not very clear about it.

The state of Ubuntu 22.04 is also unclear. It's marked as BETA on one, but not the other. I wonder if that's a mistake. (for reference, on this page).

I asked that in his feedback forum thread, and I also reached out to him by private message when the recent vulnerability came out (because as a packager he'd be interested, likely), but he didn't reply. The PM is still in my outbox here on the forum, actually. I don't know if he's on vacation, or ... ? It also doesn't help that the forum's e-mail notifications seem broken. I haven't had one for a long time.

Officially:
Ubuntu 22.04 has GA status for Zimbra 10.1.1.
RHEL 9 has GA status for Zimbra 10.1.2.

[siliconalchemy]

I used to feel exactly like what you have articulated about the build process when 9.0 FOSS binaries were no longer provided. It appeared complicated and coupled with bad documentation that made building a source release feel more like trial and error than reproducible at times. That changed when Adrian came up with some helper scripts to generate the Maldua builds. Previous to that, we had Ian's build script that would generate a build from the current branch in github. Both of them then produced binaries for those that didn't want to build their own releases from source code. A major concern I had and I believe they also had was that we have had companies jump in and claim they will support the FOSS builds and then it was silence and they never seem to share the build process they used.

...

In other words, we have tools that can generate reproducible FOSS builds. We have at least 2 builders that are providing full binaries and have documented how they build their binaries. This uncertainty of building FOSS was a major reason I wanted to understand the build process and ended up with this build_zimbra.sh script to verify that FOSS wasn't dead. I also documented this in the wiki page below. One other piece of the puzzle is that Zimbra still provides the repositories for the 3rd party open source binaries (nginx,ldap, etc) meaning that most if not all of the FOSS binaries and builds will use these when you execute install.sh to install or update your FOSS zimbra.

Your point of verification of the security patches for FOSS is valid. It would seem like any tool we come up with could be abused by bad actors.

Ref: https://wiki.zimbra.com/wiki/JDunphy-Co ... mbraScript

Thank you very much for your work, and to Ian/Adrian. You're absolutely right, being able to reliably/consistently/transparently build FOSS packages is the most important thing to document and enable.

I've always been puzzled by Zimbra (and it's various owners) approach to opensource. They've never been remotely interested in fostering cooperation/engagement with opensource developers, at times it's felt like they've actively put roadblocks in the way to prevent it. I suspect they have to keep doing it to a large degree from the legacy of the intial release, having a large integrated footprint of opensource components, and at this point it would be too difficult to change the licensing. Hence separately developing isolated components like the modern UI that is completely closed source, and retro closing Zimbra Desktop (which I guess they've migrated to modern UI?). In recent years it feels like Zimbra is simply a tool used for investment companies to generate income/increase sale value as an asset, hence the decreasing interest in the opensource side of things.

In looking at FOSS options for the few very small instances that I still run, it looks like there are the following options:

- Techfiles.online packages (Ian?)
- Maldua packages (Adrian?)
- Intalio repo
- Self-build from the aforementioned scripts

I ruled two options out immediately:
- Intalio repo. I think it's great that this outfit produces repo based installs for FOSS builds. They are updated rapidly after Zimbra releases, usually within a day or two. However, there is zero transparency so we have absolutely no clue what is in these builds. That alone excludes these builds for me.
- Techonline packages. It looks like the author of these has been instrumental in the ability to build FOSS packages so we owe a lot to him for doing so - many thanks. However, they're only updated sporadically and not nearly enough to use in a production environment. There have been some fairly serious security issues raised in the recent past and so having packages updated in a good time frame is vital, so this 100% excludes these builds for me.

- Maldua packages. These are actively and frequently updated, so this is what I've chosen to currently use. However saying that, reading through his threads there is still quite a bit of confusion around tags, where bugs/security issues are fixed etc. For example I've just installed 10.1.2 but the Maldua packages were generated on the 24th Sep, whereas the Zimbra release was done on Oct 8th. Do the Maldua packages contain all the fixes that Zimbra call 10.1.2? Zimbra used to issue a definitive build number that was easy to identify the cutoff point of a release, but since the move to git and then 9.x and 10.x releases, it looks like the exact revision of each release is a more murky thing to pin down exactly? But clearly there's a huge amount of thought and effort that goes into these releases, a huge thanks to those involved.
- Self-build. I run Ubuntu 22.04 LTS across most of my servers so was forced to build some 'legacy' 20.04 VMs to run Zimbra as it took them so long to support 22.04 (in fact after 24.04 LTS was released!). So I don't have build/development resources to hand to build Zimbra. I guess now 22.04 builds are finally a thing I can upgrade and start to build releases myself which is the ideal way to do it, although I guess I'll still have to wait for you guys to upgrade the builds scripts sometimes to support the new tags/builds.

A huge thanks for everyone's work recently on getting 10.x FOSS builds a reality - I waited for years to upgrade from 8.8.15 as there weren't really good options. Sadly it looks like what community there was around Zimbra has diminished to a very small remaining few. I wonder if this is around the prevalence of cloud based services, or about Zimbra itself? I still can't see any opensource alternatives to the level of Zimbra. Is the world really going to lose the ability to run an opensource email system and keep sovereignty over our own private communications?

[JDunphy]

- Self-build. I run Ubuntu 22.04 LTS across most of my servers so was forced to build some 'legacy' 20.04 VMs to run Zimbra as it took them so long to support 22.04 (in fact after 24.04 LTS was released!). So I don't have build/development resources to hand to build Zimbra. I guess now 22.04 builds are finally a thing I can upgrade and start to build releases myself which is the ideal way to do it, although I guess I'll still have to wait for you guys to upgrade the builds scripts sometimes to support the new tags/builds.

A few additional points... @liverpoolfcfan has tested updates from 8.8.15 all the way through 10 as part of a method to name a version string to identify the FOSS builds. That gives us a very high level of confidence that FOSS builds can be built and updated in a continuous fashion across versions. He folded those naming changes into the build_zimbra.sh script which also is keeping a history of the previous builds.

He also provided a tag build script back in June 24, 2024 for version 10.1 which eventually showed up months later as we were not supporting that new release at the time in the script. That eventually led to a rewrite of how we generate tags which is different than Adrian's tag helpers since we pull the tags remotely from every repository vs cloning and then doing it locally per repository. That is good news because we have 2 tools that appear to generate identical tag output but are quite different in their implementation and we still have the Zimbra README.md file with the manual steps that tells you exactly the tags you would need to create your own manual build so in theory you should never have to wait for developers to update the scripts to support the latest tags. For a conceptual model of these tag helpers - think this for each repository and then creating a list (descending) that can be passed to build.pl:

Code: Select all

% cd zm_build (or any zimbra repository)
% git tag |grep "10.1"

As far as build_zimbra.sh, it only takes about 10 seconds now to create the correct tag list. Tags are specific to repositories. The script will pull the appropriate tags when building. We don't even cache the tag files which is how it was previously done. It's fairly simple... if you have 54 repositories with your version 10.1 build and you find tag 10.1.2 somewhere than Zimbra's build.pl script will always attempt that 1st for each repository before trying the next tag 10.1.1 in the build list when cloning the repository. So these tags helpers scripts will most likely in the future with continued sane git tagging from Zimbra become simple decrementer from N to 0 for the tags and whatever version you are trying to build. The actual build script to build zimbra and create that tarball using this tag list is what... 4 lines which I show in my first post above.

As for building your own FOSS releases, the biggest hurdle to build the latest tag version which we hope matches the Network version of the release is:

* having a free github account with your public key stored from the build machine with that github account
* having a supported linux os release from Zimbra installed (that is what they test against)

You don't have to be root so create a directory and do:

Code: Select all

% mkdir my_build; cd my_build
% /build_zimbra.sh --init
% ./build_zimbra.sh --version 10.1

Next time you want to build the latest release after a new patch you do the following again.

Code: Select all

% ./build_zimbra --version 10.1

The dependency logic of setting up that initial build environment is documented in the Zimbra README.md file so you can do it manually with apt/yum/dnf/etc or have build_zimbra.sh --init which clones Ian's build script and runs this command:

Code: Select all

% ./zimbra-build-helper.sh --install-deps

I hope your thread gives more admin's confidence to build their own releases when they want to and know that a tag build is a tag build. If you do it manually via what Zimbra recommends on their README.md page or you use maldua binaries or you use another method such as build_zimbra.sh then the code should be the same provided the tag list is similar. What is NOT the same is the output from:

Code: Select all

% zmcontrol -v

We have no actual method of duplicating the version string that the NETWORK version has at this point so each FOSS builder came up with a scheme. We would have had to patch the source code and that is not something these builder scripts are doing so we use the existing parameters provided by build.pl to encode that information.

HTH,

Jim

[CSylvain]

I put on other since I use my own Release.

[adrian.gibanel.btactic]

Maldua seems to be the most frequently updated, but I'm completely confused about what of the many versions on the download page to use, and what actually covers whatever serious security issue has been released today about Zimbra.

He waits with calling a release 'stable' for two weeks. His 'recent' means 'testing', sort of.

That's right you have: Stable, Recent (Sort of testing) and Experimental (Sort of beta).
It's nice to have this sort of feedback.
I might end up renaming the current download as 'Archive'.
A new download page (the default one) would only have the very latest Stable releases (with a dropdown like Zimbra does) and a link to the Archive.

[adrian.gibanel.btactic]

even then there's confusion about which tags mean what, when.

I installed a 'recent' 10.1.1 when the security vulnerability CVE-2024-45519 in postjournal came out. But yes, there was nothing specially indicated about it containing a security patch. That would be hard to do, because Zimbra is also not very clear about it.

Your point of verification of the security patches for FOSS is valid. It would seem like any tool we come up with could be abused by bad actors.

1) So... the official policy about Maldua builds which, apparently it's not written anywhere in the downloads page is to track NE tags as close as possible. Usually NE tags appears in Github one week prior to NE release and we build from there.

I will have to add a bold message that clarifies it. Our versions matches NE versions. E.g. Maldua ZCS 10.1.2 versions matches Synacord ZCS NE 10.1.2 regarding its FOSS parts and vulnerability fixes.

2) Do the Github tags that appear one week prior to NE release contain whatever security patch ends up in the NE release with the same tag?

We do not know. We asked about that in viewtopic.php?p=313525#p313525 but we got no answer in reply.

Again... an official announcement about a new open source version (not available as a binary) from Synacor is a must. Either that or a clarification that once a new tag appears in the Github repo it means that in less than 24 hours that tag will be updated through all of the repos.

Right now I am writing this message offline (forums.zimbra.org I forgot my password does not work) and it is October 10th, 2024. zm-mailbox repo already has a 10.1.3 tag ( d3367a24f789e991caa7690299d8e0fff15664d2 ) but there is no official NE 10.1.3 release. There is also a 10.0.11 tag but not a 9.0.0.p42 tag.

3) Currently the build script generates an snippet of commands that explain how the build has been built (the usual ./build.pl invocation).

I might end up adding to the current zcs-*.tgz, zcs-*.tgz.md5 and zcs-*.tgz.sha256 files a file that shows the HEAD of each repo as a tag and as a commit such as:

Code: Select all

zm-build 10.1.1 2c9e44931f70170191e4df1f8b5eae789a917f76
zm-mailbox 10.1.2 d72b429781ecb874b0dfccfd31c7eebc0b31d1fa

.

It could also be used as an snapshot of the different github repos for a given tag.
That way I will be sure about having to rebuild a version or not.