Maldua's Zimbra 10 FOSS Build system

[adrian.gibanel.btactic]

HI Adrian,

I did end up redoing the tag building using the remote method vs local cloning with git tags. About 11-13 seconds vs 1:18-1:20 which is fast enough that I am now doing it dynamically inside build_zimbra.sh instead of calling an external tag helper and using cached files containing prebuilt tags with your helper scripts. It also simplifies the builds for novices as they only need to know --version option to reliably build new releases for themselves on any of the supported platforms.

Any link to the code or its repo so that I can take a deeper dive into it?

Thank you!

[JDunphy]

HI Adrian,

I did end up redoing the tag building using the remote method vs local cloning with git tags. About 11-13 seconds vs 1:18-1:20 which is fast enough that I am now doing it dynamically inside build_zimbra.sh instead of calling an external tag helper and using cached files containing prebuilt tags with your helper scripts. It also simplifies the builds for novices as they only need to know --version option to reliably build new releases for themselves on any of the supported platforms.

Any link to the code or its repo so that I can take a deeper dive into it?

Thank you!

It is what I posted here but looping with the repos so we could do the comparison with the tag helper. I used an inline perl call in my bash function to grab the repos since we needed to include those instruction repo lists.

Link to github for this can be found here:
https://github.com/JimDunphy/build_zimbra.sh
or described here:
https://wiki.zimbra.com/wiki/JDunphy-Co ... mbraScript

testing directory has some unit tests... build_tags_test.sh is stand-alone and was what I integrated into build_zimbra.sh ... It would be similar to your tag helper in function as a standalone.

My suspicion is that we are headed to iterating tags from 1 to N so none of this complexity will be necessary after December 31, 2024 given 9 will be EOL.

Jim

[adrian.gibanel.btactic]

Not sure if it's worth an update to this thread but here it goes.

I have added a script helper to ease maintainer releases so that in one go you can build every build for each of the distros and each of one of the main version families.
It actually needs more improvement so that it doesn't need to be edited each time a new set of releases is built.

Here: maintainer-release.sh.

[adrian.gibanel.btactic]

As you might now the current Maldua releases do not feature a custom repo.
It's a bit of a hassle having to download and install the tgz installer again and again.

The zimbra-patch support? issue has brought up the idea of a custom repo (including only core packages) and associated maldua-zimbra-patch packages.

We still do not know what are the minimal commands you need to issue to apply the new packages onto a running system (once you have upgraded the core packages). Feedback is welcome either here or on the mentioned issue above.

One way of simulating that hypothetical new repo would be to unpack the tgz installer for a new version and installing manually the packages which are found in the packages directory.

[adrian.gibanel.btactic]

After a bunch of tests, it looks like the ONLY possibility for identifying unique builds in the GUI is the BUILD_NUMBER, which is back to where Adrian started.

So... I have crafted Zimbra Foss Builders List in order to be able to match build numbers and Zimbra builders (or build script authors).

Feel free to update as needed .

[adrian.gibanel.btactic]

So... ZCS 10.1.5 has been published by Synacor some weeks ago without releasing their vulnerability fix to the Github repos.

---

We already maintain an OnlyOffice pseudo fork (It's a pseudo fork because not all the repos are cloned, and it's not up to date) with unlimited connections. If you check the public big howto you can learn how to fetch OnlyOffice (upstream) updated branches, fetch bTactic patches, push everything onto your own Github repo and then build onlyOffice binaries from there.

It is based on three points:

• Reusing some specific commits that are needed for unlimited connections (akin to patch non-public CVEs)
• Hacking the official build system so that some specific packages are fetched from custom Github repos
• Using tags with specific suffixes so that unlimited connections related commits are actually used.

---

Going back to build Zimbra,... I'm not very sure if a new organisation (acme-open-patch) should be created, if we should create repos with suffixes (zm-web-client-acme) (see below on the notes) or if we should go with repos and tags with special suffixes (10.1.5-acme).

In any case you can check GIT_OVERRIDES on config.build.in where you can override for a given repo its remote url, branch and tag. I guess you are supposed to copy config.build.in as config.build so that it's taken onto account.

There might also be an alternate way of doing the same thing from the ./build.pl arguments too.

So I think this is the way forward:

• Maintain our own forks of Zimbra repos with our own crafted vulnerability fixes.
• Create branches or tags each a time a new release is published with the crafted-vulnerability fixes being cherry-pick on the top.
• Publish documentation on how to build Zimbra with all of the above applied.

If anyone has any experience on using these override switches, feedback is welcome.

I am busy with other projects so I won't be able to tackle on this in the short term.

Notes:

• This is not about having an equivalent to the zimbra-patch package .
• Zimbra has recently implemented an improvement to zm-build itself so that you can have a common suffix to the modified repos. E.g. zm-web-client-acme.

[adrian.gibanel.btactic]

I am busy with other projects so I won't be able to tackle on this in the short term.

Thanks to zmcontrol I have made a quick draft on how to implement this.

[adrian.gibanel.btactic]

@JDunphy... Are you able to build 9.0.0.p44 with your alternate way of getting tags?

The current v0.0.1 of zimbra-tag-helper is unable to build 9.0.0.p44 because that tag is only found only in zm-build repo. In a normal release zm-web-client would also have had a tag there.

Anyways, it's rather easy to fix on my side... I'm just curious to see if you trigger the same problem.

[JDunphy]

@JDunphy... Are you able to build 9.0.0.p44 with your alternate way of getting tags?

The current v0.0.1 of zimbra-tag-helper is unable to build 9.0.0.p44 because that tag is only found only in zm-build repo. In a normal release zm-web-client would also have had a tag there.

Anyways, it's rather easy to fix on my side... I'm just curious to see if you trigger the same problem.

That would be the same here, If you specify p44 and there isn't any repository with p44 than ./build_zimbra.sh would tell you that there is not p44 and stop. However if you do it like this:

Code: Select all

% ./build_zimbra.sh --version 9.0

Then yes it will build latest but there would be no p44 in the --git-default-tag list because p44 exists only in zm-build.git.

Jim

[adrian.gibanel.btactic]

@JDunphy... Are you able to build 9.0.0.p44 with your alternate way of getting tags?

The current v0.0.1 of zimbra-tag-helper is unable to build 9.0.0.p44 because that tag is only found only in zm-build repo. In a normal release zm-web-client would also have had a tag there.

Anyways, it's rather easy to fix on my side... I'm just curious to see if you trigger the same problem.

That would be the same here, If you specify p44 and there isn't any repository with p44 than ./build_zimbra.sh would tell you that there is not p44 and stop. However if you do it like this:

Just in case it's helpful here there is the fix for zimbra-tag-helper.