Maldua's Pimbra - Patched Zimbra

[JDunphy]

Yes this is a mess. I chose 9.0 because it seemed to be worst case example.

I build any version of a release from 8.8.15 to 10.1 and I think I mentioned this before but for zm-build, if someone gives me a version 9.0.0.p40 to build, I don't start with the latest 9.0 tag of zm-build.git but look for p40 and if it isn't there then I start walking backwards until I find the tag which for 9.0.0.p40 was p38 for zm-build. That is how the README.md file told us and I thought it was beneficial to have an independent tool that could generate this given they didn't always keep up the documentation.

Code: Select all

% ./build_zimbra.sh --dry-run --version 9.0.0.p40 
#!/bin/sh
git clone --depth 1 --branch "9.0.0.p38" "git@github.com:Zimbra/zm-build.git"
cd zm-build
ENV_CACHE_CLEAR_FLAG=true ./build.pl --ant-options -DskipTests=true --git-default-tag="9.0.0.p40,9.0.0.p39,9.0.0.p38,9.0.0.p37,9.0.0.p36,9.0.0.p34,9.0.0.p33,9.0.0.P33,9.0.0.p32.1,9.0.0.p32,9.0.0.p30,9.0.0.p29,9.0.0.p28,9.0.0.p27,9.0.0.p26,9.0.0.p25,9.0.0.p24.1,9.0.0.p24,9.0.0.p23,9.0.0.p22,9.0.0.p21,9.0.0.p20,9.0.0.p19,9.0.0.p18,9.0.0.p17,9.0.0.p16,9.0.0.p15,9.0.0.p14,9.0.0.p13,9.0.0.p12,9.0.0.p11,9.0.0.p10,9.0.0.p9,9.0.0.p8,9.0.0.p7,9.0.0.p6,9.0.0.p5,9.0.0.p4,9.0.0.p3,9.0.0.p2,9.0.0.p1,9.0.0" --build-release-no="9.0.0" --build-type=FOSS --build-release="KEPLER_T090000p40C090000p38FOSS" --build-thirdparty-server=files.zimbra.com --no-interactive --build-release-candidate=GA 

That also fixed some issues where the latest zm-build repository would no longer build earlier releases .... but that ended when they started removing/deleting actual repositories so one can only go back so far now.

None of this is your problem but I liked your idea of a patched zimbra and wanted to understand how the repositories were cloned and tagged.

One nagging concern is pulling a repository with updated patches that do not have a tag associated with it then we add our tag and now include those patches. You are very careful but it shows how much work might be involved to verify incomplete schema patches and other things don't end up in the release. I don't even know if that scenario is possible but I have been known to worry about things that don't or can't happen.

Jim

[adrian.gibanel.btactic]

and I think I mentioned this before but for zm-build, if someone gives me a version 9.0.0.p40 to build, I don't start with the latest 9.0 tag of zm-build.git but look for p40 and if it isn't there then I start walking backwards until I find the tag which for 9.0.0.p40 was p38 for zm-build. That is how the README.md file told us and I thought it was beneficial to have an independent tool that could generate this given they didn't always keep up the documentation.

Maldua's zimbra-tag-helper goes the other way around. The tags are ordered from more recent to least recent and then you find the target tag.
Also Maldua's zimbra-tag-helper does not only check for non zm-build repos linked from the most recent zm-build tag but also from old ones. I wonder if that's the difference with your tool that makes you think that some repositories have been removed.

In any way it's true that it should be a good idea to snapshot old versions of Zimbra in a similar way to archive.org just in case. E.g. 9.0.0.pX is going to be EOL in threee months.

None of this is your problem but I liked your idea of a patched zimbra and wanted to understand how the repositories were cloned and tagged.

Yeah, I'm sure you now do . I will improve the public explanation in the Pimbra repos later on with some bits of this thread.

One nagging concern is pulling a repository with updated patches that do not have a tag associated with it then we add our tag and now include those patches. You are very careful but it shows how much work might be involved to verify incomplete schema patches and other things don't end up in the release. I don't even know if that scenario is possible but I have been known to worry about things that don't or can't happen.

Jim

Well, once again, it's not written anywhere... but the main policy about Pimbra (quite similar to maldua-foss-builder) is that we build the Foss releases as similar as possible as their Zimbra NE counterparts.
So... if there is a x.y.z-maldua tag then it means that it has been branched out from a recent upstream tag.
It's quite easy to branch out from a recent upstream tag where you would never have incomplete schema patches in the first place because they are stable tags so to speak.
The idea here is not to add new features or branch out from develop branches.

[JDunphy]

We first saw them change files in the instructions directory in zm-build.git ... can't remember tag where that happened. They removed 2 or 3 repositories from files in the instruction directory initially... eventually they completely removed those repositories from github which broke some really early release builds on 9.0 and 8.8.15.

I also see exactly what you were seeing for latest tag as that is different logic and do it the same as you before your recent change to include the latest tag from zm-build repository to --get-default-tag which I am still holding out and not doing. Again using 9.0.0 as the example since it clearly shows there is no repositories tagged with p44 used in the build of the ZCS tarball and also shows they are not updating version 9.0 so no reason to build a new tarball unless you want to include the pimbra repositories.

Code: Select all

./build_zimbra.sh --dry-run --version 9.0 
***** pimbra_tag [9.0.0.p44] LATEST_TAG [9.0.0.p43] *****
#!/bin/sh
git clone --depth 1 --branch "9.0.0.p44" "git@github.com:Zimbra/zm-build.git"
cd zm-build
ENV_CACHE_CLEAR_FLAG=true ./build.pl --ant-options -DskipTests=true --git-default-tag="9.0.0.p43,9.0.0.p42,9.0.0.p41,9.0.0.p40,9.0.0.p39,9.0.0.p38,9.0.0.p37,9.0.0.p36,9.0.0.p34,9.0.0.p33,9.0.0.P33,9.0.0.p32.1,9.0.0.p32,9.0.0.p30,9.0.0.p29,9.0.0.p28,9.0.0.p27,9.0.0.p26,9.0.0.p25,9.0.0.p24.1,9.0.0.p24,9.0.0.p23,9.0.0.p22,9.0.0.p21,9.0.0.p20,9.0.0.p19,9.0.0.p18,9.0.0.p17,9.0.0.p16,9.0.0.p15,9.0.0.p14,9.0.0.p13,9.0.0.p12,9.0.0.p11,9.0.0.p10,9.0.0.p9,9.0.0.p8,9.0.0.p7,9.0.0.p6,9.0.0.p5,9.0.0.p4,9.0.0.p3,9.0.0.p2,9.0.0.p1,9.0.0" --build-release-no="9.0.0" --build-type=FOSS --build-release="KEPLER_T090000p43C090000p44FOSS" --build-thirdparty-server=files.zimbra.com --no-interactive --build-release-candidate=GA   --git-overrides maldua-pimbra.url-prefix="git@github.com:maldua-pimbra" --git-overrides zm-web-client.remote="maldua-pimbra" --git-overrides zm-web-client.tag="9.0.0.p44-maldua"

Jim

[zmcontrol]

Yeah, I saw that branch but... you know... it's a bit tricky. A branch it's supposed to be used as a work-in-progress. Why should we imply that the 'hotfix/10.1.6' branch should have ended onto '10.1.6' tag as is ?

adrian.gibanel.btactic,

It is currently not complete as is, it's missing the 10.1.5 commits.
I did confirm that other than the missing 10.1.5 commits the hotfix/10.1.6 branch is complete, so nothing to imply here.

Either way not a big deal, just a friendly suggestion since you had put in the time to set up pimbra.
It would be trivial to create a 10.1.6-maldua tag so users building 10.1.6 FOSS would have a proper zm-web-client repo matching 10.1.6 NE.
This project could end up being crucial to the FOSS community going forward.
Much thanks for setting up pimbra!

[adrian.gibanel.btactic]

Yeah, I saw that branch but... you know... it's a bit tricky. A branch it's supposed to be used as a work-in-progress. Why should we imply that the 'hotfix/10.1.6' branch should have ended onto '10.1.6' tag as is ?

adrian.gibanel.btactic,

It is currently not complete as is, it's missing the 10.1.5 commits.
I did confirm that other than the missing 10.1.5 commits the hotfix/10.1.6 branch is complete, so nothing to imply here.

I see. So you're saying hotfix/10.1.6 branch plus the recreated 10.1.5 commits equals NE 10.1.6 without adding any NE-specific bits.
I didn't know that at the build time.
Be sure to give similar feedback for the next 10.1.7 release so that I can pick the right branch/commit where to apply the recreated 10.1.5 commits.
Thank you!

Either way not a big deal, just a friendly suggestion since you had put in the time to set up pimbra.
It would be trivial to create a 10.1.6-maldua tag so users building 10.1.6 FOSS would have a proper zm-web-client repo matching 10.1.6 NE.

The 10.1.6-maldua tag for maldua-pimbra-config repo (which points to 10.1.5-maldua in zm-web-client repo) it's already there and I'm not going to force-push it.

This project could end up being crucial to the FOSS community going forward.
Much thanks for setting up pimbra!

I should resume the zimbra-mirror project instead so that we don't lose any repos but the workflows have been paused for a while, probably because Zimbra was force-pushing branches and not making it easy to replicate their repos. Not that I'm going to do that soon so if anyone want to volunteer go ahead.

[adrian.gibanel.btactic]

In response to the problems described on FOSS - Upgrading to 10.1.7 breaks nginx I have created a 10.1.7 tag so that you can apply Pimbra patches in such a way that commits related to modern-UI-only Zulip chat integration are reverted and don't cause nginx to fail to start.

I haven't tested to build with this Pimbra 10.1.7 tag myself though so feedback is welcome.

Check the first post on this thread if you want to know how to use Pimbra in your build.

---

Update: As it might happen with all of the Pimbra patches in the end this was properly patched and in the end it was not needed.

[adrian.gibanel.btactic]

I wanted to keep you updated with how I am updating Pimbra repos right now.

So our original contract was that when a Zimbra version needs those security fixes you can check Maldua repos for its tag. E.g.: for 10.1.5 version you can download https://github.com/maldua-pimbra/maldua-pimbra-config/raw/refs/tags/10.1.5/config.build and use it.

As embargos come and go and are not complete it's not easy to determine when to put the right tag.
Additionally from the Maldua zimbra-foss-builder calling a release like 10.1.5 (with the original Zimbra tag) does not seem fair because it's not reflecting that it's not completely upstream.

So... in the end.. we are going to have tags (with their own config.build) which includes pX in its name to reflect the Patch/Pimbra nature in them. E.g.: You now have 10.1.9.p1 config.build.

Sometimes you have 10.1.7.p1 with some of the patches and, then later on, you might need 10.1.7.p2 to reflect that you are applying more security patches from newer releases (which are partially embargoed).

How have I updated zimbra-foss-builder for this?

Well, the final user requests to build 10.1.9.p1 with pimbra support enabled. What zimbra-foss-builder does under the hood is to remove the patch part from the version (It becomes 10.1.9) and asks for the usual '10.1.9,10.1.8,...10.0.0' tag list. It currently puts 10.1.9.p1 at the very start so that it is: '10.1.9.p1,10.1.9,10.1.8,...10.0.0' but I am quite sure this is not actually needed.

Related commits from zimbra-tag-helper which I also modified:
- zm-build-tags-arguments-from-file.sh: Add pimbra support for extra tags.
- zm-build-tags-arguments.sh: Add pimbra support for extra tags.

So if you want to accomodate this new way of naming Pimbra tags into either your build wrapper or your manual build process it's just about:

- Downloading config.build (and optionally merge with your existing config.build) based on the complete tag with the tag (10.1.9.p1)
- Do everything else with the patch part stripped out (10.1.9)

This new way of patching would interfere with old 9.0.0.pXX releases but those are EOL so we should be fine regarding that.