Maldua's Pimbra - Patched Zimbra

[JDunphy]

Yes this is a mess. I chose 9.0 because it seemed to be worst case example.

I build any version of a release from 8.8.15 to 10.1 and I think I mentioned this before but for zm-build, if someone gives me a version 9.0.0.p40 to build, I don't start with the latest 9.0 tag of zm-build.git but look for p40 and if it isn't there then I start walking backwards until I find the tag which for 9.0.0.p40 was p38 for zm-build. That is how the README.md file told us and I thought it was beneficial to have an independent tool that could generate this given they didn't always keep up the documentation.

Code: Select all

% ./build_zimbra.sh --dry-run --version 9.0.0.p40 
#!/bin/sh
git clone --depth 1 --branch "9.0.0.p38" "git@github.com:Zimbra/zm-build.git"
cd zm-build
ENV_CACHE_CLEAR_FLAG=true ./build.pl --ant-options -DskipTests=true --git-default-tag="9.0.0.p40,9.0.0.p39,9.0.0.p38,9.0.0.p37,9.0.0.p36,9.0.0.p34,9.0.0.p33,9.0.0.P33,9.0.0.p32.1,9.0.0.p32,9.0.0.p30,9.0.0.p29,9.0.0.p28,9.0.0.p27,9.0.0.p26,9.0.0.p25,9.0.0.p24.1,9.0.0.p24,9.0.0.p23,9.0.0.p22,9.0.0.p21,9.0.0.p20,9.0.0.p19,9.0.0.p18,9.0.0.p17,9.0.0.p16,9.0.0.p15,9.0.0.p14,9.0.0.p13,9.0.0.p12,9.0.0.p11,9.0.0.p10,9.0.0.p9,9.0.0.p8,9.0.0.p7,9.0.0.p6,9.0.0.p5,9.0.0.p4,9.0.0.p3,9.0.0.p2,9.0.0.p1,9.0.0" --build-release-no="9.0.0" --build-type=FOSS --build-release="KEPLER_T090000p40C090000p38FOSS" --build-thirdparty-server=files.zimbra.com --no-interactive --build-release-candidate=GA 

That also fixed some issues where the latest zm-build repository would no longer build earlier releases .... but that ended when they started removing/deleting actual repositories so one can only go back so far now.

None of this is your problem but I liked your idea of a patched zimbra and wanted to understand how the repositories were cloned and tagged.

One nagging concern is pulling a repository with updated patches that do not have a tag associated with it then we add our tag and now include those patches. You are very careful but it shows how much work might be involved to verify incomplete schema patches and other things don't end up in the release. I don't even know if that scenario is possible but I have been known to worry about things that don't or can't happen.

Jim

[adrian.gibanel.btactic]

and I think I mentioned this before but for zm-build, if someone gives me a version 9.0.0.p40 to build, I don't start with the latest 9.0 tag of zm-build.git but look for p40 and if it isn't there then I start walking backwards until I find the tag which for 9.0.0.p40 was p38 for zm-build. That is how the README.md file told us and I thought it was beneficial to have an independent tool that could generate this given they didn't always keep up the documentation.

Maldua's zimbra-tag-helper goes the other way around. The tags are ordered from more recent to least recent and then you find the target tag.
Also Maldua's zimbra-tag-helper does not only check for non zm-build repos linked from the most recent zm-build tag but also from old ones. I wonder if that's the difference with your tool that makes you think that some repositories have been removed.

In any way it's true that it should be a good idea to snapshot old versions of Zimbra in a similar way to archive.org just in case. E.g. 9.0.0.pX is going to be EOL in threee months.

None of this is your problem but I liked your idea of a patched zimbra and wanted to understand how the repositories were cloned and tagged.

Yeah, I'm sure you now do . I will improve the public explanation in the Pimbra repos later on with some bits of this thread.

One nagging concern is pulling a repository with updated patches that do not have a tag associated with it then we add our tag and now include those patches. You are very careful but it shows how much work might be involved to verify incomplete schema patches and other things don't end up in the release. I don't even know if that scenario is possible but I have been known to worry about things that don't or can't happen.

Jim

Well, once again, it's not written anywhere... but the main policy about Pimbra (quite similar to maldua-foss-builder) is that we build the Foss releases as similar as possible as their Zimbra NE counterparts.
So... if there is a x.y.z-maldua tag then it means that it has been branched out from a recent upstream tag.
It's quite easy to branch out from a recent upstream tag where you would never have incomplete schema patches in the first place because they are stable tags so to speak.
The idea here is not to add new features or branch out from develop branches.

[JDunphy]

We first saw them change files in the instructions directory in zm-build.git ... can't remember tag where that happened. They removed 2 or 3 repositories from files in the instruction directory initially... eventually they completely removed those repositories from github which broke some really early release builds on 9.0 and 8.8.15.

I also see exactly what you were seeing for latest tag as that is different logic and do it the same as you before your recent change to include the latest tag from zm-build repository to --get-default-tag which I am still holding out and not doing. Again using 9.0.0 as the example since it clearly shows there is no repositories tagged with p44 used in the build of the ZCS tarball and also shows they are not updating version 9.0 so no reason to build a new tarball unless you want to include the pimbra repositories.

Code: Select all

./build_zimbra.sh --dry-run --version 9.0 
***** pimbra_tag [9.0.0.p44] LATEST_TAG [9.0.0.p43] *****
#!/bin/sh
git clone --depth 1 --branch "9.0.0.p44" "git@github.com:Zimbra/zm-build.git"
cd zm-build
ENV_CACHE_CLEAR_FLAG=true ./build.pl --ant-options -DskipTests=true --git-default-tag="9.0.0.p43,9.0.0.p42,9.0.0.p41,9.0.0.p40,9.0.0.p39,9.0.0.p38,9.0.0.p37,9.0.0.p36,9.0.0.p34,9.0.0.p33,9.0.0.P33,9.0.0.p32.1,9.0.0.p32,9.0.0.p30,9.0.0.p29,9.0.0.p28,9.0.0.p27,9.0.0.p26,9.0.0.p25,9.0.0.p24.1,9.0.0.p24,9.0.0.p23,9.0.0.p22,9.0.0.p21,9.0.0.p20,9.0.0.p19,9.0.0.p18,9.0.0.p17,9.0.0.p16,9.0.0.p15,9.0.0.p14,9.0.0.p13,9.0.0.p12,9.0.0.p11,9.0.0.p10,9.0.0.p9,9.0.0.p8,9.0.0.p7,9.0.0.p6,9.0.0.p5,9.0.0.p4,9.0.0.p3,9.0.0.p2,9.0.0.p1,9.0.0" --build-release-no="9.0.0" --build-type=FOSS --build-release="KEPLER_T090000p43C090000p44FOSS" --build-thirdparty-server=files.zimbra.com --no-interactive --build-release-candidate=GA   --git-overrides maldua-pimbra.url-prefix="git@github.com:maldua-pimbra" --git-overrides zm-web-client.remote="maldua-pimbra" --git-overrides zm-web-client.tag="9.0.0.p44-maldua"

Jim

[zmcontrol]

Yeah, I saw that branch but... you know... it's a bit tricky. A branch it's supposed to be used as a work-in-progress. Why should we imply that the 'hotfix/10.1.6' branch should have ended onto '10.1.6' tag as is ?

adrian.gibanel.btactic,

It is currently not complete as is, it's missing the 10.1.5 commits.
I did confirm that other than the missing 10.1.5 commits the hotfix/10.1.6 branch is complete, so nothing to imply here.

Either way not a big deal, just a friendly suggestion since you had put in the time to set up pimbra.
It would be trivial to create a 10.1.6-maldua tag so users building 10.1.6 FOSS would have a proper zm-web-client repo matching 10.1.6 NE.
This project could end up being crucial to the FOSS community going forward.
Much thanks for setting up pimbra!

[adrian.gibanel.btactic]

Yeah, I saw that branch but... you know... it's a bit tricky. A branch it's supposed to be used as a work-in-progress. Why should we imply that the 'hotfix/10.1.6' branch should have ended onto '10.1.6' tag as is ?

adrian.gibanel.btactic,

It is currently not complete as is, it's missing the 10.1.5 commits.
I did confirm that other than the missing 10.1.5 commits the hotfix/10.1.6 branch is complete, so nothing to imply here.

I see. So you're saying hotfix/10.1.6 branch plus the recreated 10.1.5 commits equals NE 10.1.6 without adding any NE-specific bits.
I didn't know that at the build time.
Be sure to give similar feedback for the next 10.1.7 release so that I can pick the right branch/commit where to apply the recreated 10.1.5 commits.
Thank you!

Either way not a big deal, just a friendly suggestion since you had put in the time to set up pimbra.
It would be trivial to create a 10.1.6-maldua tag so users building 10.1.6 FOSS would have a proper zm-web-client repo matching 10.1.6 NE.

The 10.1.6-maldua tag for maldua-pimbra-config repo (which points to 10.1.5-maldua in zm-web-client repo) it's already there and I'm not going to force-push it.

This project could end up being crucial to the FOSS community going forward.
Much thanks for setting up pimbra!

I should resume the zimbra-mirror project instead so that we don't lose any repos but the workflows have been paused for a while, probably because Zimbra was force-pushing branches and not making it easy to replicate their repos. Not that I'm going to do that soon so if anyone want to volunteer go ahead.

[adrian.gibanel.btactic]

In response to the problems described on FOSS - Upgrading to 10.1.7 breaks nginx I have created a 10.1.7 tag so that you can apply Pimbra patches in such a way that commits related to modern-UI-only Zulip chat integration are reverted and don't cause nginx to fail to start.

I haven't tested to build with this Pimbra 10.1.7 tag myself though so feedback is welcome.

Check the first post on this thread if you want to know how to use Pimbra in your build.