ZCS FOSS 10.1.12 - Upcoming FOSS release issues

[adrian.gibanel.btactic]

If you keep a track of Github repos commits and recently pushed tags you can deduce that ZCS NE 10.1.12 release (and its ZCS NE 10.0.18 counterpart) is going to be released soon.

But this release seems to have many issues:

- First of all we are waiting for the ZCS 10.1.10 embargoed commits which are taking many more days than the usual 60 days of embargo. These are:
-- Addressed a Cross-Site Request Forgery (CSRF) vulnerability in the ResetPasswordRequest SOAP operation by enforcing CSRF token validation.
-- A security fix has been applied to require a valid auth token before allowing 2FA modifications, preventing unauthorized changes. (This one I'm not sure if it's found in main Zimbra or in the 2FA NE extension which wouldn't be published anyways).

- Regarding zm-zcs-lib repo:
-- zm-zcs-lib 10.0.18 commits has ZBUG-4023:Upgraded jetty to 9.4.57.v20241219.
-- zm-zcs-lib 10.1.13 commits do exist and have both ZBUG-4023 and ZBUG-4572-Upgrade-Apache-HttpClient-library-to-4.5.14-for-zm-zcs-lib .
-- No 10.1.12 tag has been pushed.

So... was 10.1.12 tag pushed as 10.1.13 by mistake?
Or maybe 10.1.13 was pushed way earlier (actually this is not a problem if that's the case) ?
I actually remember having seen 10.1.12 tags some days ago which I guess have been force-pushed as 10.1.13. Or maybe my grey-matter memory is faulty.

The reports below won't be so much detailed:

- Regarding zm-mailbox repo:
-- develop has ZBUG-4023
-- develop has ZBUG-4572
-- No 10.0.18 tag has been pushed.
-- No 10.1.12 tag has been pushed.

- Regarding packages, zm-build, zm-jetty-conf, repos:
-- 10.0.18 has ZBUG-4023.
-- 10.1.13 has ZBUG-4023.
-- No 10.1.12 tag has been pushed.

- Regarding zm-bulkprovision-admin-zimlet, zm-helptooltip-zimlet, zm-nginx-lookup-store, zm-proxy-config-admin-zimlet, zm-ssdb-ephemeral-store, zm-versioncheck-admin-zimlet, zm-viewmail-admin-zimlet repos:
-- 10.1.13 has 10.1.12 ZCS-17537: ldapsdk version update to 7.0.3 commit.
-- No 10.0.18 tag has been pushed.
-- No 10.1.12 tag has been pushed.

- Regarding zm-zcs, zm-oauth-social, antisamy, zm-zimlets repos:
-- 10.1.13 has ZBUG-4572.
-- No 10.0.18 tag has been pushed.
-- No 10.1.12 tag has been pushed.


- Regarding zm-admin-console repo:
-- 10.0.18 has ZBUG-4023.
-- No 10.1.12 tag has been pushed.

- Regarding zm-ajax, zm-admin-ajax repo:
-- 10.0.18 has ZBUG-5081
-- No 10.1.12 tag has been pushed.

- Regarding zm-web-client repo:
-- develop has ZBUG-4023
-- No 10.0.18 tag has been pushed.
-- No 10.1.12 tag has been pushed.

- Assuming 10.1.12 will have Ubuntu24 which I'm not so sure right now, well, repos with ZCS-17694 in their develop branches such as zm-core-utils should also have their tags uploaded.

So... yes... you are right... I'm probably complaining too early about all of these small details.
Some of them might not relevant as of some bug fix is applied to 10.1.x but not 10.0.x (or viceversa) because the latter one is not affected.
Some of them might have been fixed anyways just a few days earlier than the NE release.
Some of them might actually be specific commits from 10.1.13 which, as has happened other times, future tags has been published in the Zimbra Github repos earlier and we should just ignore it.

In any case I prefer to complain now and I give some time for this to be fixed (including 10.1.10 and 10.0.16 embargoed commits ) rather than wait for the ZCS NE releases and be disappointed when I check the Zimbra Github repos again. Hopefully I can save myself writing a ZCS FOSS 10.1.12 - Security fixes and embargoed commits thread.

[rainer_d]

10.1.12 is out now - and it's a fix for the chat proxy.

Isn't that NE-only anyway? Or is there an OSS version of chat?

[adrian.gibanel.btactic]

10.1.12 is out now - and it's a fix for the chat proxy.

Isn't that NE-only anyway? Or is there an OSS version of chat?

Thank you for the update!

ZCS 10.1.12 having only NE changes makes a lot of sense. That's why there was not any 10.1.2 tag being pushed to the public repos. So... no need for additional feedback from Synacor after all.

ZCS NE 10.0.18 was not released (makes sense because it was announced to be EOL). However the 10.0.18 tag is in the repos with this zm-zcs-lib ZBUG-4023:Upgraded jetty to 9.4.57.v20241219 additional commit.

I ask myself if I would have eventually to add this change as 10.0.16.p1 or 10.0.16.p2 once they lift the 10.0.16 embargo. Don't think so.

I think that we can conclude that if we find in the public repos a two releases ahead tag and the one release ahead tag is missing... maybe the latter one is going to be NE only.

[ghen]

Maybe it's a new policy to separate critical security patch releases from the usual new feature / incremental update releases?

That would be a Really Good Thing, and allow us to properly prioritise the deployment of patch releases. So 10.1.12 for the critical Chat proxy issue, followed by 10.1.13 for the jetty upgrade and other new stuff, so as a customer you're not forced to deploy those in a rush just to get that security fix?

[BradC]

Maybe it's a new policy to separate critical security patch releases from the usual new feature / incremental update releases?

I can't see how. Let's say I'm at 10.1.12. The 10.1.13 release is a pile of new features and bugs I don't want. Then 10.1.14 comes along with another urgent security fix. There's no way to install 10.1.14 without getting the contents of 10.1.13 that I've been avoiding until they've fixed it properly.

I think what we've seen for the last 2 patches is "We're not ready to push out the next patch, just put out a single-fix patch in the meantime". That's why there were a pile of recent commits to LDAP attributes changing "from 10.1.12" to "from 10.1.13". Stuff planned for the next patch that's not ready, but a security vulnerability severe enough to warrant a rapid response.

[ghen]

In that scenario, the 10.1.13 update isn't urgent (assuming you're already on 10.1.12), you can take your time to test it and deploy it. And by the time 10.1.14 arrives with a hypothetical urgent security fix, it again isn't burdened by irrelevant other changes.

That's certainly better than 10.1.x coming out with urgent security fixes and various other stuff that makes an urgent upgrade less controlled and more risky.

[BradC]

In that scenario, the 10.1.13 update isn't urgent (assuming you're already on 10.1.12), you can take your time to test it and deploy it. And by the time 10.1.14 arrives with a hypothetical urgent security fix, it again isn't burdened by irrelevant other changes.

I see what you are saying, but at points we've been 2 or 3 patches behind due to significant bugs. It's not a matter of just taking time to test and deploy, but taking time to wait for fixes in "the next patch" before deploying. Anyway, it's always going to be sequential so we're always going to be in the situation.

[ghen]

Yes, we're in the same position, and we have often manually merged security fixes directly onto our servers to avoid full patch installations.
But still, separate bugfix vs. generic improvement patch releases would also allow to spot the relevant changes more easily. Now it's often hard to tell which bits are relevant and which are not.

[adrian.gibanel.btactic]

Maybe it's a new policy to separate critical security patch releases from the usual new feature / incremental update releases?

I'm not sure where I read it but it's expected that Zimbra allows you to subscribe to either a bugfix-only repo or a bugfix+enhacements repo in the future. Not the exact words but that's the idea.

That's a lot of work.

- You have to maintain and differentiate two different Zimbras from the support point of view. Maybe two names, maybe checking zmcontrol -v ?
- Installer should be updated to ask you which mode you want (or maybe two installers). Also write an additional command to switch between modes.
- Update related documentation.
- Then you have to maintain two repos for Ubuntu packages and two repos for RHEL packages.
- Then write down announcements in the wiki page... with two different announcements?
- Same in the blog page.
- Then the email that are being sent when there are email version updates... they should also take into account this dual scenario.

In any case if you check the Zimbra page history you can see how in the 8.6.x era when they didn't even have package repos... they updated and published like 4 or 5 versions at the same time. So... some experience they have.

If that useful idea regarding Zimbra sysadmins that don't want to be updating so often... is going to be implemented or if it's going to be buried into a drawer as other proposed ideas in the past... it's to be seen.

If you ask me I bet that they will be able to implement it. Despite the Foss-related rants I might have I have noticed an improvement on the Zimbra development quality. Less and less patches/versions are breaking stuff that previously worked.

[ghen]

You are probably referring to this: https://blog.zimbra.com/2023/09/extendi ... ent-153456, which sounded just like two separate release branches, eg. 10.0.x and 10.1.x (or 11.x).

Beginning with the release of Zimbra Daffodil (Ver.10), Zimbra has formalized a three-year general support period for major releases.

The three-year cycle has been implemented to ensure optimally secured and up-to-date third-party components in Zimbra continue to work as intended. We feel this is critical to maintaining the security of the Zimbra software. Starting in 2024, two paths are available to customers catering to their tolerance for changes.

– Stable Path (low impact) – contains only necessary changes, including security and critical bug fixes.
– Feature Path (medium impact) – contains some changes which may cause different behavior, including new features.

Stable/feature path will be on Zimbra 10.

Unfortunately this promise never materialized, and the people that announced it are gone now.