IMPORTANT: DO NOT let system package manger update zimbra-jetty-distribution w/o 10.1.13

Ask questions about your setup or get help installing ZCS server (ZD section below).
User avatar
jered
Advanced member
Advanced member
Posts: 116
Joined: Sat Sep 13, 2014 12:35 am
Location: Somerville, MA

IMPORTANT: DO NOT let system package manger update zimbra-jetty-distribution w/o 10.1.13

Post by jered »

My system package manager pulled the Zimbra FOSS packages from the repo updates coincident with 10.1.13, which caused zmmailboxd and the web apps to fail to start.

This was due to the update of zimbra-jetty-distribution from 9.4.46 to 9.4.57; zmmailboxd.out has lots of errors that boil down to:

Code: Select all

org.xml.sax.SAXParseException; Unable to parse: file:///opt/zimbra/jetty_base/etc/jetty.xml
Caused by: org.xml.sax.SAXParseException; lineNumber: 26; columnNumber: 24; Attribute value "pool" of type ID must be unique within the document.
This is resolved by a zm-jetty-conf commit (https://github.com/Zimbra/zm-jetty-conf ... 34dbeba4bb) that is part of 10.1.13.

For the time being, it's important to pin the version of zimbra-jetty-distribution; I suppose one could also manually fixup jetty.xml.in as per the repo.

Zimbra upgrade instructions specifically do instruct never updating the FOSS packages separate from a full Zimbra upgrade, but this usually is not a compatibility issue -- this update will, for example, break a new install of 10.1.12 (or earlier).
User avatar
halfgaar
Outstanding Member
Outstanding Member
Posts: 252
Joined: Sat Sep 13, 2014 12:54 am
Location: Netherlands
ZCS/ZD Version: Ubuntu 22.04, Maldua/Btactic FOSS
Contact:

Re: IMPORTANT: DO NOT let system package manger update zimbra-jetty-distribution w/o 10.1.13

Post by halfgaar »

There are other packages that break if you just do the apt stuff. I have learned you just can't do that. Which means now I'm in a bind, because my FOSS doesn't have 10.1.13.

As for the clam-av fixes that would be installed, I disabled clamav a long time ago.
Consider seriously: because of the history of exploits: block Zimbra web interface with VPN, firewall or HTTP proxy.
User avatar
jered
Advanced member
Advanced member
Posts: 116
Joined: Sat Sep 13, 2014 12:35 am
Location: Somerville, MA

Re: IMPORTANT: DO NOT let system package manger update zimbra-jetty-distribution w/o 10.1.13

Post by jered »

halfgaar wrote: Thu Nov 06, 2025 7:44 pm There are other packages that break if you just do the apt stuff. I have learned you just can't do that. Which means now I'm in a bind, because my FOSS doesn't have 10.1.13.
What other breaking changes have you noticed with the 10.1.13 FOSS packages? Rolling back and pinning zimbra-jetty-distribution was sufficient to resolve my problems. I have not yet seen any issues with the other major update (openssl), and I do not use Zimbra's clamav.
User avatar
jered
Advanced member
Advanced member
Posts: 116
Joined: Sat Sep 13, 2014 12:35 am
Location: Somerville, MA

Re: IMPORTANT: DO NOT let system package manger update zimbra-jetty-distribution w/o 10.1.13

Post by jered »

BTW, the fundamental bug here is that the .spec and .deb files that are part of the zm-build repo specify FOSS component versions that are >= the corresponding release component. This is normal, but is a problem for a system like Zimbra where all the components are tightly coupled. I would strongly suggest that Synacor modify the package definitions to specify only the fixed corresponding version number of a FOSS component build -- this would be a helpful bug to file by someone with access, as it's a pretty simple fix.
User avatar
halfgaar
Outstanding Member
Outstanding Member
Posts: 252
Joined: Sat Sep 13, 2014 12:54 am
Location: Netherlands
ZCS/ZD Version: Ubuntu 22.04, Maldua/Btactic FOSS
Contact:

Re: IMPORTANT: DO NOT let system package manger update zimbra-jetty-distribution w/o 10.1.13

Post by halfgaar »

jered wrote: Thu Nov 06, 2025 10:58 pm What other breaking changes have you noticed with the 10.1.13 FOSS packages? Rolling back and pinning zimbra-jetty-distribution was sufficient to resolve my problems. I have not yet seen any issues with the other major update (openssl), and I do not use Zimbra's clamav.
That wasn't specifically 10.1.13, but the Nginx proxy is known to break. The normal deb packages that apt would install break, because necessary work that the zimbra installer does is required as well.
jered wrote: Thu Nov 06, 2025 11:01 pm BTW, the fundamental bug here is that the .spec and .deb files that are part of the zm-build repo specify FOSS component versions that are >= the corresponding release component. This is normal, but is a problem for a system like Zimbra where all the components are tightly coupled. I would strongly suggest that Synacor modify the package definitions to specify only the fixed corresponding version number of a FOSS component build -- this would be a helpful bug to file by someone with access, as it's a pretty simple fix.
I'm a bit confused here. By FOSS, you seem to mean the 3rd party dependencies pulled in by apt? The term FOSS is in my head mostly as 'the FOSS build of Zimbra' (from Maldua in my case).

Do you mean that all dependencies (the packages from the apt repo repo.zimbra.com) are pulled in as '==' dependency by packages installed by the installer? So like this >= would be == :

Code: Select all

# aptitude why zimbra-jetty-distribution
i   zimbra-store Depends zimbra-jetty-distribution (>= 9.4.46.v20220331-2.u22)
I'm actually not sure how that will interact with the fact that zimbra-jetty-distribution is manually installed:

Code: Select all

# aptitude show zimbra-jetty-distribution
Package: zimbra-jetty-distribution       
Version: 9.4.57.v20241219-2.u22
New: yes
State: installed (9.4.46.v20220331-2.u22), upgrade available (9.4.57.v20241219-2.u22)
Automatically installed: no
Priority: optional
Section: utils
Maintainer: Zimbra Packaging Services <packaging-devel@zimbra.com>
Architecture: amd64
Uncompressed Size: 12,0 M
Depends: libc6 (>= 2.4), zimbra-store-base
Description: Zimbra's jetty-distribution
 
Homepage: https://www.eclipse.org/jetty
Will apt try to update it and fail, or will it just stick to the version required by 'zimbra-store'?
Last edited by halfgaar on Fri Nov 07, 2025 11:15 am, edited 1 time in total.
Consider seriously: because of the history of exploits: block Zimbra web interface with VPN, firewall or HTTP proxy.
User avatar
adrian.gibanel.btactic
Outstanding Member
Outstanding Member
Posts: 582
Joined: Thu Jan 30, 2014 11:13 am
Contact:

snapshot Zimbra Binary Packages repos idea

Post by adrian.gibanel.btactic »

This thread is interesting.

It's like replicating snapshot.debian.org but for the different zimbra repos is something worth considering.

Something like:

Code: Select all

deb     [arch=amd64] https://repo.snapshot-for-zimbra.org/archive/debian/apt/87/20091004T111800Z/ jammy zimbra
deb-src [arch=amd64] https://repo.snapshot-for-zimbra.org/archive/debian/apt/87/20091004T111800Z/ jammy zimbra
deb     [arch=amd64] https://repo.snapshot-for-zimbra.org/archive/debian/apt/1000/20091004T111800Z/ jammy zimbra
deb     [arch=amd64] https://repo.snapshot-for-zimbra.org/archive/debian/apt/1010/20091004T111800Z/ jammy zimbra
Extra: I am currently working on a zimbra tracker that would track Zimbra Github repos changes.
JBravo
Posts: 12
Joined: Mon Feb 10, 2025 11:36 pm

Re: IMPORTANT: DO NOT let system package manger update zimbra-jetty-distribution w/o 10.1.13

Post by JBravo »

Sadly, I did not catch this thread before dnf-automatic upgraded those packages.

Code: Select all

[root@zimb log]# yum history info 69
Updating Subscription Management repositories.
Transaction ID : 69
Begin time     : Fri 07 Nov 2025 06:50:40 GMT
Begin rpmdb    : 1503:d51249888272f881ae0255774a18bc6769c9a35e
End time       : Fri 07 Nov 2025 06:51:21 GMT (41 seconds)
End rpmdb      : 1504:e23d25dd0da4a96cde934b7faf141acb5641a9fb
User           : System <unset>
Return-Code    : Success
Releasever     : 8
Command Line   : 
Comment        : 
Packages Altered:
    Install  xxhash-libs-0.8.2-1.el8.x86_64                         @rhel-8-for-x86_64-appstream-rpms
    Upgrade  tigervnc-1.15.0-8.el8_10.x86_64                        @rhel-8-for-x86_64-appstream-rpms
    Upgraded tigervnc-1.15.0-7.el8_10.x86_64                        @@System
    Upgrade  tigervnc-icons-1.15.0-8.el8_10.noarch                  @rhel-8-for-x86_64-appstream-rpms
    Upgraded tigervnc-icons-1.15.0-7.el8_10.noarch                  @@System
    Upgrade  tigervnc-license-1.15.0-8.el8_10.noarch                @rhel-8-for-x86_64-appstream-rpms
    Upgraded tigervnc-license-1.15.0-7.el8_10.noarch                @@System
    Upgrade  zimbra-clamav-1.4.3-1zimbra8.8b4.el8.x86_64            @zimbra-1000-oss
    Upgraded zimbra-clamav-1.0.8-1zimbra8.8b4.el8.x86_64            @@System
    Upgrade  zimbra-clamav-db-1.0.0-1zimbra8.7b3.el8.x86_64         @zimbra-1000-oss
    Upgraded zimbra-clamav-db-1.0.0-1zimbra8.7b2.el8.x86_64         @@System
    Upgrade  zimbra-clamav-libs-1.4.3-1zimbra8.8b4.el8.x86_64       @zimbra-1000-oss
    Upgraded zimbra-clamav-libs-1.0.8-1zimbra8.8b4.el8.x86_64       @@System
    Upgrade  zimbra-jetty-distribution-9.4.57.v20241219-2.r8.x86_64 @zimbra-1000-oss
    Upgraded zimbra-jetty-distribution-9.4.46.v20220331-2.r8.x86_64 @@System
    Upgrade  zimbra-core-components-10.1.5-1zimbra10.0b1.el8.x86_64 @zimbra-1010-oss
    Upgraded zimbra-core-components-10.1.3-1zimbra10.0b1.el8.x86_64 @@System
    Upgrade  zimbra-ldap-components-10.1.2-1zimbra10.0b1.el8.x86_64 @zimbra-1010-oss
    Upgraded zimbra-ldap-components-10.1.0-1zimbra10.0b1.el8.x86_64 @@System
    Upgrade  zimbra-mta-components-10.1.4-1zimbra8.8b1.el8.x86_64   @zimbra-1010-oss
    Upgraded zimbra-mta-components-10.1.2-1zimbra8.8b1.el8.x86_64   @@System
    Upgrade  zimbra-openssl-3.5.1-1zimbra8.8b1.el8.x86_64           @zimbra-1010-oss
    Upgraded zimbra-openssl-3.0.9-1zimbra8.8b1.el8.x86_64           @@System
    Upgrade  zimbra-openssl-libs-3.5.1-1zimbra8.8b1.el8.x86_64      @zimbra-1010-oss
    Upgraded zimbra-openssl-libs-3.0.9-1zimbra8.8b1.el8.x86_64      @@System
    Upgrade  zimbra-postfix-3.6.14-1zimbra8.7b6.el8.x86_64          @zimbra-1010-oss
    Upgraded zimbra-postfix-3.6.14-1zimbra8.7b5.el8.x86_64          @@System
    Upgrade  zimbra-rsync-3.4.1-1zimbra8.7b2.el8.x86_64             @zimbra-1010-oss
    Upgraded zimbra-rsync-3.4.1-1zimbra8.7b1.el8.x86_64             @@System
Scriptlet output:
   1 Nov 07, 2025 6:50:51 AM org.python.google.common.base.internal.Finalizer getInheritableThreadLocalsField
   2 INFO: Couldn't access Thread.inheritableThreadLocals. Reference finalizer threads will inherit thread local values.
   3 stty: 'standard input': Inappropriate ioctl for device
Now the mailbox service wont start and same with the webapp services. I see the errors like:
Caused by: org.xml.sax.SAXParseException; lineNumber: 26; columnNumber: 24; Attribute value "pool" of type ID must be unique within the document.
Caused by: org.xml.sax.SAXParseException; Unable to parse: file:///opt/zimbra/jetty_base/etc/jetty.xml
Caused by: java.security.PrivilegedActionException: org.xml.sax.SAXParseException; Unable to parse: file:///opt/zimbra/jetty_base/etc/jetty.xml

What is the fix here? Im running maldua's 10.1.9 on RHEL 8
User avatar
adrian.gibanel.btactic
Outstanding Member
Outstanding Member
Posts: 582
Joined: Thu Jan 30, 2014 11:13 am
Contact:

Re: IMPORTANT: DO NOT let system package manger update zimbra-jetty-distribution w/o 10.1.13

Post by adrian.gibanel.btactic »

JBravo wrote: Fri Nov 07, 2025 12:37 pm Sadly, I did not catch this thread before dnf-automatic upgraded those packages.

Code: Select all

[root@zimb log]# yum history info 69
Updating Subscription Management repositories.
Transaction ID : 69
Begin time     : Fri 07 Nov 2025 06:50:40 GMT
Begin rpmdb    : 1503:d51249888272f881ae0255774a18bc6769c9a35e
End time       : Fri 07 Nov 2025 06:51:21 GMT (41 seconds)
End rpmdb      : 1504:e23d25dd0da4a96cde934b7faf141acb5641a9fb
User           : System <unset>
Return-Code    : Success
Releasever     : 8
Command Line   : 
Comment        : 
Packages Altered:
    Install  xxhash-libs-0.8.2-1.el8.x86_64                         @rhel-8-for-x86_64-appstream-rpms
    Upgrade  tigervnc-1.15.0-8.el8_10.x86_64                        @rhel-8-for-x86_64-appstream-rpms
    Upgraded tigervnc-1.15.0-7.el8_10.x86_64                        @@System
    Upgrade  tigervnc-icons-1.15.0-8.el8_10.noarch                  @rhel-8-for-x86_64-appstream-rpms
    Upgraded tigervnc-icons-1.15.0-7.el8_10.noarch                  @@System
    Upgrade  tigervnc-license-1.15.0-8.el8_10.noarch                @rhel-8-for-x86_64-appstream-rpms
    Upgraded tigervnc-license-1.15.0-7.el8_10.noarch                @@System
    Upgrade  zimbra-clamav-1.4.3-1zimbra8.8b4.el8.x86_64            @zimbra-1000-oss
    Upgraded zimbra-clamav-1.0.8-1zimbra8.8b4.el8.x86_64            @@System
    Upgrade  zimbra-clamav-db-1.0.0-1zimbra8.7b3.el8.x86_64         @zimbra-1000-oss
    Upgraded zimbra-clamav-db-1.0.0-1zimbra8.7b2.el8.x86_64         @@System
    Upgrade  zimbra-clamav-libs-1.4.3-1zimbra8.8b4.el8.x86_64       @zimbra-1000-oss
    Upgraded zimbra-clamav-libs-1.0.8-1zimbra8.8b4.el8.x86_64       @@System
    Upgrade  zimbra-jetty-distribution-9.4.57.v20241219-2.r8.x86_64 @zimbra-1000-oss
    Upgraded zimbra-jetty-distribution-9.4.46.v20220331-2.r8.x86_64 @@System
    Upgrade  zimbra-core-components-10.1.5-1zimbra10.0b1.el8.x86_64 @zimbra-1010-oss
    Upgraded zimbra-core-components-10.1.3-1zimbra10.0b1.el8.x86_64 @@System
    Upgrade  zimbra-ldap-components-10.1.2-1zimbra10.0b1.el8.x86_64 @zimbra-1010-oss
    Upgraded zimbra-ldap-components-10.1.0-1zimbra10.0b1.el8.x86_64 @@System
    Upgrade  zimbra-mta-components-10.1.4-1zimbra8.8b1.el8.x86_64   @zimbra-1010-oss
    Upgraded zimbra-mta-components-10.1.2-1zimbra8.8b1.el8.x86_64   @@System
    Upgrade  zimbra-openssl-3.5.1-1zimbra8.8b1.el8.x86_64           @zimbra-1010-oss
    Upgraded zimbra-openssl-3.0.9-1zimbra8.8b1.el8.x86_64           @@System
    Upgrade  zimbra-openssl-libs-3.5.1-1zimbra8.8b1.el8.x86_64      @zimbra-1010-oss
    Upgraded zimbra-openssl-libs-3.0.9-1zimbra8.8b1.el8.x86_64      @@System
    Upgrade  zimbra-postfix-3.6.14-1zimbra8.7b6.el8.x86_64          @zimbra-1010-oss
    Upgraded zimbra-postfix-3.6.14-1zimbra8.7b5.el8.x86_64          @@System
    Upgrade  zimbra-rsync-3.4.1-1zimbra8.7b2.el8.x86_64             @zimbra-1010-oss
    Upgraded zimbra-rsync-3.4.1-1zimbra8.7b1.el8.x86_64             @@System
Scriptlet output:
   1 Nov 07, 2025 6:50:51 AM org.python.google.common.base.internal.Finalizer getInheritableThreadLocalsField
   2 INFO: Couldn't access Thread.inheritableThreadLocals. Reference finalizer threads will inherit thread local values.
   3 stty: 'standard input': Inappropriate ioctl for device
Now the mailbox service wont start and same with the webapp services. I see the errors like:
Caused by: org.xml.sax.SAXParseException; lineNumber: 26; columnNumber: 24; Attribute value "pool" of type ID must be unique within the document.
Caused by: org.xml.sax.SAXParseException; Unable to parse: file:///opt/zimbra/jetty_base/etc/jetty.xml
Caused by: java.security.PrivilegedActionException: org.xml.sax.SAXParseException; Unable to parse: file:///opt/zimbra/jetty_base/etc/jetty.xml

What is the fix here? Im running maldua's 10.1.9 on RHEL 8
Maybe you can try to adapt the apt pinning from After apt upgrade mailboxd and jetty don´t work. Resolved: downgrade zimbra-jetty-distribution thread to a yum/rpm pinning which I have no idea if it exists.
User avatar
jered
Advanced member
Advanced member
Posts: 116
Joined: Sat Sep 13, 2014 12:35 am
Location: Somerville, MA

Re: IMPORTANT: DO NOT let system package manger update zimbra-jetty-distribution w/o 10.1.13

Post by jered »

JBravo wrote: Fri Nov 07, 2025 12:37 pm What is the fix here? Im running maldua's 10.1.9 on RHEL 8
The fix is to just downgrade to zimbra-jetty-distribution-9.4.46.v20220331-2.r8 with your package manager -- it's still in the repo.

I think the command is "apt install zimbra-jetty-distribution=9.4.46.v20220331-2.r8". Then use apt-pinning to hold that version until future notice.

On RHEL-likes, the equivalent is "dnf install zimbra-jetty-distribution-9.4.46.v20220331-2.r8" and then use the "dnf versionlock" plugin.
User avatar
jered
Advanced member
Advanced member
Posts: 116
Joined: Sat Sep 13, 2014 12:35 am
Location: Somerville, MA

Re: snapshot Zimbra Binary Packages repos idea

Post by jered »

adrian.gibanel.btactic wrote: Fri Nov 07, 2025 9:10 am This thread is interesting.

It's like replicating snapshot.debian.org but for the different zimbra repos is something worth considering.
While this is not a bad idea, this doesn't directly help with the problem. The older package versions are still published in the repo -- it's that the spec/deb files do not pin the required versions.

This sort of breaking change with dependency update doesn't happen very often, but it's really frustrating when it does.
Post Reply