8.8.15 Patch 40 GA Release

Ask questions about your setup or get help installing ZCS server (ZD section below).
imanudin11
Outstanding Member
Outstanding Member
Posts: 304
Joined: Sat Sep 13, 2014 2:23 am
ZCS/ZD Version: Release 8.8.15.GA.3829.UBUNTU16.64
Contact:

Re: 8.8.15 Patch 40 GA Release

Post by imanudin11 »

rwalcott wrote: Thu Jun 01, 2023 2:07 pm Updated a single-server instance of 8.8.15 P38 / CentOS 7 to P40, after the upgrade all services are shown to be running and the base web UI loads up. But when a user try's to login a message saying "A network service error has occurred" appears, also the admin page shows up blank when we try to access it through the admin url. Inspecting the page shows 503 (service Unavailable) errors.

Here is what we see in the logs:

Code: Select all

2023-05-31 22:47:55,838 ERROR [qtp439928219-42:https://mail.example.com/] [] webclient - Unable to get domain config
com.zimbra.common.service.ServiceException: error while proxying request to target server: Service Unavailable

Code: Select all

2023-05-31 18:53:27,660 WARN  [qtp439928219-23:https://mail.example.com/] [] webclient - system failure: error while proxying request to target server: Service Unavailable
com.zimbra.common.service.ServiceException: system failure: error while proxying request to target server: Service Unavailable
Has anyone else experience this problem?
I have similar problem. I have tested on my prod email server and patch goes well (zimbra 9 NE). However, when I patch on my client's server, I got problem like yours. Because I have not found a solution, I decide to revert all package that changes to the previous version. This is what I do on my Zimbra 8.8.15 on top Ubuntu 20

Code: Select all

apt install zimbra-patch=8.8.15.1680534965.p39-1.u20 zimbra-common-core-jar=8.8.15.1679569995-1.u20 zimbra-mbox-war=8.8.15.1655458176-1.u20 zimbra-common-core-libs=8.8.15.1663926687-1.u20 zimbra-mbox-webclient-war=8.8.15.1680786603-1.u20
Restart zimbra services and all zimbra function work again
**

Best Regards,
Ahmad Imanudin - Sharing is Beautiful !
Personal Blog [EN] :http://www.imanudin.net
User avatar
oetiker
Outstanding Member
Outstanding Member
Posts: 275
Joined: Fri Mar 07, 2014 1:05 pm
Location: Switzerland
ZCS/ZD Version: Release 10.0.6.GA.4518.UBUNTU20_64
Contact:

Re: 8.8.15 Patch 40 GA Release

Post by oetiker »

Hi

I did the upgrade on a single node Ubuntu 18.04 VM (p39-p40), All looks good ...
Release 8.8.15.GA.3869.UBUNTU18.64 UBUNTU18_64 NETWORK edition, Patch 8.8.15_P40.

(still need to have a filesystem that has capabilities otherwise you have to save the slapd startup
script and the sudo ldap config in place)

Code: Select all

/opt/zimbra/bin/ldap
/etc/sudoers.d/02_zimbra-ldap
rwalcott
Posts: 2
Joined: Thu Jun 01, 2023 1:48 pm

Re: 8.8.15 Patch 40 GA Release

Post by rwalcott »

rwalcott wrote: Thu Jun 01, 2023 2:07 pm Updated a single-server instance of 8.8.15 P38 / CentOS 7 to P40, after the upgrade all services are shown to be running and the base web UI loads up. But when a user try's to login a message saying "A network service error has occurred" appears, also the admin page shows up blank when we try to access it through the admin url. Inspecting the page shows 503 (service Unavailable) errors.

Here is what we see in the logs:

Code: Select all

2023-05-31 22:47:55,838 ERROR [qtp439928219-42:https://mail.example.com/] [] webclient - Unable to get domain config
com.zimbra.common.service.ServiceException: error while proxying request to target server: Service Unavailable

Code: Select all

2023-05-31 18:53:27,660 WARN  [qtp439928219-23:https://mail.example.com/] [] webclient - system failure: error while proxying request to target server: Service Unavailable
com.zimbra.common.service.ServiceException: system failure: error while proxying request to target server: Service Unavailable
Has anyone else experience this problem?
After a couple days of troubleshooting we narrowed down the issue to the

Code: Select all

zimbra-mbox-war-8.8.15.1684213151-1.r7
package.
Once we downgraded to

Code: Select all

zimbra-mbox-war-8.8.15.1655458176
the issue was resolved. Will have to wait and see if the issue gets addressed before we try another upgrade to patch 40 again.
halfgaar
Advanced member
Advanced member
Posts: 171
Joined: Sat Sep 13, 2014 12:54 am
Location: Netherlands
ZCS/ZD Version: Ubuntu 18.04, 8.8.15_P43
Contact:

Re: 8.8.15 Patch 40 GA Release

Post by halfgaar »

Maybe it should be reported as bug?
Consider seriously: because of the history of exploits: block Zimbra web interface with VPN, firewall or HTTP proxy.
mishomvp
Posts: 5
Joined: Fri Nov 24, 2017 12:25 am

Re: 8.8.15 Patch 40 GA Release

Post by mishomvp »

Hello,

I have the same problem on one of the servers. On two of the servers the patch went through and everything works (for now), but on another it doesn't want to and I had to downgrade.
All servers are running Rocky Linux 8.8.
User avatar
L. Mark Stone
Ambassador
Ambassador
Posts: 2796
Joined: Wed Oct 09, 2013 11:35 am
Location: Portland, Maine, US
ZCS/ZD Version: 10.0.6 Network Edition
Contact:

Re: 8.8.15 Patch 40 GA Release

Post by L. Mark Stone »

Zimbra multi-server farm running 8.8.15 Patch 39 on a mix of Ubuntu 16, 18 and 20 successfully upgraded to Patch 40 with no apparent issues.
___________________________________
L. Mark Stone
Mission Critical Email - Zimbra VAR/BSP/Training Partner https://www.missioncriticalemail.com/
AWS Certified Solutions Architect-Associate
User avatar
barrydegraaff
Zimbra Employee
Zimbra Employee
Posts: 242
Joined: Tue Jun 17, 2014 3:31 am
Contact:

Re: 8.8.15 Patch 40 GA Release

Post by barrydegraaff »

imanudin11 wrote: Fri Jun 02, 2023 2:54 am
rwalcott wrote: Thu Jun 01, 2023 2:07 pm Here is what we see in the logs:

Code: Select all

2023-05-31 22:47:55,838 ERROR [qtp439928219-42:https://mail.example.com/] [] webclient - Unable to get domain config
com.zimbra.common.service.ServiceException: error while proxying request to target server: Service Unavailable
If you can try and re-apply the update, and share additional logs from /opt/zimbra/log/zmmailboxd.out and /opt/zimbra/log/mailbox.log then we can see what the issue may be.
--
Barry de Graaff
Email: barry.degraaff [at] synacor [dot] com
Admin of Zimbra-Community Github: https://github.com/orgs/Zimbra-Community/ and the
Zimlet Gallery https://gallery.zetalliance.org/extend/
gommatermica
Posts: 1
Joined: Wed Jun 07, 2023 6:48 am

Re: 8.8.15 Patch 40 GA Release

Post by gommatermica »

Hi,
i have same problem, some server no update problem, some with the same error reported above.

All server are Centos 7
Release 8.8.15_GA_3869.RHEL7_64_20190917004220 RHEL7_64 FOSS edition, Patch 8.8.15_P39 to P40

I had to downgrade

Code: Select all

yum downgrade zimbra-patch zimbra-mbox-war
the issue was resolved.

Log:

mailbox.log

2023-06-03 15:52:45,546 INFO [main] [] misc - DoSFilter: Configured whitelist IPs = 197.0.0.100,127.0.0.1,::1,0:0:0:0:0:0:0:1
2023-06-03 15:52:45,639 INFO [main] [] misc - Initialized with /service:max=80%, /zimbra:max=15%, /zimbraAdmin:max=5%
2023-06-03 15:52:45,640 INFO [main] [] misc - Thread pool was configured to max=250
2023-06-03 15:52:47,628 INFO [main] [] misc - DoSFilter: Configured whitelist IPs = 197.0.0.100,127.0.0.1,::1,0:0:0:0:0:0:0:1
2023-06-03 15:52:47,631 INFO [main] [] misc - Initialized with /service:max=80%, /zimbra:max=15%, /zimbraAdmin:max=5%
2023-06-03 15:52:47,631 INFO [main] [] misc - Thread pool was configured to max=250
2023-06-03 15:57:04,360 INFO [qtp1461149300-50:https://mail.example.it/] [] AuthProvider - Adding auth provider: zimbra com.zimbra.cs.service.ZimbraAuthProvider
2023-06-03 15:57:04,381 INFO [qtp1461149300-50:https://mail.example.it/] [] AuthProvider - Adding auth provider: sampleoauth com.zimbra.cs.service.ZimbraAuthProviderForOAuth
2023-06-03 15:57:04,745 ERROR [qtp1461149300-50:https://mail.example.it/] [] webclient - Unable to get domain config
com.zimbra.common.service.ServiceException: error while proxying request to target server: Service Unavailable
at com.zimbra.common.service.ServiceException.PROXY_ERROR(ServiceException.java:409) ~[zimbracommon.jar:8.8.15_GA_4545]
at com.zimbra.common.soap.SoapHttpTransport.invoke(SoapHttpTransport.java:326) ~[zimbracommon.jar:8.8.15_GA_4545]
at com.zimbra.common.soap.SoapHttpTransport.invoke(SoapHttpTransport.java:196) ~[zimbracommon.jar:8.8.15_GA_4545]
at com.zimbra.common.soap.SoapHttpTransport.invoke(SoapHttpTransport.java:189) ~[zimbracommon.jar:8.8.15_GA_4545]
at com.zimbra.common.soap.SoapTransport.invoke(SoapTransport.java:447) ~[zimbracommon.jar:8.8.15_GA_4545]
.
.
2023-06-03 15:57:10,692 ERROR [qtp1461149300-50:https://mail.example.it/] [] webclient - Error during GetDomainInfoRequest invocation
com.zimbra.common.service.ServiceException: error while proxying request to target server: Service Unavailable
at com.zimbra.common.service.ServiceException.PROXY_ERROR(ServiceException.java:409) ~[zimbracommon.jar:8.8.15_GA_4545]
at com.zimbra.common.soap.SoapHttpTransport.invoke(SoapHttpTransport.java:326) ~[zimbracommon.jar:8.8.15_GA_4545]
at com.zimbra.common.soap.SoapHttpTransport.invoke(SoapHttpTransport.java:196) ~[zimbracommon.jar:8.8.15_GA_4545]
at com.zimbra.common.soap.SoapHttpTransport.invoke(SoapHttpTransport.java:189) ~[zimbracommon.jar:8.8.15_GA_4545]
at com.zimbra.common.soap.SoapTransport.invoke(SoapTransport.java:447) ~[zimbracommon.jar:8.8.15_GA_4545]
.
.
2023-06-03 15:57:25,099 WARN [qtp1461149300-55:https://mail.example.it/] [] webclient - system failure: error while proxying request to target server: Service Unavailable
com.zimbra.common.service.ServiceException: system failure: error while proxying request to target server: Service Unavailable
at com.zimbra.common.service.ServiceException.FAILURE(ServiceException.java:292) ~[zimbracommon.jar:8.8.15_GA_4545]
at com.zimbra.client.ZMailbox.invoke(ZMailbox.java:1050) ~[zimbraclient.jar:8.8.15_GA_4545]
at com.zimbra.client.ZMailbox.invokeJaxb(ZMailbox.java:979) ~[zimbraclient.jar:8.8.15_GA_4545]
at com.zimbra.client.ZMailbox.authByPassword(ZMailbox.java:881) ~[zimbraclient.jar:8.8.15_GA_4545]
at com.zimbra.client.ZMailbox.<init>(ZMailbox.java:753) ~[zimbraclient.jar:8.8.15_GA_4545]
at com.zimbra.client.ZMailbox.getMailbox(ZMailbox.java:675) ~[zimbraclient.jar:8.8.15_GA_4545]
at com.zimbra.cs.taglib.tag.LoginTag.doTag(LoginTag.java:199) ~[zm-taglib-8.8.15.1676624209.jar:8.8.15_GA_4545]
.
.

no error in zmmailboxd.out

[7802687.804s][info][gc] GC(1238) Pause Young (Normal) (G1 Evacuation Pause) 1157M->570M(3994M) 315.673ms
[7835620.665s][info][gc] GC(1239) Pause Young (Normal) (G1 Evacuation Pause) 1664M->572M(3994M) 686.990ms
[7846069.851s][info][gc] GC(1240) Pause Young (Normal) (G1 Evacuation Pause) 1158M->573M(3994M) 3904.698ms
[7865305.786s][info][gc] GC(1241) Pause Young (Normal) (G1 Evacuation Pause) 1155M->571M(3994M) 525.979ms
[7882735.178s][info][gc] GC(1242) Pause Young (Normal) (G1 Evacuation Pause) 1157M->572M(3994M) 919.901ms

Thank you.
User avatar
JDunphy
Outstanding Member
Outstanding Member
Posts: 889
Joined: Fri Sep 12, 2014 11:18 pm
Location: Victoria, BC
ZCS/ZD Version: 9.0.0_P39 NETWORK Edition

Re: 8.8.15 Patch 40 GA Release

Post by JDunphy »

Code: Select all

% zmcontrol -v
Release 8.8.15_GA_3953.RHEL8_64_20200629025823 RHEL8_64 NETWORK edition, Patch 8.8.15_P40.
I believe this is a problem for us now. It has to do with forwarding an inline image that was uuencoded.

An example of this would be from a company like Marriott Hotels when they send the statement of your stay:

Here is the section which is their logo:

Code: Select all

---1Pnk2fld3
Content-Type: image/jpg
Content-Disposition: inline; filename="Body_Logo.jpg"
Content-Transfer-Encoding: uuencode
Content-Id: <Body_Logo.jpg>

M_]C_X``02D9)1@`!`0$!+`$L``#_[0`L4&AO=&]S:&]P(#,N,``X0DE-`^T`
M`````!`!+`````$``0$L`````0`!_^%+_6AT='`Z+R]N<RYA9&]B92YC;VTO
M>&%P+S$N,"\`/#]X<&%C:V5T(&)E9VEN/2+ON[\B(&ED/2)7-4TP37!#96AI
M2'IR95-Z3E1C>FMC.60B/SX*/'@Z>&UP;65T82!X;6QN<SIX/2)A9&]B93IN
M<SIM971A+R(@>#IX;7!T:STB061O8F4@6$U0($-O<F4@-2XS+6,P,3$@-C8N
You can not forward this as you will get:

Code: Select all

com.zimbra.common.service.ServiceException: system failure: IOExecption
...
...
Caused by: com.sun.mail.util.DecodingException: UUDecoder: Missing begin
...
Work around:
1) view email
2) save attachment
3) new email
4) add attachment
5) send will now work

What doesn't work:
1) forward email
2) redirect email
3) edit as new and send

Support ticket is open but what a weird one. I can't remember the last inline uuencoded image I was looking for in an email. They are normally all base64 encoded.

We can decode it manually from the shell but zimbra seems to have a problem with it. I am leaning toward calling this a bug given this customer stays at Marriott about 200 days a year and this would be normal email activity for them. The assistant makes the reservations and uses her email... she then forwards statements to the person that stayed as an FYI.

PS... to be clear, there is an attached pdf which is the customer statement of days stayed, charges, etc and a logo that is an inline image. Because of that inline image, it is stuck in the users inbox with no way to send it on to another user. I went back and verified that Marriott Hotels have been doing this for years. What is new is the customer gets a network service error when they click forward - not even sent yet. Does anyone know anything about this behavior? We went from P38 to P40 so don't know when this behavior first appeared. I won't bother asking why Marriott Hotels sends out emails like this. ;-)

Jim
User avatar
barrydegraaff
Zimbra Employee
Zimbra Employee
Posts: 242
Joined: Tue Jun 17, 2014 3:31 am
Contact:

Re: 8.8.15 Patch 40 GA Release

Post by barrydegraaff »

JDunphy wrote: Wed Jun 07, 2023 5:08 pm

Code: Select all

% zmcontrol -v
Release 8.8.15_GA_3953.RHEL8_64_20200629025823 RHEL8_64 NETWORK edition, Patch 8.8.15_P40.
I believe this is a problem for us now. It has to do with forwarding an inline image that was uuencoded.

An example of this would be from a company like Marriott Hotels when they send the statement of your stay:

Support ticket is open but what a weird one. I can't remember the last inline uuencoded image I was looking for in an email. They are normally all base64 encoded.

We can decode it manually from the shell but zimbra seems to have a problem with it. I am leaning toward calling this a bug given this customer stays at Marriott about 200 days a year and this would be normal email activity for them. The assistant makes the reservations and uses her email... she then forwards statements to the person that stayed as an FYI.

PS... to be clear, there is an attached pdf which is the customer statement of days stayed, charges, etc and a logo that is an inline image. Because of that inline image, it is stuck in the users inbox with no way to send it on to another user. I went back and verified that Marriott Hotels have been doing this for years. What is new is the customer gets a network service error when they click forward - not even sent yet. Does anyone know anything about this behavior? We went from P38 to P40 so don't know when this behavior first appeared. I won't bother asking why Marriott Hotels sends out emails like this. ;-)

Jim
I have reproduced your issue. The inline image is formatted wrongly and that throws an exception when Zimbra tries to create the new message because the Mime parser chokes on it. It tries to decode the uuencoded part, which fails. So the sending party has a bug...

In my opinion it is ALSO a bug that Zimbra throws an system failure: IOExecption, I think that this com.sun.mail.util.DecodingException should be dealt with in code, probably by removing the image from the message whenever a wrongly formatted image is being forwarded. Or at least show a more user friendly error.

This issue appears the same on all versions of Zimbra.

Since whatever data present in the uuencoded image needs to go through the appropriate decoder, I do not see how one could exploit this, or in other words I do not believe it is a security issue. It either decodes and it is an image, or it does not decode, and it throws an error.

I will file a bug for you.
--
Barry de Graaff
Email: barry.degraaff [at] synacor [dot] com
Admin of Zimbra-Community Github: https://github.com/orgs/Zimbra-Community/ and the
Zimlet Gallery https://gallery.zetalliance.org/extend/
Post Reply