8.8.15 Patch 40 GA Release

Post by **barrydegraaff** » Thu Jun 08, 2023 1:35 pm

I have filed ZBUG-3457 Improve error handling on bad uuencoded inline images

gabrieles · Post by **gabrieles** » Thu Jun 08, 2023 3:07 pm

Hi, we patched four mailstores U18 ZCS8.8.15 Network from P33 to P40 with no apparent issues.
Then we patched another mailstore U18 ZCS8.8.15 Network from P30 to P40. The only issue was the deletion of a zimlet-related jar that prevented zmmailboxd to start, but replacing it in place solved the problem, so actually there are no apparent issues neither for this server.

Klug · Post by **Klug** » Sun Jun 11, 2023 11:36 am

I tried to upgrade a server.
It's an old server with a couple domains for friends and testing (was 8.0, maybe even older, on CentOS6, upgraded to last possible 8.8.15 on CentOS6 as years passed then moved and upgraded to 8.8.15P39 on Ubuntu 20.04).

I encountered the same issue than over people did.
I tracked it down to nginx-lookup going 503:

Code: Select all

$ zmcontrol -v
Release 8.8.15.GA.4179.UBUNTU20.64 UBUNTU20_64 FOSS edition, Patch 8.8.15_P40.
$ curl -v -k https://127.0.0.1:7072/service/extension/nginx-lookup
*   Trying 127.0.0.1...
* Connected to 127.0.0.1 (127.0.0.1) port 7072 (#0)
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /opt/zimbra/common/share/curl/ca-bundle.crt
  CApath: none
* (304) (OUT), TLS handshake, Client hello (1):
* (304) (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: CN=*.domain.tld
*  start date: 25 00:00:00 2023 GMT
*  expire date: 23:59:59 2024 GMT
*  issuer: C=FR; ST=Paris; L=Paris; O=Gandi; CN=Gandi Standard SSL CA 2
*  SSL certificate verify ok.
> GET /service/extension/nginx-lookup HTTP/1.1
> Host: 127.0.0.1:7072
> User-Agent: curl/7.49.1
> Accept: */*
>
< HTTP/1.1 503 Service Unavailable
< Cache-Control: must-revalidate,no-cache,no-store
< Content-Type: text/html;charset=iso-8859-1
< Content-Length: 417
<
<html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 503 Service Unavailable</title>
</head>
<body><h2>HTTP ERROR 503 Service Unavailable</h2>
<table>
<tr><th>URI:</th><td>/service/extension/nginx-lookup</td></tr>
<tr><th>STATUS:</th><td>503</td></tr>
<tr><th>MESSAGE:</th><td>Service Unavailable</td></tr>
<tr><th>SERVLET:</th><td>-</td></tr>
</table>

</body>
</html>
* Connection #0 to host 127.0.0.1 left intact

I found out several setup issues on my side, linked to previous upgrades ; proxy was badly setup.
I fixed all these ("ports" and "proxyports" were swapped, some ciphers not disabled, etc).
The 503 issue was still there.

So:

Code: Select all

apt install zimbra-patch=8.8.15.1680534965.p39-1.u20 zimbra-mbox-war=8.8.15.1655458176-1.u20

The downgrade fixed the issue for me too (like others in this thread).

halfgaar · Post by **halfgaar** » Sun Jun 11, 2023 11:04 pm

Something similar has happened before: Blank admin page and nginx-lookup 503 error after updating from 8.8.15 p2 to p30

They removed 'adpassword' from '/opt/zimbra/lib/ext'. For people who are affected by this, can you list the contents of that dir? I haven't tried the patch yet, but mine is:

Code: Select all

# lh /opt/zimbra/lib/ext
total 60K
drwxrwxr-x 2 root root 4,0K apr 14 06:36 clamscanner
drwxrwxr-x 2 root root 4,0K apr 14 06:36 com_zimbra_bulkprovision
drwxrwxr-x 2 root root 4,0K apr 14 06:36 com_zimbra_cert_manager
drwxrwxr-x 2 root root 4,0K apr 14 06:37 com_zimbra_clientuploader
drwxrwxr-x 2 root root 4,0K apr 14 06:36 com_zimbra_ssdb_ephemeral_store
drwxrwxr-x 2 root root 4,0K mrt 29  2019 mitel
drwxrwxr-x 2 root root 4,0K apr 14 06:36 nginx-lookup
drwxrwxr-x 2 root root 4,0K mrt  2  2020 openidconsumer
drwxrwxr-x 2 root root 4,0K mrt 29  2019 twofactorauth
drwxrwxr-x 2 root root 4,0K mrt  2  2020 zimbraadminversioncheck
drwxrwxr-x 2 root root 4,0K mrt 29  2019 zimbra-freebusy
drwxrwxr-x 2 root root 4,0K apr 14 06:36 zimbraldaputils
drwxrwxr-x 2 root root 4,0K mrt 29  2019 zimbra-license
drwxrwxr-x 2 root root 4,0K apr 14 06:36 zm-gql
drwxrwxr-x 2 root root 4,0K apr 14 06:36 zm-oauth-social

Klug · Post by **Klug** » Mon Jun 12, 2023 7:41 am

I though of this (there once was the same issue with owncloud zimlet) and triple checked yesterday, compared to a "standard" OSE install but found notfhing that should not be in /opt/zimbra/lib/ext.

However I do remember using (years ago) Zextras Migration Tools to do some exports of the server.
There might be some left files somewhere else (not in /opt/zimbra/lib/ext).
I did found some config files (/opt/zimbra/conf/zextras/) but no binary.

Here's my current folder (patched but downgraded):

Code: Select all

$ ll /opt/zimbra/lib/ext
total 64
drwxrwxr-x 16 root root 4096 Jun 11 11:44 ./
drwxrwxr-x  6 root root 4096 May 18 09:53 ../
drwxrwxr-x  2 root root 4096 Jun 11 13:25 clamscanner/
drwxrwxr-x  2 root root 4096 Jun 11 13:25 com_zimbra_bulkprovision/
drwxrwxr-x  2 root root 4096 Jun 11 13:25 com_zimbra_cert_manager/
drwxrwxr-x  2 root root 4096 Jun 11 13:25 com_zimbra_ssdb_ephemeral_store/
drwxrwxr-x  2 root root 4096 Nov 18  2021 mitel/
drwxrwxr-x  2 root root 4096 Jun 11 13:25 nginx-lookup/
drwxrwxr-x  2 root root 4096 May 18 09:52 openidconsumer/
drwxrwxr-x  2 root root 4096 Nov 18  2021 twofactorauth/
drwxrwxr-x  2 root root 4096 May 18 09:52 zimbraadminversioncheck/
drwxrwxr-x  2 root root 4096 Nov 18  2021 zimbra-freebusy/
drwxrwxr-x  2 root root 4096 Jun 11 13:25 zimbraldaputils/
drwxrwxr-x  2 root root 4096 Nov 18  2021 zimbra-license/
drwxrwxr-x  2 root root 4096 Jun 11 13:25 zm-gql/
drwxrwxr-x  2 root root 4096 Jun 11 13:25 zm-oauth-social/

ghen · Post by **ghen** » Mon Jun 12, 2023 11:55 am

The zimbra-mbox-war package that you've been rolling back contains the following changes in Patch 40:

Remove unused JSP file which may bypass the Preauth verification (CVE-2023-29382)
The Apache CXF package has been upgraded to version 3.5.5 to fix SSRF vulnerability (CVE-2022-46364)
The Spring Core package has been upgraded to version 6.0.8 to fix multiple vulnerabilities (CVE-2022-22970)

I don't believe the first item to be the problem, since we already applied this (removed sfdc_preauth.jsp) on our P37 production systems.
So maybe one of the other two...

L. Mark Stone · Post by **L. Mark Stone** » Mon Jun 12, 2023 11:57 am

In reviewing this thread (we have an on-premises customer who had this same issue), is my observation correct that only RedHat-flavoured distros succumbed to this issue; Ubuntu distros did not?

Please post if you are running an Ubuntu distro that had this error.

Thanks,
Mark

halfgaar · Post by **halfgaar** » Mon Jun 12, 2023 1:45 pm

There was a report of an Ubuntu 20 installation that has the problem:

viewtopic.php?p=309643#p309643

L. Mark Stone · Post by **L. Mark Stone** » Mon Jun 12, 2023 1:53 pm

halfgaar wrote: ↑Mon Jun 12, 2023 1:45 pm There was a report of an Ubuntu 20 installation that has the problem:

viewtopic.php?p=309643#p309643

I missed that; thanks!

halfgaar · Post by **halfgaar** » Mon Jun 12, 2023 11:19 pm

What is the next step? I haven't upgraded yet, because of this. Luckily, my web interface is behind a proxy for me, so I can afford to wait.

If a bug is to be reported, the maintainers are probably going to want to have more information than just the 503. We need the actual error message from the service/url that is not reachable.

Klug didn't restore a full machine image, so perhaps there is still some unseen stuff in the logs?

The process on port 7072 is simply 'java'. For me, that's currently pid 27153 ('netstat -l -n -t -p|grep 7072') and these are the open files with the word 'log' in it:

Code: Select all

# /proc/27153/fd
# lh|grep -Fi log
l-wx------ 1 root root 64 apr 14 06:43 1 -> /opt/zimbra/log/zmmailboxd.out
lr-x------ 1 root root 64 apr 14 06:43 102 -> /opt/zimbra/jetty_base/common/lib/log4j-api-2.17.1.jar
lr-x------ 1 root root 64 apr 14 06:43 103 -> /opt/zimbra/jetty_base/common/lib/log4j-core-2.17.1.jar
lr-x------ 1 root root 64 apr 14 06:43 171 -> /opt/zimbra/jetty_base/webapps/service/WEB-INF/lib/log4j-slf4j-impl-2.17.1.jar
l-wx------ 1 root root 64 apr 14 06:43 2 -> /opt/zimbra/log/zmmailboxd.out
l-wx------ 1 root root 64 apr 14 06:43 203 -> /opt/zimbra/log/ews.log
l-wx------ 1 root root 64 apr 14 06:43 204 -> /opt/zimbra/log/audit.log
l-wx------ 1 root root 64 apr 14 06:43 205 -> /opt/zimbra/log/mailbox.log
l-wx------ 1 root root 64 apr 14 06:43 206 -> /opt/zimbra/log/synctrace.log
l-wx------ 1 root root 64 apr 14 06:43 207 -> /opt/zimbra/log/searchstat.log
l-wx------ 1 root root 64 apr 14 06:43 209 -> /opt/zimbra/log/wbxml.log
l-wx------ 1 root root 64 apr 14 06:43 210 -> /opt/zimbra/log/syncstate.log
l-wx------ 1 root root 64 apr 14 06:43 211 -> /opt/zimbra/log/sync.log
l-wx------ 1 root root 64 apr 14 06:43 212 -> /opt/zimbra/log/activity.log
lrwx------ 1 root root 64 apr 14 06:43 269 -> /opt/zimbra/redolog/redo.log
l-wx------ 1 root root 64 mei  7 01:59 3 -> /opt/zimbra/log/access_log.2023-06-12
l-wx------ 1 root root 64 jun  5 01:46 388 -> /opt/zimbra/log/gc.log
lr-x------ 1 root root 64 apr 14 06:43 53 -> /opt/zimbra/jetty_base/common/lib/apache-log4j-extras-1.0.jar
lr-x------ 1 root root 64 apr 14 06:43 66 -> /opt/zimbra/jetty_base/common/lib/commons-logging-1.1.1.jar
l-wx------ 1 root root 64 apr 14 06:43 8 -> /opt/zimbra/log/trace_log.2023_06_13

Is there anything useful when you cd to /opt/zimbra/log and do 'grep -li nginx-lookup'. If I use 'silver searcher' and type 'ag -l nginx-lookup', I get (many versions of) these files:

Code: Select all

access_log
mailbox.log
trace_log

The trace log wasn't mentioned before in this thread.

Zimbra Forums

8.8.15 Patch 40 GA Release

Re: 8.8.15 Patch 40 GA Release

Re: 8.8.15 Patch 40 GA Release

Re: 8.8.15 Patch 40 GA Release

Re: 8.8.15 Patch 40 GA Release

Re: 8.8.15 Patch 40 GA Release

Re: 8.8.15 Patch 40 GA Release

Re: 8.8.15 Patch 40 GA Release

Re: 8.8.15 Patch 40 GA Release

Re: 8.8.15 Patch 40 GA Release

Re: 8.8.15 Patch 40 GA Release