Empty response from nginx after default install

Zimbra Collaboration 8.8 Beta
User avatar
msquadrat
Advanced member
Advanced member
Posts: 183
Joined: Mon Oct 14, 2013 10:09 am

Empty response from nginx after default install

Post by msquadrat »

I just finished a quick default install of the Beta 4 (zcs-8.8.2_GA_1822.UBUNTU16_64.20170809063438). With default I mean that I spun up an AWS instance, fixed /etc/hosts, ran install.sh and accepted all defaults (except for the domain and admin password of course).

I was greeted with a server which doesn't answer on port 80 (looks like https is the default, redirect would be a better choice I think) and a server which doesn't reply on port 443 either:

Code: Select all

$ curl -vkI https://ec2-52-57-41-128.eu-central-1.compute.amazonaws.com/
*   Trying 52.57.41.128...
* Connected to ec2-52-57-41-128.eu-central-1.compute.amazonaws.com (52.57.41.128) port 443 (#0)
* found 173 certificates in /etc/ssl/certs/ca-certificates.crt
* found 697 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* SSL connection using TLS1.2 / ECDHE_RSA_AES_128_GCM_SHA256
*        server certificate verification SKIPPED
*        server certificate status verification SKIPPED
*        common name: ip-172-31-4-202.eu-central-1.compute.internal (does not match 'ec2-52-57-41-128.eu-central-1.compute.amazonaws.com')
*        server certificate expiration date OK
*        server certificate activation date OK
*        certificate public key: RSA
*        certificate version: #3
*        subject: OU=Zimbra Collaboration Server,CN=ip-172-31-4-202.eu-central-1.compute.internal
*        start date: Wed, 30 Aug 2017 16:20:00 GMT
*        expire date: Mon, 29 Aug 2022 16:20:00 GMT
*        issuer: O=CA,OU=Zimbra Collaboration Server,CN=ip-172-31-4-202.eu-central-1.compute.internal
*        compression: NULL
* ALPN, server accepted to use http/1.1
> HEAD / HTTP/1.1
> Host: ec2-52-57-41-128.eu-central-1.compute.amazonaws.com
> User-Agent: curl/7.47.0
> Accept: */*
> 
* Empty reply from server
* Connection #0 to host ec2-52-57-41-128.eu-central-1.compute.amazonaws.com left intact
curl: (52) Empty reply from server
All I have in nginx.log is some odd SSL error.

Code: Select all

2017/08/30 16:23:55 [notice] 1830#0: Using 32KiB of shared memory for upstream_fair
2017/08/30 16:23:55 [notice] 1830#0: using the "epoll" event method
2017/08/30 16:23:55 [notice] 1830#0: nginx/1.7.1
2017/08/30 16:23:55 [notice] 1830#0: OS: Linux 4.4.0-1022-aws
2017/08/30 16:23:55 [notice] 1830#0: getrlimit(RLIMIT_NOFILE): 524288:524288
2017/08/30 16:23:55 [notice] 1831#0: start worker processes
2017/08/30 16:23:55 [notice] 1831#0: start worker process 1832
2017/08/30 16:23:55 [notice] 1831#0: start worker process 1833
2017/08/30 16:23:55 [notice] 1831#0: start worker process 1834
2017/08/30 16:23:55 [notice] 1831#0: start worker process 1835
2017/08/30 16:23:55 [info] 1835#0: memcache: 1/1 connections initialized
2017/08/30 16:23:55 [info] 1834#0: memcache: 1/1 connections initialized
2017/08/30 16:23:55 [info] 1833#0: memcache: 1/1 connections initialized
2017/08/30 16:23:55 [info] 1832#0: memcache: 1/1 connections initialized
2017/08/30 16:27:02 [info] 1835#0: *5 SSL_read() failed (SSL: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca:SSL alert number 48) while waiting for request, client: 213.209.99.168, server: 0.0.0.0:443
2017/08/30 16:28:21 [info] 1832#0: *46 client closed connection while waiting for request, client: 213.209.99.168, server: 0.0.0.0:443
2017/08/30 16:28:28 [info] 1832#0: *48 client closed connection while waiting for request, client: 213.209.99.168, server: 0.0.0.0:443
2017/08/30 16:28:33 [info] 1832#0: *50 client closed connection while waiting for request, client: 213.209.99.168, server: 0.0.0.0:443
2017/08/30 16:29:03 [info] 1832#0: *52 client closed connection while waiting for request, client: 213.209.99.168, server: 0.0.0.0:443
2017/08/30 16:30:03 [info] 1832#0: *54 client closed connection while waiting for request, client: 213.209.99.168, server: 0.0.0.0:443
This sounds a bit like the issue jlan421 posted in that other thread.

I'm about to leave, will have a look tomorrow.
User avatar
msquadrat
Advanced member
Advanced member
Posts: 183
Joined: Mon Oct 14, 2013 10:09 am

Re: Empty response from nginx after default install

Post by msquadrat »

Odd, after setting zimbraReverseProxyLogLevel to debug all I see is that nginx fails with the internal status code 444 which just means that it closes the connection without sending any reply (well, duh):

Code: Select all

2017/08/31 10:49:36 [debug] 9962#0: *5 http process request line
2017/08/31 10:49:36 [debug] 9962#0: *5 http request line: "HEAD / HTTP/1.1"
2017/08/31 10:49:36 [debug] 9962#0: *5 http uri: "/"
2017/08/31 10:49:36 [debug] 9962#0: *5 http args: ""
2017/08/31 10:49:36 [debug] 9962#0: *5 http exten: ""
2017/08/31 10:49:36 [debug] 9962#0: *5 http process request header line
2017/08/31 10:49:36 [debug] 9962#0: *5 http header: "Host: ec2-52-57-41-128.eu-central-1.compute.amazonaws.com"
2017/08/31 10:49:36 [debug] 9962#0: *5 http header: "User-Agent: curl/7.47.0"
2017/08/31 10:49:36 [debug] 9962#0: *5 http header: "Accept: */*"
2017/08/31 10:49:36 [debug] 9962#0: *5 http header done
2017/08/31 10:49:36 [debug] 9962#0: *5 event timer del: 16: 1504176636798
2017/08/31 10:49:36 [debug] 9962#0: *5 rewrite phase: 0
2017/08/31 10:49:36 [debug] 9962#0: *5 posix_memalign: 00000000014CCFB0:4096 @16
2017/08/31 10:49:36 [debug] 9962#0: *5 http finalize request: 444, "/?" a:1, c:1
2017/08/31 10:49:36 [debug] 9962#0: *5 http terminate request count:1
2017/08/31 10:49:36 [debug] 9962#0: *5 http terminate cleanup count:1 blk:0
2017/08/31 10:49:36 [debug] 9962#0: *5 http posted request: "/?"
2017/08/31 10:49:36 [debug] 9962#0: *5 http terminate handler count:1
2017/08/31 10:49:36 [debug] 9962#0: *5 http request count:1 blk:0
2017/08/31 10:49:36 [debug] 9962#0: *5 http close request
User avatar
msquadrat
Advanced member
Advanced member
Posts: 183
Joined: Mon Oct 14, 2013 10:09 am

Re: Empty response from nginx after default install

Post by msquadrat »

Ok, got it: Since the external hostname is different to the internal one I have to set zimbraVirtualHostname on the domain. After restarting the proxy, everything works ok.

This is a regression from at least 8.6 which did work when the zmhostname was different to the Host header and no virtual hosting was used. I guess this issue was introduced with the SNI support in 8.7. It is extra confusing that nginx just closes the connection and doesn't return a HTTP error instead (there's probably no default_server created by the templates).
User avatar
msquadrat
Advanced member
Advanced member
Posts: 183
Joined: Mon Oct 14, 2013 10:09 am

Re: Empty response from nginx after default install

Post by msquadrat »

I filed bug [bug]108299[/bug].
User avatar
ppearl
Advanced member
Advanced member
Posts: 114
Joined: Thu May 15, 2014 7:36 am

Re: Empty response from nginx after default install

Post by ppearl »

The behavior noted here seems to be due to the changes introduced to fix [bug]107963[/bug]
User avatar
msquadrat
Advanced member
Advanced member
Posts: 183
Joined: Mon Oct 14, 2013 10:09 am

Re: Empty response from nginx after default install

Post by msquadrat »

ppearl wrote:The behavior noted here seems to be due to the changes introduced to fix [bug]107963[/bug]
Well, duh, I should have grepped for 'return 444'.

The fix/workaround implemented in that bug breaks the out of the box behaviour of Zimbra; I couldn't make my test instance work after almost two days of tinkering around.

If I understand the public part of [bug]107963[/bug] correctly is the actual issue in the backend, ie. mailboxd, and the nginx hack is just a workaround?

I don't really see how this could be fixed without making zimbraVirtualHostname mandatory and set by the installer.
User avatar
msquadrat
Advanced member
Advanced member
Posts: 183
Joined: Mon Oct 14, 2013 10:09 am

Re: Empty response from nginx after default install

Post by msquadrat »

Another thing: 444 (ie. TCP close) is a really bad behaving and hard to debug behaviour. I replaced the
return 444
with a
return 421
(the status code which fits best I think) and while this generates an empty page because nginx doesn't have an error page for that, at least I get something to work with.

What do you think? Should I open a separate bug or pull request for this small change?
User avatar
msquadrat
Advanced member
Advanced member
Posts: 183
Joined: Mon Oct 14, 2013 10:09 am

Re: Empty response from nginx after default install

Post by msquadrat »

msquadrat wrote:Another thing: 444 (ie. TCP close) is a really bad behaving and hard to debug behaviour. I replaced the
return 444
with a
return 421
(the status code which fits best I think) and while this generates an empty page because nginx doesn't have an error page for that, at least I get something to work with.
Correction: Re-reading RFC 7230 section 5.4 this should probably just be a 400:
A server MUST respond with a 400 (Bad Request) status code to any
HTTP/1.1 request message that lacks a Host header field and to any
request message that contains more than one Host header field or a
Host header field with an invalid field-value.
What do you think? Should I open a separate bug or pull request for this small change?
User avatar
ppearl
Advanced member
Advanced member
Posts: 114
Joined: Thu May 15, 2014 7:36 am

Re: Empty response from nginx after default install

Post by ppearl »

FYI, Malte submitted this pull request https://github.com/Zimbra/zm-nginx-conf/pull/5.
moren
Posts: 27
Joined: Wed Jul 23, 2014 8:39 am
ZCS/ZD Version: 8.7.11_P10

Re: Empty response from nginx after default install

Post by moren »

Same problem here:
viewtopic.php?f=15&t=62695
Locked