SSL_write failed SSL: 32: Broken pipe while proxying

Discuss your pilot or production implementation with other Zimbra admins or our engineers.
Labsy
Outstanding Member
Outstanding Member
Posts: 411
Joined: Sat Sep 13, 2014 12:52 am

SSL_write failed SSL: 32: Broken pipe while proxying

Post by Labsy »

Hi,

sorry for opening new thread, but this is driving me crazy.
After migrating from Ubuntu 12.04 to Ubuntu 14.04 and upgrading afterwards ZCS 8.0.9 to ZCS 8.8.6, I get strange issues.
Clients work OK for few days, then all of the sudden they report cannot access IMAP or POP3 mail with mail client. Then agin it works for some time, and refueses again.

Might it be some Split-DNS issue, like internally resolving to publicDNS then after cache expired resolving internally? Or vice versa?
Or some Firewall issue? On Ubuntu itself I have Firewall disabled. There's pfSense in front of ZCS, but shows no blocked evens in monitroed time slot.
Ideas welcome!

Code: Select all

-- Connection OK --
nginx.log
2018/02/23 07:46:36 [info] 3013#0: *139135 client <clientIP>:41624 connected to 0.0.0.0:993
2018/02/23 07:46:36 [info] 3013#0: *139135 client logged in, client: <clientIP>:41624, server: 0.0.0.0:993, login: "<client@email>", upstream: 10.10.11.50:7143 (<clientIP>:41624->10.10.11.50:993) <=> (10.10.11.50:52824->10.10.11.50:7143)
mailbox.log
2018-02-23 07:46:36,752 INFO  [ImapServer-70] [ip=10.10.11.50;oip=<clientIP>;via=com.samsung.android.email.provider,10.10.11.50(nginx/1.7.1);ua=Zimbra/8.8.6_GA_1906;] imap - ID elapsed=1
2018-02-23 07:46:36,753 INFO  [ImapServer-70] [name=<client@email>;ip=10.10.11.50;oip=<clientIP>;via=com.samsung.android.email.provider,10.10.11.50(nginx/1.7.1);ua=Zimbra/8.8.6_GA_1906;] imap - user <client@email> authenticated, mechanism=LOGIN
2018-02-23 07:46:36,753 INFO  [ImapServer-70] [name=<client@email>;ip=10.10.11.50;oip=<clientIP>;via=com.samsung.android.email.provider,10.10.11.50(nginx/1.7.1);ua=Zimbra/8.8.6_GA_1906;] imap - LOGIN elapsed=1
2018-02-23 07:46:36,803 INFO  [ImapServer-64] [name=<client@email>;ip=10.10.11.50;oip=<clientIP>;via=com.samsung.android.email.provider,10.10.11.50(nginx/1.7.1);ua=Zimbra/8.8.6_GA_1906;] imap - NAMESPACE elapsed=0
2018-02-23 07:46:36,837 INFO  [ImapServer-69] [name=<client@email>;ip=10.10.11.50;oip=<clientIP>;via=com.samsung.android.email.provider,10.10.11.50(nginx/1.7.1);ua=Zimbra/8.8.6_GA_1906;] imap - ENABLE elapsed=0
2018-02-23 07:46:36,874 INFO  [ImapServer-64] [name=<client@email>;ip=10.10.11.50;oip=<clientIP>;via=com.samsung.android.email.provider,10.10.11.50(nginx/1.7.1);ua=Zimbra/8.8.6_GA_1906;] imap - ENABLE elapsed=0
2018-02-23 07:46:36,925 INFO  [ImapServer-64] [name=<client@email>;ip=10.10.11.50;oip=<clientIP>;via=com.samsung.android.email.provider,10.10.11.50(nginx/1.7.1);ua=Zimbra/8.8.6_GA_1906;] imap - copying message data from existing session: INBOX
2018-02-23 07:46:36,926 INFO  [ImapServer-64] [name=<client@email>;ip=10.10.11.50;oip=<clientIP>;via=com.samsung.android.email.provider,10.10.11.50(nginx/1.7.1);ua=Zimbra/8.8.6_GA_1906;] imap - selected folder INBOX
2018-02-23 07:46:36,929 INFO  [ImapServer-64] [name=<client@email>;ip=10.10.11.50;oip=<clientIP>;via=com.samsung.android.email.provider,10.10.11.50(nginx/1.7.1);ua=Zimbra/8.8.6_GA_1906;] imap - SELECT elapsed=11

-- Connection FAILED --
nginx.log
2018/02/23 08:02:52 [info] 3013#0: *139135 proxied session done, client: <clientIP>:41624, server: 0.0.0.0:993, login: "<client@email>", upstream: 10.10.11.50:7143 (->10.10.11.50:993) <=> (10.10.11.50:52824->10.10.11.50:7143)
2018/02/23 08:02:52 [info] 3013#0: *139135 SSL_write() failed (SSL:) (32: Broken pipe) while proxying, client: <clientIP>:41624, server: 0.0.0.0:993, login: "<client@email>", upstream: 10.10.11.50:7143 (->10.10.11.50:993) <=> (->)
mailbox.log
2018-02-23 08:01:31,762 INFO  [ImapServer-70] [name=<client@email>;mid=696;ip=10.10.11.50;oip=<clientIP>;via=com.samsung.android.email.provider,10.10.11.50(nginx/1.7.1);ua=Zimbra/8.8.6_GA_1906;] imap - NOOP elapsed=0
2018-02-23 08:01:31,796 INFO  [ImapServer-69] [name=<client@email>;mid=696;ip=10.10.11.50;oip=<clientIP>;via=com.samsung.android.email.provider,10.10.11.50(nginx/1.7.1);ua=Zimbra/8.8.6_GA_1906;] imap - copying message data from existing session: INBOX
2018-02-23 08:01:31,797 INFO  [ImapServer-69] [name=<client@email>;mid=696;ip=10.10.11.50;oip=<clientIP>;via=com.samsung.android.email.provider,10.10.11.50(nginx/1.7.1);ua=Zimbra/8.8.6_GA_1906;] imap - selected folder INBOX
2018-02-23 08:01:31,802 INFO  [ImapServer-69] [name=<client@email>;mid=696;ip=10.10.11.50;oip=<clientIP>;via=com.samsung.android.email.provider,10.10.11.50(nginx/1.7.1);ua=Zimbra/8.8.6_GA_1906;] imap - SELECT elapsed=10
Labsy
Outstanding Member
Outstanding Member
Posts: 411
Joined: Sat Sep 13, 2014 12:52 am

Re: SSL_write failed SSL: 32: Broken pipe while proxying

Post by Labsy »

May I ask you guyz with ZCS 8.8.6 to check your nginx.log against same issue? Just to confirm I indeed have a problem.
When I run (as root user):

Code: Select all

 grep "SSL_write() failed" /opt/zimbra/log/nginx.log
I get a lot of those errors for different users and from different client IP addresses:

Code: Select all

2018/02/24 11:37:49 [info] 3013#0: *183475 SSL_write() failed (SSL:) (32: Broken pipe) while proxying, client: <CleintIP>:49932, server: 0.0.0.0:993, login: "<Client@email>", upstream: 10.10.11.50:7143 (->10.10.11.50:993) <=> (->)
When I grep nginx proxy session ID, I can see one additional LOG entry (for the above session).
This particular session is from my Samsung S7 android mail client. It is working fine, but still errors bother me:

Code: Select all

 grep 183475 /opt/zimbra/log/nginx.log
2018/02/24 11:37:49 [info] 3013#0: *183475 proxied session done, client: <CleintIP>:49932, server: 0.0.0.0:993, login: "<Client@email>", upstream: 10.10.11.50:7143 (->10.10.11.50:993) <=> (10.10.11.50:58464->10.10.11.50:7143)
2018/02/24 11:37:49 [info] 3013#0: *183475 SSL_write() failed (SSL:) (32: Broken pipe) while proxying, client: <CleintIP>:49932, server: 0.0.0.0:993, login: "<Client@email>", upstream: 10.10.11.50:7143 (->10.10.11.50:993) <=> (->)
Can somebody confirm or deny existence of such errors on your systems?
User avatar
L. Mark Stone
Ambassador
Ambassador
Posts: 2796
Joined: Wed Oct 09, 2013 11:35 am
Location: Portland, Maine, US
ZCS/ZD Version: 10.0.6 Network Edition
Contact:

Re: SSL_write failed SSL: 32: Broken pipe while proxying

Post by L. Mark Stone »

___________________________________
L. Mark Stone
Mission Critical Email - Zimbra VAR/BSP/Training Partner https://www.missioncriticalemail.com/
AWS Certified Solutions Architect-Associate
Labsy
Outstanding Member
Outstanding Member
Posts: 411
Joined: Sat Sep 13, 2014 12:52 am

Re: SSL_write failed SSL: 32: Broken pipe while proxying

Post by Labsy »

L. Mark Stone wrote:Have you applied The Hotfix?
Yes, I did. ZCS version now reads 8.8.6_GA_1906.NETWORK

BTW...I've examined /opt/zimbra/conf/nginx/... config files for https, imap, pop3 etc, and they all contain references to .key and .crt files, which do not exist.
Might that be somehow related to my issue?

Code: Select all

    ssl_certificate         /opt/zimbra/conf/nginx.crt;
    ssl_certificate_key     /opt/zimbra/conf/nginx.key;
User avatar
L. Mark Stone
Ambassador
Ambassador
Posts: 2796
Joined: Wed Oct 09, 2013 11:35 am
Location: Portland, Maine, US
ZCS/ZD Version: 10.0.6 Network Edition
Contact:

Re: SSL_write failed SSL: 32: Broken pipe while proxying

Post by L. Mark Stone »

Single server?

Let's make sure DNS is OK first (since you brought this up).

Please post the outputs of the following (easiest if run as root on the Zimbra server)

cat /etc/hosts
cat /etc/resolv.conf
cat /etc/hostname
ifconfig

And then for the fqdn of the Zimbra server listed in /etc/hosts, please run:

dig <fqdn of the Zimbra server>

And then for the IP address of the Zimbra server listed in /etc/hosts -- which should be the same as what you get from the dig command above -- please run:

host <ip address from the dig command>

That will get us started.

Mark
___________________________________
L. Mark Stone
Mission Critical Email - Zimbra VAR/BSP/Training Partner https://www.missioncriticalemail.com/
AWS Certified Solutions Architect-Associate
Labsy
Outstanding Member
Outstanding Member
Posts: 411
Joined: Sat Sep 13, 2014 12:52 am

Re: SSL_write failed SSL: 32: Broken pipe while proxying

Post by Labsy »

Hi Mark,

thank you for kicking-in, but I am pretty sure all those hostname/IP/splitDNS is fine. But I wanna be surprised just to find out the solution.
So here is my output. For privacy purposes I changed actual FQDN and Zimbra name with "myzimbra.mydomain.com" in the actual output:

Code: Select all

 cat /etc/hosts
127.0.0.1       localhost
10.10.11.50     myzimbra.mydomain.com   myzimbra
# The following lines are desirable for IPv6 capable hosts
::1     localhost ip6-localhost ip6-loopback
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters


 cat /etc/resolv.conf
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
nameserver 10.10.11.1
nameserver 8.8.8.8
search mydomain.com


 cat /etc/hostname
myzimbra.mydomain.com


 ifconfig
eth0      Link encap:Ethernet  HWaddr 00:0c:29:1a:dd:76
          inet addr:10.10.11.50  Bcast:10.10.11.255  Mask:255.255.255.0
          inet6 addr: fe80::20c:29ff:fe1a:dd76/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:9309773 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8735802 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:4857334994 (4.8 GB)  TX bytes:15738187355 (15.7 GB)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:47189712 errors:0 dropped:0 overruns:0 frame:0
          TX packets:47189712 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1
          RX bytes:42898949423 (42.8 GB)  TX bytes:42898949423 (42.8 GB)


		  
 dig myzimbra.mydomain.com

; <<>> DiG 9.9.5-3ubuntu0.17-Ubuntu <<>> myzimbra.mydomain.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 20509
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;myzimbra.mydomain.com.         IN      A

;; ANSWER SECTION:
myzimbra.mydomain.com.  3600    IN      A       10.10.11.50

;; Query time: 0 msec
;; SERVER: 10.10.11.1#53(10.10.11.1)
;; WHEN: Sat Feb 24 18:26:19 CET 2018
;; MSG SIZE  rcvd: 66


 host 10.10.11.50
50.11.10.10.in-addr.arpa domain name pointer myzimbra.mydomain.com.

User avatar
L. Mark Stone
Ambassador
Ambassador
Posts: 2796
Joined: Wed Oct 09, 2013 11:35 am
Location: Portland, Maine, US
ZCS/ZD Version: 10.0.6 Network Edition
Contact:

Re: SSL_write failed SSL: 32: Broken pipe while proxying

Post by L. Mark Stone »

Well, you've got some configuration issues there that I have seen cause a variety of issues.

First, /etc/hosts' s first line should be:

127.0.0.1 localhost.localdomain localhost

Next, Google's primary DNS server (8.8.8.8) will not resolve your Zimbra server's private IP address. The OS uses all of the nameserver listings in /etc/resolv.conf, so some of the time Zimbra's components when doing a DNS lookup will resolve themselves correctly with their actual private IP address, and sometimes not. Depending on the router at the gateway, Zimbra's components looking to reach another Zimbra component on its public IP address may not succeed.

Third, not all of Zimbra's components do well with both IPv4 and IPv6 enabled. If you've installed Zimbra to use just IPv4, then you should remove IPv6 from the operating system, because components like BIND9 have default settings that resolve both IPv6 and IPv4.

Intermittent connectivity issues between various Zimbra components in my experience almost always are DNS related.

Similarly, nginx usually either works or it doesn't, but never intermittently, unless there is a resource issue or a DNS issue.

To be clear, I'm not saying you don't have an nginx issue, but I am saying you have some DNS configuration issues that, at the very least, could be getting in the way of you accurately diagnosing your nginx issue.

Hope that helps,
Mark
___________________________________
L. Mark Stone
Mission Critical Email - Zimbra VAR/BSP/Training Partner https://www.missioncriticalemail.com/
AWS Certified Solutions Architect-Associate
Labsy
Outstanding Member
Outstanding Member
Posts: 411
Joined: Sat Sep 13, 2014 12:52 am

Re: SSL_write failed SSL: 32: Broken pipe while proxying

Post by Labsy »

L. Mark Stone wrote: First, /etc/hosts' s first line should be:

127.0.0.1 localhost.localdomain localhost
Mark, thank you for all hints. I'll resolve all them.
Just here above - did you mean literally "localhost.localdomain" or should I fill my "myzimbra.mydomain.com"?

Regarding DNS resolving:
My ROUTER, which is 10.10.11.1, has an entry to resolve Zimbra's LAN IP to "myzimbra.mydomain.com" and vice versa. Like SplitDNS, but on router.
Is this OK?

**EDIT**
I've done all the changes you suggested.
Rebooted server and the issue remains.
I also see my account and my computer having SSL_wrinte() failed error, but I can send and receive mail from my client.
Regardless...I'd still like to resolve it.
....or get some feedback from others, if they too see this in nginx.log.
Labsy
Outstanding Member
Outstanding Member
Posts: 411
Joined: Sat Sep 13, 2014 12:52 am

Re: SSL_write failed SSL: 32: Broken pipe while proxying

Post by Labsy »

After some digging, it seems like indeed I have some IMAP-related problems on backend after upgrade/migration.
Here my other problem with IMAP retreival of external accounts, possibly related to the issue:
viewtopic.php?f=15&t=63705&p=281875#p281875
User avatar
JDunphy
Outstanding Member
Outstanding Member
Posts: 889
Joined: Fri Sep 12, 2014 11:18 pm
Location: Victoria, BC
ZCS/ZD Version: 9.0.0_P39 NETWORK Edition

Re: SSL_write failed SSL: 32: Broken pipe while proxying

Post by JDunphy »

Labsy wrote: BTW...I've examined /opt/zimbra/conf/nginx/... config files for https, imap, pop3 etc, and they all contain references to .key and .crt files, which do not exist.
Might that be somehow related to my issue?

Code: Select all

   
    ssl_certificate         /opt/zimbra/conf/nginx.crt;
    ssl_certificate_key     /opt/zimbra/conf/nginx.key;
I don't have a good picture what is going on. So lets try a few things to get a picture and maybe something will jump out for you.

Code: Select all

apt-get install lsof   # if you don't already have lsof
# lsof -i :993
COMMAND   PID   USER   FD   TYPE   DEVICE SIZE/OFF NODE NAME
nginx   21785 zimbra    7u  IPv4 19503411      0t0  TCP *:imaps (LISTEN)
nginx   21787 zimbra    7u  IPv4 19503411      0t0  TCP *:imaps (LISTEN)
nginx   21788 zimbra    7u  IPv4 19503411      0t0  TCP *:imaps (LISTEN)
nginx   21789 zimbra    7u  IPv4 19503411      0t0  TCP *:imaps (LISTEN)
nginx   21790 zimbra    7u  IPv4 19503411      0t0  TCP *:imaps (LISTEN)
...
# lsof -i:7993
COMMAND   PID   USER   FD   TYPE   DEVICE SIZE/OFF NODE NAME
java    21527 zimbra  109u  IPv4 19506672      0t0  TCP *:7993 (LISTEN)
# lsof -i:7143
COMMAND   PID   USER   FD   TYPE   DEVICE SIZE/OFF NODE NAME
java    21527 zimbra  108u  IPv4 19506671      0t0  TCP *:7143 (LISTEN)
I might do a connection directly to 993 to verify tls on 993

Code: Select all

openssl s_client -connect localhost:993 
and you could do something directly to the backend imap server - should be identical

Code: Select all

telnet localhost 7143 
and 
openssl s_client -connect localhost:7993
Note: imap command examples - https://delog.wordpress.com/2011/05/10/ ... g-openssl/

Observe the certs, handshakes, etc. Look for any errors.

I would also look for tcp errors, unusual number of connections, failures, etc. Are we blowing through some limits, etc.

Code: Select all

netstat -s
During a failure scenario... do you see excessive FIN_WAIT2

Code: Select all

netstat -na | egrep '(FIN_WAIT)'
Didn't 8.8 introduce a beta imap service? Are you running that? From the release notes: "(BETA) Decoupled IMAP Service - Improves email reliability and SLAs by optionally deploying IMAP as a separate service."

I would try to verify that you don't have something quick firing connections to nginx (rate limits, etc https://www.nginx.com/blog/tuning-nginx/ ) and therefore your imap server ... ie) error from internal polling to an external client off the webclient on the same machine, etc.

Grasping at straws here. Who doesn't love a mystery! :-)
Post Reply