Revert upgrade / roll-back to old ZCS server

Ask questions about your setup or get help installing ZCS server (ZD section below).
Labsy
Outstanding Member
Outstanding Member
Posts: 411
Joined: Sat Sep 13, 2014 12:52 am

Re: Revert upgrade / roll-back to old ZCS server

Post by Labsy »

JDunphy wrote:That looks really good. We will get this fixed. I have a lot of trust on your install at this point.
I really like this :)
I am a bit tired today, so I'll jump on your suggestion tomorrow and get back with results. Thank you, JDunphy, for really extensive help!
User avatar
JDunphy
Outstanding Member
Outstanding Member
Posts: 889
Joined: Fri Sep 12, 2014 11:18 pm
Location: Victoria, BC
ZCS/ZD Version: 9.0.0_P39 NETWORK Edition

Re: Revert upgrade / roll-back to old ZCS server

Post by JDunphy »

Labsy wrote:Even after re-upgrade and re-patch same "weird" problems, for example:

EXTERNAL ACCOUNTS
Users A, B and C have configured in Webmail access to additional external account INFO via imap.
IT works flawlessly for user A and B, but in C user's Webmail he get's error:

Account "INFO" Failed
system failure: Folder sync failed, system failure: Synchronization of folder '/INFO/INBOX' failed, system failure: Server returned no response for UID FETCH 1659 BODY.PEEK[]


nginx.log

Code: Select all

2018/03/02 02:51:27 [info] 3021#0: *449 upstream sent invalid response: "NO LOGIN failed" while reading response from upstream, client: 10.10.11.50:58392, server: 0.0.0.0:993, login: "info@domain.com", upstream: 10.10.11.50:7143 (10.10.11.50:58392->10.10.11.50:993) <=> (10.10.11.50:52118->10.10.11.50:7143)
mailbox.log

Code: Select all

2018-03-02 02:51:27,215 INFO  [ImapServer-1] [ip=10.10.11.50;oip=10.10.11.50;via=10.10.11.50(nginx/1.7.1);ua=Zimbra/8.8.6_GA_1906;] imap - authentication failed for [info@domain.com] (invalid password)
com.zimbra.common.service.ServiceException: system failure: Unable to connect to IMAP server: LdapDataSource{id=fe917564-07da-4729-aca3-a3cd1e940809, type=imap, enabled=true, name=INFO, host=myzimbra.myserver.com, port=993, connectionType=ssl, username=info@domain.com, folderId=58579, smtpEnabled=false, smtpHost=null, smtpPort=-1}
But I copy-pasted password myself all the same in user's A, B and C Webmail. It's not invalid, no way, as it works for user A and B.
But if user C looks inside INFO folder, all INFO mail is there.
And even better - it pops up above error only once per session, upon logon. Afterwards it works fine.
What the hack? Where the hack?
Hmmm. There is no lack of challenge on this one. :-)

Would this external account be on this same zimbra server as user A,B, and C? or is it another external imap server? Also, failing to fetch a message seems to indicate an imap client cache issue. I use to see it all the time with outlook users and if we could find the message and deleting it or moving it out of the way was our standard go to. In some cases, you could have them go to their source account, move the folder or files then have them sync. Finally, put the folder or files back and imap/pop starting syncing again.

From above, that login fail on port 7143 ... That is imap but port 993 is imaps in the corresponding entry. Could user C have imap as the external access method and users A and B have imaps? Have you attempted to disable imap and pop3 to force users to their secure counterparts.

Sometimes deleting the external profile in user C account and recreating it can fix these sort of problems but I am still looking for root cause and repair. I still need to check the bug database for clues.

I am also wondering if data corruption or the update has created an extra internal entries for this external account internally for user C or if you have any database errors? check: /opt/zimbra/log/mysql_error.log
User avatar
JDunphy
Outstanding Member
Outstanding Member
Posts: 889
Joined: Fri Sep 12, 2014 11:18 pm
Location: Victoria, BC
ZCS/ZD Version: 9.0.0_P39 NETWORK Edition

Re: Revert upgrade / roll-back to old ZCS server

Post by JDunphy »

Labsy wrote:One weird detail about blacklists and numerous account lockouts, which bursted out after upgrade.
I said it might be coincidence, but even I started receiving spam to some strictly administrative mail accounts.
For example, I have admin mail aliases for different logins, like admin.cpanel1@subdomain.domain.com and this is in use for more than 10 years only for login to 1 control panel. No other usage on this email and for 10 years I received only mails from this control panel.
But noe, after upgrade, I started receiving spam on numerous such emails, which are not published anywhere.
Very weird, not have any explanation and it's too weird to be coincidence.
I think it is unrelated with the upgrade.

It could be an indication of a bigger problem either at the source of that email account or various users that have this information. I tend to put FW's on internet connected hosts with services because I want choke points and early warning in my networks. If you do a netstat -na |grep LISTEN ... All those wildcard addresses to various services give me pause. A programmer could bind just the network address but they didn't with much of the zimbra services so you have additional paths in from ip aliases and other interfaces on the host. I can allow just the incoming services on the ip address I expect and deny an entire class of problems. Locking down port 7071 to only trusted address space fixes a lot of problems also. There was an exploit years ago, where an attacker could create an account. https://www.exploit-db.com/exploits/30085/ Version 8.0.9 was not immune.
I feel that we are making progress with zimbra on the security side... or I want to feel that way. LOL
Labsy
Outstanding Member
Outstanding Member
Posts: 411
Joined: Sat Sep 13, 2014 12:52 am

Re: Revert upgrade / roll-back to old ZCS server

Post by Labsy »

Weird about those mailbox.log errors:

Code: Select all

2018-03-02 02:51:27,215 INFO  [ImapServer-1] [ip=10.10.11.50;oip=10.10.11.50;via=10.10.11.50(nginx/1.7.1);ua=Zimbra/8.8.6_GA_1906;] imap - authentication failed for [info@domain.com] (invalid password)
com.zimbra.common.service.ServiceException: system failure: Unable to connect to IMAP server: LdapDataSource{id=fe917564-07da-4729-aca3-a3cd1e940809, type=imap, enabled=true, name=INFO, host=myzimbra.myserver.com, port=993, connectionType=ssl, username=info@domain.com, folderId=58579, smtpEnabled=false, smtpHost=null, smtpPort=-1}
is that today I checked again all 3 Webmialboxes, and today all receive the External account via IMAP fine. No problems from user's perspective.
And yes, they are all 3 on the same server, the external account is also on same server, and they all 3 use IMAP 993 port to retreive external account.
And I checked all company accounts, and those 3 are the only ones, who have this particular external account added into Webmail.

....but mailbox.log simply keeps filing up with above mentioned errors.
Labsy
Outstanding Member
Outstanding Member
Posts: 411
Joined: Sat Sep 13, 2014 12:52 am

Re: Revert upgrade / roll-back to old ZCS server

Post by Labsy »

Just to double check - is it Java on the other side of IMAP pipe?
Nginx proxying to Java?

Code: Select all

 netstat -nap |grep 993
tcp        0      0 0.0.0.0:7993            0.0.0.0:*               LISTEN      20439/java
tcp        0      0 0.0.0.0:993             0.0.0.0:*               LISTEN      3018/nginx.conf
And maybe proxy-related:
Should I try "use SSL for upstream connections"? It's about whether to use SSL from nginx to internal Java.
Now it is DISABLED.
Labsy
Outstanding Member
Outstanding Member
Posts: 411
Joined: Sat Sep 13, 2014 12:52 am

Re: Revert upgrade / roll-back to old ZCS server

Post by Labsy »

Hmmm...it looks like ciphers are reduced. Is this normal?
Might it be Androids having problems because of that?
And SSL_write() failed (32: Broken pipe) errors filling nginx.log due to some cipher restrictions?

Code: Select all

zmprov gacf | egrep -i 'mailsslp|proxy|ciphers' | egrep -i 'ssl|cipher|enabled:|port:|mode'
zimbraAdminProxyPort: 9071
zimbraImapProxyBindPort: 143
zimbraImapSSLProxyBindPort: 993
zimbraMailProxyPort: 0
zimbraMailSSLPort: 0
zimbraMailSSLProxyClientCertPort: 3443
zimbraMailSSLProxyPort: 443
zimbraMtaLmtpTlsCiphers: export
zimbraMtaLmtpTlsMandatoryCiphers: medium
zimbraMtaSmtpTlsCiphers: export
zimbraMtaSmtpTlsMandatoryCiphers: medium
zimbraMtaSmtpdTlsCiphers: export
zimbraMtaSmtpdTlsMandatoryCiphers: medium
zimbraPop3ProxyBindPort: 110
zimbraPop3SSLProxyBindPort: 995
zimbraReverseProxyAdminEnabled: FALSE
zimbraReverseProxyClientCertMode: off
zimbraReverseProxyDnsLookupInServerEnabled: TRUE
zimbraReverseProxyHttpEnabled: TRUE
zimbraReverseProxyHttpSSLPortAttribute: zimbraMailSSLPort
zimbraReverseProxyImapSSLPortAttribute: zimbraImapSSLBindPort
zimbraReverseProxyImapSaslGssapiEnabled: FALSE
zimbraReverseProxyImapSaslPlainEnabled: TRUE
zimbraReverseProxyImapStartTlsMode: only
zimbraReverseProxyMailEnabled: TRUE
zimbraReverseProxyMailImapEnabled: TRUE
zimbraReverseProxyMailImapsEnabled: TRUE
zimbraReverseProxyMailMode: https
zimbraReverseProxyMailPop3Enabled: TRUE
zimbraReverseProxyMailPop3sEnabled: TRUE
zimbraReverseProxyPop3SSLPortAttribute: zimbraPop3SSLBindPort
zimbraReverseProxyPop3SaslGssapiEnabled: FALSE
zimbraReverseProxyPop3SaslPlainEnabled: TRUE
zimbraReverseProxyPop3StartTlsMode: only
zimbraReverseProxySNIEnabled: FALSE
zimbraReverseProxySSLCiphers: EECDH:EDH:SHA256:SHA384:!RC4:HIGH:!aNULL:!MD5:!kEDH:!AD:!SSLv2:!NULL:!3DES
zimbraReverseProxySSLECDHCurve: prime256v1
zimbraReverseProxySSLProtocols: TLSv1
zimbraReverseProxySSLProtocols: TLSv1.1
zimbraReverseProxySSLProtocols: TLSv1.2
zimbraReverseProxySSLSessionCacheSize: 10m
zimbraReverseProxySSLSessionTimeout: 10m
zimbraReverseProxySSLToUpstreamEnabled: FALSE
zimbraReverseProxyStrictServerNameEnabled: TRUE
zimbraReverseProxyXmppBoshEnabled: FALSE
zimbraReverseProxyXmppBoshSSL: FALSE
zimbraReverseProxyZmlookupCachingEnabled: TRUE
zimbraSSLExcludeCipherSuites: .*_RC4_.*
and

Code: Select all

zmprov gs `zmhostname` zimbraMtaSmtpTlsMandatoryCiphers
zimbraMtaSmtpTlsMandatoryCiphers: medium
User avatar
JDunphy
Outstanding Member
Outstanding Member
Posts: 889
Joined: Fri Sep 12, 2014 11:18 pm
Location: Victoria, BC
ZCS/ZD Version: 9.0.0_P39 NETWORK Edition

Re: Revert upgrade / roll-back to old ZCS server

Post by JDunphy »

Labsy wrote:Just to double check - is it Java on the other side of IMAP pipe?
Nginx proxying to Java?

Code: Select all

 netstat -nap |grep 993
tcp        0      0 0.0.0.0:7993            0.0.0.0:*               LISTEN      20439/java
tcp        0      0 0.0.0.0:993             0.0.0.0:*               LISTEN      3018/nginx.conf
Yes. mailboxd ... You can verify by looking here.

Code: Select all

$ cat /opt/zimbra/log/zmmailboxd_java.pid
Do you know the polling interval for that external account across these accounts. Normally, we just share the folder vs using an external account poll on our servers but this would look like the ip address of the zimbra server connecting I would guess to the front end.

I am really not sure what to think at this point.

Perhaps try looking at the output from this to see if anything looks odd given your server usage.

Code: Select all

zmprov gs `zmhostname`  | egrep -i ('Imap|throttle|filter|white)'
User avatar
JDunphy
Outstanding Member
Outstanding Member
Posts: 889
Joined: Fri Sep 12, 2014 11:18 pm
Location: Victoria, BC
ZCS/ZD Version: 9.0.0_P39 NETWORK Edition

Re: Revert upgrade / roll-back to old ZCS server

Post by JDunphy »

Labsy wrote:Hmmm...it looks like ciphers are reduced. Is this normal?
Might it be Androids having problems because of that?
And SSL_write() failed (32: Broken pipe) errors filling nginx.log due to some cipher restrictions?
Yes that is quite reduced from what I have on my 8.7+ version.
User avatar
JDunphy
Outstanding Member
Outstanding Member
Posts: 889
Joined: Fri Sep 12, 2014 11:18 pm
Location: Victoria, BC
ZCS/ZD Version: 9.0.0_P39 NETWORK Edition

Re: Revert upgrade / roll-back to old ZCS server

Post by JDunphy »

Try pointing those android clients to this https://www.ssllabs.com/ssltest/viewMyClient.html and compare against the server.
Labsy
Outstanding Member
Outstanding Member
Posts: 411
Joined: Sat Sep 13, 2014 12:52 am

Re: Revert upgrade / roll-back to old ZCS server

Post by Labsy »

It seems like there's some restriction on upstream Java. I have a feeling that nginx proxies connection to upstream, and from time to time upstream refuses it, like hitting the max connections limit.
Becasue if I examine logs of ONE Webmail client, who has configured IMAP retreival of ONE other email (all on same server), I see:
- some 3-5 OK logins
- then one FAILED login
- and it goes on and on, more OK logins and less FAILED, but still there are failed ones:

Code: Select all

3-5 of those OK in series:
2018/03/03 01:36:44 [info] 13749#0: *2332 client 10.10.11.50:54874 connected to 0.0.0.0:993
2018/03/03 01:36:44 [info] 13749#0: *2332 client logged in, client: 10.10.11.50:54874, server: 0.0.0.0:993, login: "info@domain.com", upstream: 10.10.11.50:7143 (10.10.11.50:54874->10.10.11.50:993) <=> (10.10.11.50:48600->10.10.11.50:7143)
2018/03/03 01:36:44 [info] 13749#0: *2332 proxied session done, client: 10.10.11.50:54874, server: 0.0.0.0:993, login: "info@domain.com", upstream: 10.10.11.50:7143 (10.10.11.50:54874->10.10.11.50:993) <=> (10.10.11.50:48600->10.10.11.50:7143)
...then followed by 1 FAILED like this:
2018/03/03 01:44:04 [info] 13747#0: *2547 client 10.10.11.50:55304 connected to 0.0.0.0:993
2018/03/03 01:44:04 [info] 13747#0: *2547 upstream sent invalid response: "NO LOGIN failed" while reading response from upstream, client: 10.10.11.50:55304, server: 0.0.0.0:993, login: "info@domain.com", upstream: 10.10.11.50:7143 (10.10.11.50:55304->10.10.11.50:993) <=> (10.10.11.50:49030->10.10.11.50:7143)
I do have trusted netoworks/mynetowrks 127.0.0.1 and 10.10.11.50.
Post Reply