Page 1 of 3

Revert upgrade / roll-back to old ZCS server

Posted: Thu Mar 01, 2018 10:25 am
by Labsy
Hi,

I have numerous issues after I did migration + upgrade:
- Ubuntu 12.04 --> migrated to new server with Ubuntu 14.04 (moved over same ZCS version)
- then on new server upgraded ZCS 8.0.9 --> to 8.8.6

Multiple issues on new server: Random, sporadic account lockouts, SSL_write pipe broken errors, multiple android devices cannot connect sporadically, and most of all, CONTIUOUS BLACKLISTING issues. Accounts get locked, tens of thousands NDRs on multiple accounts as response to spam outbursts on different clients and different domains. My old server has NEVER been on ANY blacklist for past 6 years for sure! Now in 14 days it's 4th blacklisting today. I simply cannot cope with all those problems anymore.

Please, advise how to ROLL-BACK to old server.
I have the old Ubuntu 12.04 with ZCS 8.0.9 fully functional VM, just shut-down.
What I would need somehow to:
- add 1 new user account to old ZCS
- reflect back to old server all mail and calendar items

Re: Revert upgrade / roll-back to old ZCS server

Posted: Thu Mar 01, 2018 4:47 pm
by scantec
Do you have the hotfix/patch for 8.8.6 released recently?

The blacklist problem is probably a compromised account on the system - just coincidence with the upgrade - or not..

In last resort If you can restore your system to the state of your old 8.0.9 you can probably move to another version of Zimbra more stable like 8.7.11 - of course you'd need to find a way to import data correctly

Re: Revert upgrade / roll-back to old ZCS server

Posted: Thu Mar 01, 2018 5:43 pm
by JDunphy
Yikes. I didn't realize it was that bad. I went back and looked at your topics over the past few years to see if anything popped out.

I am wondering if we need to verify a few things. What do others link of this plan?

1) make sure you don't use any non supported repositories
2) update your ubuntu box to latest patch levels
3) run install.sh (update) again for 8.8.6 in the hopes it brings in any new zimbra libraries and executable, etc. During this, make sure you are not enabling any of the new beta services like imap yet, etc
4) re-patch 8.8.6

It sounds like you might have been on ESXi at one time so take a snapshot if that is the case before proceeding.

One of the obvious changes I think from where you started with 8.09 and ubuntu 12 to where you are now would be the support for TLS 1.0, TLS 1.1 so re-enable those if you have really old clients that need to connect. What did sslabs tell you about your cert and ciphers and which clients could connect? Those should be EOL but if you need those clients to connect than you have to look at all your options and mitigate as best as you can. Other things became more capable such as https://wiki.zimbra.com/wiki/DoSFilter ... It is pretty easy to lock out ip ranges if you have users typing bad passwords or under dictionary attacks, etc. If you have remote offices, this can cause some odd behavior for all the users when this happens. I have had to raise the limits because of guy who could not seem to type his password correctly every monday morning and was locking out the office. I posted a perl script a while back I use to check the count of login failures against ip's, protocols to give me a heads up who to call or ip's to watch closer. We run some external VPN access servers so we can lock down the ip range of mobile services to those trusted VPN ip ranges for 587, 995, 993, etc

I saw you mentioned that some accounts were sending spam in earlier posts so you may have active dictionary attacks ongoing. Lots of people use fail2ban, I use a few iptable rules and an ipset that does this automatically using match-set, recent, hitcount, add-set. I have the ipset expire after 5mins to slow them. I have a few different ipset's with progressively longer timeouts that escalate this timeout depending on connection frequency.

Last thing I can think of is the blacklisting problem... if the ip address has changed, you need to do all the things required for the big services like Yahoo, outlook.com, google, aol, hotmail, etc. Sometimes its registering ip's though special web forms, sometimes it's requesting removal from BL's if you have been given a new ip range... sometimes its running spf, dmarc, etc, etc. I don't have a handy 1 place link to what has to be done. There is nothing in 8.8+ or 8.7 or 8.6, etc that would make this more prone to blacklisting. That generally comes from sending spam from your ip range with the exception of new ip space. While there are 100's of BLs, there are only a few that matter IMHO. This is what I use https://mxtoolbox.com/blacklists.aspx to check.

Re: Revert upgrade / roll-back to old ZCS server

Posted: Fri Mar 02, 2018 12:48 am
by ccelis5215
@JDunphy men.., a hell of knwoledge, an enormous piece of advice.

@labsy sorry for you., Sorry, sometimes.that's it is . .. I've follow some of your posts., you are a seasoned sysadmin and of course zimbra / admin,etc,, just keep going.

ccelis

Re: Revert upgrade / roll-back to old ZCS server

Posted: Fri Mar 02, 2018 12:59 am
by Labsy
Thank you all for ideas and thoughts. LEt me just add some maybe critical points and answer the open questions.

Yes, I am on ESX all the time. The old (2 weeks old) ZCS 8.0.9 on Ubuntu 12.04 is also on ESX, shut down and getting older every day.

Regarding TLS 1.0 and 1.1 on new ZCS 8.8.6 + Patch I think I have both them ENABLED. SSL Labs shows it should be supported by all clients, back to XP w/ SP3 and Android 2.3.7. That covers pretty much all clients I have.

Regarding blacklists and account lockouts...maybe it's just a coincidence, but dealing with 10000 queue length every few days due to some account breach, and each time different client. Well, this never happened to me in past 6 or 7 years using any previous Zimbra server. But still could be coincidence.

Regarding Fail2ban...I did not like the idea having additional firewall on ZCS box beside existing pfSense/OPENSense firewall. So I quit installling fail2ban, because of lack of knowledge to make it work in conjunction with pfSense/OPENSense firewall. I would like my firewall to ban offending IPs for entire server "cluster", not just for Zimbra server.
Beside, when I was last dealing with compromised account yesterday, I had some 100/minute failed login attempts via Zimbra Webmail by some malicious scripts (obviously...who else could type so fast?). How to deal with those, as ZCS reveals it's own IP in logs?

When going to 8.8.6 + patch I did NOT select any new features, like new IMAP, Chat or Zimbra drive.
BUT I HAD A LOT OF PROBLEMS injecting memcached and nginx into new 8.8.6. PRevious 8.0.9 did not have any of those, and when I upgraded, those two simply did not start. Tried many different methods, so I do not remember, which method worked, but it took me few hours to bring those two up and running. Maybe I broke something during the process, very likely.
ANOTHER possible problem was migration of config, because old ZCS 8.0.9 was directly opened to internet and had public IP on nic adapter, while new server where I migrated to is now behind firewall, having LAN IP. Of course, I configured split DNS on firewall and adjusted settings accordingly, but still there might be something somewhere inside, causing troubles.

Regarding repositories...this is what's in use:

Code: Select all

/etc/apt/sources.list:deb http://si.archive.ubuntu.com/ubuntu/ trusty main restricted
/etc/apt/sources.list:deb-src http://si.archive.ubuntu.com/ubuntu/ trusty main restricted
/etc/apt/sources.list:deb http://si.archive.ubuntu.com/ubuntu/ trusty-updates main restricted
/etc/apt/sources.list:deb-src http://si.archive.ubuntu.com/ubuntu/ trusty-updates main restricted
/etc/apt/sources.list:deb http://si.archive.ubuntu.com/ubuntu/ trusty universe
/etc/apt/sources.list:deb-src http://si.archive.ubuntu.com/ubuntu/ trusty universe
/etc/apt/sources.list:deb http://si.archive.ubuntu.com/ubuntu/ trusty-updates universe
/etc/apt/sources.list:deb-src http://si.archive.ubuntu.com/ubuntu/ trusty-updates universe
/etc/apt/sources.list:deb http://si.archive.ubuntu.com/ubuntu/ trusty multiverse
/etc/apt/sources.list:deb-src http://si.archive.ubuntu.com/ubuntu/ trusty multiverse
/etc/apt/sources.list:deb http://si.archive.ubuntu.com/ubuntu/ trusty-updates multiverse
/etc/apt/sources.list:deb-src http://si.archive.ubuntu.com/ubuntu/ trusty-updates multiverse
/etc/apt/sources.list:deb http://si.archive.ubuntu.com/ubuntu/ trusty-backports main restricted universe multiverse
/etc/apt/sources.list:deb-src http://si.archive.ubuntu.com/ubuntu/ trusty-backports main restricted universe multiverse
/etc/apt/sources.list:deb http://security.ubuntu.com/ubuntu trusty-security main restricted
/etc/apt/sources.list:deb-src http://security.ubuntu.com/ubuntu trusty-security main restricted
/etc/apt/sources.list:deb http://security.ubuntu.com/ubuntu trusty-security universe
/etc/apt/sources.list:deb-src http://security.ubuntu.com/ubuntu trusty-security universe
/etc/apt/sources.list:deb http://security.ubuntu.com/ubuntu trusty-security multiverse
/etc/apt/sources.list:deb-src http://security.ubuntu.com/ubuntu trusty-security multiverse
/etc/apt/sources.list.d/zimbra.list:deb     [arch=amd64] https://repo.zimbra.com/apt/87 trusty zimbra
/etc/apt/sources.list.d/zimbra.list:deb-src [arch=amd64] https://repo.zimbra.com/apt/87 trusty zimbra
I also updated right now existing Ubuntu 14.04 to latest, reinstalled ZCS 8.8.6 and re-patched it.
Wait until tomorrow and see what we've got.

BTW...one fail during re-install:

Code: Select all

zimlets
com_sol1_chromefix...failed. This may impact system functionality.

Re: Revert upgrade / roll-back to old ZCS server

Posted: Fri Mar 02, 2018 1:01 am
by Labsy
ccelis5215 wrote:@JDunphy men.., a hell of knwoledge, an enormous piece of advice.

@labsy sorry for you., Sorry, sometimes.that's it is . .. I've follow some of your posts., you are a seasoned sysadmin and of course zimbra / admin,etc,, just keep going.

ccelis
Must agree to the highest degree about JDunphy...that's what knowledge is about. Thank you!
And thank you Ccelis for nice words :)

Re: Revert upgrade / roll-back to old ZCS server

Posted: Fri Mar 02, 2018 2:04 am
by Labsy
Even after re-upgrade and re-patch same "weird" problems, for example:

EXTERNAL ACCOUNTS
Users A, B and C have configured in Webmail access to additional external account INFO via imap.
IT works flawlessly for user A and B, but in C user's Webmail he get's error:

Account "INFO" Failed
system failure: Folder sync failed, system failure: Synchronization of folder '/INFO/INBOX' failed, system failure: Server returned no response for UID FETCH 1659 BODY.PEEK[]


nginx.log

Code: Select all

2018/03/02 02:51:27 [info] 3021#0: *449 upstream sent invalid response: "NO LOGIN failed" while reading response from upstream, client: 10.10.11.50:58392, server: 0.0.0.0:993, login: "info@domain.com", upstream: 10.10.11.50:7143 (10.10.11.50:58392->10.10.11.50:993) <=> (10.10.11.50:52118->10.10.11.50:7143)
mailbox.log

Code: Select all

2018-03-02 02:51:27,215 INFO  [ImapServer-1] [ip=10.10.11.50;oip=10.10.11.50;via=10.10.11.50(nginx/1.7.1);ua=Zimbra/8.8.6_GA_1906;] imap - authentication failed for [info@domain.com] (invalid password)
com.zimbra.common.service.ServiceException: system failure: Unable to connect to IMAP server: LdapDataSource{id=fe917564-07da-4729-aca3-a3cd1e940809, type=imap, enabled=true, name=INFO, host=myzimbra.myserver.com, port=993, connectionType=ssl, username=info@domain.com, folderId=58579, smtpEnabled=false, smtpHost=null, smtpPort=-1}
But I copy-pasted password myself all the same in user's A, B and C Webmail. It's not invalid, no way, as it works for user A and B.
But if user C looks inside INFO folder, all INFO mail is there.
And even better - it pops up above error only once per session, upon logon. Afterwards it works fine.
What the hack? Where the hack?

Re: Revert upgrade / roll-back to old ZCS server

Posted: Fri Mar 02, 2018 12:06 pm
by scantec
"BUT I HAD A LOT OF PROBLEMS injecting memcached and nginx into new 8.8.6. PRevious 8.0.9 did not have any of those, and when I upgraded, those two simply did not start."

You should have upgraded to 8.6 first and install proxy + memcached correctly - some of your problems may be due to what you stated

Re: Revert upgrade / roll-back to old ZCS server

Posted: Fri Mar 02, 2018 12:51 pm
by Labsy
One weird detail about blacklists and numerous account lockouts, which bursted out after upgrade.
I said it might be coincidence, but even I started receiving spam to some strictly administrative mail accounts.
For example, I have admin mail aliases for different logins, like admin.cpanel1@subdomain.domain.com and this is in use for more than 10 years only for login to 1 control panel. No other usage on this email and for 10 years I received only mails from this control panel.
But noe, after upgrade, I started receiving spam on numerous such emails, which are not published anywhere.
Very weird, not have any explanation and it's too weird to be coincidence.

Re: Revert upgrade / roll-back to old ZCS server

Posted: Fri Mar 02, 2018 1:14 pm
by JDunphy
That looks really good. We will get this fixed. I have a lot of trust on your install at this point.
Labsy wrote: Beside, when I was last dealing with compromised account yesterday, I had some 100/minute failed login attempts via Zimbra Webmail by some malicious scripts (obviously...who else could type so fast?). How to deal with those, as ZCS reveals it's own IP in logs?
Tell nginx to pass on the ip through because now we are behind a proxy. Change X.X.X.X to the ip address of your zmhostname. See https://wiki.zimbra.com/wiki/Log_Files and the section Logging the Originating IP

Code: Select all

% zmprov mcf +zimbraMailTrustedIP 127.0.0.1 +zimbraMailTrustedIP X.X.X.X
% zmmailboxdctl restart
Labsy wrote: When going to 8.8.6 + patch I did NOT select any new features, like new IMAP, Chat or Zimbra drive.
BUT I HAD A LOT OF PROBLEMS injecting memcached and nginx into new 8.8.6. PRevious 8.0.9 did not have any of those, and when I upgraded, those two simply did not start. Tried many different methods, so I do not remember, which method worked, but it took me few hours to bring those two up and running. Maybe I broke something during the process, very likely.
ANOTHER possible problem was migration of config, because old ZCS 8.0.9 was directly opened to internet and had public IP on nic adapter, while new server where I migrated to is now behind firewall, having LAN IP. Of course, I configured split DNS on firewall and adjusted settings accordingly, but still there might be something somewhere inside, causing troubles.
I am going through my notes of my installs and there was a ton of changes I had to do by hand as I look back. I wonder if we have 2 ip entries somewhere given that ip change... I remember that fact for later but that could be a hint. Note: I like to make sure files is first lookup in /etc/nsswitch.conf for hosts. Then I add my all my ip's in /etc/hosts. I still do split DNS to be comprehensive but I use the same staging server to test various customer installs just with /etc/hosts. If you do similar, double check you don't have an old ip address in there.
Labsy wrote: I also updated right now existing Ubuntu 14.04 to latest, reinstalled ZCS 8.8.6 and re-patched it.
Wait until tomorrow and see what we've got.

BTW...one fail during re-install:

Code: Select all

zimlets
com_sol1_chromefix...failed. This may impact system functionality.
That looks really good.
BTW, that zimlet appears to be something from zimbra 8 so you can disable it or uninstall it I would think. https://github.com/sol1/chromefix-zimle It appears to fix some style sheet stuff so not a problem. I will look at your other posts next but we should be able to get this working better.