How to Upgrade openssl to OpenSSL 1.0.1h in zimbra 8.0.3

Discuss your pilot or production implementation with other Zimbra admins or our engineers.
fnoopdogg
Posts: 9
Joined: Wed Oct 29, 2014 3:05 am

How to Upgrade openssl to OpenSSL 1.0.1h in zimbra 8.0.3

Post by fnoopdogg »

Thanks Sebastian, this works fine. To be fair, CentOS isn't an officially supported platform so I guess the usual --platform-override got overlooked on this patch.
metux
Advanced member
Advanced member
Posts: 146
Joined: Mon Jul 28, 2014 6:21 pm

How to Upgrade openssl to OpenSSL 1.0.1h in zimbra 8.0.3

Post by metux »

*FACEPALM*
fnoopdogg
Posts: 9
Joined: Wed Oct 29, 2014 3:05 am

How to Upgrade openssl to OpenSSL 1.0.1h in zimbra 8.0.3

Post by fnoopdogg »

Hi Metux
You raise some interesting points (that have been raised before by others, by the look of it), although not sure the way you're raising them is conducive to working with Zimbra to make any improvements.
[quote user="metux"]

The --no-check-certificate does not "EXPLICITLY SWITCH OFF TRANSPORT SECURITY"...it just allows the communication to proceed even when handshake fails

Which _EXACTLY_ means NO SECURITY - at this point anybody in the middle can send his own cert and so has taken over the encrypted channel.
[/quote]
It doesn't mean 'NO SECURITY' by any means.  SSL/TLS is primarily meant to secure the transport of data - to stop people eavesdropping on sensitive information (credit cards, private data etc) - which is still fully active here.  Of course part of that is authentication of the source, so in this case disabling the certificate chain checks does reduce security and opens the possibility of MiM attacks.  However, this would require either authoritative DNS control or control of some part of the unpredictable route between you and Zimbra/AWS infrastructure, and the ability to do MiM attack in realtime.  It's also no more or less secure than the distribution of the main Zimbra packages themselves (all done through http, not https), even if they did produce a full point release to address the SSL issues.  

However, this is certainly not 'optimal' on the part of Zimbra.   The certificate for files.zimbra.com should be fixed so it has a proper chain of authority.  Either that or just don't use ssl to download these patches, it' not really adding much in the way of security currently.  It would be nice for Zimbra to take the process a little more seriously, and to respond quicker to these quite public and serious issues.


[quote user="metux"]


it does even has a MD5 checksum verification

This doesn't add any security whatsoever, just protects against accidental file transfer errors.
[/quote]
That's not true, MD5 checksums are a widespread and important security measure - checksums are regularly published by major software vendors to verify the integrity of software downloads, and checking them is frequently a mandatory requirement within large corporates for downloading external software.  However, for the checksums to be of any practical use they need to be hosted on a different, secure site, otherwise they can be replaced/hijacked just as the distribution software can be.  The way they are used in this ssl patch script doesn't add any security.



[quote user="metux"]
Pretty trivial:



The session key handshake is encrypted with the server's pubkey/privkey (which is found in the cert).
As the server's pubkey usually isn't known at client side, it's signed by an CA (the signature is also in the cert).

If the cert cannot be verified, anybody in the middle can replace it with it's own one and so pretend being
the real server and serve anything it likes. In your case, that fake server would send a patched libssl.so binary
which contains certain backdoors to compromise the target machine.

Voila! dozens of Zimbra instances, running in hi profile large organisations easily taken over and having
full access to their mail traffic.
[/quote]
This is somewhat true, although due to the nature of this kind of attack there would numerous other easier ways to achieve the same thing.  Most compromised software are usually compromised at the source, ie. by people breaking into to the source or distribution servers (openssl itself has been hacked this way in the past, although independently hosted MD5 checksums showed that the distribution software wasn't affected).  And as the main software is distributed over http, like a lot of software is, this is somewhat of a moot point anyhoo.  Most software is like this, including the majority of Linux OS distributions.  If you look at the vast majority of OS software/update sources, they are all http.  So it's highly likely the base OS OpenSSL packages of most Zimbra customers are also distributed over 'insecure channels'.

[quote user="metux"]
Actually, the whole approach of replacing single files by such scripts is a totally dumb idea in the first place.
The correct way is to just to do a fresh rebuild and publish the new packages in a proper (signed!) repo.

I tried discussing that with your devs, but turned out as a waste of my precious time.
[/quote]

I have to somewhat agree with this.  The trouble is that Zimbra is a massive, delicate install and large corporates who are likely most at need of this patch need a lot of planning, notice and change control to do updates like this.  Emergency security updates are normally only approved if they are as unobtrusive as possible.  Speaking from personal experience, I would be considerably more comfortable using these small Openssl-specific patches for a quick response than I would be doing an entire upgrade.  And I would imagine (hope!) that Zimbra QA of full releases, even point releases, is quite a large task.
But in the long term it would be a good idea for Zimbra to break the packages down further into smaller units that can be updated easier, and it would be a lot easier if they provided repos.

As to the issue of not using underlying OS packages, I suspect historically it was probably to keep control over the stability and (ironically) security of the Zimbra platform.  Many moons ago I used to roll my own mail systems and I'm quite familiar with a lot of the underlying components that Zimbra uses.  It's a very large and complex ecosystem, and some of them are very, very sensitive to very specific versions of the supporting libraries, otherwise you start getting all sorts of stability issues.  And there used to be a larger variety of OS versions supported with a wide variety of support library versions.  It was (and is) probably much easier to develop and QA a known baseline across the board.  As with anything, there are pros and cons.
metux
Advanced member
Advanced member
Posts: 146
Joined: Mon Jul 28, 2014 6:21 pm

How to Upgrade openssl to OpenSSL 1.0.1h in zimbra 8.0.3

Post by metux »

[quote]

You raise some interesting points (that have been raised before by others, by the look of it), although not sure the way you're raising them is conducive to working with Zimbra to make any improvements.

[/quote]



Well, I've tried the diplomatic way for *YEARS* - but it didn't work at all. So this way is the only one left.



[quote]

It doesn't mean 'NO SECURITY' by any means

[/quote]



Security that's _easily_ breakable is no security at all. That kind of attack (cert forgery for mitm-attack) is so standard that it's even builtin in lots of network analysis tools.



[quote]

SSL/TLS is primarily meant to secure the transport of data - to stop people eavesdropping on sensitive information (credit cards, private data etc) - which is still fully active here.

[/quote]



But without verified certificates, it completely fails to do so.



And in this scenario here, we dont need any secrecy, but authority - we dont need to protect against eavesdropping, but code injection.

Exactly what's NOT provided w/o verified certs.



So, it's completely useless here - no security at all.



[quote]

Of course part of that is authentication of the source, so in this case disabling the certificate chain checks does reduce security and opens the possibility of MiM attacks.

[/quote]



Actually, it really invites them, considering whats downloaded here, and who the affected users are.



[quote]

However, this would require either authoritative DNS control or control of some part of the unpredictable route between you and Zimbra/AWS infrastructure, and the ability to do MiM attack in realtime.

[/quote]



Quite trivial, if you're already on the routing path, like [tag:NSA] or the telcos.

And yes: considering where Zimbra is used, this is a _primary_ target of the NSA.



[quote]

It's also no more or less secure than the distribution of the main Zimbra packages themselves

[/quote]



Sure, that's also completely insecure.

Yet another reason for NOT trusting the downloaded binaries and better compile the source on your own.

(or take it from some trustworthy distro - if there was one)



[quote]

However, this is certainly not 'optimal' on the part of Zimbra.

[/quote]



It's a desaster and yet another big warn sign that Zimbra folks are simply incompetent in IT security.

(certainly, far not the only one).



[quote]

Either that or just don't use ssl to download these patches, it' not really adding much in the way of security currently.

[/quote]



Sure, as long as they don't have properly signed repositories, it's all completely insecure.



[quote]

It would be nice for Zimbra to take the process a little more seriously, and to respond quicker to these quite public and serious issues.

[/quote]



It would be nice if Zimbra folks take IT security seriously at all.



[quote]

That's not true, MD5 checksums are a widespread and important security measure - checksums are regularly published by major software vendors to verify the integrity of software downloads, and checking them is frequently a mandatory requirement within large corporates for downloading external software.

[/quote]



Trusting md5 for security (instead of safety) is simply dumb. It has been broken long ago.

The way to go is SHA256, at least.



[quote]

However, for the checksums to be of any practical use they need to be hosted on a different, secure site, otherwise they can be replaced/hijacked just as the distribution software can be.

[/quote]



Or properly signed. Exactly what all decent package managers do.

But Zimbra folks are really resistant against those well-proven technologies.



[quote]

This is somewhat true, although due to the nature of this kind of attack there would numerous other easier ways to achieve the same thing.

[/quote]



Actually it's a pretty easy attack vector, considering the primary enemy are organisations like the [tag:NSA], which have the infrastructure to run such attacks automatically.



[quote]

Most compromised software are usually compromised at the source, ie. by people breaking into to the source or distribution servers (openssl itself has been hacked this way in the past, although independently hosted MD5 checksums showed that the distribution software wasn't affected).

[/quote]



With decent source control technologies (eg. git+signed tags), such attacks are pretty hard to do undetected.

If you still wanna succeed, you'll need to get the right privkeys. Many years ago, Debian folks did the bad mistake

of having them on an publically available server - they (and others) have learned and switched them to safe locations.



[quote]

And as the main software is distributed over http, like a lot of software is, this is somewhat of a moot point anyhoo.

[/quote]



With proper signature validation, that's no problem at all.

For example, apt repos are usually published via plain http or ftp - but the individual packages and metadata are

properly signed with the repo key. So, you only need to verify the repo pubkey _once_ (which can be easily done

via other trust networks).





Yet again: if Zimbra folks would simply use the distro's package management infrastructures properly, the

whole problem wouldn't exist at all.



Oh, and if they wouldn't do this _useless_ bundling of their own openssl library, these kind of security

upgrades wouldn't have been necessary in the first place - as the affected system would already be upgraded

via the distro, even before somebody @Zimbra shows the first reaction.
metux
Advanced member
Advanced member
Posts: 146
Joined: Mon Jul 28, 2014 6:21 pm

How to Upgrade openssl to OpenSSL 1.0.1h in zimbra 8.0.3

Post by metux »

[quote]

The trouble is that Zimbra is a massive, delicate install and large corporates who are likely most at need of this patch need a lot of planning, notice and change control to do updates like this.

[/quote]



With proper package management, this would be pretty easy.

And for such security hotfixes, there shouldn't be any planning required, nor any manual intervention at all (-> unattendet-upgrades).



Yet again, if openssl wasn't bundled, the problem wouldn't even exist in the first place, as distros already handle that.



[quote]

Speaking from personal experience, I would be considerably more comfortable using these small Openssl-specific patches for a quick response than I would be doing an entire upgrade.

[/quote]



No, you seriously don't wanna do such manual steps and so harm the system consistency. _especially_ if you need auditing.



The problem comes from a different area - beginning with source code handling (these guys dont even have a decent SCM,

just play around with the horrible P4).



In case of such criticial updates, you'd simply do you patch in the maintenance branch for certain release (which *ONLY* gets

such criticial fixes), rebuild and publish the final package in the corresponding repos. For example, 8.0.3 will be updated by 8.0.3.1.

The package manager then will automatically update when necessary. (if somebody _really_ needs even finer control, he still

can use his own moderated repo, pinning, etc - just the usual package manager methods for such cases).



I've done a lot of Zimbra customizations (in fact, acting as the chief sw architect for one of the largest Zimbra integrators in Europe)

for hi-profile installations (eg. banking). And the lack of proper packaging is really a big pain. For Zimbra extensions it's even worse:

there's NO RELIABLE deployment mechanism whatsoever - that's why I've invented zmpkg.



[quote]

And I would imagine (hope!) that Zimbra QA of full releases, even point releases, is quite a large task.

[/quote]



Indeed it is, because they don't have proper release engineering.

Just look at the mainlines of certain major releases, or even the maintenance branches. They don't even compile.

With each new commit, there's big chance of breaking everything, because they dont care about quality on source/SCM level.



If they would do proper SCM workflows and release engineering (as I already suggested *YEARS* ago), such things

wouldn't happen in the first place. But who am I that they might listen to me ? Just one of the guys with the deepest

Zimbra development and customization experience outside the company, at least in Europe.



[quote]

But in the long term it would be a good idea for Zimbra to break the packages down further into smaller units that can be updated easier, and it would be a lot easier if they provided repos.

[/quote]



Yes, and also drop those 3dparty stuff which is not _absolutely_ necessary. Could have been done *YEARS* ago.

(already supplied patches, but all my input just gets refused or ignored).



[quote]

As to the issue of not using underlying OS packages, I suspect historically it was probably to keep control over the stability and (ironically) security of the Zimbra platform.

[/quote]



Deep in ancient history.

I already had such discussions w/ certain Zimbra devs and they seriously picked out 15 years old issues as excuse.



[quote]

Many moons ago I used to roll my own mail systems and I'm quite familiar with a lot of the underlying components that Zimbra uses. It's a very large and complex ecosystem, and some of them are very, very sensitive to very specific versions of the supporting libraries, otherwise you start getting all sorts of stability issues.

[/quote]



Well, in recent years, such problems are pretty rare, if you take some care.



Okay, for the java stuff, it's sometimes a bit different there, as many folks in java community don't even care about API stability

(even change APIs in _minor_ releases, etc). But enterprise distros like Debian have a quite good handling of such issues

(eg. MVCC packaging, etc).



[quote]

And there used to be a larger variety of OS versions supported with a wide variety of support library versions.

[/quote]



Long ago. With Helix it were only a few deb and rpm based ones. Most of them quite equal to each other, regarding the libraries in question.



[quote]

It was (and is) probably much easier to develop and QA a known baseline across the board.

[/quote]



No, it wasn't - they just _believe_ so. Of course, w/o any proper packaging and SCM, you're lost anyways.

I've got over 15 years hands-on experience in such cases, and been through all of that.
User avatar
ccelis5215
Outstanding Member
Outstanding Member
Posts: 632
Joined: Sat Sep 13, 2014 2:04 am
Location: Caracas - Venezuela
ZCS/ZD Version: 8.8.15.GA.3869.UBUNTU18.64 P12

How to Upgrade openssl to OpenSSL 1.0.1h in zimbra 8.0.3

Post by ccelis5215 »

I agree with Metux rants, as i follow this forums and his team work in his company.



There's no reason, technically speaking, to not follow Metux arguments.



ccelis
revanth1226
Posts: 11
Joined: Sat Sep 13, 2014 3:07 am

How to Upgrade openssl to OpenSSL 1.0.1h in zimbra 8.0.3

Post by revanth1226 »

This works fine , but It has downloaded openssl-1.0.1d instead of openssl-1.0.1h.

I need to upgrade to openssl-1.0.1h , server is already with openssl-1.0.1d.



Below are the script result



[root@zim01 tmp]# ./zmopenssl-updater.sh

++ whoami

+ '[' xroot '!=' xroot ']'

+ SSL[0]=1.0.1d

+ SSL[1]=1.0.1e

+ SSL[2]=1.0.1e

+ SSL[3]=1.0.1e

+ SSL[4]=1.0.1f

++ su - zimbra -c 'zmcontrol -v'

+ VERSION='Release 8.0.3_GA_5664.RHEL6_64_20130305090204 CentOS6_64 FOSS edition.'

+ [[ Release 8.0.3_GA_5664.RHEL6_64_20130305090204 CentOS6_64 FOSS edition. == *ZCA* ]]

++ echo Release 8.0.3_GA_5664.RHEL6_64_20130305090204 CentOS6_64 FOSS edition.

++ cut '-d ' -f2

+ VERSION=8.0.3_GA_5664.RHEL6_64_20130305090204

++ echo 8.0.3_GA_5664.RHEL6_64_20130305090204

++ sed 's/_.*//'

+ VERSION=8.0.3

++ echo 8.0.3

++ cut -d. -f1

+ MAJOR=8

++ echo 8.0.3

++ cut -d. -f2

+ MINOR=0

++ echo 8.0.3

++ cut -d. -f3

+ PATCH=3

+ '[' 8 -ne 8 ']'

+ '[' 0 -ne 0 ']'

+ '[' 3 -lt 3 ']'

++ expr 3 - 3

+ ARPATCH=0

+ SSL_VERSION=1.0.1d

+ '[' '!' -d /opt/zimbra/openssl-1.0.1d ']'

++ which egrep

+ EGREP=/bin/egrep

+ '[' x/bin/egrep = x ']'

+ /bin/egrep 'OpenSSL 1.0.1h' /opt/zimbra/openssl-1.0.1d/lib/libssl.so.1.0.0

+ RC=1

+ '[' 1 -eq 0 ']'

+ PLAT=RHEL6_64

+ cd /tmp

+ '[' -d openssl/RHEL6_64 ']'

+ mkdir -p openssl/RHEL6_64

+ cd openssl/RHEL6_64

++ which wget

+ WGET=/usr/bin/wget

+ '[' x/usr/bin/wget = x ']'

++ which md5sum

+ MD5SUM=/usr/bin/md5sum

+ '[' x/usr/bin/md5sum = x ']'

+ echo 'Downloading patched openssl'

Downloading patched openssl

+ wget --no-check-certificate https://files.zimbra.com/downloads/8.0. ... 1.0.1d.tgz

+ RC=0

+ '[' 0 -ne 0 ']'

+ wget --no-check-certificate https://files.zimbra.com/downloads/8.0. ... tgz.md5sum

+ RC=0

+ '[' 0 -ne 0 ']'

+ echo -n 'Validating patched openssl: '

Validating patched openssl: ++ /usr/bin/md5sum openssl-1.0.1d.tgz

+ DOWNLOAD_SUM='3bad23cc4b09cf2391046ca39a8b779f openssl-1.0.1d.tgz'

++ cat openssl-1.0.1d.tgz.md5sum

+ GOOD_SUM='3bad23cc4b09cf2391046ca39a8b779f openssl-1.0.1d.tgz'

+ '[' '3bad23cc4b09cf2391046ca39a8b779f openssl-1.0.1d.tgz' = '3bad23cc4b09cf2391046ca39a8b779f openssl-1.0.1d.tgz' ']'

+ echo success

success





Kindly suggest how to upgrade to openssl-1.0.1h.



Let me know below steps work , with out using script.



update system to latest openssl with yum update openssl

cd /opt/zimbra/openssl/lib

unlink libssl.so

mv libssl.so.1.0.0 orig.libssl.so.1.0.0

ln -sf /usr/lib64/libssl.so.1.0.1e ./libssl.so.1.0.0

ln -sf /usr/lib64/libssl.so.1.0.1e ./libssl.so

service zimbra restart
User avatar
pup_seba
Outstanding Member
Outstanding Member
Posts: 687
Joined: Sat Sep 13, 2014 2:43 am
Location: Tarragona - Spain
Contact:

How to Upgrade openssl to OpenSSL 1.0.1h in zimbra 8.0.3

Post by pup_seba »

SSL_VERSION=1.0.1h
fnoopdogg
Posts: 9
Joined: Wed Oct 29, 2014 3:05 am

How to Upgrade openssl to OpenSSL 1.0.1h in zimbra 8.0.3

Post by fnoopdogg »

Gosh, I've fed a gremlin after midnight.. Metux, I'm not going to respond to you point by point because I don't want to encourage you any further, but I'll just say this: you have some good points, but you also have some misunderstandings, inaccuracies and you're not acknowledging that there is no one 'correct' way of accomplishing an end goal, and to see others point of view or approach. At this point your rants are counterproductive - they will never accomplish what you want - and you're just trolling public forums. It's not your product, or your place to tell Zimbra what to do - if you don't like it, go use something else and leave the rest of us who are excited by this project to use it and positively move it forward.

However, as Zimbra has now opensourced the code, you can fork the project (and have done so, by the look of it). Zimbra has given you, and everyone else, this massive gift, so instead of constantly publicly insulting them, why don't you just get on with fixing the issues in your fork, and hopefully contributing back to the project? It looks like the new owners are a lot more open to participation and community than before. Why don't you pick one or two small issues, and calmly, rationally and methodically approach them one by one by submitting patches and opening a discussion in the developer forum?
metux
Advanced member
Advanced member
Posts: 146
Joined: Mon Jul 28, 2014 6:21 pm

How to Upgrade openssl to OpenSSL 1.0.1h in zimbra 8.0.3

Post by metux »

[quote]

you have some good points, but you also have some misunderstandings, inaccuracies and you're not acknowledging that there is no one 'correct'

[/quote]



Of course, there're always several ways to reach a goal. But Zimbra folks just picked the worst one and refuse to think about anything else.

This situation is persistant over many years now.



Back some years ago, I really tried to explain these things in-depth, politely and diplomatic. But over the years, my diplomancy reserviour for that simply ran dry. Now I just name the problem and leave them alone with that (as they dont listen anyways).



[quote]

and to see others point of view or approach.

[/quote]



I already know their PoV, analyzed and discussed that many times, shown up the fundamental problems with that, provided better solutions.

They just ignored everything, so I won't waste my time repeating it over and over again. Instead I just it right on my own.

(for now, open-zimbra is just a fun project, but our group is growing...)



[quote]

At this point your rants are counterproductive - they will never accomplish what you want

[/quote]



No, they're not. It helps opening the eyes for others (not Zimbra devs - I've already given them up anyways) and raises interest for the OpenZimbra project.



By the way: we had the same problems w/ OOffice ... we all know where it went ...



[quote]

- and you're just trolling public forums.

[/quote]



Yes! In wide open public, where it gets the max audience. Exactly as intended.

(direct conversations with Zimbra devs have been proven to be useless, even @ bugzilla).



My goal is not trying to convice Zimbra devs of anything(as its useless anyways), but instead collecting more people who are as disappointed as myself and increase the community of OpenZimbra.



And yes! We're going for a full fork. And in this fork, we're also going to implement NE ($$$) features on opensource (->free) technology,

so there won't be any reason for purchasing NE anymore (several of my prev clients won't be able to use recent NE versions, as we

can't customize anymore in the way we need it).



[quote]

Zimbra has given you, and everyone else, this massive gift, so instead of constantly publicly insulting them,

[/quote]



The mentioned problems also affect (in even worse way) the NE version, where my clients pay much $$$ for.

(several 10k of mailboxes). But Zimbra has left us standing in the rain w/ criticial issues so many times (and

produces _unnecessary_ operating overhead by simply _not_ packaging properly and having to decent

release engineering), that my reservoir of politeness simply went dry.



[quote]

why don't you just get on with fixing the issues in your fork, and hopefully contributing back to the project?

[/quote]



Already doing so (as much as my precious time allows - it's just a fun project right now). But Zimbra doesn't

even provide the exact sources for releases (just a bunch of branches which dont even compile - no such

thing like release tags, etc), so we don't even have a clear baseline



[quote]

and hopefully contributing back to the project?

[/quote]



Forget about it. They won't do it. Tried so for many years.

I've always been careful about providing patches that can be easily cherry-picked into the mainline,

but they didn't even have a look at it. They even refuse that the mentioned problems exist at all.



[quote]

It looks like the new owners are a lot more open to participation and community than before.

[/quote]



That's what they always tell. But the practical experience with the Zimbra devs shown no difference

to the previous owner. (well, to be fair, certain community managers seem to show some interest,

but that doesn't affect the devs, which remain as ignorant as they've been for *years*).



[quote]

Why don't you pick one or two small issues, and calmly, rationally and methodically approach them one by one by submitting patches and opening a discussion in the developer forum?

[/quote]



Been through this a dozen times. Turned out as a waste of time, everything just got ignored.

I won't waste my precious spare time with such useless attempts anymore.
Post Reply