Horrible security "patching"

Discuss your pilot or production implementation with other Zimbra admins or our engineers.
Post Reply
10119metux
Advanced member
Advanced member
Posts: 75
Joined: Sat Sep 13, 2014 2:29 am

Horrible security "patching"

Post by 10119metux »

Hi folks,
I'm really getting fed up w/ your way of doing security "patches".

Things like this:

We">https://www.zimbra.com/forums/announcem ... ility.html
We already had the same mess w/ the heartbleed fix. A script which downloads some tarball and replaces individual files,

completely outside the package manager, so defeating the whole purpose of package management.
First of all, that script is totally insecure. Okay, you've now learned to use https instead, but w/ explicitly switching off

certificate check, this becomes nothing but a bad joke. Oh, and why that md5 file ? I hope you dont think it should

give you more security, because it does NOT.
You really wanna invite the NSA to abuse your "patching" for injecting backdoors ?!
:mad:
And now for the approach of replacing single files, instead of just shipping new packages:
Do you understand what the concept of package management it is meant for ? Because I really doubt that :P
Package management isn't just a fancy way for transporting bundles of files to some target system, it is

the *CORE* of software deployment on modern *nix platforms, which operators use all the day.. The main

purpose is CONSISTENCY and easy tracking of installed files and package versions (of course, including

_fully automatic_ checking for available packages and updates, downloads, signature checking, deployment,

and removal, etc).
And it also adds another - fully automatic - security layer, as the package manager checks package signatures

against the repo and validates the repo against the authorized keys. If done correctly (which is pretty easy),

it automatically makes sure that only properly authenticated packages are installed (assuming you've told

it to trust the repo in question).
Another _vital_ aspect:
Operators usually don't have the time for reading tons upgade notifications on a daily basis and run upgrades

for dozens of packages on each single machine manually. That's what we have distros for (and for professional

commercial software, we just expect the vendor to do his homework properly!). And when it comes to security

fixes, that's what mechanisms like 'unattended-upgrades' are for, which deploy security fixes fully-automatically,

w/o having to wait for an operator to trigger the upgrade manually.
:mad: :mad: :mad:


This is a pretty typical problem of many commercial vendors (also seen it at a lot of my clients - many even don't

know how to write proper Makefiles or use a decent SCM): they might produce pretty good software, but

deployment tends to be a total mess.
This mess is getting worse and worse over the years (well, I'm pretty curious how the situation becomes w/ JP,

haven't got access to the source for a review yet ...), and it _really_ should be cleaned up soon, otherwise I

see even more people switching to other (really opensource!) solutions like Kolab !
I can understand that such fundamental infrastructure things don't easily fit into product plans, and hilevel

decision makes usually don't know anything about that. But these things are really importan for stability

as well as TCO. It's like trying to build a skyscraper on thin sand ... either it's extremly expensive or it

will fall apart sooner or later ... ;-o
In case you're lacking proper resources (which is my _strong_ impression for years now), don't hesistate

to call me. I can arrange a whole team w/ very deep Zimbra knowledge (we're probably the ones who

do the deepest / most complex Zimbra customizations in whole Europe, if not world wide). Besides the

architect, who eg. developed zmpkg (which would by myself :p), we have a whole team of very

Zimbra-experienced developers, several system integrators / operators, lots of testers, etc - everything

that's needed to roll the whole show on our own :P
After all these years, I still belive that Zimbra has great potential, but it needs a change in development

management approaches. The mentioned deployment issues is just one part, the lack of a _real_

OSS community is another one (I guess, I already mentioned the problems w/ your SCM infrastructure

and workflows, which is really expensive ;-o)


cu
phoenix
Ambassador
Ambassador
Posts: 27278
Joined: Fri Sep 12, 2014 9:56 pm
Location: Liverpool, England

Horrible security "patching"

Post by phoenix »

Instead of ranting in the forums you'd be far better providing a well described description and possible solution to the 'problem' in bugzilla, possibly an RFE.
Regards

Bill

Rspamd: A high performance spamassassin replacement

Per ardua ad astra
bandaram
Posts: 1
Joined: Sat Sep 13, 2014 3:18 am

Horrible security "patching"

Post by bandaram »

I agree with Bill here. Zimbra being an open source product, we would love to leverage the experience and commitment of the community for the benefit of everyone. Obviously 'metux' has good ideas and passion for improvement, so I would like to work with these ideas if anyone is willing to start a discussion and help with implementation.
User avatar
quanah
Zimbra Alumni
Zimbra Alumni
Posts: 1668
Joined: Fri Sep 12, 2014 10:33 pm
Contact:

Horrible security "patching"

Post by quanah »

--
Quanah Gibson-Mount
Product Architect, Symas http://www.symas.com/
OpenLDAP Core team http://www.openldap.org/project/
10119metux
Advanced member
Advanced member
Posts: 75
Joined: Sat Sep 13, 2014 2:29 am

Horrible security "patching"

Post by 10119metux »

[quote user="10330phoenix"]Instead of ranting in the forums you'd be far better providing a well described description and possible solution to the 'problem' in bugzilla, possibly an RFE.[/QUOTE]
Also did so quite some time ago.
There's not much to explain, as it's pretty trivial and obvious:

in case of such an hotfix case occours, fork off at the latest release, fix the bug, then rebuild and publish that.
And, of course, for automatic updates (which should be the standard for such immediate hotfixes),

put the packages into an repo and tell the world it's url, so package manager and usual tools

like `unattended-upgrades` (which every decent operator activates) will automatically handle

the security fix upgrade.
What is so complicated here ?
User avatar
quanah
Zimbra Alumni
Zimbra Alumni
Posts: 1668
Joined: Fri Sep 12, 2014 10:33 pm
Contact:

Horrible security "patching"

Post by quanah »

As you yourself note, you've already asked this, and it has been answered, in the bug. It's also been answered offline for you in direct email as well. I don't see much point to your posting here other than some need to grandstand. But just in case a 3rd time is a charm:
The installer is monolithic and ancient (virtually unchanged since about 2005). It's not currently possible with the way it is designed to use repositories. This has already been identified as an issue that needs immediate attention, as noted in https://bugzilla.zimbra.com/show_bug.cgi?id=87788. The plan is to rework the installer as part of the ZCS 8.6 release. One issue using repositories does not address is our clients that have servers with no external internet access. The current installer resolves this by including everything it needs as part of the package. I.e., we are going to likely still need to bundle a "complete" installer vs one that relies on repositories. Either way, you are already quite aware of the fact that Zimbra knows this is an issue, and that Zimbra already has plans to address it. Ranting on the forums does nothing in terms of resolving an already identified issue.
--Quanah
--
Quanah Gibson-Mount
Product Architect, Symas http://www.symas.com/
OpenLDAP Core team http://www.openldap.org/project/
10119metux
Advanced member
Advanced member
Posts: 75
Joined: Sat Sep 13, 2014 2:29 am

Horrible security "patching"

Post by 10119metux »

[quote user="quanah"]As you yourself note, you've already asked this, and it has been answered, in the bug.

[/QUOTE]
Not completely answered.
But that's not the main issue here. You already have a build system, which creates packages.

Even it's not yet suited for initial deployment or larger upgrades (like from helix to ironmaiden),

but certainly enough for such relatively small things.
In this case, you just had to run a usual rebuild and publish the packages to some repo,

and point people to it. And at this point you also optionally could even have added your repo url

and key to the system.
Having to run the installer on initial deployments and major upgrades, is still acceptable.

But such security-fixes should really run automatically.
Instead of writing a script that downloads some tarball and directly unpacks it completely

outside the package management, you could instead at least have it download upgraded

.deb/.rpm and install it - this at least would have left the package manager database

in an consistent state.
> It's not currently possible with the way it is designed to use repositories.
Simply put the upgraded .deb's / .rpm's in a repository and point people to it.

The installer doesn't have much to do with that. (in the case in question, it didn't

even have to be run)
After that point, the actual packages can be removed from the installer tarball - the installer

then just needs to add the repositories, and change certain calls (eg. on Debian, call apt

instead of dpkg).
Yet again: I'm currently just talking about upgrades within an major release line.
[QUOTE]

The plan is to rework the installer as part of the ZCS 8.6 release.

[/QUOTE]
So, in several years ? ;-o
[QUOTE]

One issue using repositories does not address is our clients that have servers with no external internet access.

[/QUOTE]
Mirroring package repositories is rather trivial. Can be done w/ some lines of shell script.

Same for automatically generating an installer tarball that includes the repo mirror.

(local mirrors aren't that unusual, in fact quite standard for decades, eg. w/ CD installations)
[QUOTE]

I.e., we are going to likely still need to bundle a "complete" installer vs one that relies on repositories.

[/QUOTE]
Pretty trivial, can be automatically generated, pretty easily.
User avatar
quanah
Zimbra Alumni
Zimbra Alumni
Posts: 1668
Joined: Fri Sep 12, 2014 10:33 pm
Contact:

Horrible security "patching"

Post by quanah »

It would take distributing zimbra-core, which is a massive package full of a million different things. Installing a new version of this on top of an existing install could wreak havoc for customers that apply their own custom patches on top of what Zimbra ships (and there are plenty that do that). It is not a workable solution. Again, the current system is perfect, but it also isolates the the update to what's required without touching large portions of the product unnecessarily.
--
Quanah Gibson-Mount
Product Architect, Symas http://www.symas.com/
OpenLDAP Core team http://www.openldap.org/project/
10119metux
Advanced member
Advanced member
Posts: 75
Joined: Sat Sep 13, 2014 2:29 am

Horrible security "patching"

Post by 10119metux »

> It would take distributing zimbra-core, which is a massive package full of a million different things.
In case of such an security fix, most of these things would remain the same. Where's the problem ?
> Installing a new version of this on top of an existing install could wreak havoc for customers that

> apply their own custom patches on top of what Zimbra ships (and there are plenty that do that).
If course, they'll have to rebase their customizations onto the new version (which in that case

would be an no-op).
By the way: we're one of these people doing pretty many core customizations.

And we have pretty much automatized the whole process. (on minor upgrades, it's just a

matter of a few minutes manual stuff)
Oh, and why havent't you split it already in several smaller packages ?

Even with the horribly broken IronMaiden buildsystem, it's not that difficult (once you've got

the build system working at all ;-o). You're already building several packages - just add

add another one ...
EDIT: with some package manager magic, you could even split it out within an minor

upgrade - without overwriting the other stuff ...
Post Reply