ldap database went from 97meg to 86gig

samgann · Post by **samgann** » Fri Nov 23, 2012 10:56 pm

ok I guess I understand now.

The 86 gig is allocated. Why does the data base need that much if the database is less that 200 meg? I now you said that rsync is not a great way to backup the database. Why does a rsync backup that file as a 86 gig file. Because the space is 86 gigs allocated ? I need to find another way to backup my servers. I have a main server and a backup server. Can you suggest another way? I think this whole thread is about this issue of backing up and not understanding the allocation of disk space from the 8.01 upgrade. The upgrade took alot of people off gaurd when they tried to backup there servers.
Thanks for your support

looking forward to a reply on a new way to backup the servers

Sam

quanah · Post by **quanah** » Fri Nov 23, 2012 11:28 pm

See https://bugzilla.zimbra.com/show_bug.cgi?id=78781#c4

kevindods · Post by **kevindods** » Sat Nov 24, 2012 9:12 am

I think if you use virtual instances of Zimbra then taking a snapshot of the server before updating anything is a nice simple way of doing it.
Working on server metal directly these things are not so simple. Generally the process was to rsync the /opt/zimbra directory once, then stop Zimbra services do another rsync with LDAP and all other services stopped to produce consistent files that updated the initial rsync.
For many FOSS users this approach works well especially in smaller businesses and is relatively simple to script and cron job and is very reliable
I think that Quanah has explained clearly and helpfully how to backup the LDAP database in a reliable way, thanks Quanah - that is good and useful.
What none of us seem to quite understand why the DB has 80GB provisioned when most DBs are a few hundred MB at most. If the DB was closer to expected maximums the issues we experienced would not arise at all. Perhaps if the focus is on server instances in virtualized cloud environments with cloud storage, block devices and simple snapshots then these concerns many smaller users experience do not become apparent.
RSYNC is a perfectly reasonable way to backup a system that has no open DBs or files and that is how it has been used. Things change and that is often good, so long as the change continues to help. The idea of dealing better with the LDAP DB is a great idea, but why provision 80GB for all DBs and adversely affect other supporting processes that work in a wider application when there is no need.
That last point is the one I am trying to understand, is there a clear need to provision 80GB of DB space for something that is usually no where near that size. Is it not better to fit the purpose as needed and provision more appropriate sizes for the DB and avoid the hassle. I can't answer that which is why I wanted Quanah to explain the drive to provision the 80GB not say 1GB or 500MB. Can you explain that Quanah?
The file data.mdb reports its size as 80GB and that is not the actual used space, but it means to all FS based utilities it will be seen as 80GB. There is no checking for available space so that capacity may not be available even at creation and that will cause problems if it ever did get to 80GB or whenever the space available is consumed.
Can you explain Zimbra thinking here? I really just want to get this understood and move on but the logic is not yet working through to my headspace

EDIT

Quanah - quick thought we may be at crossed purpose - if you, yourself, stop all ldap services and cp the data.mdb file are you actually saying you think it will only copy say, 7MB data and not copy 80GB? From my tests it really does spend ages and copy 80GB of nothing and create a genuine 80GB file.

magnus.moren · Post by **magnus.moren** » Sat Nov 24, 2012 9:42 am

The correct method of backing up is explained in
Could">https://bugzilla.zimbra.com/show_bug.cgi?id=78781#c4.
Could this rsync option help to keep the size of the file not to be expanded ?
-S, --sparse

Try to handle sparse files efficiently so they take up less

space on the destination. Conflicts with --inplace because it’s

not possible to overwrite data in a sparse fashion.

kevindods · Post by **kevindods** » Sat Nov 24, 2012 11:28 am

I wonder if this is just a sparse file? normally a util like cp will detect that and work with it but this one seems a little different since cp sees it as a normal 80GB file.

winepress · Post by **winepress** » Sat Nov 24, 2012 12:12 pm

Dear Zimbra,
I think you're completely missing the point. Why on earth is a 5.5MB / 3.3MB / whatever size file being reported as 80GB? This thread feels like some sort of police interrogation where we're being made to believe that it's okay to cover up an apparent error, and if we're made to feel stupid enough, we'll believe it. This is not a professional or logical way to approach a problem.
Look, if the file size is 3.3MB, then it should be reported to the OS as the same. That's just common sense. Otherwise, if it really is an 80GB file, then just say so and make sure it's consistent among all other file utilities. 3.3MB does not equal 80GB, and our bosses and customers will not accept explanations like the one you're giving.
Secondly, best practice doesn't always equal real world. For instance, we have a mixed environment of FOSS and Commercial software. Our backup software is in fact commercial and will back up an entire linux box. We use this software to backup all of our machines in a best-practice method (disk-to-disk-to-tape-to-offsite-to-archive). However, our software doesn't have native Zimbra support, so it sees 80GB, not 3.3MB.
Our request here is that you work with OpenLDAP to come up with a viable solution. Again, consistency is key. For my business, this is a critical problem, and if I must have a static file size, I need some way to adjust it down to something more reasonable. My database is 4 years old and only 3.3MB. So to have it inflated to 80GB is unreasonable.
Regards,

Kevin

phoenix · Post by **phoenix** » Sat Nov 24, 2012 1:21 pm

[quote user="winepress"]Our request here is that you work with OpenLDAP to come up with a viable solution.[/QUOTE]I think you'll find that Zimbra is 'in touch' with OpenLDAP, check out the names of the Contributors to the OpenLDAP project. The solution you've been given for backing-up the mdb is a workable solution, it just needs incorporating into your current backup procedure for Zimbra.
[quote user="winepress"]So to have it inflated to 80GB is unreasonable.[/QUOTE]Your database is not inflated to 80GB, as has been mentioned before that is allocated space not used space.

winepress · Post by **winepress** » Sat Nov 24, 2012 2:03 pm

Greetings Bill,
[quote user="10330phoenix"]I think you'll find that Zimbra is 'in touch' with OpenLDAP, check out the names of the Contributors to the OpenLDAP project. The solution you've been given for backing-up the mdb is a workable solution, it just needs incorporating into your current backup procedure for Zimbra.[/QUOTE]
Agreed, it's a workable solution, but a breaking one for FOSS users. It should have been introduced in 8.0.0, the major release, not 8.0.1, a very minor release meant for bug fixes only. Revisioning systems like this were put in place so sysadmins can decide when to implement updates vs. upgrades.
[quote user="10330phoenix"]Your database is not inflated to 80GB, as has been mentioned before that is allocated space not used space.[/QUOTE]
With all due respect, you're not saying anything of substance here. If I own 5 acres of land, all of that land belongs to me, even though my house may only occupy 3000 sq ft. The rest of the world has no rights to my 5 acres, not just my house. So here, in real-world scenarios, I have to treat "allocated space" as used space. If I don't, I'd be extremely irresponsible and could bring harm to the data I'm supposed to protect.
This wouldn't be a problem if the default size wasn't so huge. And it appears to be one-way only. I'd be satisfied if I could change my "allocated space" down to 64 MB, but I couldn't find any way to do that. Do you know of a way?
Regards,

Kevin

quanah · Post by **quanah** » Mon Nov 26, 2012 2:04 pm

[quote user="winepress"]

This wouldn't be a problem if the default size wasn't so huge. And it appears to be one-way only. I'd be satisfied if I could change my "allocated space" down to 64 MB, but I couldn't find any way to do that. Do you know of a way?
Regards,

Kevin[/QUOTE]
It is trivial.
Do the following as the zimbra user:
Change the LC key for the max size of the database. Let zmconfigd push that to the OpenLDAP config DB (just watch zimbra.log).

After that is done, stop ldap, zmslapcat your DB, move /opt/zimbra/data/ldap/mdb aside, recreate /opt/zimbra/data/ldap/mdb/db

Re-run slapadd to reload your DB. It will now only show an allocation of MAXSIZE

Make sure maxsize is large enough to hold your DB + plenty of room for growth.

quanah · Post by **quanah** » Mon Nov 26, 2012 2:25 pm

[quote user="kevindods"]

What none of us seem to quite understand why the DB has 80GB provisioned when most DBs are a few hundred MB at most. If the DB was closer to expected maximums the issues we experienced would not arise at all. Perhaps if the focus is on server instances in virtualized cloud environments with cloud storage, block devices and simple snapshots then these concerns many smaller users experience do not become apparent.

[/QUOTE]
It is provisioned at 80GB because MDB has a maxsize limit. To allow upgrading from an unknown size, I had to choose some number that I was sure would allow anyone to upgrade without issue. I know there is no one who uses zimbra with an 80GB database.
[QUOTE]

RSYNC is a perfectly reasonable way to backup a system that has no open DBs or files and that is how it has been used. Things change and that is often good, so long as the change continues to help. The idea of dealing better with the LDAP DB is a great idea, but why provision 80GB for all DBs and adversely affect other supporting processes that work in a wider application when there is no need.

[/QUOTE]
Rsync is not, and never has been, a valid way of backing up the LDAP server. The fact that this is referenced in numerous location is a mistake that should be corrected. Using rsync is the wrong thing to do in any circumstance, regardless of the 80GB copying issue. In any case, there is nothing holding you to using the 80GB maximum data size. This is why there is a localconfig key in ZCS 8 that allows you to adjust the maxsize, and this is why that key is documented on the Zimbra Wiki for OpenLDAP tuning in ZCS 8.
[QUOTE]

That last point is the one I am trying to understand, is there a clear need to provision 80GB of DB space for something that is usually no where near that size. Is it not better to fit the purpose as needed and provision more appropriate sizes for the DB and avoid the hassle. I can't answer that which is why I wanted Quanah to explain the drive to provision the 80GB not say 1GB or 500MB. Can you explain that Quanah?

[/QUOTE]
As noted above -- We needed some maximum that we could ensure any migration would allow for. There are certainly installations with multi-gigabyte LDAP databases. I have a 16GB one from a customer I use for testing internally. I'm fairly positive no one has an 80GB size database.
Again, no one is forcing you to maintain the 80GB maximum, and the tuning key was explicitly added so that people who had smaller DBs could adjust for it accordingly. However, you must *always* keep your maxsize larger than your total database size, or data loss will occur.
--Quanah