Also read:
https://wiki.zimbra.com/wiki/OpenLDAP_P ... Tuning_8.0
ldap database went from 97meg to 86gig
ldap database went from 97meg to 86gig
--
Quanah Gibson-Mount
Product Architect, Symas http://www.symas.com/
OpenLDAP Core team http://www.openldap.org/project/
Quanah Gibson-Mount
Product Architect, Symas http://www.symas.com/
OpenLDAP Core team http://www.openldap.org/project/
ldap database went from 97meg to 86gig
Ok, the worst scenario has now happened to me. 
I hadn't noticed all that and just backed up my Zimbra server with duplicity like I always did. It doesn't handle sparse files well, unfortunately.
But as the backups are compressed, and such a sparse contains mainly zeros, there wasn't much of an increase in the backup sizes. It took longer than before, but I didn't think a lot about that.
Today the time has come that I would have had to restore a backup of my server. But that doesn't work as the complete storage size of the server is 60GB, so it isn't possible to write an 80 GB file there.
I'm now doing a completely new setup of Zimbra. Not many users here, so that's not so much of a problem; more so the question if and how I will be able to restore the users' emails from my backups (I'll try to find answers on that later when the server is up and running again).
I really wish there had been a BIG RED notice about that when I upgraded from 8.0.0 to 8.0.2. :-/
BTW, if the size of the sparse file is reduced and backed up using a method that isn't aware of sparse files – will the file still be valied when it is restored from a backup, now that it isn't a sparse file anymore?
I hadn't noticed all that and just backed up my Zimbra server with duplicity like I always did. It doesn't handle sparse files well, unfortunately.
But as the backups are compressed, and such a sparse contains mainly zeros, there wasn't much of an increase in the backup sizes. It took longer than before, but I didn't think a lot about that.
Today the time has come that I would have had to restore a backup of my server. But that doesn't work as the complete storage size of the server is 60GB, so it isn't possible to write an 80 GB file there.
I'm now doing a completely new setup of Zimbra. Not many users here, so that's not so much of a problem; more so the question if and how I will be able to restore the users' emails from my backups (I'll try to find answers on that later when the server is up and running again).
I really wish there had been a BIG RED notice about that when I upgraded from 8.0.0 to 8.0.2. :-/
BTW, if the size of the sparse file is reduced and backed up using a method that isn't aware of sparse files – will the file still be valied when it is restored from a backup, now that it isn't a sparse file anymore?
ldap database went from 97meg to 86gig
Yes. You can use mdb_copy to copy the database to its actual size, and back that up. Or use /opt/zimbra/libexec/zmslapcat to export the DB to LDIF.
If you have a backup with the 80GB file on it, you can still use mdb_copy of that data.mdb file to copy it to its actual size and restore it on your server...
If you have a backup with the 80GB file on it, you can still use mdb_copy of that data.mdb file to copy it to its actual size and restore it on your server...
--
Quanah Gibson-Mount
Product Architect, Symas http://www.symas.com/
OpenLDAP Core team http://www.openldap.org/project/
Quanah Gibson-Mount
Product Architect, Symas http://www.symas.com/
OpenLDAP Core team http://www.openldap.org/project/
ldap database went from 97meg to 86gig
You might also find https://wiki.zimbra.com/wiki/Importing_ ... to_replica useful.
--
Quanah Gibson-Mount
Product Architect, Symas http://www.symas.com/
OpenLDAP Core team http://www.openldap.org/project/
Quanah Gibson-Mount
Product Architect, Symas http://www.symas.com/
OpenLDAP Core team http://www.openldap.org/project/
ldap database went from 97meg to 86gig
Huh, where did the reply go that I posted here yesterday? Seems I forgot to post it as I could restore it as saved content in this reply box here.
Anyway, thanks for the tips.
With an Ubuntu VM on my Mac I managed to restore the backup, then used mdb_copy there, created a new full backup from this changed version and then restored that on my server, which is now back to normal operation.
And, of course, I now changed the maximum size of the ldap database to 64M as described farther up in the thread.
Phew.
Anyway, thanks for the tips.
With an Ubuntu VM on my Mac I managed to restore the backup, then used mdb_copy there, created a new full backup from this changed version and then restored that on my server, which is now back to normal operation.
And, of course, I now changed the maximum size of the ldap database to 64M as described farther up in the thread.
Phew.
ldap database went from 97meg to 86gig
I will note that changing the max size is an extremely bad idea, particularly with any release
You should fix your backup procedure to account for the sparse file, not break the max size.
You should fix your backup procedure to account for the sparse file, not break the max size.
--
Quanah Gibson-Mount
Product Architect, Symas http://www.symas.com/
OpenLDAP Core team http://www.openldap.org/project/
Quanah Gibson-Mount
Product Architect, Symas http://www.symas.com/
OpenLDAP Core team http://www.openldap.org/project/
ldap database went from 97meg to 86gig
[quote user="quanah"]I will note that changing the max size is an extremely bad idea, particularly with any release
You should fix your backup procedure to account for the sparse file, not break the max size.[/QUOTE]
Well serisously but i (and i beleive most foss users) will find this kinda offending.
Let me straight out some facts.
Zimbra name itself as a leading in Opensource collaboration - matter of fact - its not really opensource.
Zimbra refuses to give FOSS users the same backup NE have. Bringing out a Server Product - make it opensource - then refuse to give a backup
thats a joke itself, its a bad guys marketing strategy, its irresponsible and a big deal and a good reason why zimbra its even remotly that popular as exchange is.
We could go there deeper now but lets move on.
Now the only way to backup Zimbra in a quick way is to hot/cold backup it like with rsync or similar for foss.
anything else is NOT relyable - anykind of exporting / reimporting and stuff makes things more complicated and faulty - its defenetly not recommended, say what you want.
even better all this because you decide to allocate a file with 86 gig - this isnt a improvement - its a quick and dirty solution.
why not letting the setupscript determine the size and allocate double of current size ? not abig deal.
no instead of addmitting you made a very big, sloppy mistake you tell users to adjust their backup to this nonsense solution.
Nonsense because majority of servers never gonna reach even remotly close to that size and btw a daily maintance script could check DB size and adjust it.
and btw about the backup. iam running NE too. i do not use the zimbras backup solution - i simpyl dont.
simply because theres no real mechanism to send the backups straight offsite. without a proper errormanagement (like connection aborted and stuff like that) or integrity checks, email reports, extended sheduling, extended retention policys, encryption and so in its not a real backupsolution
only way to use it would be let it run as a prebackup script - then use a real backupsoftware to backup the backup which increases the size of needed diskspace
also not a real good solution.
thats why i backup foss and NE with our own software which compress/encrypt/delta on the fly and replicate backups over 2 datacenters. i need todo it offline (or better i do 5 hotbackups during the day and one cold one in the night)
So at the end the rsync or similar method is still the only real realistic option if you really need it relyable. so dont tell us we did wrong after you guys made in a MINOR VERSION such a big change - very sloppy one anyway.
cmon serisously minimum requirements- zimbra core files 10 gig - zimbra database a few gigs (depeding what users aer saving) - zimbra ldap 86gig - kiddn right ?
- and no that issue didnt hit me at all - i didnt upgrade yet - and i always have half a terra space there - still the way zimbra responded - the way zimbra threats foss users (who pay a lot in their time testing this buggy product, supporting and spreading it) is just disgusting.
and any programmer who things simply making a file ,which is average between 30 and 300meg, a 86gig file really should think about a new carrer - maybe fishermen or at blackwater - not even oracle would ever came up with such a number.
You should fix your backup procedure to account for the sparse file, not break the max size.[/QUOTE]
Well serisously but i (and i beleive most foss users) will find this kinda offending.
Let me straight out some facts.
Zimbra name itself as a leading in Opensource collaboration - matter of fact - its not really opensource.
Zimbra refuses to give FOSS users the same backup NE have. Bringing out a Server Product - make it opensource - then refuse to give a backup
thats a joke itself, its a bad guys marketing strategy, its irresponsible and a big deal and a good reason why zimbra its even remotly that popular as exchange is.
We could go there deeper now but lets move on.
Now the only way to backup Zimbra in a quick way is to hot/cold backup it like with rsync or similar for foss.
anything else is NOT relyable - anykind of exporting / reimporting and stuff makes things more complicated and faulty - its defenetly not recommended, say what you want.
even better all this because you decide to allocate a file with 86 gig - this isnt a improvement - its a quick and dirty solution.
why not letting the setupscript determine the size and allocate double of current size ? not abig deal.
no instead of addmitting you made a very big, sloppy mistake you tell users to adjust their backup to this nonsense solution.
Nonsense because majority of servers never gonna reach even remotly close to that size and btw a daily maintance script could check DB size and adjust it.
and btw about the backup. iam running NE too. i do not use the zimbras backup solution - i simpyl dont.
simply because theres no real mechanism to send the backups straight offsite. without a proper errormanagement (like connection aborted and stuff like that) or integrity checks, email reports, extended sheduling, extended retention policys, encryption and so in its not a real backupsolution
only way to use it would be let it run as a prebackup script - then use a real backupsoftware to backup the backup which increases the size of needed diskspace
also not a real good solution.
thats why i backup foss and NE with our own software which compress/encrypt/delta on the fly and replicate backups over 2 datacenters. i need todo it offline (or better i do 5 hotbackups during the day and one cold one in the night)
So at the end the rsync or similar method is still the only real realistic option if you really need it relyable. so dont tell us we did wrong after you guys made in a MINOR VERSION such a big change - very sloppy one anyway.
cmon serisously minimum requirements- zimbra core files 10 gig - zimbra database a few gigs (depeding what users aer saving) - zimbra ldap 86gig - kiddn right ?
- and no that issue didnt hit me at all - i didnt upgrade yet - and i always have half a terra space there - still the way zimbra responded - the way zimbra threats foss users (who pay a lot in their time testing this buggy product, supporting and spreading it) is just disgusting.
and any programmer who things simply making a file ,which is average between 30 and 300meg, a 86gig file really should think about a new carrer - maybe fishermen or at blackwater - not even oracle would ever came up with such a number.
ldap database went from 97meg to 86gig
I'm going to ignore your misguided rant, since you clearly haven't read all the details of this thread. I will however point you at the specific notes about MDB I made:
As">https://wiki.zimbra.com/wiki/OpenLDAP_P ... tes_on_MDB
As for the backup solution for LDAP, it's trivial and you have two choices with 8.x, where you only had one with 7.x and prior, which I see as an improvement.
a) You can use zmslapcat to make a hot backup of the DB in LDIF format. This is what NE uses for backups of LDAP, but as you note, it does require reloading the LDAP server.
b) You can use mdb_copy to make a hot backup of the database and rsync that. This avoids needing to do a reload of the LDAP server, but will be larger in size than the LDIF output.
Either way, rsync (for the LDAP server) is not a valid solution unless slapd is stopped, regardless of release. If you want to use rsync reliably with ZCS8, then you need to stop slapd, and use the -S (sparse files) option to rsync. With ZCS 7 and previous, you would need to stop slapd, run db_recover, and then rsync. Either way, you were doing invalid backups from the get go. Just because that finally caught up to you is not Zimbra's fault.
There was no "ill-advised" change between minor releases, either. If you install any of the ZCS 8 series, you will see that the maxsize has *always* been set at 80GB.
--Quanah
As">https://wiki.zimbra.com/wiki/OpenLDAP_P ... tes_on_MDB
As for the backup solution for LDAP, it's trivial and you have two choices with 8.x, where you only had one with 7.x and prior, which I see as an improvement.
a) You can use zmslapcat to make a hot backup of the DB in LDIF format. This is what NE uses for backups of LDAP, but as you note, it does require reloading the LDAP server.
b) You can use mdb_copy to make a hot backup of the database and rsync that. This avoids needing to do a reload of the LDAP server, but will be larger in size than the LDIF output.
Either way, rsync (for the LDAP server) is not a valid solution unless slapd is stopped, regardless of release. If you want to use rsync reliably with ZCS8, then you need to stop slapd, and use the -S (sparse files) option to rsync. With ZCS 7 and previous, you would need to stop slapd, run db_recover, and then rsync. Either way, you were doing invalid backups from the get go. Just because that finally caught up to you is not Zimbra's fault.
There was no "ill-advised" change between minor releases, either. If you install any of the ZCS 8 series, you will see that the maxsize has *always* been set at 80GB.
--Quanah
--
Quanah Gibson-Mount
Product Architect, Symas http://www.symas.com/
OpenLDAP Core team http://www.openldap.org/project/
Quanah Gibson-Mount
Product Architect, Symas http://www.symas.com/
OpenLDAP Core team http://www.openldap.org/project/
ldap database went from 97meg to 86gig
Or course shutting down the server is the only way to backup by rsync (or in my case a different backupsoftware filebased)
and it is actually still the only real relyable desaster recovery option for NE too.
after 13 years expiernce with such systems (like exchange) the mboxbackup option is never a guarantee and the way zimbra does is even worse because there no plugins for any backupsoftware outthere to backup directly on the fly. so its nessesary to backup within a mounted relyable storage option or on the same disks and then backup it again.
-
so thats the reason why the rsync (or similar) method is critical even for ne if you need desaster recovery.
the maxsize in 8.0 had no impact but in the minor update it has.
creating a sparsefile of 86 gig breaks easily any kind of backup - even with sparsefile option it will take way longer.
for example our offline (cold) backup is set to a maxtime of 4 minutes a day - which is plenty given we do hotbackup in advance to reduce the amout of data we need to backup offline.
point is - making a 86 sparsefile is a very bad idea - it has only impact at a minor upgrade which makes it even worse.
alone the pratice of doing that is undescribleable - period.
and any serisous developer will agree with that. there are plenty of choices todo that better.
more options ... and so on .. not gonna change the fact that you HARDCODED a 86 gig sparsefile
which could make issues in a lot of ways not only backups
and instead of saying ok we screwed up - no you says your users using a bad method of backups - you personally descriped promoting rsync as a backupmethod as a bad idea
this is simple not true - noone ever said rsync hotbackups are a good idea - its always - hotbackup to reduce the downtime - coldbackup for the real backup. which is a good way.
...
PS: just a sidenote - hotbackups of ldap wont do any good for any 3rd party backupmethod (rsync,..) anyway.
we all know if we work lowlevel (below zimbra api - so on system file and database level) we need a snapshot of mysql, ldapt and store at the same time
no real way of doing that relyable with any kind of hotbackup method except those working on the api level (zimbras maybe even xetras but i dont know their solution yet)
so promoting beeing abel to getting a hot copy of the ldap as a new feature - well it maybe needed for zimbra itself but for users i cant see any real world use for that. we always backup cold - or - as a vm snapshot
---
and btw: you never explained why it shoudl be a good idea to place a 86 gig file on users disks anyway.
you never explained why its not adjusted based on usage and check it every day or something like that. make it 10 times bigger than the actual size thats fine - but 86 gig means for a user with 200mb db its 430 time bigger as needed - dont sell us this crap as a good solution. it does make no sense.
specially if you configured your storage on different locations and have a relatively small zimbra main disk/storage - or for smal users like we have in this thread with a total of 60 gig space for the hole installation.
-
for me it would be crap either - we use a cluster both with mirrored raid arrays - backup that to another cluster of 3 machines each with mirrored disk - going into replication to another one with 2 machines.
so i end up with wasting 1.2 TB diskspace (all the mirrors counted) for a sparsefile - even i can copy them they are reserving diskspace - really make no sense
and it is actually still the only real relyable desaster recovery option for NE too.
after 13 years expiernce with such systems (like exchange) the mboxbackup option is never a guarantee and the way zimbra does is even worse because there no plugins for any backupsoftware outthere to backup directly on the fly. so its nessesary to backup within a mounted relyable storage option or on the same disks and then backup it again.
-
so thats the reason why the rsync (or similar) method is critical even for ne if you need desaster recovery.
the maxsize in 8.0 had no impact but in the minor update it has.
creating a sparsefile of 86 gig breaks easily any kind of backup - even with sparsefile option it will take way longer.
for example our offline (cold) backup is set to a maxtime of 4 minutes a day - which is plenty given we do hotbackup in advance to reduce the amout of data we need to backup offline.
point is - making a 86 sparsefile is a very bad idea - it has only impact at a minor upgrade which makes it even worse.
alone the pratice of doing that is undescribleable - period.
and any serisous developer will agree with that. there are plenty of choices todo that better.
more options ... and so on .. not gonna change the fact that you HARDCODED a 86 gig sparsefile
which could make issues in a lot of ways not only backups
and instead of saying ok we screwed up - no you says your users using a bad method of backups - you personally descriped promoting rsync as a backupmethod as a bad idea
this is simple not true - noone ever said rsync hotbackups are a good idea - its always - hotbackup to reduce the downtime - coldbackup for the real backup. which is a good way.
...
PS: just a sidenote - hotbackups of ldap wont do any good for any 3rd party backupmethod (rsync,..) anyway.
we all know if we work lowlevel (below zimbra api - so on system file and database level) we need a snapshot of mysql, ldapt and store at the same time
no real way of doing that relyable with any kind of hotbackup method except those working on the api level (zimbras maybe even xetras but i dont know their solution yet)
so promoting beeing abel to getting a hot copy of the ldap as a new feature - well it maybe needed for zimbra itself but for users i cant see any real world use for that. we always backup cold - or - as a vm snapshot
---
and btw: you never explained why it shoudl be a good idea to place a 86 gig file on users disks anyway.
you never explained why its not adjusted based on usage and check it every day or something like that. make it 10 times bigger than the actual size thats fine - but 86 gig means for a user with 200mb db its 430 time bigger as needed - dont sell us this crap as a good solution. it does make no sense.
specially if you configured your storage on different locations and have a relatively small zimbra main disk/storage - or for smal users like we have in this thread with a total of 60 gig space for the hole installation.
-
for me it would be crap either - we use a cluster both with mirrored raid arrays - backup that to another cluster of 3 machines each with mirrored disk - going into replication to another one with 2 machines.
so i end up with wasting 1.2 TB diskspace (all the mirrors counted) for a sparsefile - even i can copy them they are reserving diskspace - really make no sense
ldap database went from 97meg to 86gig
The problem is, I don't have any indication of what size and end-user's DB will be when upgrading from ZCS
What you should really be asking, is why do I advise against making it smaller than 80GB in ZCS
In">https://bugzilla.zimbra.com/show_bug.cgi?id=80141
In any case, whether or not you are doing hot or cold backups via rsync, it has always been required to take steps before starting the rsync for it to be valid. Again, with ZCS7 or prior, one needs to run db_recover in the LDAP DB directory prior to starting rsync. In ZCS8, if you want to avoid rsyncing the sparse file, then use mdb_copy to copy the DB to a non-sparse file first, and then rsync that. It is a trivial solution that shouldn't take more than 5 minutes of your time to script.
--Quanah
What you should really be asking, is why do I advise against making it smaller than 80GB in ZCS
In">https://bugzilla.zimbra.com/show_bug.cgi?id=80141
In any case, whether or not you are doing hot or cold backups via rsync, it has always been required to take steps before starting the rsync for it to be valid. Again, with ZCS7 or prior, one needs to run db_recover in the LDAP DB directory prior to starting rsync. In ZCS8, if you want to avoid rsyncing the sparse file, then use mdb_copy to copy the DB to a non-sparse file first, and then rsync that. It is a trivial solution that shouldn't take more than 5 minutes of your time to script.
--Quanah
--
Quanah Gibson-Mount
Product Architect, Symas http://www.symas.com/
OpenLDAP Core team http://www.openldap.org/project/
Quanah Gibson-Mount
Product Architect, Symas http://www.symas.com/
OpenLDAP Core team http://www.openldap.org/project/