shrink the data.mdb sparse file
- wentum
- Advanced member
- Posts: 53
- Joined: Fri Apr 04, 2014 10:49 am
- Location: Pforzheim (Germany)
- ZCS/ZD Version: Release 9.0.0.GA.3924 _P30
- Contact:
shrink the data.mdb sparse file
Hello folks,
first of all a happy new year to all of you!
Having some spare time recently I decided to deal with this 80GB sparse file data.mdb.
The goal is to generate a sparse file much smaller to make veeam recovery possible again (don't want to keep 80GB empty file system on every zimbra instance...).
Let me also mention that we're here only dealing with realy small zimbra deployments, so real data in data.mdb is 50MB max!
I read those tuning tips in the wiki and changed parameters ldap_db_maxsize, ldap_accesslog_maxsize to 5368709120.
But this won't change the sparse file though it is said here https://cloudsubversion.wordpress.com/2 ... file-size/
OK, most things in life don't work automagically so i tried to backup data.mdb and regenerate it using syslint's way https://syslint.com/blog/tutorial/solve ... in-zimbra/
But surprise, sparse file ist still 80GB!
Can you give me some hints!?
Regards
Joerg
Zimra is Release 8.8.15.GA.3829.UBUNTU14.64 UBUNTU14_64 NETWORK edition, Patch 8.8.15_P17
first of all a happy new year to all of you!
Having some spare time recently I decided to deal with this 80GB sparse file data.mdb.
The goal is to generate a sparse file much smaller to make veeam recovery possible again (don't want to keep 80GB empty file system on every zimbra instance...).
Let me also mention that we're here only dealing with realy small zimbra deployments, so real data in data.mdb is 50MB max!
I read those tuning tips in the wiki and changed parameters ldap_db_maxsize, ldap_accesslog_maxsize to 5368709120.
But this won't change the sparse file though it is said here https://cloudsubversion.wordpress.com/2 ... file-size/
OK, most things in life don't work automagically so i tried to backup data.mdb and regenerate it using syslint's way https://syslint.com/blog/tutorial/solve ... in-zimbra/
But surprise, sparse file ist still 80GB!
Can you give me some hints!?
Regards
Joerg
Zimra is Release 8.8.15.GA.3829.UBUNTU14.64 UBUNTU14_64 NETWORK edition, Patch 8.8.15_P17
-
- Outstanding Member
- Posts: 258
- Joined: Thu May 12, 2016 1:56 pm
- Location: Belgium
- ZCS/ZD Version: 9.0.0
Re: shrink the data.mdb sparse file
You don't actually need 80 GB of filesystem space to host this file, it is just a max addressable size. You can see the actual size with `du`.
If you want a non-sparse copy eg. for backup purposes, use /opt/zimbra/common/bin/mdb_copy (optionally with -c to skip free pages).
If you want a non-sparse copy eg. for backup purposes, use /opt/zimbra/common/bin/mdb_copy (optionally with -c to skip free pages).
- wentum
- Advanced member
- Posts: 53
- Joined: Fri Apr 04, 2014 10:49 am
- Location: Pforzheim (Germany)
- ZCS/ZD Version: Release 9.0.0.GA.3924 _P30
- Contact:
Re: shrink the data.mdb sparse file
Hi ghen,ghen wrote:You don't actually need 80 GB of filesystem space to host this file, it is just a max addressable size. You can see the actual size with `du`.
If you want a non-sparse copy eg. for backup purposes, use /opt/zimbra/common/bin/mdb_copy (optionally with -c to skip free pages).
I am aware that data.mdb is a sparse file and is 'under nomal circumstances' not 80GB big... but Veeam (and other backup systems) can't handle sparse files correctly. So when it comes to recovery veeam will recovery a regular file (not a sparse file) that will be 80GB big! And when there is no 80GB free space on your /opt/zimbra filesystem you will be out of luck and recovery will fail!
So my wish is to generate a let's say 5GB sparse file now (for our small deployments this is by fare enough) and when it comes to recovery veeam will generate a 5GB regular file that will probably fit onto my filesystem. Then after recovery I think I can change that regular file back into a sparse one in a second step.
Hope this clarifies what the problem is and what I intend to achive.
Regards
Joerg
Re: shrink the data.mdb sparse file
you might use the mdb_copy command: https://wiki.zimbra.com/wiki/OpenLDAP_P ... Tuning_8.0
-
- Elite member
- Posts: 1096
- Joined: Sat Sep 13, 2014 12:47 am
Re: shrink the data.mdb sparse file
That document says "Note: The copied MDB file will *not* be a sparse file. It will be the actual size of the database." so I don't think it will helpbulletxt wrote:you might use the mdb_copy command: https://wiki.zimbra.com/wiki/OpenLDAP_P ... Tuning_8.0
- L. Mark Stone
- Ambassador
- Posts: 2796
- Joined: Wed Oct 09, 2013 11:35 am
- Location: Portland, Maine, US
- ZCS/ZD Version: 10.0.6 Network Edition
- Contact:
Re: shrink the data.mdb sparse file
The problem here is Veeam, which is ignorant of sparse files and upon restore will create an ordinary 80GB file when it restores the LDAP directory.
From experience I have found, as wonderful a product as Veeam is, it is inappropriate for Zimbra live system backups, as are all so-called "crash-consistent" backup tools.
Why?
Because Zimbra has a number of databases that keep their working set in RAM, which Veeam does not backup.
Consequently, even if the restored server actually starts (it doesn't reliably), MariaDB will roll back a lot of transactions (so you'll have missing and orphaned blobs); some mailbox indexes will be corrupted; LDAP may not start, (or in the case of replication, replication may be non-functional on restore), and; you may/will have a host of other problems related to losing the "harmony" between LDAP, MariaDB and the blobs on disk.
I love Veeam... but I had a long talk with a senior Veeam engineer at the 2018 AWS Summit in NYC about this, and he agreed that Veeam, indeed any "crash-consistent" backup tool, is not a good Zimbra backup nor Disaster Recovery solution.
Hope that helps,
Mark
From experience I have found, as wonderful a product as Veeam is, it is inappropriate for Zimbra live system backups, as are all so-called "crash-consistent" backup tools.
Why?
Because Zimbra has a number of databases that keep their working set in RAM, which Veeam does not backup.
Consequently, even if the restored server actually starts (it doesn't reliably), MariaDB will roll back a lot of transactions (so you'll have missing and orphaned blobs); some mailbox indexes will be corrupted; LDAP may not start, (or in the case of replication, replication may be non-functional on restore), and; you may/will have a host of other problems related to losing the "harmony" between LDAP, MariaDB and the blobs on disk.
I love Veeam... but I had a long talk with a senior Veeam engineer at the 2018 AWS Summit in NYC about this, and he agreed that Veeam, indeed any "crash-consistent" backup tool, is not a good Zimbra backup nor Disaster Recovery solution.
Hope that helps,
Mark
___________________________________
L. Mark Stone
Mission Critical Email - Zimbra VAR/BSP/Training Partner https://www.missioncriticalemail.com/
AWS Certified Solutions Architect-Associate
L. Mark Stone
Mission Critical Email - Zimbra VAR/BSP/Training Partner https://www.missioncriticalemail.com/
AWS Certified Solutions Architect-Associate
- wentum
- Advanced member
- Posts: 53
- Joined: Fri Apr 04, 2014 10:49 am
- Location: Pforzheim (Germany)
- ZCS/ZD Version: Release 9.0.0.GA.3924 _P30
- Contact:
Re: shrink the data.mdb sparse file
Hi Mark,L. Mark Stone wrote:The problem here is Veeam, which is ignorant of sparse files and upon restore will create an ordinary 80GB file when it restores the LDAP directory.
From experience I have found, as wonderful a product as Veeam is, it is inappropriate for Zimbra live system backups, as are all so-called "crash-consistent" backup tools.
Why?
Because Zimbra has a number of databases that keep their working set in RAM, which Veeam does not backup.
Consequently, even if the restored server actually starts (it doesn't reliably), MariaDB will roll back a lot of transactions (so you'll have missing and orphaned blobs); some mailbox indexes will be corrupted; LDAP may not start, (or in the case of replication, replication may be non-functional on restore), and; you may/will have a host of other problems related to losing the "harmony" between LDAP, MariaDB and the blobs on disk.
I love Veeam... but I had a long talk with a senior Veeam engineer at the 2018 AWS Summit in NYC about this, and he agreed that Veeam, indeed any "crash-consistent" backup tool, is not a good Zimbra backup nor Disaster Recovery solution.
Hope that helps,
Mark
to be honest it was your briliant article here https://www.missioncriticalemail.com/20 ... reference/ that made me aware of the Veeam (and others...) problem. So, thanks a lot for that blog post!
First of all our customers are allmost all Network Edition Customer so we do have a valid backup... but I'm a little paranoid...(it's getting worse the older I become) I DO REALY like second chances/options.
Second is those servers are quite small and internal ones, meaning I can put those Zimbra systems down for backup purposes and then I think veeam is an option again!
Besides of this sparse file problem...
The offical zimbra documentation here https://wiki.zimbra.com/wiki/OpenLDAP_P ... Tuning_8.0 says the sparse file can be shrinked (" If it is desired to reduce the maxsize of the database to trim down the size of the sparse file, this can be controlled via the following two localconfig keys: ldap_db_maxsize ldap_accesslog_maxsize"). But that didn't work out...
So from my point of view either the certified documentation is wrong or it's a a bug...
Regards
Joerg
- wentum
- Advanced member
- Posts: 53
- Joined: Fri Apr 04, 2014 10:49 am
- Location: Pforzheim (Germany)
- ZCS/ZD Version: Release 9.0.0.GA.3924 _P30
- Contact:
Re: shrink the data.mdb sparse file
Hi folks,
small update from my side.
Did some backup and recovery testing over the last few days...
Testing environment:
- Zimbra 8.8.15P17 (on Ubuntu 14.04 LTS 64bit, ext4 filesytem, virtual machine)
- Vsphere 6.5 U3
- Veeam Backup&Recovery 9.5 U4
- Synology Active Backup for Business V 2.1.1-1125
Testing process:
- Backup running Zimbra VM (vmware quiescence enabled)
- Restoring Zimbra VM
- Check the results
Results:
- Both solutions restored Zimbra VM without issues
- After both restores the data.mdb sparse file was STILL a sparse file
- all Zimbra services were starting without issues after both restores
Conclusion:
- you can use Veeam and Synology Active Backup for Business to backup a Zimbra VM
- the sparse file will be a sparse file after restore
- you should stop zimbra before you backup your VM if you can. Otherwise you could get into trouble because of databases in RAM as Mark explained here viewtopic.php?f=15&t=69129#p300150
HTH
Joerg
small update from my side.
Did some backup and recovery testing over the last few days...
Testing environment:
- Zimbra 8.8.15P17 (on Ubuntu 14.04 LTS 64bit, ext4 filesytem, virtual machine)
- Vsphere 6.5 U3
- Veeam Backup&Recovery 9.5 U4
- Synology Active Backup for Business V 2.1.1-1125
Testing process:
- Backup running Zimbra VM (vmware quiescence enabled)
- Restoring Zimbra VM
- Check the results
Results:
- Both solutions restored Zimbra VM without issues
- After both restores the data.mdb sparse file was STILL a sparse file
- all Zimbra services were starting without issues after both restores
Conclusion:
- you can use Veeam and Synology Active Backup for Business to backup a Zimbra VM
- the sparse file will be a sparse file after restore
- you should stop zimbra before you backup your VM if you can. Otherwise you could get into trouble because of databases in RAM as Mark explained here viewtopic.php?f=15&t=69129#p300150
HTH
Joerg
Re: shrink the data.mdb sparse file
Hi Wentum, you already mention After both restores the data.mdb sparse file was STILL a sparse file but have you successfully change the size of the file?
If you can, is it showing the new resized size when you ls -l ?
Thanks/Gokalp
If you can, is it showing the new resized size when you ls -l ?
Thanks/Gokalp
- wentum
- Advanced member
- Posts: 53
- Joined: Fri Apr 04, 2014 10:49 am
- Location: Pforzheim (Germany)
- ZCS/ZD Version: Release 9.0.0.GA.3924 _P30
- Contact:
Re: shrink the data.mdb sparse file
Hello gcakici2,gcakici2 wrote:Hi Wentum, you already mention After both restores the data.mdb sparse file was STILL a sparse file but have you successfully change the size of the file?
If you can, is it showing the new resized size when you ls -l ?
Thanks/Gokalp
no I actually did not succeed in shrinking the sparse file yet. It is still 80GB big no matter what I put in ldap_db_maxsize and ldap_accesslog_maxsize. I fear that these parameters do refer only to the logging/messaging part of zimbra and have nothing to do with file sizes. But this is a guess.
So, yes I'm still looking for a method to shrink the sparse file but the pressure isn't as high as before...
Very disappointing that verfied zimbra documentation is wrong (see my reply to Mark on Jan, 13th 2021)!
Maybe Jorge de la Cruz can give us a hint, he's the one named there as last person who updated the article...
Regards
Joerg