shrink the data.mdb sparse file

Discuss your pilot or production implementation with other Zimbra admins or our engineers.
User avatar
wentum
Advanced member
Advanced member
Posts: 53
Joined: Fri Apr 04, 2014 10:49 am
Location: Pforzheim (Germany)
ZCS/ZD Version: Release 9.0.0.GA.3924 _P30
Contact:

shrink the data.mdb sparse file

Post by wentum »

Hello folks,

first of all a happy new year to all of you!

Having some spare time recently I decided to deal with this 80GB sparse file data.mdb.
The goal is to generate a sparse file much smaller to make veeam recovery possible again (don't want to keep 80GB empty file system on every zimbra instance...).
Let me also mention that we're here only dealing with realy small zimbra deployments, so real data in data.mdb is 50MB max!

I read those tuning tips in the wiki and changed parameters ldap_db_maxsize, ldap_accesslog_maxsize to 5368709120.
But this won't change the sparse file though it is said here https://cloudsubversion.wordpress.com/2 ... file-size/

OK, most things in life don't work automagically so i tried to backup data.mdb and regenerate it using syslint's way https://syslint.com/blog/tutorial/solve ... in-zimbra/
But surprise, sparse file ist still 80GB!

Can you give me some hints!?

Regards
Joerg

Zimra is Release 8.8.15.GA.3829.UBUNTU14.64 UBUNTU14_64 NETWORK edition, Patch 8.8.15_P17
ghen
Outstanding Member
Outstanding Member
Posts: 258
Joined: Thu May 12, 2016 1:56 pm
Location: Belgium
ZCS/ZD Version: 9.0.0

Re: shrink the data.mdb sparse file

Post by ghen »

You don't actually need 80 GB of filesystem space to host this file, it is just a max addressable size. You can see the actual size with `du`.

If you want a non-sparse copy eg. for backup purposes, use /opt/zimbra/common/bin/mdb_copy (optionally with -c to skip free pages).
User avatar
wentum
Advanced member
Advanced member
Posts: 53
Joined: Fri Apr 04, 2014 10:49 am
Location: Pforzheim (Germany)
ZCS/ZD Version: Release 9.0.0.GA.3924 _P30
Contact:

Re: shrink the data.mdb sparse file

Post by wentum »

ghen wrote:You don't actually need 80 GB of filesystem space to host this file, it is just a max addressable size. You can see the actual size with `du`.

If you want a non-sparse copy eg. for backup purposes, use /opt/zimbra/common/bin/mdb_copy (optionally with -c to skip free pages).
Hi ghen,
I am aware that data.mdb is a sparse file and is 'under nomal circumstances' not 80GB big... but Veeam (and other backup systems) can't handle sparse files correctly. So when it comes to recovery veeam will recovery a regular file (not a sparse file) that will be 80GB big! And when there is no 80GB free space on your /opt/zimbra filesystem you will be out of luck and recovery will fail!

So my wish is to generate a let's say 5GB sparse file now (for our small deployments this is by fare enough) and when it comes to recovery veeam will generate a 5GB regular file that will probably fit onto my filesystem. Then after recovery I think I can change that regular file back into a sparse one in a second step.

Hope this clarifies what the problem is and what I intend to achive.

Regards
Joerg
bulletxt
Advanced member
Advanced member
Posts: 81
Joined: Sat Sep 13, 2014 1:08 am

Re: shrink the data.mdb sparse file

Post by bulletxt »

you might use the mdb_copy command: https://wiki.zimbra.com/wiki/OpenLDAP_P ... Tuning_8.0
liverpoolfcfan
Elite member
Elite member
Posts: 1096
Joined: Sat Sep 13, 2014 12:47 am

Re: shrink the data.mdb sparse file

Post by liverpoolfcfan »

bulletxt wrote:you might use the mdb_copy command: https://wiki.zimbra.com/wiki/OpenLDAP_P ... Tuning_8.0
That document says "Note: The copied MDB file will *not* be a sparse file. It will be the actual size of the database." so I don't think it will help
User avatar
L. Mark Stone
Ambassador
Ambassador
Posts: 2796
Joined: Wed Oct 09, 2013 11:35 am
Location: Portland, Maine, US
ZCS/ZD Version: 10.0.6 Network Edition
Contact:

Re: shrink the data.mdb sparse file

Post by L. Mark Stone »

The problem here is Veeam, which is ignorant of sparse files and upon restore will create an ordinary 80GB file when it restores the LDAP directory.

From experience I have found, as wonderful a product as Veeam is, it is inappropriate for Zimbra live system backups, as are all so-called "crash-consistent" backup tools.

Why?

Because Zimbra has a number of databases that keep their working set in RAM, which Veeam does not backup.

Consequently, even if the restored server actually starts (it doesn't reliably), MariaDB will roll back a lot of transactions (so you'll have missing and orphaned blobs); some mailbox indexes will be corrupted; LDAP may not start, (or in the case of replication, replication may be non-functional on restore), and; you may/will have a host of other problems related to losing the "harmony" between LDAP, MariaDB and the blobs on disk.

I love Veeam... but I had a long talk with a senior Veeam engineer at the 2018 AWS Summit in NYC about this, and he agreed that Veeam, indeed any "crash-consistent" backup tool, is not a good Zimbra backup nor Disaster Recovery solution.

Hope that helps,
Mark
___________________________________
L. Mark Stone
Mission Critical Email - Zimbra VAR/BSP/Training Partner https://www.missioncriticalemail.com/
AWS Certified Solutions Architect-Associate
User avatar
wentum
Advanced member
Advanced member
Posts: 53
Joined: Fri Apr 04, 2014 10:49 am
Location: Pforzheim (Germany)
ZCS/ZD Version: Release 9.0.0.GA.3924 _P30
Contact:

Re: shrink the data.mdb sparse file

Post by wentum »

L. Mark Stone wrote:The problem here is Veeam, which is ignorant of sparse files and upon restore will create an ordinary 80GB file when it restores the LDAP directory.

From experience I have found, as wonderful a product as Veeam is, it is inappropriate for Zimbra live system backups, as are all so-called "crash-consistent" backup tools.

Why?

Because Zimbra has a number of databases that keep their working set in RAM, which Veeam does not backup.

Consequently, even if the restored server actually starts (it doesn't reliably), MariaDB will roll back a lot of transactions (so you'll have missing and orphaned blobs); some mailbox indexes will be corrupted; LDAP may not start, (or in the case of replication, replication may be non-functional on restore), and; you may/will have a host of other problems related to losing the "harmony" between LDAP, MariaDB and the blobs on disk.

I love Veeam... but I had a long talk with a senior Veeam engineer at the 2018 AWS Summit in NYC about this, and he agreed that Veeam, indeed any "crash-consistent" backup tool, is not a good Zimbra backup nor Disaster Recovery solution.

Hope that helps,
Mark
Hi Mark,

to be honest it was your briliant article here https://www.missioncriticalemail.com/20 ... reference/ that made me aware of the Veeam (and others...) problem. So, thanks a lot for that blog post!

First of all our customers are allmost all Network Edition Customer so we do have a valid backup... but I'm a little paranoid...(it's getting worse the older I become) I DO REALY like second chances/options.
Second is those servers are quite small and internal ones, meaning I can put those Zimbra systems down for backup purposes and then I think veeam is an option again!
Besides of this sparse file problem...

The offical zimbra documentation here https://wiki.zimbra.com/wiki/OpenLDAP_P ... Tuning_8.0 says the sparse file can be shrinked (" If it is desired to reduce the maxsize of the database to trim down the size of the sparse file, this can be controlled via the following two localconfig keys: ldap_db_maxsize ldap_accesslog_maxsize"). But that didn't work out...
So from my point of view either the certified documentation is wrong or it's a a bug...

Regards
Joerg
User avatar
wentum
Advanced member
Advanced member
Posts: 53
Joined: Fri Apr 04, 2014 10:49 am
Location: Pforzheim (Germany)
ZCS/ZD Version: Release 9.0.0.GA.3924 _P30
Contact:

Re: shrink the data.mdb sparse file

Post by wentum »

Hi folks,
small update from my side.

Did some backup and recovery testing over the last few days...

Testing environment:
- Zimbra 8.8.15P17 (on Ubuntu 14.04 LTS 64bit, ext4 filesytem, virtual machine)
- Vsphere 6.5 U3
- Veeam Backup&Recovery 9.5 U4
- Synology Active Backup for Business V 2.1.1-1125

Testing process:
- Backup running Zimbra VM (vmware quiescence enabled)
- Restoring Zimbra VM
- Check the results

Results:
- Both solutions restored Zimbra VM without issues
- After both restores the data.mdb sparse file was STILL a sparse file
- all Zimbra services were starting without issues after both restores

Conclusion:
- you can use Veeam and Synology Active Backup for Business to backup a Zimbra VM
- the sparse file will be a sparse file after restore
- you should stop zimbra before you backup your VM if you can. Otherwise you could get into trouble because of databases in RAM as Mark explained here viewtopic.php?f=15&t=69129#p300150

HTH
Joerg
gcakici2
Posts: 4
Joined: Mon Jan 18, 2021 9:45 pm

Re: shrink the data.mdb sparse file

Post by gcakici2 »

Hi Wentum, you already mention After both restores the data.mdb sparse file was STILL a sparse file but have you successfully change the size of the file?

If you can, is it showing the new resized size when you ls -l ?

Thanks/Gokalp
User avatar
wentum
Advanced member
Advanced member
Posts: 53
Joined: Fri Apr 04, 2014 10:49 am
Location: Pforzheim (Germany)
ZCS/ZD Version: Release 9.0.0.GA.3924 _P30
Contact:

Re: shrink the data.mdb sparse file

Post by wentum »

gcakici2 wrote:Hi Wentum, you already mention After both restores the data.mdb sparse file was STILL a sparse file but have you successfully change the size of the file?

If you can, is it showing the new resized size when you ls -l ?

Thanks/Gokalp
Hello gcakici2,

no I actually did not succeed in shrinking the sparse file yet. It is still 80GB big no matter what I put in ldap_db_maxsize and ldap_accesslog_maxsize. I fear that these parameters do refer only to the logging/messaging part of zimbra and have nothing to do with file sizes. But this is a guess.
So, yes I'm still looking for a method to shrink the sparse file but the pressure isn't as high as before...
Very disappointing that verfied zimbra documentation is wrong (see my reply to Mark on Jan, 13th 2021)!

Maybe Jorge de la Cruz can give us a hint, he's the one named there as last person who updated the article...

Regards
Joerg
Post Reply