shrink the data.mdb sparse file

Discuss your pilot or production implementation with other Zimbra admins or our engineers.
wentum
Posts: 24
Joined: Fri Apr 04, 2014 10:49 am

shrink the data.mdb sparse file

Postby wentum » Fri Jan 08, 2021 9:51 am

Hello folks,

first of all a happy new year to all of you!

Having some spare time recently I decided to deal with this 80GB sparse file data.mdb.
The goal is to generate a sparse file much smaller to make veeam recovery possible again (don't want to keep 80GB empty file system on every zimbra instance...).
Let me also mention that we're here only dealing with realy small zimbra deployments, so real data in data.mdb is 50MB max!

I read those tuning tips in the wiki and changed parameters ldap_db_maxsize, ldap_accesslog_maxsize to 5368709120.
But this won't change the sparse file though it is said here https://cloudsubversion.wordpress.com/2015/06/18/changeshrink-zimbra-ldap-db-file-size/

OK, most things in life don't work automagically so i tried to backup data.mdb and regenerate it using syslint's way https://syslint.com/blog/tutorial/solved-critical-ldap-primary-mdb-database-is-90-full-in-zimbra/
But surprise, sparse file ist still 80GB!

Can you give me some hints!?

Regards
Joerg

Zimra is Release 8.8.15.GA.3829.UBUNTU14.64 UBUNTU14_64 NETWORK edition, Patch 8.8.15_P17


ghen
Posts: 19
Joined: Thu May 12, 2016 1:56 pm
Location: Belgium

Re: shrink the data.mdb sparse file

Postby ghen » Fri Jan 08, 2021 12:30 pm

You don't actually need 80 GB of filesystem space to host this file, it is just a max addressable size. You can see the actual size with `du`.

If you want a non-sparse copy eg. for backup purposes, use /opt/zimbra/common/bin/mdb_copy (optionally with -c to skip free pages).
wentum
Posts: 24
Joined: Fri Apr 04, 2014 10:49 am

Re: shrink the data.mdb sparse file

Postby wentum » Fri Jan 08, 2021 3:31 pm

ghen wrote:You don't actually need 80 GB of filesystem space to host this file, it is just a max addressable size. You can see the actual size with `du`.

If you want a non-sparse copy eg. for backup purposes, use /opt/zimbra/common/bin/mdb_copy (optionally with -c to skip free pages).


Hi ghen,
I am aware that data.mdb is a sparse file and is 'under nomal circumstances' not 80GB big... but Veeam (and other backup systems) can't handle sparse files correctly. So when it comes to recovery veeam will recovery a regular file (not a sparse file) that will be 80GB big! And when there is no 80GB free space on your /opt/zimbra filesystem you will be out of luck and recovery will fail!

So my wish is to generate a let's say 5GB sparse file now (for our small deployments this is by fare enough) and when it comes to recovery veeam will generate a 5GB regular file that will probably fit onto my filesystem. Then after recovery I think I can change that regular file back into a sparse one in a second step.

Hope this clarifies what the problem is and what I intend to achive.

Regards
Joerg
bulletxt
Posts: 37
Joined: Sat Sep 13, 2014 1:08 am

Re: shrink the data.mdb sparse file

Postby bulletxt » Sat Jan 09, 2021 12:34 pm

you might use the mdb_copy command: https://wiki.zimbra.com/wiki/OpenLDAP_P ... Tuning_8.0
liverpoolfcfan
Outstanding Member
Outstanding Member
Posts: 957
Joined: Sat Sep 13, 2014 12:47 am

Re: shrink the data.mdb sparse file

Postby liverpoolfcfan » Sat Jan 09, 2021 1:06 pm

bulletxt wrote:you might use the mdb_copy command: https://wiki.zimbra.com/wiki/OpenLDAP_P ... Tuning_8.0

That document says "Note: The copied MDB file will *not* be a sparse file. It will be the actual size of the database." so I don't think it will help
User avatar
L. Mark Stone
Elite member
Elite member
Posts: 2215
Joined: Wed Oct 09, 2013 11:35 am
Location: Portland, Maine, US
ZCS/ZD Version: 8.8.15 Network Edition
Contact:

Re: shrink the data.mdb sparse file

Postby L. Mark Stone » Mon Jan 11, 2021 3:33 pm

The problem here is Veeam, which is ignorant of sparse files and upon restore will create an ordinary 80GB file when it restores the LDAP directory.

From experience I have found, as wonderful a product as Veeam is, it is inappropriate for Zimbra live system backups, as are all so-called "crash-consistent" backup tools.

Why?

Because Zimbra has a number of databases that keep their working set in RAM, which Veeam does not backup.

Consequently, even if the restored server actually starts (it doesn't reliably), MariaDB will roll back a lot of transactions (so you'll have missing and orphaned blobs); some mailbox indexes will be corrupted; LDAP may not start, (or in the case of replication, replication may be non-functional on restore), and; you may/will have a host of other problems related to losing the "harmony" between LDAP, MariaDB and the blobs on disk.

I love Veeam... but I had a long talk with a senior Veeam engineer at the 2018 AWS Summit in NYC about this, and he agreed that Veeam, indeed any "crash-consistent" backup tool, is not a good Zimbra backup nor Disaster Recovery solution.

Hope that helps,
Mark
___________________________________
L. Mark Stone
Mission Critical Email - Zimbra VAR/BSP/Training Partner https://www.missioncriticalemail.com/
Zeta Alliance http://www.zetalliance.org/
wentum
Posts: 24
Joined: Fri Apr 04, 2014 10:49 am

Re: shrink the data.mdb sparse file

Postby wentum » Wed Jan 13, 2021 10:51 am

L. Mark Stone wrote:The problem here is Veeam, which is ignorant of sparse files and upon restore will create an ordinary 80GB file when it restores the LDAP directory.

From experience I have found, as wonderful a product as Veeam is, it is inappropriate for Zimbra live system backups, as are all so-called "crash-consistent" backup tools.

Why?

Because Zimbra has a number of databases that keep their working set in RAM, which Veeam does not backup.

Consequently, even if the restored server actually starts (it doesn't reliably), MariaDB will roll back a lot of transactions (so you'll have missing and orphaned blobs); some mailbox indexes will be corrupted; LDAP may not start, (or in the case of replication, replication may be non-functional on restore), and; you may/will have a host of other problems related to losing the "harmony" between LDAP, MariaDB and the blobs on disk.

I love Veeam... but I had a long talk with a senior Veeam engineer at the 2018 AWS Summit in NYC about this, and he agreed that Veeam, indeed any "crash-consistent" backup tool, is not a good Zimbra backup nor Disaster Recovery solution.

Hope that helps,
Mark


Hi Mark,

to be honest it was your briliant article here https://www.missioncriticalemail.com/2020/11/16/zimbra-aws-architectural-reference/ that made me aware of the Veeam (and others...) problem. So, thanks a lot for that blog post!

First of all our customers are allmost all Network Edition Customer so we do have a valid backup... but I'm a little paranoid...(it's getting worse the older I become) I DO REALY like second chances/options.
Second is those servers are quite small and internal ones, meaning I can put those Zimbra systems down for backup purposes and then I think veeam is an option again!
Besides of this sparse file problem...

The offical zimbra documentation here https://wiki.zimbra.com/wiki/OpenLDAP_Performance_Tuning_8.0 says the sparse file can be shrinked (" If it is desired to reduce the maxsize of the database to trim down the size of the sparse file, this can be controlled via the following two localconfig keys: ldap_db_maxsize ldap_accesslog_maxsize"). But that didn't work out...
So from my point of view either the certified documentation is wrong or it's a a bug...

Regards
Joerg
wentum
Posts: 24
Joined: Fri Apr 04, 2014 10:49 am

Re: shrink the data.mdb sparse file

Postby wentum » Fri Jan 15, 2021 10:07 am

Hi folks,
small update from my side.

Did some backup and recovery testing over the last few days...

Testing environment:
- Zimbra 8.8.15P17 (on Ubuntu 14.04 LTS 64bit, ext4 filesytem, virtual machine)
- Vsphere 6.5 U3
- Veeam Backup&Recovery 9.5 U4
- Synology Active Backup for Business V 2.1.1-1125

Testing process:
- Backup running Zimbra VM (vmware quiescence enabled)
- Restoring Zimbra VM
- Check the results

Results:
- Both solutions restored Zimbra VM without issues
- After both restores the data.mdb sparse file was STILL a sparse file
- all Zimbra services were starting without issues after both restores

Conclusion:
- you can use Veeam and Synology Active Backup for Business to backup a Zimbra VM
- the sparse file will be a sparse file after restore
- you should stop zimbra before you backup your VM if you can. Otherwise you could get into trouble because of databases in RAM as Mark explained here https://forums.zimbra.org/viewtopic.php?f=15&t=69129#p300150

HTH
Joerg
gcakici2
Posts: 1
Joined: Mon Jan 18, 2021 9:45 pm

Re: shrink the data.mdb sparse file

Postby gcakici2 » Mon Jan 18, 2021 9:51 pm

Hi Wentum, you already mention After both restores the data.mdb sparse file was STILL a sparse file but have you successfully change the size of the file?

If you can, is it showing the new resized size when you ls -l ?

Thanks/Gokalp

Return to “Administrators”

Who is online

Users browsing this forum: No registered users and 9 guests