[SOLVED] Yet Another Backup Script for Community Version

Discuss your pilot or production implementation with other Zimbra admins or our engineers.
LaFong
Outstanding Member
Outstanding Member
Posts: 221
Joined: Fri Sep 12, 2014 11:54 pm

[SOLVED] Yet Another Backup Script for Community Version

Post by LaFong »

[quote user="jjeffers1"]After some troubleshooting I discovered the /store directory is being passed up by dar because it is a symlink. I currently have the store directory symlinked to a larger disk on the server.
Mysql and other directories are symlinked as well but to other directories under /opt/zimbra.
I was under the impression mysql was not being backed up but it was because it was symlinked to mysql-standard-5.1.55-pc-linux-gnu-i686-glibc23.
But back to the issue with /store which is truly symlinked outside of /opt/zimbra in my environment. I am looking to the DAR documentation for a solution and apparently DAR should be able to handle and symlinks based on this exerpt from the DAR's features section:
Any ideas how to make DAR recognize symlinks?[/QUOTE]

Heh. I had started a reply about symlinks, but decided you wouldn't be doing that just for a trial, and inferred mysql was empty on your test restore. :)
Dar does not follow symlinks to directories out of its root path (which for this script is $SYNC_DIR). You will have to change the script to point to any other directories you want in the dar. I would suggest the following changes:

1) Create a new DAR_OPTS options line near the top of the script, right under RSYNC_OPTS:



RSYNC_OPTS="-aHK --delete --stats --exclude=*.pid" # leave these unless you are sure you need something else

DAR_OPTS="-R / -g path/to/SYNC_DIR -g path/to/store -g path/to/db"

Note the lack of a leading slash for the -g directories. All paths are relative to the -R root. You will have to use / for dar root, if you want to get it all in one go. The side effect of this is that the path of the files within the dar will be different from the script default. In a dar -l listing, instead of (scroll to right)



[Saved] [-----] drwxrwxr-x 501 zimbra 0 Thu Aug 4 15:02:10 2011 logger

[Saved] [-----] drwxr-xr-x zimbra zimbra 0 Thu Aug 4 15:02:10 2011 logger/db

[Saved] [-----] drwxrwxr-x zimbra zimbra 0 Sun Mar 25 04:04:09 2012 logger/db/data

Assuming /backup/sync is your SYNC_DIR, you will see (scroll to right)



[Saved] [-----] drwxrwxr-x 501 zimbra 0 Thu Aug 4 15:02:10 2011 backup/sync/logger

[Saved] [-----] drwxr-xr-x zimbra zimbra 0 Thu Aug 4 15:02:10 2011 backup/sync/logger/db

[Saved] [-----] drwxrwxr-x zimbra zimbra 0 Sun Mar 25 04:04:09 2012 backup/sync/logger/db/data

Notice that SYNC_DIR precedes the rest of the path, using this modification. Similarly, the store and db directories would have a path relative to / and not /opt/zimbra. For that reason, on restores you should be positive that your restore destination is NOT / if you do not want to overwrite an existing SYNC_DIR, db, and store.
2) You will also need to remove the "-R `pwd`" root setting at the 4 places in the script where $DAR_BIN -c gets called, and add $DAR_OPTS to the command:



nice -19 $DAR_BIN $DAR_OPTS -K bf:$KEY -s $ARCHIVESIZE -z$COMPRESS -Z "*.gz" -Z "*.zip"

-Z "*.bz2" -Z "*.tgz" -Z "*.zgz" -Z "*.jar" -Z "*.tiff"

-Z "*.jpg" -Z "*.png" -Z "*.gif" -Z "*.jpeg"

-c "$ARCHIVEDIR""$ARCHIVENAME" -Q

Or, skip DAR_OPTS and just make all the changes in all 4 places:



nice -19 $DAR_BIN -K bf:$KEY -s $ARCHIVESIZE -z$COMPRESS -Z "*.gz" -Z "*.zip"

-Z "*.bz2" -Z "*.tgz" -Z "*.zgz" -Z "*.jar" -Z "*.tiff"

-Z "*.jpg" -Z "*.png" -Z "*.gif" -Z "*.jpeg" -R "/"

-g "path/to/SYNC_DIR" -g "path/to/db" -g "/path/to/store"

-c "$ARCHIVEDIR""$ARCHIVENAME" -Q

I know that tar can follow symlinks, but that has its own complications.
LaFong
Outstanding Member
Outstanding Member
Posts: 221
Joined: Fri Sep 12, 2014 11:54 pm

[SOLVED] Yet Another Backup Script for Community Version

Post by LaFong »

[QUOTE]

Yes, you can hard link symbolic links!

[/QUOTE]Oh, and this just means that symlinks take up an inode, and hard links can point to that inode. And hard links have to be on the same filesystem (i.e. partition) as the inode they're pointing at, so it would not help your issue.
savez
Posts: 9
Joined: Sat Sep 13, 2014 2:13 am

[SOLVED] Yet Another Backup Script for Community Version

Post by savez »

hello LaFOng,

I read your post
voelvo ask if I could move the files and instructions to use your backup script to zimbraCE ....


Thank you and good job you're good
LaFong
Outstanding Member
Outstanding Member
Posts: 221
Joined: Fri Sep 12, 2014 11:54 pm

[SOLVED] Yet Another Backup Script for Community Version

Post by LaFong »

[quote]HI,
I'VE BEEN COMBING THRU POSTS LOOKING FOR A GOOD UNDERSTANDING OF EXACTLY WHAT IS REQUIRED FOR A "TRUE", "FULL", "DISASTER RECOVERY" BACKUP.
FULL COLD COPY OF ALL OF "/OPT/ZIMBRA" SEEMS TO BE THE GUIDANCE, BUT -- IS IT REALLY 'ALL'?

[/QUOTE]

ALMOST. MANY/MOST INSTALLATIONS WILL ALSO REQUIRE SPLIT DNS. THAT WILL REQUIRE BIND OR OTHER SETUP, WHICH VARIES A BIT DEPENDING ON THE DISTRO AND DNS SERVER. YOU WOULD WANT TO BACKUP THAT CONFIG. YOU'LL ALSO WANT TO BACKUP ANY CONFIG OF SSH, FAIL2BAN, HOSTS, PASSWD, SHADOW, INIT.D, RESOLV.CONF, ETC. THESE ARE IN /ETC, USUALLY. EASIEST TO BACKUP ALL OF /ETC.
[QUOTE]

IN PARTICULAR, I'M INTERESTED IN UNDERSTANDING WHETHER THE 'SPAECIAL FILES' -- AKA SOCKETS, DEVICES, PIPES, FIFOS, BLOCKS, ETC ETC -- ARE REQUIRED TO BE BACKED UP, OR CAN THEY ALL (OR SOME PORTION OF THEM) BE EXCLUDED FROM THE BACKUP?
I'VE JUST RECENTLY ASKED THIS QUESTION HERE,

BUT">HTTPS://WWW.ZIMBRA.COM/FORUMS/ADMINISTR ... STORE.HTML
BUT, GIVEN THE BROAD INTEREST IN THIS (RATHER LONG) THREAD RE: GENERAL BACKUP ISSUES, I THOUGHT I'D ASK HERE AS WELL.[/QUOTE]

THEY COULD BE EXCLUDED. THIS SCRIPT DOES NOT BACK THEM UP. IF YOU ARE USING A DIFFERENT BACKUP METHOD, WHICH IS NOT SPECIFIC TO ZIMBRA, THERE IS LITTLE REASON TO GO OUT OF YOUR WAY TO EXCLUDE THEM. THEY TAKE UP LITTLE ROOM COMPARED TO USER DATA. IF YOU ARE JUST DOING ZIMBRA BACKUP, THE DISASTER RESTORE WOULD GO LIKE THIS:



INSTALL OS. (SAME OS VERSION)

INSTALL DNS.

RESTORE DNS CONFIG.

RESTORE MISC. CONFIG (SSH, HOSTS, ETC.)

INSTALL ZIMBRA (EXACT SAME VERSION)

STOP ZIMBRA.

RESTORE /OPT/ZIMBRA.

START ZIMBRA.


THERE ARE EDGE CASES, E.G. WHERE PEOPLE USE SYMLINKS FOR /OPT ZIMBRA/STORE AND PUT THE STORE ON A DIFFERENT PARTITION, BUT THIS IS NOT COMMON.
liverpoolfcfan
Elite member
Elite member
Posts: 1112
Joined: Sat Sep 13, 2014 12:47 am

[SOLVED] Yet Another Backup Script for Community Version

Post by liverpoolfcfan »

Each night when the DB backup is done I get the following line in the middle of all the mailbox backups
E-Mail User DB LDAP zimbraId

------ ------------ ---- -------

...

mboxgroup .ldif fbd258fb-f8c8-461e-84fa-754d642b5f21

...


There is no associated email address - or mailbox number
Can anyone suggest what I need to do to clean this up ?
Thanks.
LaFong
Outstanding Member
Outstanding Member
Posts: 221
Joined: Fri Sep 12, 2014 11:54 pm

[SOLVED] Yet Another Backup Script for Community Version

Post by LaFong »

[quote user="liverpoolfcfan"]Each night when the DB backup is done I get the following line in the middle of all the mailbox backups
E-Mail User DB LDAP zimbraId

------ ------------ ---- -------

...

mboxgroup .ldif fbd258fb-f8c8-461e-84fa-754d642b5f21

...

There is no associated email address - or mailbox number
Can anyone suggest what I need to do to clean this up ?
[/QUOTE]
Try the commands below to identify the user.



sudo su - zimbra

source ~/bin/zmshutil ; zmsetvars

ldapsearch -x -H ldap://"$HOSTNAME" -D "$zimbra_ldap_userdn" -w "$zimbra_ldap_password" '(&(objectClass=zimbraAccount)(zimbraId=fbd258fb-f8c8-461e-84fa-754d642b5f21))'

raid3868
Posts: 4
Joined: Sat Sep 13, 2014 2:34 am

[SOLVED] Yet Another Backup Script for Community Version

Post by raid3868 »

Hi,
I m very new in zimbra and still testing, currently i have installed zimbra and is working,

at this stage i m testing dor backup and restore. I have download this download script,

backup working good. But i m using fetchmail with schedule every 15min to fetch thirty party pop server, it work with no problem. but come to backup it need to add some script

to disable fetchmail before backup.
fetchmail crontab is running as user admin.
Can someone help how can i add script to disable fetchmail crontab before shutting down zimbra as i saw that this backup has disable zimbra crontab. but i don't understand how to add the script to. can someone pls help.
thks
blason
Outstanding Member
Outstanding Member
Posts: 265
Joined: Sat Sep 13, 2014 2:13 am

[SOLVED] Yet Another Backup Script for Community Version

Post by blason »

OK, I have been using this script for quiet some time and failed to understand how do I disable dar file function so that script would not dar the files. This is actually consuming my space since I am running very low on space hence I have written small script to delete those files but If I done that script failed to take a DIFF backup not sure which parameter or file it verifies and proceed with backup for that particular week.
Any clue? How do I completely disable dar functions?
LaFong
Outstanding Member
Outstanding Member
Posts: 221
Joined: Fri Sep 12, 2014 11:54 pm

[SOLVED] Yet Another Backup Script for Community Version

Post by LaFong »

[quote user="blason"]OK,

Any clue? How do I completely disable dar functions?[/QUOTE]
If you don't want dar, there is little reason to use this script. Just use a basic rsync script. You will not have dated backups unless you use a separate backup tool on your SYNC_DIR.
LaFong
Outstanding Member
Outstanding Member
Posts: 221
Joined: Fri Sep 12, 2014 11:54 pm

[SOLVED] Yet Another Backup Script for Community Version

Post by LaFong »

[quote user="raid3868"]Hi,

Can someone help how can i add script to disable fetchmail crontab before shutting down zimbra as i saw that this backup has disable zimbra crontab. but i don't understand how to add the script to. can someone pls help.

[/QUOTE]
Possible solutions include setting your fetchmail crontab to exclude your backup crontab time, or change one of the scripts to use pgrep or similar to detect whether the other script is running.
Post Reply