[SOLVED] Yet Another Backup Script for Community Version

Discuss your pilot or production implementation with other Zimbra admins or our engineers.
heinzg
Advanced member
Advanced member
Posts: 83
Joined: Fri Sep 12, 2014 11:16 pm

[SOLVED] Yet Another Backup Script for Community Version

Post by heinzg »

Hi osiris!
sorry to hear that it did not work out of the box for you. why is it that things that work well on Ubuntu just don't work the same on RH.
Could you maybe send me your version info for:

bash, su an your env setup e.g. /etc/profile, /etc/bash.bashrc, and ~/.bashrc. or maybe you want to migrate your system to Ubuntu? :D
Fun aside.

I changed the script's sudo execution to su and now pack the zmcontrol strings into a $CMD variable and then pass it onto su -
And from your output I see that there is a problem with the with the way the $CMD construct is being passed onto su.... Nothing wild!
On lines 148 to 152 is this $CMD construct, which I think you should have a look at maybe there is a quote sign in the $CMD variable which is not liked by su or zmcontrol
here is a script to diagnose what is going wrong in the variable construct and to test the stop and start logic in the script. You just have to uncomment the section you want to test.


#!/bin/bash
ZM_HOME=/opt/zimbra/
ZM_USER=`ps -ef | grep "$ZM_HOME" | grep "java" | grep -v "zmmailboxdmgr" | awk '{print $1}' | head -n 1`

if [ -z $ZM_USER ]

then

echo "Unable to determan the zimbra user"

cat $LOG | mail -s "Zimbra backup error on `hostname --fqdn`" $EMAIL

exit 1

elif [ $ZM_USER = "root" ]

then

echo "Zimbra user should never be root! I'll die now..."

# send email....

cat $LOG | mail -s "Zimbra backup error on `hostname --fqdn`" $EMAIL

exit 1

fi
# Command short form

ZIM_STOP="$ZM_USER -c $ZM_HOME"bin/zmcontrol" shutdown"

ZIM_START="$ZM_USER -c $ZM_HOME"bin/zmcontrol" start"

ZIM_STATUS="$ZM_USER -c $ZM_HOME"bin/zmcontrol" status"

ZIM_STAT_START="$ZM_USER -c $ZM_HOME"bin/zmstatctl" start"

ZIM_STAT_STOP="$ZM_USER -c $ZM_HOME"bin/zmstatctl" stop"
echo

echo $ZIM_STOP

echo

echo $ZIM_START

echo

echo $ZIM_STATUS

echo

echo $ZIM_STAT_START

echo

echo $ZIM_STAT_STOP

echo



########## Stopping Zimbra

# echo "Stopping the Zimbra server"

# sleep 2 && sync

# su - $ZIM_STOP

# if [ "$?" -eq "1" ]

# then

# echo "zmcontrol shutdown had an error!"

# # send email....

# cat $LOG | mail -s "Zimbra backup error on `hostname --fqdn`" $EMAIL

# else

# echo

# echo "Zimbra server has stopped "

# fi

#

#...................................................................................

# echo

########## Starting the Zimbra server again

# su - $ZIM_START

# if [ "$?" -ne "0" ]

# then

# echo "There was an error starting Zimbra!"

# # send email....

# cat $LOG | mail -s "Zimbra backup error on `hostname --fqdn`" $EMAIL

# else

# echo

# echo "Zimbra has started"

# fi

#

# sleep 15

#...................................................................................

######### quick hack to start the stats component

# # On my system I have 7 pid files whe all is running well, you might have less depending on your config

# SPIDS=`ls -A -1 $STATPIDBASE | wc -l`

# if [ "$SPIDS" != "7" ]

# then

# echo "I am expecting 7 pids for Stats, rebooting Stats subsystem!"

# su - $ZIM_STAT_STOP

# if [ "$?" -ne "0" ]

# then

# echo "Stopping stats failed!"

# # send email....

# cat $LOG | mail -s "Zimbra backup error on `hostname --fqdn`" $EMAIL

# else

# echo "Stats have been stopped"

# fi

#

# sleep 2

# su - $ZIM_STAT_START

# if [ "$?" -ne "0" ]

# then

# echo "Starting stats failed!"

# # send email....

# cat $LOG | mail -s "Zimbra backup error on `hostname --fqdn`" $EMAIL

# else

# echo "Stats started..."

# fi

# else

# echo "Stats seems to be running fine..."

# fi

#

# # Status Check to see that is running

# sleep 2

# STATUS=`su - $ZIM_STATUS | grep Stopped`

# if [ "$STATUS" ]

# then

# echo $STATUS

# # send email....

# cat $LOG | mail -s "Zimbra backup error on `hostname --fqdn`" $EMAIL

# fi


You can also send my a private message then we can try diagnose this via skype.... I would really like this scrip to be more compatible with other distros and to work without changing code in the logic section of the script.
Cheers heinzg
heinzg
Advanced member
Advanced member
Posts: 83
Joined: Fri Sep 12, 2014 11:16 pm

[SOLVED] Yet Another Backup Script for Community Version

Post by heinzg »

HI ALL!
JUST TO UPDATE. THE SCRIPT DOES NOT WORK UNMODIFIED ON RED HAT.
IF YOU ARE HAVING PROBLEMS YOU'LL HAVE TO CHANGE ALL SHORT FORM CONSTRUCTS E.G.
SU - $ZIM_STOP

SU - $ZIM_START

SU - $ZIM_STATUS

SU - $ZIM_STAT_START

SU - $ZIM_STAT_STOP
TO THE FULL COMMAND E.G.

SU - ZIMBRA -C ZMCONTROL SHUTDOWN

SU - ZIMBRA -C ZMCONTROL START


SORRY FOR THIS, I WILL FINE A BETTER SOLUTION.
P.S. PLEASE POST YOUR EXPERIENCE WITH THE SCRIPT I.E. DISTRO / WORKS "OUT OF THE BOX" OR NOT




UBUNTU 6.06.1 LTS / WORKS OUT OF THE BOX

RED HAT ENTERPRISE 5 / DOES NOT



CHEERS
Stergil
Advanced member
Advanced member
Posts: 69
Joined: Fri Sep 12, 2014 10:34 pm

[SOLVED] Yet Another Backup Script for Community Version

Post by Stergil »

Thanks for the work you have put into the script.
I've downloaded the latest version offered by your site and tried it out.
I also needed to install mailutils and struggled with that for a few. I didn't want to install the exim packages that apt-get suggested when trying to install the mailutils package. To get through this I came across this forum post which worked perfectly.
I was happy to see the link your script provided when it failed to detect dar support, since dar was new to me.
I also thought that dar was hanging, getting hung up on dar file creation. However, that was just my machine. It took ~5min for the dar process alone to finish. So if anyone else is on older hardware have some patience :)
FWIW It's running on an athlon xp 1.8ghz with 1gig ram.
I may wind up using a variant of your script for the rsyncing and process stopping/starting. The dar archives aren't really needed for my setup. I have a tape backup on the network which just pulls data from the /backups slice. So the rsync'd dir works like a champ for me.
I'm pretty much the only user on my server for a small home-office installation. My uncompressed install is only ~2gig.
If you find others that are in the same situation maybe a -nodar flag or something would be useful.
Almost forgot, running Ubuntu 6.06 .
Edit: BTW, I needed to touch create the log file to get the script to start. After a successful backup I checked the log to find it empty.
I also just tried a dar -t on the created archive and got the following

root@zimbra:/opt/share/bin/backups/dar# dar -t 09_Zimbra_Backup_03-March-2008_FULL.1.dar

Warning, 09_Zimbra_Backup_03-March-2008_FULL.1.dar seems more to be a slice name than a base name. Do you want to replace it by 09_Zimbra_Backup_03-March-2008_FULL ? [return = OK | esc = cancel]

Escaping...

OK, keeping 09_Zimbra_Backup_03-March-2008_FULL.1.dar as basename

./09_Zimbra_Backup_03-March-2008_FULL.1.dar.1.dar is required for further operation, please provide the file. [return = OK | esc = cancel]


Also, the TOO_MEDIA_DIR was empty after the backup was finished.
Stergil
Advanced member
Advanced member
Posts: 69
Joined: Fri Sep 12, 2014 10:34 pm

[SOLVED] Yet Another Backup Script for Community Version

Post by Stergil »

Well it looks like dar may be getting hung up afterall. When I thought the previous backup was hungup on the dar process I ran a -t on the archive in another terminal. I eventually returned to the original terminal window and saw the output summary and at that point thought it finished.
This time around I've not done anything and it's been sitting here for more than 10minutes.



PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

1152 root 39 19 15836 11m 2016 R 64.1 1.6 10:21.07 dar


It's late and I should sleep, I'll let it run overnight to see if it's going to finish on its own. The CPU utilization keeps fluctuating which seems to hint that it's not hung. But requiring more than now 12min of processing time just to create a single archive on a local hard disk seems excessive.
heinzg
Advanced member
Advanced member
Posts: 83
Joined: Fri Sep 12, 2014 11:16 pm

[SOLVED] Yet Another Backup Script for Community Version

Post by heinzg »

Hi Stergil!
Thanks for your post, and feed back!
To answer some of your questions in short and to ask some questions to.
Good I see you got mail installed and working. Did the script not print out the following when not finding mailx?

"For a "Howto" install mailx without postfix please visit the link below"

http://www.zimbra.com/forums/administra ... #post70636"



About dar. Dar is my choice of archive due to the the list of features it has in comparison to tar, and the list is well long, e.g. one of the features of dar is the ability to recover single files from the archive with out having to unpack the whole archive. It also has encryption builtin which I will add to the script in the next release for better security. After all we don't want our tapes and dvd being copied by others with intentions we might find offensive!
But the main reason to use dar is for the neat way it make full and then differential backups. Meaning you save a lot of space on your storage media. I have setup dar to make compressed backups archives (gzip -9 level) you can change this be changing COMPRESS=9 to a lower amount.

This is also one of the reasons why it seams that dar is hanging on your system. At over 2GB of data it takes quite a bit of time to create the initial catalogue, and secondly the script is running dar as a background proc in non interactive mode. So it is normal for dar to sit there for a bit seeming not to do anything.

On your spec system and data volume I can imagine that with compression on full and all IO on the same disk, that it can take up the good part of 35 to 50 min for a full backup.

After making a Full backup you now only have to make differential backups to capture and save only file that have changed from those in the full backup archive, plus it is a lot faster now. I would guess on your system depending on the amount of spam you get a day 5 to 10 min and about 60MB data. So to recap you save lots of space on long term storage media through compression and differential backups, Dar has also been instructed not to compress common compressed files thus a performance increase for you.

As for a increased in performance you would have to put your SYNC_DIR on an other disk than where your ARCHIVEDIR is, the basic IO story.

OH yes the script starts dar with “nice 19” (change 19 -> 0 for more speed) which is the lowest priority... not to hinder Zimbra server from doing important mailing work :-)
About there not being any files in TOO_MEDIA_DIR has to do with my logic of doing the backups. The script works on a weekly rotation e.g. On Monday morning it makes a full backup and then every 24 hour there after a differential backup. Until it gets to Monday morning again where it will then transfer the backups from the last week to TOO_MEDIA_DIR where you can then write it to tape or DVD what ever. But the full backup archive should stay in ARCHIVEDIR as dar compares the files from the SYNC_DIR with those in the full backup archive to then create a differential backup file.

So if you use my crontab suggestion you will find files in TOO_MEDIA_DIR one week after running the script for the first time.
The bit about the LOG file. This is sort of a hack if you wish... as the script is/or should mainly be used as a root corn job you should redirect all out put from the script into the same file as stated in LOG=.

Crontab example (run the script with -h to see the right statement for your LOG entry in the contab):

0 3 * * 1 /bin/sh /path to script/zmbac.sh -f > /path to log dir/file.log 2>&1

0 3 * * 2-7 /bin/sh /root/zmbac.sh -d >> /path to log dir/file.log 2>&1



If you don't do the redirect then you will land up this mails with a subject but with no body, which should actually contain the contents of the log file.
A sample print out of the weekly full backup should look something like this, note this system also took 38 min for the task to complete (time for a new office server...):


Full Backup started at: 03:00
Old backups found...old week= 08 current week= 09
/data/Backup/zimbra_dars/08_Zimbra_Backup_01-March-2008_DIFF.1.dar moved to /data/Backup/burn/

/data/Backup/zimbra_dars/08_Zimbra_Backup_02-March-2008_DIFF.1.dar moved to /data/Backup/burn/

/data/Backup/zimbra_dars/08_Zimbra_Backup_25-February-2008_FULL.1.dar moved to /data/Backup/burn/

/data/Backup/zimbra_dars/08_Zimbra_Backup_26-February-2008_DIFF.1.dar moved to /data/Backup/burn/

/data/Backup/zimbra_dars/08_Zimbra_Backup_27-February-2008_DIFF.1.dar moved to /data/Backup/burn/

/data/Backup/zimbra_dars/08_Zimbra_Backup_28-February-2008_DIFF.1.dar moved to /data/Backup/burn/

/data/Backup/zimbra_dars/08_Zimbra_Backup_29-February-2008_DIFF.1.dar moved to /data/Backup/burn/
Printing date: 03-March-2008 and software version: *into /opt/zimbra/DATE_VERSION.txt
Doing a hotsync of /opt/zimbra/ to /mnt/fakebackup/

sync went ok
Stopping the Zimbra server

Host my.mail.host

Stopping stats...Done

Stopping mta...Done

Stopping spell...Done

Stopping snmp...Done

Stopping archiving...Done

Stopping antivirus...Done

Stopping antispam...Done

Stopping imapproxy...Done

Stopping mailbox...Done

Stopping logger...Done

Stopping ldap...Done
Zimbra server has stopped
kill any left over zimbra procs
Doing a fast cold sync of /opt/zimbra/ & /mnt/fakebackup/ while the Zimbra server is down

sync went ok
Host my.mail.host

Starting ldap...Done.

Starting logger...Done.

Starting mailbox...Done.

Starting antispam...Done.

Starting antivirus...Done.

Starting snmp...Done.

Starting spell...Done.

Starting mta...Done.

Starting stats...Done.
Zimbra has started

Stats seems to be running fine...
Writing a full backup called:09_Zimbra_Backup_03-March-2008_FULL in: /data/Backup/zimbra_dars/ with file sizes of max: 4695M


*--------------------------------------------

*35430 inode(s) saved

*with 1454 hard link(s) recorded

*0 inode(s) not saved (no file change)

*0 inode(s) failed to save (filesystem error)

*0 files(s) ignored (excluded by filters)

*0 files(s) recorded as deleted from reference backup

*--------------------------------------------

*Total number of file considered: 35430

*--------------------------------------------

*EA saved for 0 file(s)

*--------------------------------------------

Archive Created successfully


:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::

Full Zimbra Backup ended at: 03:38

The the full backup took Hr:0 Min:38 Sec:39 to complete

:::::::::::::::::::Cheers Osoffice.de for the script:::::::::::::::::::::::



I hope this helps you some what!
Best regards

Heinzg
heinzg
Advanced member
Advanced member
Posts: 83
Joined: Fri Sep 12, 2014 11:16 pm

[SOLVED] Yet Another Backup Script for Community Version

Post by heinzg »

Hi
Added a Redhat compatible version on the download site, for now. Till the next release is out where all distros should be compatible (I hope).
Cheers

heinzg
P.S. thank you for your feed back!
madunix
Advanced member
Advanced member
Posts: 65
Joined: Fri Sep 12, 2014 11:21 pm

[SOLVED] Yet Another Backup Script for Community Version

Post by madunix »

Can you please explain in details how to use it on Redhat RHEL5 64bit with zimbra Opensource version, to backup and restore of mail boxes using the above script.
Stergil
Advanced member
Advanced member
Posts: 69
Joined: Fri Sep 12, 2014 10:34 pm

[SOLVED] Yet Another Backup Script for Community Version

Post by Stergil »

[quote user="heinzg"]Good I see you got mail installed and working. Did the script not print out the following when not finding mailx?

"For a "Howto" install mailx without postfix please visit the link below"

http://www.zimbra.com/forums/administra ... #post70636"



[/QUOTE]

Hmm, not that I recall seeing. Grant it it was about 4am and I could have missed it :o
I didn't realize the benefits of dar, thanks for the info. You were right the full backup took about an hour, but the differentials only took ~10min.
So far things seem to be working fine. Differentials are being created, etc.
I appreciate the extra explanation on things, it was very helpful.
dwmtractor
Outstanding Member
Outstanding Member
Posts: 993
Joined: Fri Sep 12, 2014 10:41 pm

[SOLVED] Yet Another Backup Script for Community Version

Post by dwmtractor »

heinzg,
I'm not enough of a shell script guru to analyze all you have done, but from what I have read of it so far, my hat is off to you for an extremely well-thought-out script. It's great to have the development site you linked, but if I could presume to make a suggestion, it'd be nice if you would post either the script, or at least a brief discussion with link to your development page, on the wiki under the open-source backup section (here) so that when people are reading about backup methodologies, this one is added to the mix. This will make it more likely (1) that those who need it find your script, and (2) that you'll get a wider variety of people testing and providing feedback to you. Kinda what we call a win-win! :D
Kudos for your hard work!
Dan
cornbread
Advanced member
Advanced member
Posts: 80
Joined: Fri Sep 12, 2014 10:46 pm

[SOLVED] Yet Another Backup Script for Community Version

Post by cornbread »

How do I do this? I already have the destination set up to do automatic ssh authentication from the machine. Which part of the script would I modify to do rsync via ssh?
Post Reply