[SOLVED] Yet Another Backup Script for Community Version

Discuss your pilot or production implementation with other Zimbra admins or our engineers.
heinzg
Advanced member
Advanced member
Posts: 83
Joined: Fri Sep 12, 2014 11:16 pm

[SOLVED] Yet Another Backup Script for Community Version

Post by heinzg »

[quote user="pinoy_steal"]@heinzg
Hi Sir, We have 1 linux box for zimbra and 1 windows box that act as backup server. My idea are all archive copy of zimbra backup are deliver to windows box. It is possible with this script sir. Thanks.[/QUOTE]
Hi pinoy_steal,
This script support shipping the backup files to another system that has a ssh server installed and you have configured it to use PubkeyAuthentication. So that you can login with out a password.
Then YES you can save your back up on a DOS server that does the above :-)
having a quick look at sourceforge reviled the following... sshwindows.sourceforge.net package you might wana have a look at.
Cheers

Heinzg
sem
Posts: 29
Joined: Sat Sep 13, 2014 12:56 am

[SOLVED] Yet Another Backup Script for Community Version

Post by sem »

[quote user="pinoy_steal"]@heinzg
Hi Sir, We have 1 linux box for zimbra and 1 windows box that act as backup server. My idea are all archive copy of zimbra backup are deliver to windows box. It is possible with this script sir. Thanks.[/QUOTE]

I have mapped a drive using SMB on the zimbra server to a windows share on a fileserver. This is then used as the backup location for the script. The windows server is backed up offsite each night so zimbra is included. Works really well.
sem
heinzg
Advanced member
Advanced member
Posts: 83
Joined: Fri Sep 12, 2014 11:16 pm

[SOLVED] Yet Another Backup Script for Community Version

Post by heinzg »

[quote user="sem"]I have mapped a drive using SMB on the zimbra server to a windows share on a fileserver. This is then used as the backup location for the script. The windows server is backed up offsite each night so zimbra is included. Works really well.
sem[/QUOTE]
Hi,
Wow Simple... I did not think of this version!

My next suggestion would have been to reinstalling Windos server with a real OS :D
P.S. what is your motivation to do this all together, save the backups to a windos box?
Cheers

Heinzg
sem
Posts: 29
Joined: Sat Sep 13, 2014 12:56 am

[SOLVED] Yet Another Backup Script for Community Version

Post by sem »

[quote user="heinzg"]Hi,
Wow Simple... I did not think of this version!

My next suggestion would have been to reinstalling Windos server with a real OS :D
P.S. what is your motivation to do this all together, save the backups to a windos box?
Cheers

Heinzg[/QUOTE]
Hi - I always go for the simple approach :)
Motivation - the client has a windows 2008 server as their main file storage/ad controller. As it's being backed up to removable storage it seemed the obvious choice.
Thanks for the great script.
sem
LaFong
Outstanding Member
Outstanding Member
Posts: 221
Joined: Fri Sep 12, 2014 11:54 pm

[SOLVED] Yet Another Backup Script for Community Version

Post by LaFong »

SOME MIGHT FIND SOME USE FOR MY CHANGES. I DON'T WRITE TO DVDS, BUT JUST BACKUP TO ANOTHER PARTITION. THEN OUR NETWORK BACKUP SOFTWARE BACKS UP THE BACKUP. I MADE THESE CHANGES:




CHANGED THE DATE COMMAND TO USE %V, ISO WEEK NUMBER. THIS DEALS WITH END OF YEAR BETTER THAN %W, FOR ME.

INCREASED THE ARCHIVE SIZE. BECAUSE I AM NOT BURNING DVDS, I JUST WANT A SINGLE ARCHIVE FILE.

ADDED CODE TO DELETE THE ARCHIVED BACKUPS FROM TWO WEEKS AGO. THIS GETS RUN DURING THE WEEKLY FULL.

CHANGED BACKUPDATE TO A MORE READABLE FORMAT, E.G. 23-MARCH-2010 INSTEAD OF 20100323

TRIVIAL FILE AND DIRECTORY NAME CHANGES.


I DID SOME TESTING WITH THE 2-WEEK DELETE, BUT MAKE NO GUARANTEES.
[ROOT@MAIL SBIN]# DIFF -U ZMBAC.0.8.SH.ORIG ZMBAC.SH

--- ZMBAC.0.8.SH.ORIG 2010-02-14 20:19:22.000000000 -0700

+++ ZMBAC.SH 2010-03-23 18:15:24.000000000 -0600

@@ -61,25 +61,25 @@

#--- DIRECTORIES ---#

# PLEASE ADD THE TRAILING "/" TO DIRECTORIES!

ZM_HOME=/OPT/ZIMBRA/ # WHERE ZIMBRA LIVES

-SYNC_DIR=/TMP/FAKEBACKUP/ # INTERMEDIATE DIR FOR HOT/COLD SYNCS. MUST HAVE AT LEAST AS MUCH FREE SPACE AS ZM_HOME CONSUMES

-ARCHIVEDIR=/BACKUP/ZIMBRA_DARS/ # WHERE TO STORE FINAL BACKUPS

-TO_MEDIA_DIR=/BACKUP/BURN/

+SYNC_DIR=/BACKUP/SYNC/ # INTERMEDIATE DIR FOR HOT/COLD SYNCS. MUST HAVE AT LEAST AS MUCH FREE SPACE AS ZM_HOME CONSUMES

+ARCHIVEDIR=/BACKUP/CURRENT/ # WHERE TO STORE FINAL BACKUPS

+TO_MEDIA_DIR=/BACKUP/PREVIOUS/



#--- PROGRAM OPTIONS ---#

RSYNC_OPTS="-AHK --DELETE --EXCLUDE=*.PID" # LEAVE THESE UNLESS YOU ARE SURE YOU NEED SOMETHING ELSE



#--- ARCHIVE NAMES ---#

-BACKUPNAME="ZIMBRABACKUP" # WHAT YOU WANT YOUR BACKUPS CALLED

+BACKUPNAME="ZIMBRA_BACKUP" # WHAT YOU WANT YOUR BACKUPS CALLED

FULL_PREFIX="FULL" # PREFIX USED FOR FULL BACKUPS

DIFF_PREFIX="DIFF" # PREFIX USED FOR DIFFERENTIAL BACKUPS

-BACKUPDATE=`DATE +%Y%M%D` # DATE FORMAT USED IN ARCHIVE NAMES

-# BACKUPWEEK=`DATE +%W` # WEEK PREFIX USED FOR BACKUP WEEKLY ROTATION AND NAMING

+BACKUPDATE=`DATE +%D-%B-%Y` # DATE FORMAT USED IN ARCHIVE NAMES

+# BACKUPWEEK=`DATE +%V` # WEEK PREFIX USED FOR BACKUP WEEKLY ROTATION AND NAMING

# VJS - COMMENTED OUT ABOVE, AND ADDED BELOW TO ALLOW WEEKLY BASELINE TO BE ANY DAY OF WEEK.

# NEEDS FULL SET OF TESTS TO BE RUN, INCLUDING ARCHIVING OLD WEEKS DATA, ETC.

BACKUPWEEKFILE="/OPT/ZIMBRA/BACKUP.WEEK"

CASE $1 IN

-F | --FULL)

-DATE +%W > $BACKUPWEEKFILE

+DATE +%V > $BACKUPWEEKFILE

;;

-D | --DIFF)

IF [ ! -E "$BACKUPWEEKFILE" ]

@@ -95,7 +95,7 @@

BACKUPWEEK=`CAT $BACKUPWEEKFILE` # WEEK PREFIX USED FOR BACKUP WEEKLY ROTATION AND NAMING



#--- ARCHIVE SIZE ---#

-ARCHIVESIZE="4395M" # STORAGE MEDIA SIZE, FOR FULL-BACKUP ARCHIVING

+ARCHIVESIZE="110000M" # STORAGE MEDIA SIZE, FOR FULL-BACKUP ARCHIVING

# VJS - UNCOMMENT NEXT LINE FOR TESTING MULTI-DAR SCENARIO ON SMALL MAIL STORE.

#ARCHIVESIZE="50M" # STORAGE MEDIA SIZE, FOR FULL-BACKUP ARCHIVING

COMPRESS="9" # VALID ANSWERS ARE 1 - 9 ( 9 = BEST )

@@ -107,9 +107,9 @@

PASSFILE="NOREAD" # THE FILE CONTAINING THE PASSWORD HASH



#--- LOG SETTINGS ---#

-EMAIL="UESR@DOMAIN.LTD" # THE ADDRESS TO SEND LOGS TO

+EMAIL="IT@SOMEWHERE.COM" # THE ADDRESS TO SEND LOGS TO

# VJS - ADDED CC EMAIL TO AVOID ERRORS SENDING REPORT

-EMAILCC="USER@DOMAIN.LTD" # ANOTHER ADDRESS TO SEND TO, BLANK FOR NONE

+#EMAILCC="USER@DOMAIN.LTD" # ANOTHER ADDRESS TO SEND TO, BLANK FOR NONE

LOG="/VAR/LOG/ZIM_BACKUP.LOG" # LOG LOCATION

# VJS - ADDED CHEWITT SUGGESTION FOR OPTIONAL LISTING ON REPORT EMAIL

ATTACHLIST="NO" # ATTACH BACKUP FILE OR NOT?

@@ -622,7 +622,41 @@

FI

DONE

FI

+ # REMOVE PREVIOUS WEEK OF BACKUPS FROM 2 WEEKS AGO

+ TWO_WEEKS_AGO=$(EXPR "$BACKUPWEEK" - 2)

+ LASTYEAR=$(EXPR `DATE +%Y` - 1)

+ FINALWEEK=`DATE -D "$LASTYEAR"1228 +%V`

+ IF [ "$BACKUPWEEK" -EQ "1" ] ; THEN

+ TWO_WEEKS_AGO=$(EXPR "$FINALWEEK" - 1)

+ ELIF [ "$BACKUPWEEK" -EQ "2" ] ; THEN

+ TWO_WEEKS_AGO="$FINALWEEK"

+ ELIF [ "$TWO_WEEKS_AGO" -LT "10" ] ; THEN

+ TWO_WEEKS_AGO=0"$TWO_WEEKS_AGO"

+ FI

+ TWO_WEEK_BACKUP=`LS -A -1 "$TO_MEDIA_DIR" | GREP "$TWO_WEEKS_AGO" | GREP "$BACKUPNAME" | CUT -D _ -F1 | HEAD -N1`

+ IF [ -Z "$TWO_WEEK_BACKUP" ]

+ THEN

+ ECHO

+ ECHO "NO TWO-WEEK OLD BACKUPS FOUND"

+ ECHO

+ ELSE

+ ECHO

+ ECHO "TWO-WEEK OLD BACKUP FOUND...OLD WEEK= "$TWO_WEEKS_AGO" CURRENT WEEK= $BACKUPWEEK"

+ ECHO

+ FOR I IN `LS -A -1 "$TO_MEDIA_DIR""$TWO_WEEKS_AGO"_"$BACKUPNAME"*`

+ DO

+ RM $I

+ IF [ "$?" -NE "0" ]

+ THEN

+ ECHO "ERROR DURING DELETE!"

+ ELSE

+ ECHO "$I DELETED"

+ FI

+ DONE

+ ECHO

+ FI

FI

+

# CHECKING FOR A BACKUP FILE COLLISIONS. CREATING DATE AND MARKER FILE.

CURRENTNAME1=`LS -A -1 "$ARCHIVEDIR""$ARCHIVENAME"*DAR 2> /DEV/NULL | HEAD -QN1`

IF [ -F "$CURRENTNAME1" ]
pinoy_steal
Advanced member
Advanced member
Posts: 63
Joined: Sat Sep 13, 2014 12:51 am

[SOLVED] Yet Another Backup Script for Community Version

Post by pinoy_steal »

@heinzg

Maybe I try your suggestion
@sem

You mean from Zimbra Box I just map windows share using samba. How about "SSH REMOTE DR COPY" option on this script. Im sorry if I have a lot of question. Im still experimenting Zimbra and I'm very very interesting on this. Thanks.
sem
Posts: 29
Joined: Sat Sep 13, 2014 12:56 am

[SOLVED] Yet Another Backup Script for Community Version

Post by sem »

[quote user="pinoy_steal"]@heinzg

Maybe I try your suggestion
@sem

You mean from Zimbra Box I just map windows share using samba. How about "SSH REMOTE DR COPY" option on this script. Im sorry if I have a lot of question. Im still experimenting Zimbra and I'm very very interesting on this. Thanks.[/QUOTE]
Hi. Yes, I stuck a permanent mount to the windows share in /mnt and use this. On the windows server, I created a zimbra user with permissions to a Zimbra directory that's included in the daily backup. Within here, I have 2 folders - ZimbraBackups and ToMedia - so the script puts the daily dar files into the ZimbraBackups directory and then on a Monday (start of backup cycle) it moves the previous week backups into the ToMedia directory. The client can/will clear these down every couple of weeks.
My main issue with the community edition was the backup - I built a server to be as resilient as possible with a dedicated /opt raid1+lvm configuration. But this script solved the problem of reliable, scheduled and reported backups. It's ace.


sem
albsallu
Posts: 22
Joined: Sat Sep 13, 2014 12:46 am

[SOLVED] Yet Another Backup Script for Community Version

Post by albsallu »

@Sem. I will really like to do the same thing that you have done. I am currently backing up to a remote linux server onsite. We have our windows machine backed up offsite, which I will like my zimbra emails to be part of. Please email me the instructions and steps that you did.

Thanks in advance.
sem
Posts: 29
Joined: Sat Sep 13, 2014 12:56 am

[SOLVED] Yet Another Backup Script for Community Version

Post by sem »

[quote user="albsallu"]@Sem. I will really like to do the same thing that you have done. I am currently backing up to a remote linux server onsite. We have our windows machine backed up offsite, which I will like my zimbra emails to be part of. Please email me the instructions and steps that you did.

Thanks in advance.[/QUOTE]
Hi - what sort of level of instructions are you needing? All you need to do is set up a permanent mount point on your linux box to a share on your windows server -
[quote]mount -t cifs //windowsservername/zimbrashare -o username=zimbrauseronwindowsbox,password=whateveryoulike /mnt/windowsservername[/quote]
You'll also want to make a corresponding FSTAB entry so that this is mapped after a reboot of the linux box.
Then in the zmback script, set the backup location to be somewhere on the windows box. I also have a separate HDD installed (mounted as /snapshot/) in the machine and use this as the hot/cold sync disk to minimise IO issues:
[quote]

# Please add the trailing "/" to directories!

ZM_HOME=/opt/zimbra/

SYNC_DIR=/snapshot/

ARCHIVEDIR=/mnt/windowsservername/backups/

TOO_MEDIA_DIR=/mnt/windowsserverame/tomedia/

[/quote]
That's just how I do it.
sem
albsallu
Posts: 22
Joined: Sat Sep 13, 2014 12:46 am

[SOLVED] Yet Another Backup Script for Community Version

Post by albsallu »

[quote user="heinzg"]Hi albsallu!
Just having a look at the contents of the backup dir suggests something is going very wrong, as there are only 3 diff file and 28 full backup files. This is normally the other way round. Lots of DIFFs few FULLs.
The script you are using has been edited and is not the stock zmbak_v.0.8.sh one you downloaded.

I can tell this this from the spelling correction on line 768 characters 19-26. :D ( thanx )
I could imagine that unintentionally some other part of the script might have been changed causing this problem. Can you please use the stock downloaded zmbak_v.0.8.sh version and see if this is the case.
Else we need to debug some more:

If using a new fresh copy of the script don't work better then do the following:



Add 'set -x' to the script at line after line 767 like so:

echo "Unkown Backup Type. Fail."

mail_log

exit 1

fi

else

set -x

echo "Saving Unencrtyped Archive..."

set -x

if [ "$TYPE" = "full" ]

then


And then 'set +x' after line 792 like so:

if [ "$?" -ne "0" ]

then

echo "Dar had a problem!"

set +x

mail_log

exit 1


Now run the manually first with the full option and then after that with the diff option.

After that please post the debug info.
If you want you can mail me your script and I can have a closer look at it... if you want.

P.S. I Think it is safe to delete the current backup sets as i don't think they are ok.
Cheers

Heinzg[/QUOTE]
Hello Heinzg

Sorry for the late reply. The script is exactly the same one you have. No other changes made except for the backup locations which are created during the installation of the script. I hope this result will be able to tell you exactly what is going on. Base on the changes you ask me to make, this is the result:

[QUOTE]============================

Sat Mar 27 02:00:02 EDT 2010

Performing DIFF backup

============================

diff Backup started at: Sat Mar 27 02:00:02 EDT 2010

Setting date & version Marker into /opt/zimbra/DATE_VERSION.txt

Doing a hotsync of /opt/zimbra/ to /tmp/fakebackup/

Disabling the Zimbra crontab...

Stopping the Zimbra server...
Host zimbra.mail.uk.com

Stopping stats...Done.

Stopping mta...Done.

Stopping spell...Done.

Stopping snmp...Done.

Stopping archiving...Done.

Stopping antivirus...Done.

Stopping antispam...Done.

Stopping imapproxy...Done.

Stopping memcached...Done.

Stopping mailbox...Done.

Stopping logger...Done.

Stopping ldap...Done.

Doing a fast cold sync...

Reinstating Zimbra's crontab...

Starting Zimbra...

Host zimbra.mail.uk.com

Starting ldap...Done.

Starting logger...Done.

Starting mailbox...Done.

Starting antispam...Done.

Starting antivirus...Done.

Starting snmp...Done.

Starting spell...Done.

Starting mta...Done.

Starting stats...Done.

Service down time was - Hr:0 Min:3 Sec:22

Running A hack... This one to check and start Stats subsystem

Hack not needed Stats seems to be running fine...

Writing a diff backup: 12_ZimbraBackup_20100327_DIFF

into: /Backup/zimbra_dars/ with file sizes of max: 4395M

+ echo 'Saving Unencrtyped Archive...'

Saving Unencrtyped Archive...

+ set -x

+ '[' diff = full ']'

+ '[' diff = diff ']'

++ pwd

+ nice -19 /usr/bin/dar -s 4395M -z9 -Z '*.gz' -Z '*.zip' -Z '*.bz2' -Z '*.tgz' -Z '*.zgz' -Z '*.jar' -Z '*.tiff' -Z '*.jpg' -Z '*.png' -Z '*.gif' -Z '*.jpeg' -R /tmp/fakebackup -c /Backup/zimbra_dars/12_ZimbraBackup_20100327_DIFF -Q -A /Backup/zimbra_dars/12_ZimbraBackup_20100326_FULL -Q


--------------------------------------------

28498 inode(s) saved

with 5288 hard link(s) recorded

0 inode(s) changed at the moment of the backup

269550 inode(s) not saved (no inode/file change)

0 inode(s) failed to save (filesystem error)

0 inode(s) ignored (excluded by filters)

1944 inode(s) recorded as deleted from reference backup

--------------------------------------------

Total number of inode considered: 299992

--------------------------------------------

EA saved for 0 inode(s)

--------------------------------------------

+ '[' 0 -ne 0 ']'

+ cd /Backup/zimbra_dars/

++ ls -A 12_ZimbraBackup_20100327_DIFF.1.dar

+ for FILENAME in '`ls -A "$ARCHIVENAME"*`'

+ '[' -e 12_ZimbraBackup_20100327_DIFF.1.dar ']'

+ echo 'Creating MD5 Checksum for 12_ZimbraBackup_20100327_DIFF.1.dar...'

Creating MD5 Checksum for 12_ZimbraBackup_20100327_DIFF.1.dar...

+ /usr/bin/md5sum.textutils -b 12_ZimbraBackup_20100327_DIFF.1.dar

+ '[' 0 -ne 0 ']'

+ '[' yes = yes ']'

++ ls -A 12_ZimbraBackup_20100327_DIFF.1.dar 12_ZimbraBackup_20100327_DIFF.1.dar.md5

+ CPNAME='12_ZimbraBackup_20100327_DIFF.1.dar

12_ZimbraBackup_20100327_DIFF.1.dar.md5'

+ echo 'copy archive to 192.168.1.23' remote directory /zimbra_backup/

copy archive to 192.168.1.23 remote directory /zimbra_backup/

+ scp -i /root/.ssh/id_rsa 12_ZimbraBackup_20100327_DIFF.1.dar 12_ZimbraBackup_20100327_DIFF.1.dar.md5 remote@192.168.1.23:/zimbra_backup/

+ '[' 0 -ne 0 ']'

+ echo 'Creating file listing from archive...'

Creating file listing from archive...

+ '[' no = yes ']'

+ '[' yes = yes ']'

+ nice -19 /usr/bin/dar -l /Backup/zimbra_dars/12_ZimbraBackup_20100327_DIFF -Q

+ grep -v '[ ] [-----]'

+ gzip -9 /Backup/zimbra_dars/12_ZimbraBackup_20100327_DIFF.txt

+ STOPTIME=(`date +%s`)

++ date +%s

++ expr 1269671199 - 1269669602

+ RUNTIME=1597

+ hours=0

+ RUNTIME=1597

+ minutes=26

+ seconds=37

+ echo
+ echo ::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::

::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::

++ date +%H:%M

+ echo 'diff Zimbra Backup ended at: 02:26'

diff Zimbra Backup ended at: 02:26

+ echo 'Backup took Hr:0 Min:26 Sec:37 to complete'

Backup took Hr:0 Min:26 Sec:37 to complete

+ echo ':::::::::::::::Cheers Osoffice for the script:::::::::::::::::::::::'

:::::::::::::::Cheers Osoffice for the script:::::::::::::::::::::::

+ '[' no = yes ']'

+ cat /var/log/zim_backup.log

+ mail -c secondrecipient@mail.uk.com -s 'Zimbra diff Backup Log on zimbra.mail.uk.com' admin@zimbra.mail.uk.com

[/QUOTE]
Post Reply