Best regards
Stephen
OK here is the log of all the commands/actions that were performed on my server to get it back up again. I cannot stress how brilliant Quanah was in helping me to recover this situation. What an absolute superstar and very gracious in his explanations and insight. I would like to thank everyone who assisted me to get this back up. I hope the information is useful to others please feel free to PM me for any assistance.
At the start of this log the system was broken it would not start and if you have read my previous posts it was clear that the ldap system and possibly the config was broken. Partially by the power cut and probably by my running the upgrade script over itself a number of times when the LDAP system ws broken. That had in effect restored the ldap db to default thereby causing all sorts of issues with server not found 389 errors etc. The only thing i have changed are the password entries returned from the ldap lookups to confirm they were correct/existant etc. after restoring the 21st Dec ldap.bak DB.
In summary the procudure involved restoring a last known good ldap.bak file from 21st Dec which was created at upgrade time from 8.0.4. Then checking various permissions and the localconfig against the config.bak from the 21st Dec and making some changes to bring the schema upto 8.0.6.
The first red line below was after the first startup. The server was running but the Antivirus zmamavis was not running and would not start. The problem was essentially this one Upgrade Zimbra Proxy from 8.0.2 to 8.0.6 Breaks things after setting the zimbraAmavisLogLevel 1 and and checking and setting various keys to ensure the schema matched 8.0.6 not 8.0.4 and tuning the ldap db maxsize the system was restarted and came up perfectly the second red point. Then cold reboot to check if any issues and none were found. Then cold backup etc. see below. I will be looking for some help around automating the backup procedure
cd data/ldap/
ls -l
vi ldap.bak.20131221123538
ls
cd /tmp
lsred
rm q.*
cd
cd data/ldap/config/
ls
cd ..
ls
ls -l
grep zimbraThrottleWhitelist ldap.bak.20140113032714
grep zimbraDeviceCalendarSoftDeleteExcludePattern *
grep zimbraItemActionBatchSize *
zmprov gcf zimbraItemActionBatchSize
zmprov mcf zimbraItemActionBatchSize 1000
grep zimbraReverseProxyExternalRouteIncludeOriginalAuthusername *
zmprov mcf zimbraReverseProxyExternalRouteIncludeOriginalAuthusername FALSE
grep zimbraAntispamExtractionBatchSize *
zmprov mcf zimbraAntispamExtractionBatchSize 25
zmprov mcf zimbraAntispamExtractionBatchDelay 100
zmcontrol status
ps -eaf | grep zimbra
cd data/ldap/
ls
ls -l
ls -altr
ls /tmp
mv mdb mdb.broken
mkdir -p mdb/db
ls
cd config
ls
ls -altr
cd cn=config
ls
ls -altr
more olcDatabase={2}mdb.ldif
ls
cd ..
ls
cd ..
ls
ls -l
cd
./libexec/zmslapadd /opt/zibmra/data/ldap/ldap.bak.20131221123538
./libexec/zmslapadd /opt/zimbra/data/ldap/ldap.bak.20131221123538
cd
ldap start
zmlocalconfig | grep ldap | grep password
zmlocalconfig -s | grep ldap | grep password
ldapsearch -x -H ldapi:/// -D cn=config -w XXXXXXXX -b "" -s base
ldapsearch -LLL -x -H ldapi:/// -D cn=config -w XXXXXXXX -b "" -s base
ldapsearch -LLL -x -H ldapi:/// -D uid=zimbra,cn=admins,cn=zimbra -w XXXXXXXX -$
ldapsearch -LLL -x -H ldapi:/// -D uid=replica,cn=admins,cn=zimbra -w XXXXXXXX $
cd .saveconfig/
ls
cd 8.0.6.GA.5922.UBUNTU10.64/
ls -altr
cd 3
ls -altr
cat localconfig.xml
cd
zmlocalconfig
pwd
cd .saveconfig/
ls
cd 8.0.6.GA.5922.UBUNTU10.64/
ls
cd 0
ls
diff localconfig.xml /opt/zimbra/conf/localconfig.xml
cd
zmlocalconfig ssl_default_digest
cd -
ls -l
cd
zmcontrol start
zmcontrol status
zmamavisdctl start
cd conf
vi amavsid.conf.in
ls
vi amavisd.conf.in
cd
hostname
zmlocalconfig | grep host
zmprov gs zimbra.twhg.home zimbraAmavisLogLevel
cd data/ldap/
ls
ls -l
vi ldap.bak.20140113032714
zmprov mcf zimbraAmavisLogLevel 1
zmamavisdctl start
vi ldap.bak.20140113032714
zmprov mcf +zimbraStatThreadNamePrefix qtp
vi /tmp/q.1
cd /tmp
ls
mv 1.2 q.2
rm q.
diff -u q.1 q.2 | more
diff -u q.2 q.1 | more
diff -u q.2 q.1 | more
clear
cd /tmp
ls
mkdir config
cd config
ls
cd
clear
cd data/ldap/mdb/db
ls -l
du -c -h data.mdb
man rsync
ls
zmlocalconfig ldap_db_maxsize
zmlocalconfig -e ldap_db_maxsize=2147483648
ls -l
tail -f /var/log/zimbra.log
ls -l
cd
zmcontrol -v
zmprov help cos
zmprov gac
zmprov mc default zimbraMobileOutlookSyncEnabled TRUE
zmprov mc defaultExternal zimbraMobileOutlookSyncEnabled TRUE
zmprov mc default zimbraFeatureFromDisplayEnabled TRUE
zmprov mc defaultExternal zimbraFeatureFromDisplayEnabled TRUE
zmcontrol restart
zmcontrol status
zmcontrol stop
zmcontrol status
tail -f /var/log/zimbra.log
exit
sudo reboot now
After all was well I then took a full cold backup of the system, backing up the LDAP DB and config the correct way (thank you Quanah) to a backup directory in /opt/zimbra/data/ldap-backup/YYYYMMDDhhmmss. I then ran rsync outside of zimbra and created a full backup of the /opt/zimbra directory tree excluding the data/ldap/mdb/db and any previous mdb.prev directories. My system is approx. 11.3 GB and it took about half an hour. I am sure there is a better way of doing this but I did it long hand and cold to ensure I had at least a manual methodology that worked.
I would welcome any suggestions on the correct rsync options or pointers to 8.0.6 related backup scripts. We backup to a Raid 6 NAS server over the lan. I have seen many posts taking a hot rsync first and then a cold one to minimise down time. I am not quite there yet but I think I am heading in the right direction
sudo su - zimbra
zmcontrol stop
zmcontrol status
tail -f /var/log/zimbra.log
cd data/ldap
ls
rm -Rvf mdb.broken
mydir="/opt/zimbra/data/ldap-backup/`date +%Y%m%d%H%M%S`"
mkdir -p $mydir
cd data
ls -lash
cd ldap-backup
/opt/zimbra/libexec/zmslapcat $mydir
/opt/zimbra/libexec/zmslapcat -c $mydir
exit
sudo rsync -a -v -r -l -t -S -H --progress --exclude=data/ldap/mdb/db --exclude=data/ldap/mdb.prev* /opt/zimbra remote@remote:/remotezimbradir/
sudo su - zimbra
zmcontrol status
zmcontrol restart
zmcontrol status
tail -f /var/log/zimbra.log