Zimbra Forums

I am looking for some assistance in recovering our email server. It is zcs-8.0.6_GA_5922.UBUNTU10_64.20131203103719 running on Ubuntu 10.04 LTS 64 Bit. It was running fine and was upgraded in early december to this version following emails from zimbra saying there were security issues with previous versions. It has been very stable and normally works great. But we currently have no email services for the office and I cannot seem to find a solution to this. I have been up most of the night trying to find an answer to no avail.
We had a power cut and this caused the server to halt unexpectedly. On restarting the LDAP configuration/database appeared to be broken and as a result zimbra would not start. See below:


$ sudo su - zimbra

$ zmcontrol start

Host zimbra.twhg.home

Starting ldap...Done.

Unable to determine enabled services from ldap.

Enabled services read from cache. Service list may be inaccurate.

Starting zmconfigd...Failed.

Starting zmconfigd...failed.

Starting logger...Failed.

Starting logswatch...ERROR: account.NO_SUCH_SERVER (no such server: zimbra.twhg.home)

zimbra logger service is not enabled! failed.

Starting mailbox...Failed.

Starting memcached...Done.

Starting antispam...Done.

Starting antivirus...Done.

Starting opendkim...Failed.

/opt/zimbra/bin/zmopendkimctl: line 54: kill: (2929) - No such process

opendkim: /opt/zimbra/conf/opendkim.conf: ldap://zimbra.twhg.home:389/?DKIMSelector?sub?(DKIMIdentity=$d): dkimf_db_open(): Invalid credentials

Failed to start opendkim: 0

Starting snmp...Done.

Starting spell...Done.

Starting mta...Done.

Starting stats...Done.


As you can see LDAP seems to think that our mail server zimbra.twhg.home does not exist.
Can anyone help please as I cannot get this to start. I have read the forums looking for something to help but similar problems seems to be related to the Network Edition and point to directories I do not have e.g. /opt/zimbra/openldap-data.
i.e. [SOLVED] LDAP / slapd - Database environment corrupt (Issue & Solution)
Your help and assistance will be greatfully recieved by a very strung out admin
Many thanks

Stephen

Logs from both:
/var/log/zimbra.log

/opt/zimbra/log/mailbox.log
may be more revealing. Also, just on a hunch do the following:
su zimbra

zmcontrol stop

cd /opt/zimbra/log

rm *.pid -f
zmcontrol start.
If things startup, great! If not, post your log entries.
Doug

lytiedd,
Thank you for responding. After checking endless posts and looking in various config files I believe this is related to the ldap configuration. Since the server was upgraded from v7 to 8.0.4 then onto 8.0.6 it appears that my backups have not been working either, so I do not have a backup from before the crash due to it tryin gto copy an 80+ GB mdb file. Is there a way to recreate the ldap db and config from this system. I have various ldap.bak files in the /opt/zimbra/data/ldap directory that seem to contain all my accounts and user account information. If I look at the last ldap.bak file from the middle of the night after I tried to get it back up it contains none of my Server and User account details.
I have tried to run the 8.0.6 upgrade over itself and use the same options as when it was originally upgraded from 8.0.4 to make sure that no system files were corrupted it runs fine and checks out th emysql db integrity fine but then even though it seems to have the correct domain information it puts me in the main configuration menu as if this was a new installation with *** against various ldap passwords. When I check those items the passwords are all correct but show as not verified in the menus thereby not allowing me to save the config and I have to quit out to get back to the prompt.
Even if I can restore the ldap config and db from December 21st from the /opt/zimbra/data/ldap directory I will be very happy at this point. As you can see from below this all tracks back to some kind of LDAP credentials/Config/db corruption. If I look at the ldap.bak file from the 21st december all the correct information seems to be in there. This would have been created when the system was upgraded to 8.0.6 - How can I get this system back up or recreate the LDAp data. My messag estore etc. all seem to be still there? My instinct is it is all related to LDAP issues.
Excerpt from zimbra.log from when it crashed at 02:16 and then I restarted it at 02:46 and mailbox.log are attached.
Many thanks for your help.
Stephen
excerpt-mailbox.log
excerpt-zimbra.log

The logs indicate invalid credentials on your LDAP, which leads to:
LDAP: error code 49 - Invalid Credentials
Doug

[quote user="lytledd"]The logs indicate invalid credentials on your LDAP, which leads to:
LDAP: error code 49 - Invalid Credentials
Doug[/QUOTE]
Doug,
I still don't understand this or how to fix it. If I look at the config files then the correct passwords are in there. If I run the upgrade over itself it puts me in to the configration menu saying that various ldap passwords are not verified. But selecting them and entering the correct/only passwords ever used on this system results in no change. All I can do is quit out of the menu. So the system is upgraded but not configured.
Is there some way to restore the config from the backups in /opt/zimbra there are various config.xxxxxxx files one is from Dec 21st same as the ldap.bak.xxxxxxx in /opt/zimbra/data/ldap whatis the point of these backup files if you cannot restore from them. I really need some detailed help as to how to fix this as I really don't understand how this can have happened from a power cut. Can you provide some more detail please.
Many thanks

Stephen

[quote user="lytledd"]The logs indicate invalid credentials on your LDAP, which leads to:
LDAP: error code 49 - Invalid Credentials
Doug[/QUOTE]
Any idea why I can't just post in the forum. My posts seems to be waiting on a moderator to allow them!
Anyway when I run zmlocalconfig -s ldap_root_password it shows the correct password.
Help appreciated.
Stephen

I have no real answers for you, other then what I'm finding via Google. The last search I did was on zimbra ldap recovery and it turned up this:
disaster recovery help: lost ldap db to fs corruption
Doug

Doug,
Thank you, but the only ldap.bak files I have are in /opt/zimbra/data/ldap there are none in the /tmp directory, are these the same files that I can use to recreate the ldap config in /opt/zimbra/data/ldap/config ? I do not have a slapd.conf file in /opt/zimbra/conf
It seems there are 2 problems I have, there may only be one; that is the configuration at startup!!!:
1) The actual config, which I think is taken from localconfig.xml
I have various config.bak.xxxxxxx files in the opt/zimbra/directory the one from Dec 21st seems to contain al the information on my configuration server passwords networks etc. How do I restore this configuration to my server and create a new correct startup config? It must be created for a reason like this some kind of problem with the config file that can be recovered using the config.bak.xxxxxxx files.
2) Potentially some kind of issue with the ldap config and db
I have various ldap.bak.xxxxxxxxx files in /opt/zimbra/data/ldap again I have one from Dec 21st when I look in it it seems ot contain all of the information for my server domain, user and system accounts, COS settings, everthing. How can I use this file to re-create the config and or ldap DB to match the config file in 1) above. Again this must have been created for this type of reason, some kind of problem with the ldap config/db after upgrade.
Can anyone help me please. Is there anyone from Zimbra monitoring this thread that can help, I can assure you anyone can get into this situation from a simple power cut. I know its my fault that I did not monitor that the backups were running ok, they always were previously Ver 7.x but I have none of those files left. When I get this back up again I will need to work out a new backup strategy that gets around the 80G ldap db file problem that 8.0.x seems to have introduced.
As always hoping that this community can assist me in recoving this server before I get fired!!!
Below is what happens when I upgrade 8.0.6 over itself. I beleive the main problem is that it has created a default ldap db/config and therefore cannot find my server in ldap etc. I have found backups in /opt/zimbra/.saveconfig these files must be able to be used somehow to create the correct configs but I cannot find any information as to how they can be used. I beleive what I need to do is restore my ldap db and config from the Dec 21st files and then reboot the server so it is using the correct startup config and ldap db/ There have been no changes to our setup for a long time i.e. no new accounts or changes to server config for at least a year or more. So Dec 21st configs should be exactly the same as 2 days ago. I seem to be in the position that it is using a deafult ldap DB and config even though my server exists therefore the details cannot be found hence the server not found errors.
At the end it throws me in to the main configuration menu from where I cannot do anything. All passwords are correct for how the system was before the crash. When I check the passwords they are all there and zmlocalconfig -s | grep -i pass shows all passwords correct as they were before the crash.
zcs-8.0.6_GA_5922.UBUNTU10_64.20131203103719$ sudo ./install.sh
Operations logged to /tmp/install.log.3448

Checking for existing installation...

zimbra-ldap...FOUND zimbra-ldap-8.0.6.GA.5922.UBUNTU10.64

zimbra-logger...FOUND zimbra-logger-8.0.6.GA.5922.UBUNTU10.64

zimbra-mta...FOUND zimbra-mta-8.0.6.GA.5922.UBUNTU10.64

zimbra-snmp...FOUND zimbra-snmp-8.0.6.GA.5922.UBUNTU10.64

zimbra-store...FOUND zimbra-store-8.0.6.GA.5922.UBUNTU10.64

zimbra-apache...FOUND zimbra-apache-8.0.6.GA.5922.UBUNTU10.64

zimbra-spell...FOUND zimbra-spell-8.0.6.GA.5922.UBUNTU10.64

zimbra-convertd...NOT FOUND

zimbra-memcached...FOUND zimbra-memcached-8.0.6.GA.5922.UBUNTU10.64

zimbra-proxy...NOT FOUND

zimbra-archiving...NOT FOUND

zimbra-cluster...NOT FOUND

zimbra-core...FOUND zimbra-core-8.0.6.GA.5922.UBUNTU10.64

ZCS upgrade from 8.0.6 to 8.0.6 will be performed.
Saving existing configuration file to /opt/zimbra/.saveconfig


PLEASE READ THIS AGREEMENT CAREFULLY BEFORE USING THE SOFTWARE.

ZIMBRA, INC. ("ZIMBRA") WILL ONLY LICENSE THIS SOFTWARE TO YOU IF YOU

FIRST ACCEPT THE TERMS OF THIS AGREEMENT. BY DOWNLOADING OR INSTALLING

THE SOFTWARE, OR USING THE PRODUCT, YOU ARE CONSENTING TO BE BOUND BY

THIS AGREEMENT. IF YOU DO NOT AGREE TO ALL OF THE TERMS OF THIS

AGREEMENT, THEN DO NOT DOWNLOAD, INSTALL OR USE THE PRODUCT.
License Terms for the Zimbra Collaboration Suite:



Do">http://www.zimbra.com/license/zimbra_pu ... a_2.4.html

Do you agree with the terms of the software license agreement? [N] y


Do you agree with the terms of the software license agreement? [N] y
Checking for prerequisites...

FOUND: NPTL

FOUND: netcat-openbsd-1.89-3ubuntu2

FOUND: sudo-1.7.2p1-1ubuntu5.6

FOUND: libidn11-1.15-2

FOUND: libpcre3-7.8-3build1

FOUND: libgmp3c2-2:4.3.2+dfsg-1ubuntu1

FOUND: libexpat1-2.0.1-7ubuntu1.1

FOUND: libstdc++6-4.4.3-4ubuntu5.1

FOUND: libperl5.10-5.10.1-8ubuntu2.3
Checking for suggested prerequisites...

FOUND: pax

FOUND: perl-5.10.1

FOUND: sysstat

FOUND: sqlite3

Prerequisite check complete.

Checking current number of databases...
Do you want to verify message store database integrity? [Y] Y

Verifying integrity of message store databases. This may take a while.

Starting mysqld...done.

mysqld is alive

No errors found

mysqld is alive 1) Common Configuration:

+Hostname: zimbra.twhg.home

+Ldap master host: zimbra.twhg.home

+Ldap port: 389

+Ldap Admin password: set

+Secure interprocess communications: yes

+TimeZone: America/Los_Angeles

+IP Mode: ipv4

+Default SSL digest: sha1
2) zimbra-ldap: Enabled

+Create Domain: no

+Ldap root password: set

+Ldap replication password: set

******* +Ldap postfix password: Not Verified

******* +Ldap amavis password: Not Verified

+Ldap nginx password: set

******* +Ldap Bes Searcher password: Not Verified
3) zimbra-store: Enabled

+Create Admin User: no

+Anti-virus quarantine user: virus-quarantine.dijrrvf3tt@zimbra.twhg.home

+Enable automated spam training: yes

+Spam training user: spam.wpilbeoum@zimbra.twhg.home

+Non-spam(Ham) training user: ham.uejw0jxxl@zimbra.twhg.home

+SMTP host: zimbra.twhg.home

+Web server HTTP port: 80

+Web server HTTPS port: 443

+Web server mode: https

+IMAP server port: 143

+IMAP server SSL port: 993

+POP server port: 110

+POP server SSL port: 995

+Use spell check server: yes

+Spell server URL: http://zimbra.twhg.home:7780/aspell.php

******* +Enable version update checks: UNSET
4) zimbra-mta: Enabled

+MTA Auth host: zimbra.twhg.home

+Enable Spamassassin: no

+Enable Clam AV: no

+Enable OpenDKIM: no

******* +Bind password for postfix ldap user: Not Verified

******* +Bind password for amavis ldap user: Not Verified
5) zimbra-snmp: Enabled

6) zimbra-logger: Enabled

7) zimbra-spell: Enabled

8) Default Class of Service Configuration:

r) Start servers after configuration yes

s) Save config to file

x) Expand menu

q) Quit
Stopping mysqld... done.
Checking for installable packages
Found zimbra-core

Found zimbra-ldap

Found zimbra-logger

Found zimbra-mta

Found zimbra-snmp

Found zimbra-store

Found zimbra-apache

Found zimbra-spell

Found zimbra-memcached

Found zimbra-proxy


The Zimbra Collaboration Server appears already to be installed.

It can be upgraded with no effect on existing accounts,

or the current installation can be completely removed prior

to installation for a clean install.
Do you wish to upgrade? [Y] Y
Select the packages to install

Upgrading zimbra-core

Upgrading zimbra-ldap

Upgrading zimbra-logger

Upgrading zimbra-mta

Upgrading zimbra-snmp

Upgrading zimbra-store

Upgrading zimbra-apache

Upgrading zimbra-spell

Upgrading zimbra-memcached
Install zimbra-proxy [N]

Checking required space for zimbra-core

Checking space for zimbra-store
Installing:

zimbra-core

zimbra-ldap

zimbra-logger

zimbra-mta

zimbra-snmp

zimbra-store

zimbra-apache

zimbra-spell

zimbra-memcached
The system will be modified. Continue? [N] Y
Shutting down zimbra mail
Backing up the ldap database...done.
Removing existing packages
zimbra-ldap...done

zimbra-logger...done

zimbra-mta...done

zimbra-snmp...done

zimbra-store...done

zimbra-spell...done

zimbra-memcached...done

zimbra-apache...done

zimbra-core...done
Removing deployed webapp directories

Installing packages
zimbra-core......zimbra-core_8.0.6.GA.5922.UBUNTU10.64_amd64.deb...done

zimbra-ldap......zimbra-ldap_8.0.6.GA.5922.UBUNTU10.64_amd64.deb...done

zimbra-logger......zimbra-logger_8.0.6.GA.5922.UBUNTU10.64_amd64.deb...done

zimbra-mta......zimbra-mta_8.0.6.GA.5922.UBUNTU10.64_amd64.deb...done

zimbra-snmp......zimbra-snmp_8.0.6.GA.5922.UBUNTU10.64_amd64.deb...done

zimbra-store......zimbra-store_8.0.6.GA.5922.UBUNTU10.64_amd64.deb...done

zimbra-apache......zimbra-apache_8.0.6.GA.5922.UBUNTU10.64_amd64.deb...done

zimbra-spell......zimbra-spell_8.0.6.GA.5922.UBUNTU10.64_amd64.deb...done

zimbra-memcached......zimbra-memcached_8.0.6.GA.5922.UBUNTU10.64_amd64.deb...done
Setting defaults from saved config in /opt/zimbra/.saveconfig/config.save

HOSTNAME=zimbra.twhg.home

LDAPHOST=zimbra.twhg.home

LDAPPORT=389

SNMPTRAPHOST=zimbra.twhg.home

SMTPSOURCE=admin@zimbra.twhg.home

SMTPDEST=admin@zimbra.twhg.home

SNMPNOTIFY=yes

SMTPNOTIFY=yes

LDAPROOTPW=TYG6ZKeR

LDAPZIMBRAPW=TYG6ZKeR

LDAPPOSTPW=TYG6ZKeR

LDAPREPPW=TYG6ZKeR

LDAPAMAVISPW=TYG6ZKeR

LDAPNGINXPW=TYG6ZKeR

Restoring existing configuration file from /opt/zimbra/.saveconfig/localconfig.xml...done

Operations logged to /tmp/zmsetup.01142014-162022.log

Running zmldapapplyldif...done.

Checking ldap status....not running.

Starting ldap...done.

ERROR: account.NO_SUCH_SERVER (no such server: zimbra.twhg.home)

ERROR: account.NO_SUCH_SERVER (no such server: zimbra.twhg.home)

Setting defaults...ERROR: account.NO_SUCH_SERVER (no such server: zimbra.twhg.home)

done.

Setting defaults from existing config...done.

Checking for port conflicts

Setting defaults from ldap...done.

ERROR: account.NO_SUCH_SERVER (no such server: zimbra.twhg.home)

ERROR: account.NO_SUCH_SERVER (no such server: zimbra.twhg.home)
Main menu
Many thanks

Stephen

Why are you running install.sh on top of an existing install? Are you trying to break your installation even further?
Also, there is no "80GB ldap backup problem". This has been discussed in multiple times. I suggest reading OpenLDAP Performance Tuning 8.0 - Zimbra :: Wiki
At this point, there may be no telling what the actual passwords your system used were, given the fact you've re-run the installer multiple times over your existing installation, which if there was an ldap issue when you ran it, may have resulted in causing all sorts of problems. Your best bet would be to restore the ldap server from your most recent backup. If you are on a FOSS installation, I hope you had zmslapcat doing routine backups of the ldap databases to LDIF format.

[quote user="quanah"]Why are you running install.sh on top of an existing install? Are you trying to break your installation even further?
Also, there is no "80GB ldap backup problem". This has been discussed in multiple times. I suggest reading OpenLDAP Performance Tuning 8.0 - Zimbra :: Wiki
At this point, there may be no telling what the actual passwords your system used were, given the fact you've re-run the installer multiple times over your existing installation, which if there was an ldap issue when you ran it, may have resulted in causing all sorts of problems. Your best bet would be to restore the ldap server from your most recent backup. If you are on a FOSS installation, I hope you had zmslapcat doing routine backups of the ldap databases to LDIF format.[/QUOTE]
Thank you for your prompt response. No I am not trying to break my system further. I am working on a full backup of the severs .vmdk file I created yesterday so that I can get back to the broken system if things go really wrong. I have researched the 80GB problem and now understand that I need to take ldiff backups and create ldap.bak files once I get the server back up and running. My backup strategy was the same as lot of zimbra users, rsync which worked great for a long time. Clearly after 8.0.1 this is a problem that I did not pickup on. So I do not have a usable backup of my system. This is a moot point right now as I cannot get the system working The only backups I have are the ones created locally on my system by the install/upgrade script. There are various config.bak and ldap.bak and mdb.prev files on my system and files in ~/.saveconfig/8.0.6.GA.5922.UBUNTU10.64/0 that are from the 21st december when the system was first upgraded and ran perfectly until the crash 2 days ago. This is the only known good configuration infromation I have to restore from as the system was successfully upgraded using the system data from that date.
In the /opt/zimbra/data.ldap.bak.xxxxxxx file from the 21st dec it contains all the information of my system, server, system and user accounts COS setting settings etc. How can I use this ldap.bak file to restore my ldap config/db etc. as of the 21st dec. Likewise the information in /opt/zimbra/config.bak.xxxxxx from the 21st Dec also seems to contain all the correct information on my server. Is it possible to roll back my server to 8.0.4 and then re-upgrade it? My data store appears intact so presumably all email messages etc. are still there, its just the system does not know how to get at them due to config/ldap issues.
Do you agree with my analysis that I have oneor both of the issues I outlined at the top of my previous post. Can you provide any assistance as to how I can recreate my LDAP config/DB and system configs from the backups zimbra made locally.
Many thanks

Stephen
I