I need to do 'zmcontrol start' twice to get zimbra up

Ask questions about your setup or get help installing ZCS server (ZD section below).
apiening
Posts: 30
Joined: Tue Aug 30, 2016 9:57 pm

I need to do 'zmcontrol start' twice to get zimbra up

Post by apiening »

Hi there,

I have Zimbra "8.6.0.GA.1153.UBUNTU14.64 UBUNTU14_64 FOSS edition" running, and it works very well.

The issue I have is that I need to do 'zmcontrol start' twice to get zimbra starting successfully.

First attempt:

Code: Select all

$ zmcontrol start
Host myhost.com
           Starting ldap...Done.
Failed.
Started slapd: pid 30597
Error: Unable to check that slapd is listening to connections
second attempt:

Code: Select all

$ zmcontrol start
Host myhost.com
       	Starting zmconfigd...Done.
       	Starting dnscache...Done.
       	Starting logger...Done.
       	Starting mailbox...Done.
       	Starting memcached...Done.
       	Starting proxy...Done.
       	Starting amavis...Done.
       	Starting antispam...Done.
       	Starting antivirus...Done.
       	Starting opendkim...Done.
       	Starting snmp...Done.
       	Starting spell...Done.
       	Starting mta...Done.
       	Starting stats...Done.
       	Starting service webapp...Done.
       	Starting zimbra webapp...Done.
       	Starting zimbraAdmin webapp...Done.
       	Starting zimlet webapp...Done.
The first failed attempt took 37 seconds, the second took 41 seconds to bring all services up.

It looks like it did not even tried to start the ldap server since it was already running. So maybe just the check fails (Error: Unable to check that slapd is listening to connections)?

I tried to do an upgrade to the latest zimbra release but the migration script fails with the same error message because it need to restart zimbra. Since I could not "resume" the upgrade process, I needed to revert the system from my backup.

Every hint how I can solve my start issue is greatly appreciated!

Andreas
Last edited by apiening on Tue Aug 30, 2016 10:24 pm, edited 1 time in total.
User avatar
jorgedlcruz
Zimbra Alumni
Zimbra Alumni
Posts: 2782
Joined: Thu May 22, 2014 4:47 pm

Re: I need to do 'zmcontrol start' twice to get zimbra up

Post by jorgedlcruz »

Hello,
Would you mind to let us know your CPU and RAM configuration? Also which Hard drives do you have?

Best regards
Jorge de la Cruz https://jorgedelacruz.es
Systems Engineer at Veeam Software https://www.veeam.com/
apiening
Posts: 30
Joined: Tue Aug 30, 2016 9:57 pm

Re: I need to do 'zmcontrol start' twice to get zimbra up

Post by apiening »

Hi Jorge,

my CPU is a "i7-6700 CPU @ 3.40GHz" and and I have a total of 32 GB RAM in the system but for the VM running zimbra I have configured 8 GB RAM.

Code: Select all

$ free -h
             total       used       free     shared    buffers     cached
Mem:          8.0G       1.7G       6.3G       122M         0B       109M
-/+ buffers/cache:       1.6G       6.4G
Swap:         8.0G       4.6M       8.0G
The hard drives are two 4 TB drives in RAID 1 (ST4000NM0024).

Kind regards

Andreas
User avatar
jorgedlcruz
Zimbra Alumni
Zimbra Alumni
Posts: 2782
Joined: Thu May 22, 2014 4:47 pm

Re: I need to do 'zmcontrol start' twice to get zimbra up

Post by jorgedlcruz »

Do you have any error on the mailbox.log, or zimbra.log, or zmmailboxd.out?

Best regards
Jorge de la Cruz https://jorgedelacruz.es
Systems Engineer at Veeam Software https://www.veeam.com/
apiening
Posts: 30
Joined: Tue Aug 30, 2016 9:57 pm

Re: I need to do 'zmcontrol start' twice to get zimbra up

Post by apiening »

Hi,

I did a fresh restart to get current log entries from the zimbra startup.

When I grep on "error" in my /var/log/zimbra.log I got this:

Code: Select all

Aug 31 09:46:57 zimbra amavis[28640]: (28640-07) WARN: MIME::Parser error: part did not end with expected boundary; ; error: unexpected end of parts before epilogue
Aug 31 10:00:12 zimbra amavis[28640]: (28640-08) WARN: MIME::Parser error: part did not end with expected boundary; ; error: unexpected end of parts before epilogue
Aug 31 10:04:44 zimbra amavis[28667]: (28667-05) WARN: MIME::Parser error: part did not end with expected boundary; ; error: unexpected end of parts before epilogue
Aug 31 10:05:15 zimbra amavis[28649]: (28649-09) WARN: MIME::Parser error: part did not end with expected boundary; ; error: unexpected end of parts before epilogue
Aug 31 10:12:57 zimbra clamd[25079]: TCP: bind() error when trying to listen on [localhost]:3310: Cannot assign requested address
Grepping for "error" in /opt/zimbra/log/mailbox.log doesn't catch anything. When I look at this file I get tons of login entries at the INFO level, there is a WARN message:

Code: Select all

2016-08-31 10:14:09,474 WARN  [ImapSSLServer-1] [name=mylogin@mydomain.com;ip=158.101.52.72;oip=177.87.17.216;via=Mac OS X Mail/9.3 (3124),138.201.2.172(nginx/1.2.0-zimbra);ua=Zimbra/8.6.0_GA_1153;] ConfigurationFactory - No configuration found. Configuring ehcache from ehcache-failsafe.xml  found in the classpath: jar:file:/opt/zimbra/jetty-distribution-9.1.5.v20140505/webapps/service/WEB-INF/lib/ehcache-core-2.5.1.jar!/ehcache-failsafe.xml
If I grep for "error" on /opt/zimbra/log/zmmailboxd.out I get this line twice:

Code: Select all

log4j:ERROR No syslog host is set for SyslogAppedender named "SLOGGER".
Kind regards

Andreas
apiening
Posts: 30
Joined: Tue Aug 30, 2016 9:57 pm

Re: I need to do 'zmcontrol start' twice to get zimbra up

Post by apiening »

Finally, with a lot of help of the ZIMBRA cummunity on IRC, I do know what this issue is caused by.

zmcontrol start uses a method checkListening() from the script /opt/zimbra/bin/ldap to check if the LDAP server (a.k.a. slapd) successfully started. It uses the following command to do the connection test:

Code: Select all

/opt/zimbra/openldap/bin/ldapsearch -x -l 30 -b "" -s base -H ldapi:///
Well, this command fails on my system with the message "Can't contact LDAP server". I can run this command successfully as root, but it fails when I try to execut it as the user "zimbra".
The command creates a connection via the socket /opt/zimbra/data/ldap/state/run/ldapi. Unfortunately, the socket is created with write permissions for root only.

Code: Select all

$ ls -lah /opt/zimbra/data/ldap/state/run/ldapi
srwxr-xr-x 1 root root 0 Aug 31 22:35 /opt/zimbra/data/ldap/state/run/ldapi
When I do a chmod 777 on this socket, the connection succeeds.

My only workaround so far is to do something like this in another terminal or screen session:

Code: Select all

while true; do chmod 777 /opt/zimbra/data/ldap/state/run/ldapi; sleep 1; done;
While this runs, I can stop and start zimbra without any issues. The real solution however would be to either change the check command to not use the socket or to create the socket with write permissions for others / zimbra.

Does anyone know what this issue is caused by? Could this be a configuration problem on my system or is this a bug?
apiening
Posts: 30
Joined: Tue Aug 30, 2016 9:57 pm

Re: I need to do 'zmcontrol start' twice to get zimbra up

Post by apiening »

So what I tried is to leave my chown-loop running in the background in a screen session:

Code: Select all

while true; do chmod 777 /opt/zimbra/data/ldap/state/run/ldapi; sleep 1; done;
And then I did an update of zimbra from 8.6 to 8.6.0_GA_1200 with the update-patch, which worked (the failed restarts of LDAP prohibited the update beforehand). I then even successfully updated to 8.7.0_GA_1659.

However, the problem still exists with the same behaviour as before.
Knowing this and being able to get around this issue is ok to me. But I still cannot believe that I'm the only one experiencing this problem.

Should I open a bug report for this?
djzort
Posts: 17
Joined: Thu Dec 21, 2017 11:09 am

Re: I need to do 'zmcontrol start' twice to get zimbra up

Post by djzort »

This is still a problem for me in 8.7.11, has anyone spotted a fix?
djzort
Posts: 17
Joined: Thu Dec 21, 2017 11:09 am

Re: I need to do 'zmcontrol start' twice to get zimbra up

Post by djzort »

djzort
Posts: 17
Joined: Thu Dec 21, 2017 11:09 am

Re: I need to do 'zmcontrol start' twice to get zimbra up

Post by djzort »

it seems x-mod is disabled. thats what im seeing too when i add it to the start up. see https://bugzilla.redhat.com/show_bug.cgi?id=211045
Post Reply