Zimbra node starting up with errors. Please help

Discuss your pilot or production implementation with other Zimbra admins or our engineers.
Post Reply
kazishe
Posts: 3
Joined: Sat Jun 23, 2018 3:03 pm

Zimbra node starting up with errors. Please help

Post by kazishe »

Dears, My Zimbra version is Release 8.7.0.GA.1659.UBUNTU16.64 UBUNTU16_64 FOSS edition and running on Ubuntu 16.04.3 LTS (Xenial). I am really experienced on working with Zimbra, but I would appreciate if someone can show me where to look and what to look for.

On trying to start Zimbra I get the following:
zimbra@mail:~/libexec$ zmcontrol status
Connect: Unable to determine enabled services from ldap.
Enabled services read from cache. Service list may be inaccurate.
Host mail.mydomain.co.zw
sh: 1: cannot create /opt/zimbra/data/tmp/zmcontrol.status.i1p4r: Read-only file system
amavis Stopped
sh: 1: cannot create /opt/zimbra/data/tmp/zmcontrol.status.i1p4r: Read-only file system
antispam Stopped
sh: 1: cannot create /opt/zimbra/data/tmp/zmcontrol.status.i1p4r: Read-only file system
antivirus Stopped
sh: 1: cannot create /opt/zimbra/data/tmp/zmcontrol.status.i1p4r: Read-only file system
ldap Stopped
sh: 1: cannot create /opt/zimbra/data/tmp/zmcontrol.status.i1p4r: Read-only file system
logger Stopped
sh: 1: cannot create /opt/zimbra/data/tmp/zmcontrol.status.i1p4r: Read-only file system
mailbox Stopped
sh: 1: cannot create /opt/zimbra/data/tmp/zmcontrol.status.i1p4r: Read-only file system
memcached Stopped
sh: 1: cannot create /opt/zimbra/data/tmp/zmcontrol.status.i1p4r: Read-only file system
mta Stopped
sh: 1: cannot create /opt/zimbra/data/tmp/zmcontrol.status.i1p4r: Read-only file system
opendkim Stopped
sh: 1: cannot create /opt/zimbra/data/tmp/zmcontrol.status.i1p4r: Read-only file system
proxy Stopped
sh: 1: cannot create /opt/zimbra/data/tmp/zmcontrol.status.i1p4r: Read-only file system
service webapp Stopped
sh: 1: cannot create /opt/zimbra/data/tmp/zmcontrol.status.i1p4r: Read-only file system
snmp Stopped
sh: 1: cannot create /opt/zimbra/data/tmp/zmcontrol.status.i1p4r: Read-only file system
spell Stopped
sh: 1: cannot create /opt/zimbra/data/tmp/zmcontrol.status.i1p4r: Read-only file system
stats Stopped
sh: 1: cannot create /opt/zimbra/data/tmp/zmcontrol.status.i1p4r: Read-only file system
zimbra webapp Stopped
sh: 1: cannot create /opt/zimbra/data/tmp/zmcontrol.status.i1p4r: Read-only file system
zimbraAdmin webapp Stopped
sh: 1: cannot create /opt/zimbra/data/tmp/zmcontrol.status.i1p4r: Read-only file system
zimlet webapp Stopped
sh: 1: cannot create /opt/zimbra/data/tmp/zmcontrol.status.i1p4r: Read-only file system
zmconfigd Stopped


trying to start services does not even attempt much:
zmcontrol start
Host mail.mydomain.co.zw
Starting ldap...sh: 1: cannot create /opt/zimbra/data/tmp/zmcontrol.out.zFF4i: Read-only file system
Done.
cat: /opt/zimbra/data/tmp/zmcontrol.out.zFF4i: No such file or directory
Failed.


Many thanks for you help.
User avatar
L. Mark Stone
Ambassador
Ambassador
Posts: 2796
Joined: Wed Oct 09, 2013 11:35 am
Location: Portland, Maine, US
ZCS/ZD Version: 10.0.6 Network Edition
Contact:

Re: Zimbra node starting up with errors. Please help

Post by L. Mark Stone »

This is not a Zimbra issue.

In my experience, when any Linux server starts up with the file system in read-only mode, it's because there has been an "event", like a power failure, or the disk gets full, or a disk cache controller failed, etc.

The solution is either to start Linux in single-user mode to try to repair the disk, or to boot the system from a USB disk/DVD etc. to be able to repair the disk without it being mounted.

Hope that helps,
Mark
___________________________________
L. Mark Stone
Mission Critical Email - Zimbra VAR/BSP/Training Partner https://www.missioncriticalemail.com/
AWS Certified Solutions Architect-Associate
kazishe
Posts: 3
Joined: Sat Jun 23, 2018 3:03 pm

Re: Zimbra node starting up with errors. Please help

Post by kazishe »

Many thanks Mark. I guess I should have mentioned that the reboot was necessitated by the I error that I am currently getting, which I was getting before. This is the error on trying to start or restart Zimbra.

=================
zmcontrol restart
Host mail.mydomain.co.zw
Stopping zmconfigd...Done.
Stopping zimlet webapp...Done.
Stopping zimbraAdmin webapp...Done.
Stopping zimbra webapp...Done.
Stopping service webapp...Done.
Stopping stats...Done.
Stopping mta...Done.
Stopping spell...Done.
Stopping snmp...Done.
Stopping cbpolicyd...Done.
Stopping archiving...Done.
Stopping opendkim...Done.
Stopping amavis...Done.
Stopping antivirus...Done.
Stopping antispam...Done.
Stopping proxy...Done.
Stopping memcached...Done.
Stopping mailbox...Done.
Stopping logger...Done.
Stopping dnscache...Done.
Stopping ldap...Done.
Host mail.mydomain.co.zw
Starting ldap...Done.
Search error: Unable to determine enabled services from ldap.
Enabled services read from cache. Service list may be inaccurate.
Starting zmconfigd...Failed.
Starting zmconfigd...failed.


Starting logger...Failed.
Starting logswatch...[] INFO: master is down, falling back to replica...
[] FATAL: failed to initialize LDAP client
com.zimbra.cs.ldap.LdapException: LDAP error: : invalid credentials
ExceptionId:main:1529774378402:a3402632940cf418
Code:ldap.LDAP_ERROR
at com.zimbra.cs.ldap.LdapException.LDAP_ERROR(LdapException.java:90)
at com.zimbra.cs.ldap.unboundid.UBIDLdapException.mapToLdapException(UBIDLdapException.java:74)
at com.zimbra.cs.ldap.unboundid.UBIDLdapException.mapToLdapException(UBIDLdapException.java:40)
at com.zimbra.cs.ldap.unboundid.LdapConnectionPool.createConnPool(LdapConnectionPool.java:117)
at com.zimbra.cs.ldap.unboundid.LdapConnectionPool.createConnectionPool(LdapConnectionPool.java:63)
at com.zimbra.cs.ldap.unboundid.UBIDLdapContext.init(UBIDLdapContext.java:109)
at com.zimbra.cs.ldap.unboundid.UBIDLdapClient.init(UBIDLdapClient.java:39)
at com.zimbra.cs.ldap.LdapClient.getInstance(LdapClient.java:65)
at com.zimbra.cs.ldap.LdapClient.initialize(LdapClient.java:88)
at com.zimbra.cs.account.ldap.LdapProv.<init>(LdapProv.java:48)
at com.zimbra.cs.account.ldap.LdapProvisioning.<init>(LdapProvisioning.java:271)
at com.zimbra.cs.account.ldap.LdapProvisioning.<init>(LdapProvisioning.java:268)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at java.lang.Class.newInstance(Class.java:442)
at com.zimbra.cs.account.Provisioning.getInstance(Provisioning.java:287)
at com.zimbra.cs.account.Provisioning.getInstance(Provisioning.java:244)
at com.zimbra.cs.account.ProvUtil.initProvisioning(ProvUtil.java:1004)
at com.zimbra.cs.account.ProvUtil.main(ProvUtil.java:3955)
Caused by: LDAPException(resultCode=49 (invalid credentials), errorMessage='invalid credentials')
at com.unboundid.ldap.sdk.LDAPConnection.bind(LDAPConnection.java:1894)
at com.unboundid.ldap.sdk.LDAPConnectionPool.createConnection(LDAPConnectionPool.java:988)
at com.unboundid.ldap.sdk.LDAPConnectionPool.<init>(LDAPConnectionPool.java:876)
at com.unboundid.ldap.sdk.LDAPConnectionPool.<init>(LDAPConnectionPool.java:779)
at com.unboundid.ldap.sdk.LDAPConnectionPool.<init>(LDAPConnectionPool.java:726)
at com.zimbra.cs.ldap.unboundid.LdapConnectionPool.createConnPool(LdapConnectionPool.java:112)
... 17 more
zimbra logger service is not enabled! failed.


Starting mailbox...Failed.


Starting memcached...Done.
Starting proxy...Done.
Starting amavis...Done.
Starting antispam...Done.
Starting antivirus...Done.
Starting opendkim...Done.
Starting snmp...Done.
Starting spell...Done.
Starting mta...Done.
Starting stats...Done.
Starting service webapp...Failed.


Starting zimbra webapp...Done.
Starting zimbraAdmin webapp...Done.
Starting zimlet webapp...Failed.

=================

I can at least deduce that this is somewhat caused by LDAP, but I am clueless as to where to look right now.
User avatar
L. Mark Stone
Ambassador
Ambassador
Posts: 2796
Joined: Wed Oct 09, 2013 11:35 am
Location: Portland, Maine, US
ZCS/ZD Version: 10.0.6 Network Edition
Contact:

Re: Zimbra node starting up with errors. Please help

Post by L. Mark Stone »

LDAP not starting like that but other Zimbra services starting is usually related to DNS.

Zimbra’s LDAP needs to resolve itself via DNS; same for some other Zimbra sevices.

Assuming you fixed your read-only filesystem and that you have some free space on the disk, go through the DNS checks you normally would prior to doing a fresh install of Zimbra (I have some in my blog), then do a zmcontrol stop to make sure nothing is running.

At that point, just do ldap start as the Zimbra user and see what happens.

If it works, great. Do a zmcontrol restart and you are done.

If not, then start posting logs.

Mark
___________________________________
L. Mark Stone
Mission Critical Email - Zimbra VAR/BSP/Training Partner https://www.missioncriticalemail.com/
AWS Certified Solutions Architect-Associate
xenhaxumus
Posts: 17
Joined: Fri Mar 09, 2018 2:51 am

Re: Zimbra node starting up with errors. Please help

Post by xenhaxumus »

Hello, like the partner who preceded me. I have a problem with zmconfig my version of zimbra is: Release 8.7.11.GA.1854.UBUNTU16.64 UBUNTU16_64 FOSS edition. for ubuntu 16.04.04

root@mail:/home/# su - zimbra -c "zmcontrol status"
Host mail.x.x.x
antivirus Running
cbpolicyd Running
ldap Running
logger Running
mailbox Running
memcached Running
mta Running
opendkim Running
proxy Running
service webapp Running
snmp Running
spell Running
stats Running
zimbra webapp Running
zimbraAdmin webapp Running
zimlet webapp Running
zmconfigd Stopped
zmconfigd is not running.

I have done the following

Stop service Zimbra and fixperms

su - zimbra -c "zmcontrol stop"
pkill zimbra
killall zimbra
/ opt / zimbra / libexec / zmfixperms -e -v
su - zimbra -c "zmcontrol start"

And that:

The common reasons for this issue are:

Problem with IPv6
If you have IPv6 enabled, consider disabling it. Add these line to /etc/sysctl.conf to disable IPv6 on Linux.

net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1
The save the file and apply rules.

# sysctl -p
Also, comment out the entries in your / etc / hosts file.

# :: 1 localhost ip6-localhost ip6-loopback
# ff02 :: 1 ip6-allnodes
# ff02 :: 2 ip6-allrouters
Then try restart zmconfigd:

# su - zimbra
$ zmconfigdctl restart
Zimbra can not get zmconfigd service status
It could also be possible that zmconfigd is in running state but Zimbra can not get its state. For this, you may need to remove its pid file and restart the service

# cat /opt/zimbra/log/zmconfigd.pid
# rm /opt/zimbra/log/zmconfigd.pid
# su - zimbra -c "zmconfigdctl restart"
Missing nc and which commands.
If you have nc command missing, trying to start zmconfigd may result in a failure. Install it as follows.

Install nc on Ubuntu / Debian systems:

sudo apt-get install netcat-openbsd
For CentOS servers, nc command is provided by the nmap-ncat package.

sudo yum -y install nmap-ncat
After installing nc, try to restart Zimbra. Hope one the three methods work for you.

But neither does it keep giving me this:

root @ mail: / home / # ps aux | grep zimbra
root 41113 0.0 0.0 16760 984 pts / 0 S + 11:57 0:00 grep --color = auto zimbra
zimbra 54725 0.0 0.0 45276 4532? Ss jun23 0:00 / lib / systemd / systemd --user
zimbra 54730 0.0 0.0 88232 2504? S jun23 0:00 (sd-pam)
zimbra 58578 0.5 0.1 5310836 367120? Sl 06:26 1:41 / opt / zimbra / common / lib / jvm / java / bin / java -XX: ErrorFile = / opt / zimbra / log -client -Xmx256m -Dhttps.protocols = TLSv1, TLSv1.1, TLSv1 .2 -Djdk.tls.client.protocols = TLSv1, TLSv1.1, TLSv1.2 -Djava.net.preferIPv4Stack = true -Dzimbra.home = / opt / zimbra -Djava.library.path = / opt / zimbra / lib -Djava.ext.dirs = / opt / zimbra / common / lib / jvm / java / jre / lib / ext: / opt / zimbra / lib / jars: / opt / zimbra / lib / ext-common: / opt / zimbra / lib / ext / clamscanner: / opt / zimbra / lib / ext / zimbra-license: / opt / zimbra / lib / ext / twofactorauth: / opt / zimbra / lib / ext / com_zimbra_ssdb_ephemeral_store -Djava.io.tmpdir = / opt / zimbra / data / tmp -Dpython.cachedir.skip = true org.python.util.jython / opt / zimbra / libexec / zmconfigd

And

zimbra @ mail: ~ $ tail -f /var/log/zimbra.log
Jun 24 12:19:05 mail zmconfigd [53216]: All configs fetched in 0.07 seconds
Jun 24 12:19:05 mail zmconfigd [53216]: File "/ opt / zimbra / libexec / zmconfigd", line 222, in <module> myState.getMtaConfig (myConfig.configFile)
Jun 24 12:19:05 mail zmconfigd [53216]: File "/opt/zimbra/common/lib/jylibs/state.py", line 347, in getMtaConfig self.mtaconfig.load (cf, self)
Jun 24 12:19:05 mail zmconfigd [53216]: File "/opt/zimbra/common/lib/jylibs/mtaconfig.py", line 260, in load val = state.lookUpConfig (fields [2], fields [3 ])
Jun 24 12:19:05 mail zmconfigd [53216]: File "/opt/zimbra/common/lib/jylibs/state.py", line 369, in lookUpConfig lines = [self.transform (l) .strip () for l in open (tmpfile, 'r'). readlines () if self.transform (l) .strip ()]
Jun 24 12:19:05 mail zmconfigd [53216]: Sleeping ... Configuration inconsistency detected ([Errno 13] Permission denied: '/opt/zimbra/conf/zmconfigd/postfix_content_filter.cf')

Please help I urgently need to raise the services for the users. Thanks thanks
kazishe
Posts: 3
Joined: Sat Jun 23, 2018 3:03 pm

Re: Zimbra node starting up with errors. Please help

Post by kazishe »

L. Mark Stone wrote:LDAP not starting like that but other Zimbra services starting is usually related to DNS.

Zimbra’s LDAP needs to resolve itself via DNS; same for some other Zimbra sevices.

Assuming you fixed your read-only filesystem and that you have some free space on the disk, go through the DNS checks you normally would prior to doing a fresh install of Zimbra (I have some in my blog), then do a zmcontrol stop to make sure nothing is running.

At that point, just do ldap start as the Zimbra user and see what happens.

If it works, great. Do a zmcontrol restart and you are done.

If not, then start posting logs.

Mark

Many thanks Mark. Managing now to get ldap to start, but then that gives new errors.

zimbra@mail:~$ zmcontrol start
Host mail.mydomain.co.zw
Search error: Unable to determine enabled services from ldap.
Unable to determine enabled services. Cache is out of date or doesn't exist.

which some pointed to my certificates having expired, but that is not so, as seen below:
zimbra@mail:~$ zmcertmgr viewdeployedcrt
- ldap: /opt/zimbra/conf/slapd.crt
notBefore=Jun 20 07:32:56 2018 GMT
notAfter=Sep 18 07:32:56 2018 GMT
subject= /CN=mail.mydomain.co.zw
issuer= /C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
SubjectAltName=mail.mydomain.co.zw
- mailboxd: /opt/zimbra/mailboxd/etc/mailboxd.pem
notBefore=Jun 20 07:32:56 2018 GMT
notAfter=Sep 18 07:32:56 2018 GMT
subject= /CN=mail.mydomain.co.zw
issuer= /C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
SubjectAltName=mail.mydomain.co.zw
- mta: /opt/zimbra/conf/smtpd.crt
notBefore=Jun 20 07:32:56 2018 GMT
notAfter=Sep 18 07:32:56 2018 GMT
subject= /CN=mail.mydomain.co.zw
issuer= /C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
SubjectAltName=mail.mydomain.co.zw
- proxy: /opt/zimbra/conf/nginx.crt
notBefore=Jun 20 07:32:56 2018 GMT
notAfter=Sep 18 07:32:56 2018 GMT
subject= /CN=mail.mydomain.co.zw
issuer= /C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
SubjectAltName=mail.mydomain.co.zw
zimbra@mail:~$

What am I missing?
User avatar
L. Mark Stone
Ambassador
Ambassador
Posts: 2796
Joined: Wed Oct 09, 2013 11:35 am
Location: Portland, Maine, US
ZCS/ZD Version: 10.0.6 Network Edition
Contact:

Re: Zimbra node starting up with errors. Please help

Post by L. Mark Stone »

Let's start with the basics.

Please post the output of the following commands:

cat /etc/hosts
cat /etc/resolv.conf
cat /rtc/hostname
host mail.mydomain.co.zw
host <ip address given by the "host mail.mydomain.co.zw" command above>
host http://www.yahoo.com

Mark
___________________________________
L. Mark Stone
Mission Critical Email - Zimbra VAR/BSP/Training Partner https://www.missioncriticalemail.com/
AWS Certified Solutions Architect-Associate
xenhaxumus
Posts: 17
Joined: Fri Mar 09, 2018 2:51 am

Re: Zimbra node starting up with errors. Please help

Post by xenhaxumus »

Hello, excuse me.

My language is Spanish. I am translating the most understandable and clear possible to support me. I have a zimbra server, which has stopped zmconfig. I have made a series of steps but I can not start it.

by running this command tail-f /var/log/zimbra.log I get this

zmconfigd [13630]: All configs fetched in 0.09 seconds
zmconfigd [13630]: File "/ opt / zimbra / libexec / zmconfigd", line 222, in <module> myState.getMtaConfig (myConfig.configFile)
zmconfigd [13630]: File "/opt/zimbra/common/lib/jylibs/state.py", line 347, in getMtaConfig self.mtaconfig.load (cf, self)
zmconfigd [13630]: File "/opt/zimbra/common/lib/jylibs/mtaconfig.py", line 260, in load val = state.lookUpConfig (fields [2], fields [3])
Jun 25 09:42:18
zmconfigd [13630]: File "/opt/zimbra/common/lib/jylibs/state.py", line 369, in lookUpConfig lines = [self.transform (l) .strip () for l in open (tmpfile, 'r ') .readlines () if self.transform (l) .strip ()]
zmconfigd [13630]: Sleeping ... Configuration inconsistency detected ([Errno 13] Permission denied: '/opt/zimbra/conf/zmconfigd/postfix_content_filter.cf')

What I can do? I need to upload the service
odiecoranes
Posts: 15
Joined: Wed Jun 29, 2016 9:00 am

Re: Zimbra node starting up with errors. Please help

Post by odiecoranes »

Hi Everyone,

How did you solved this issue? encountered similar error.


Thanks!
Post Reply