stats and logger services restart often

[Chris-M]

On our ZCS 8.6.0 install the stats and logger services restart at least once a day. In the logs we see this:

Code: Select all

Mar 30 07:16:19 mail zmconfigd[23752]: Service status change: mail.domain.com stats changed from stopped to running
Mar 30 07:17:21 mail zmconfigd[23752]: Service status change: mail.domain.com stats changed from running to stopped
Mar 29 03:17:21 mail zmconfigd[23752]: Service status change: mail.domain.com logger changed from running to stopped
Mar 29 03:18:22 mail zmconfigd[23752]: Service status change: mail.domain.com logger changed from stopped to running
Mar 28 03:25:20 mail zmconfigd[31300]: Service status change: mail.domain.com logger changed from running to stopped
Mar 28 03:26:21 mail zmconfigd[31300]: Service status change: mail.domain.com logger changed from stopped to running
Mar 28 23:07:42 mail zmconfigd[23752]: Service status change: mail.domain.com stats changed from stopped to running
Mar 28 23:08:43 mail zmconfigd[23752]: Service status change: mail.domain.com stats changed from running to stopped

Currently the stats service is showing as stopped:

Code: Select all

[zimbra@mail log]$ zmcontrol status
Host mail.domain.com
        amavis                  Running
        antispam                Running
        antivirus               Running
        ldap                    Running
        logger                  Running
        mailbox                 Running
        mta                     Running
        opendkim                Running
        service webapp          Running
        snmp                    Running
        spell                   Running
        stats                   Stopped
        zimbra webapp           Running
        zimbraAdmin webapp      Running
        zimlet webapp           Running
        zmconfigd               Running

I have searched quite a bit but haven't found any relevant information for this issue affecting Zimbra 8.6 (only much older versions) so i'm not sure where to begin troubleshooting.

Can anyone suggest how to troubleshoot this? The Zimbra server itself is not a busy server and only serves a couple of users - the loads are very low.

Thanks in advance.

[quanah]

The logger restarts are clearly from nightly log rotation. The stats bit, what is the output of zmstatctl status?

[Chris-M]

Thanks for your response.

The output is as follows:

Code: Select all

[zimbra@mail /]$ zmstatctl status
Running: zmstat-df
Running: zmstat-mtaqueue
Running: zmstat-io
Running: zmstat-ldap
Running: zmstat-proc
Running: zmstat-mysql
Running: zmstat-cpu
Running: zmstat-vm
Running: zmstat-io-x
process 26304 in /opt/zimbra/zmstat/pid/zmstat-fd.pid not running
Running: zmstat-allprocs

Any ideas?

[quanah]

For a first pass, I'd probably do something like

Code: Select all

zmstatctl stop
rm -f /opt/zimbra/zmstat/pid/zmstat-fd.pid 
zmstatctl start
zmstatctl status

And see if that resolves it.

[Chris-M]

Thanks for your reply.

I just did as you suggested and it seems the stats process is still not running after this:

Code: Select all

[zimbra@mail /]$ zmstatctl stop
Terminating process 26283
Terminating process 26325
Terminating process 26297
Terminating process 26332
Terminating process 26275
Terminating process 26313
Terminating process 26277
Terminating process 26279
Terminating process 26281
Terminating process 26304
Terminating process 26307

[zimbra@mail /]$ rm -f /opt/zimbra/zmstat/pid/zmstat-fd.pid

[zimbra@mail /]$ zmstatctl start
Invoking: /opt/zimbra/libexec/zmstat-proc
Invoking: /opt/zimbra/libexec/zmstat-cpu
Invoking: /opt/zimbra/libexec/zmstat-vm
Invoking: /opt/zimbra/libexec/zmstat-io -x
Invoking: /opt/zimbra/libexec/zmstat-df
Invoking: /opt/zimbra/libexec/zmstat-io
Invoking: /opt/zimbra/libexec/zmstat-fd
Invoking: /opt/zimbra/libexec/zmstat-allprocs
Invoking: /opt/zimbra/libexec/zmstat-mysql
Invoking: /opt/zimbra/libexec/zmstat-mtaqueue
Invoking: /opt/zimbra/libexec/zmstat-ldap

[zimbra@mail /]$ zmstatctl status
Running: zmstat-df
Running: zmstat-mtaqueue
Running: zmstat-io
Running: zmstat-ldap
Running: zmstat-proc
Running: zmstat-mysql
Running: zmstat-cpu
Running: zmstat-vm
Running: zmstat-io-x
process 20564 in /opt/zimbra/zmstat/pid/zmstat-fd.pid not running
Running: zmstat-allprocs

[quanah]

Ok, next I would run:

Code: Select all

/opt/zimbra/libexec/zmstat-fd

and see if it reports any errors.

[Chris-M]

Output from that command is:

Code: Select all

[zimbra@mail /]$ /opt/zimbra/libexec/zmstat-fd
zmstat-fd-real.pid: Already running as pid 7015

[quanah]

Ok, if you're seeing that after running

Code: Select all

zmstatctl stop

it seems like there's either (a) a runaway process for the zmstat-fd bits, or (b) a stale pid for the zmstat-fd root portion of the process. So I'd check first to see if that process ID exists, AND that it is related to zmstat-fd. If so, then kill it, and clean up the pid file. It's possible that some other process got that pid on a reboot and it's causing the script to exit out incorrectly. If there is no process with that PID, or the PID with that process is unrelated, remove the on disk file tracking the zmstat-fd root PID, so it'll stop getting confused, and then run

Code: Select all

zmstatctl start

and see if things are happier.

[Chris-M]

Thanks so much for your replies and support.

I killed the process and then started zmstatctl and all seems good now:

Code: Select all

[root@mail /]# ps aux | grep 7015
root      7015  0.0  0.0  48816  6324 ?        S    Mar13   0:23 /usr/bin/perl -w /opt/zimbra/libexec/zmstat-fd

[root@mail /]# kill -9 7015

[zimbra@mail /]$ zmstatctl start
zmstat-df already running, skipping.
zmstat-mtaqueue already running, skipping.
zmstat-io already running, skipping.
zmstat-ldap already running, skipping.
zmstat-proc already running, skipping.
zmstat-mysql already running, skipping.
zmstat-cpu already running, skipping.
zmstat-vm already running, skipping.
zmstat-io-x already running, skipping.
zmstat-allprocs already running, skipping.
Invoking: /opt/zimbra/libexec/zmstat-fd

[zimbra@mail /]$ /opt/zimbra/libexec/zmstat-fd
zmstat-fd.pid: Already running as pid 13814

[zimbra@mail /]$ zmstatctl status
Running: zmstat-df
Running: zmstat-mtaqueue
Running: zmstat-io
Running: zmstat-ldap
Running: zmstat-proc
Running: zmstat-mysql
Running: zmstat-cpu
Running: zmstat-vm
Running: zmstat-io-x
Running: zmstat-fd
Running: zmstat-allprocs

I will monitor for the next few days to see if the service stops/starts unexpectedly.

Once again, thank you!

[quanah]

Yay, I'm glad it's finally solved!