stats service shown as not running, but actually is

Discuss your pilot or production implementation with other Zimbra admins or our engineers.
Post Reply
georgi.yankov
Posts: 7
Joined: Thu Oct 20, 2022 7:07 pm

stats service shown as not running, but actually is

Post by georgi.yankov »

Hey folks.

Running Release 8.8.15_GA_3869.RHEL7_64_20190917004220 RHEL7_64 FOSS edition, Patch 8.8.15_P35.

Previously, the installation was being started from the Init.d scripts.
Since this is 2022 and not 2002, i changed to use Systemd.

All is good, aside from zmstat for some reason being picked up as "Not running".
zmcontrol simply seems to stop seeing the service, despite the processes still running.

Code: Select all

	amavis                  Running
	antispam                Running
	antivirus               Running
	dnscache                Running
	ldap                    Running
	logger                  Running
	mailbox                 Running
	memcached               Running
	mta                     Running
	opendkim                Running
	proxy                   Running
	service webapp          Running
	snmp                    Running
	spell                   Running
	stats                   Stopped
	zimbra webapp           Running
	zimbraAdmin webapp      Running
	zimlet webapp           Running
	zmconfigd               Running
ps aux output:

Code: Select all

root     3836385  0.0  0.0 137892  2760 ?        Ss   00:34   0:01 /opt/zimbra/common/libexec/master -w
zimbra   3836447  0.0  0.0 148784  7340 ?        S    00:34   0:09 /usr/bin/perl -w /opt/zimbra/libexec/zmstat-proc
zimbra   3836449  0.0  0.0 148644  7100 ?        S    00:34   0:01 /usr/bin/perl -w /opt/zimbra/libexec/zmstat-cpu
zimbra   3836451  0.0  0.0 148640  7060 ?        S    00:34   0:01 /usr/bin/perl -w /opt/zimbra/libexec/zmstat-vm
zimbra   3836453  0.0  0.0 148644  7096 ?        S    00:34   0:01 /usr/bin/perl -w /opt/zimbra/libexec/zmstat-io -x
zimbra   3836455  0.0  0.0 148520  7032 ?        S    00:34   0:00 /usr/bin/perl -w /opt/zimbra/libexec/zmstat-df
zimbra   3836457  0.0  0.0 148516  7044 ?        S    00:34   0:01 /usr/bin/perl -w /opt/zimbra/libexec/zmstat-io
zimbra   3836459  0.0  0.0 152964  7280 ?        S    00:34   0:00 /usr/bin/perl -w /opt/zimbra/libexec/zmstat-fd
zimbra   3836465  0.1  0.0 154084  8388 ?        S    00:34   1:08 /usr/bin/perl -w /opt/zimbra/libexec/zmstat-allprocs
zimbra   3836468  0.0  0.0 148780  7348 ?        S    00:34   0:06 /usr/bin/perl -w /opt/zimbra/libexec/zmstat-mysql
zimbra   3836472  0.0  0.0 148524  6992 ?        S    00:34   0:02 /usr/bin/perl -w /opt/zimbra/libexec/zmstat-mtaqueue
zimbra   3836474  0.0  0.0 151244  7648 ?        S    00:34   0:31 /usr/bin/perl -w /opt/zimbra/libexec/zmstat-nginx
root     3837306  0.0  0.0 239396  4676 ?        S    00:34   0:00 sudo /opt/zimbra/libexec/zmstat-fd
root     3837321  0.0  0.0 152972  7324 ?        S    00:34   0:04 /usr/bin/perl -w /opt/zimbra/libexec/zmstat-fd
zimbra   3837532  0.0  0.0 108124   892 ?        S    00:34   0:01 /usr/bin/iostat -d -k 30
zimbra   3837537  0.0  0.0 152728  1424 ?        S    00:34   0:00 /usr/bin/vmstat -n -S K 30
zimbra   3837544  0.0  0.0 108124   888 ?        S    00:34   0:00 /usr/bin/iostat -d -k -x 30
zimbra   3847301  0.0  0.0 149184  4360 ?        S    00:45   0:00 /usr/bin/perl -T /opt/zimbra/common/sbin/amavis-mc
and systemd status output:

Code: Select all

   CGroup: /system.slice/zimbra.service
           ├─3832685 /opt/zimbra/common/libexec/slapd -l LOCAL0 -h ldap://redacted.net:389 ldapi:/// -F /opt/zimbra/data/ldap/config
           ├─3834514 /bin/sh /opt/zimbra/common/bin/mysqld_safe --defaults-file=/opt/zimbra/conf/my.cnf --external-locking --log-error=/opt/zimbra/log/mysql_error.log --malloc-lib=/opt/zimbra/common/lib/libtcmalloc_minimal.so --ledir=/...
           ├─3834724 /opt/zimbra/common/sbin/mysqld --defaults-file=/opt/zimbra/conf/my.cnf --basedir=/opt/zimbra/common --datadir=/opt/zimbra/db/data --plugin-dir=/opt/zimbra/common/lib/plugin --external-locking --log-error=/opt/zimbr...
           ├─3835691 /opt/zimbra/common/bin/memcached -d -P /opt/zimbra/log/memcached.pid -U 0 -p 11211
           ├─3836098 /opt/zimbra/common/sbin/clamd --config-file=/opt/zimbra/conf/clamd.conf
           ├─3836117 /opt/zimbra/common/bin/freshclam --config-file=/opt/zimbra/conf/freshclam.conf --quiet --daemon --checks=12
           ├─3836121 /opt/zimbra/common/sbin/opendkim -x /opt/zimbra/conf/opendkim.conf -u zimbra
           ├─3836122 /opt/zimbra/common/sbin/opendkim -x /opt/zimbra/conf/opendkim.conf -u zimbra
           ├─3836152 /opt/zimbra/common/bin/httpd -k start -f /opt/zimbra/conf/httpd.conf
           ├─3836153 /opt/zimbra/common/bin/rotatelogs /opt/zimbra/log/httpd_error.log.%Y-%m-%d 86400
           ├─3836155 /opt/zimbra/common/bin/rotatelogs /opt/zimbra/log/httpd_access.log.%Y-%m-%d 86400
           ├─3836160 /opt/zimbra/common/bin/httpd -k start -f /opt/zimbra/conf/httpd.conf
           ├─3836165 /opt/zimbra/common/bin/httpd -k start -f /opt/zimbra/conf/httpd.conf
           ├─3836193 /opt/zimbra/common/bin/httpd -k start -f /opt/zimbra/conf/httpd.conf
           ├─3836268 /opt/zimbra/common/sbin/saslauthd -r -a zimbra
           ├─3836270 /opt/zimbra/common/sbin/saslauthd -r -a zimbra
           ├─3836271 /opt/zimbra/common/sbin/saslauthd -r -a zimbra
           ├─3836272 /opt/zimbra/common/sbin/saslauthd -r -a zimbra
           ├─3836273 /opt/zimbra/common/sbin/saslauthd -r -a zimbra
           ├─3836447 /usr/bin/perl -w /opt/zimbra/libexec/zmstat-proc
           ├─3836449 /usr/bin/perl -w /opt/zimbra/libexec/zmstat-cpu
           ├─3836451 /usr/bin/perl -w /opt/zimbra/libexec/zmstat-vm
           ├─3836453 /usr/bin/perl -w /opt/zimbra/libexec/zmstat-io -x
           ├─3836455 /usr/bin/perl -w /opt/zimbra/libexec/zmstat-df
           ├─3836457 /usr/bin/perl -w /opt/zimbra/libexec/zmstat-io
           ├─3836459 /usr/bin/perl -w /opt/zimbra/libexec/zmstat-fd
           ├─3836465 /usr/bin/perl -w /opt/zimbra/libexec/zmstat-allprocs
           ├─3836468 /usr/bin/perl -w /opt/zimbra/libexec/zmstat-mysql
           ├─3836472 /usr/bin/perl -w /opt/zimbra/libexec/zmstat-mtaqueue
           ├─3836474 /usr/bin/perl -w /opt/zimbra/libexec/zmstat-nginx
           ├─3837532 /usr/bin/iostat -d -k 30
           ├─3837537 /usr/bin/vmstat -n -S K 30
           └─3837544 /usr/bin/iostat -d -k -x 30
I get in the zimbra.log, that the service is starting:

Code: Select all

Nov 29 11:54:18 mail zimbramon[247476]: 247476:info: Starting stats via zmcontrol
and the Watchdog seems tobe tracking it:

Code: Select all

Nov 29 11:57:03 mail zmconfigd[247573]: Tracking service stats
Not sure if it should throw an "status is OK" though.

But... zmcontrol simply does not see it as running.
Despite the lack of any messages, indicating that the process has died or something is wrong with it.

If i revert to using the Initd script, it works properly...

The systemd file is as follow;

Code: Select all


cat /etc/systemd/system/zimbra.service 
[Unit]
Description=Zimbra Collaboration Suite
After=syslog.target network.target
Conflicts=sendmail.service exim.service postfix.service

[Service]
Type=forking
User=zimbra
Group=zimbra
Environment=PERL5LIB=/opt/zimbra/common/lib/perl5/x86_64-linux-gnu-thread-multi:/opt/zimbra/common/lib/perl5
#ExecStartPre=/usr/bin/touch /var/lock/subsys/zimbra
ExecStart=/opt/zimbra/bin/zmcontrol start
ExecStop=/opt/zimbra/bin/zmcontrol stop
#ExecStopPost=/usr/bin/rm -f /var/lock/subsys/zimbra
ExecReload=/opt/zimbra/bin/zmcontrol restart
Restart=on-failure
TimeoutSec=90

StandardOutput=null

[Install]
WantedBy=multi-user.target
The only difference seems to be a check for a lock file in /var/lock/subsys/zimbra, but it does not seem to do anything at all.
georgi.yankov
Posts: 7
Joined: Thu Oct 20, 2022 7:07 pm

Re: stats service shown as not running, but actually is

Post by georgi.yankov »

Considering the amount of information provided, i expected someone to give me an advice on this :|
User avatar
JDunphy
Outstanding Member
Outstanding Member
Posts: 883
Joined: Fri Sep 12, 2014 11:18 pm
Location: Victoria, BC
ZCS/ZD Version: 9.0.0_P39 NETWORK Edition

Re: stats service shown as not running, but actually is

Post by JDunphy »

Here is a guess.

I don't see zmstat-ldap in your ps output but you have ldap listed as running with zmcontrol status.

Code: Select all

# su - zimbra
% $ zmcontrol -v
Release 8.8.15_GA_3953.RHEL8_64_20200629025823 RHEL8_64 FOSS edition, Patch 8.8.15_P35.
%  ps auxw |grep zmstat
zimbra   3289180  0.0  0.0  47808  6420 ?        S    Nov25   5:10 /usr/bin/perl -w /opt/zimbra/libexec/zmstat-proc
zimbra   3289182  0.0  0.0  47384  5884 ?        S    Nov25   0:41 /usr/bin/perl -w /opt/zimbra/libexec/zmstat-cpu
zimbra   3289184  0.0  0.0  47396  6244 ?        S    Nov25   0:53 /usr/bin/perl -w /opt/zimbra/libexec/zmstat-vm
zimbra   3289186  0.0  0.0  47404  5956 ?        S    Nov25   0:41 /usr/bin/perl -w /opt/zimbra/libexec/zmstat-io -x
zimbra   3289188  0.0  0.0  47272  5788 ?        S    Nov25   0:05 /usr/bin/perl -w /opt/zimbra/libexec/zmstat-df
zimbra   3289190  0.0  0.0  47396  5984 ?        S    Nov25   0:30 /usr/bin/perl -w /opt/zimbra/libexec/zmstat-io
zimbra   3289194  0.0  0.0  49624  4148 ?        S    Nov25   0:00 /usr/bin/perl -w /opt/zimbra/libexec/zmstat-fd
zimbra   3289201  0.0  0.1  53004  8300 ?        S    Nov25  23:50 /usr/bin/perl -w /opt/zimbra/libexec/zmstat-allprocs
zimbra   3289211  0.0  0.0  47612  6768 ?        S    Nov25   3:29 /usr/bin/perl -w /opt/zimbra/libexec/zmstat-mysql
zimbra   3289260  0.0  0.0  47272  5904 ?        S    Nov25   1:03 /usr/bin/perl -w /opt/zimbra/libexec/zmstat-mtaqueue
zimbra   3289275  0.0  0.0  50128  7884 ?        S    Nov25   6:53 /usr/bin/perl -w /opt/zimbra/libexec/zmstat-nginx
zimbra   3289299  0.0  0.1  54960 10408 ?        S    Nov25   7:59 /usr/bin/perl -w /opt/zimbra/libexec/zmstat-ldap
root     3290164  0.0  0.0 131464  4260 ?        S    Nov25   0:00 sudo /opt/zimbra/libexec/zmstat-fd
root     3290174  0.0  0.0  47376  5916 ?        S    Nov25   1:04 /usr/bin/perl -w /opt/zimbra/libexec/zmstat-fd
Perhaps check /opt/zimbra/zmstat/pid to see which pid files zmstatctl will send kill 0 $pid to determine if a process is running. Note: zmstat-fd-real is skipped in zmstatctl since it is running as root.

Code: Select all

% cd /opt/zimbra/zmstat/pid/
% ls -l
total 52
-rw-r----- 1 zimbra zimbra 8 Nov 25 16:05 zmstat-allprocs.pid
-rw-r----- 1 zimbra zimbra 8 Nov 25 16:05 zmstat-cpu.pid
-rw-r----- 1 zimbra zimbra 8 Nov 25 16:05 zmstat-df.pid
-rw-r----- 1 root   root   8 Nov 25 16:05 zmstat-fd-real.pid
-rw-r----- 1 zimbra zimbra 8 Nov 25 16:05 zmstat-fd.pid
-rw-r----- 1 zimbra zimbra 8 Nov 25 16:05 zmstat-io-x.pid
-rw-r----- 1 zimbra zimbra 8 Nov 25 16:05 zmstat-io.pid
-rw-r----- 1 zimbra zimbra 8 Nov 25 16:05 zmstat-ldap.pid
-rw-r----- 1 zimbra zimbra 8 Nov 25 16:05 zmstat-mtaqueue.pid
-rw-r----- 1 zimbra zimbra 8 Nov 25 16:05 zmstat-mysql.pid
-rw-r----- 1 zimbra zimbra 8 Nov 25 16:05 zmstat-nginx.pid
-rw-r----- 1 zimbra zimbra 8 Nov 25 16:05 zmstat-proc.pid
-rw-r----- 1 zimbra zimbra 8 Nov 25 16:05 zmstat-vm.pid
If it is /opt/zimbra/libexec/zmstat-ldap that is failing to start via systemd causing the problem, I would suspect something in environment was explicitly passed given that su - zimbra to start zmcontrol in /etc/init.d/zimbra.

Jim
georgi.yankov
Posts: 7
Joined: Thu Oct 20, 2022 7:07 pm

Re: stats service shown as not running, but actually is

Post by georgi.yankov »

Alright.
Thanks for the heads up.

I finally got some time to test this out (on holidays, lol).

Turns out you are right.
That template that i got from here:
https://github.com/Zimbra-Community/zim ... ra.service

Is not working properly.
In fact, after i started looking around, even the PERLLIB is not correct, since the actual folder does not have "gnu" in its name.

Created a PR for that:

https://github.com/Zimbra-Community/zim ... ls/pull/10
Post Reply