Services show not running in the admin page

[wauters]

When I log into my admin page it says some services are not running. when I look at the services page they are all marked with a red x.

In the CLI they all show running except proxy as I have that disabled. I have it disabled because I was having issues sending and receiving emails through my firewall. I think at this point I could turn it back on if that's my problem.

I have read some posts on how to fix this, but the few things I tried have not worked. Like restarting and such. I don't want to mess up anything so I am looking for some assistance.

Thank you!

[JDunphy]

It is safe to ignore or you can find the links in these forum to fix it... Note: the gui is also running zmcontrol status just like you did from the command line. There are lots of reasons.. sometimes its cron, sometimes it syslog, etc.... The gui interface needs to find STATUS lines in /var/log/zimbra-stats.log ... If those are missing then back track for your eventual fix until you see them.

Code: Select all

% grep STATUS /var/log/zimbra-stats.log
Mar 22 10:58:06 mail zimbramon[13794]: 13794:info: 2019-03-22 10:58:01, STATUS: mail.example.com: spell: Running
Mar 22 10:58:06 mail zimbramon[13794]: 13794:info: 2019-03-22 10:58:01, STATUS: mail.example.com: stats: Running
Mar 22 10:58:06 mail zimbramon[13794]: 13794:info: 2019-03-22 10:58:01, STATUS: mail.example.com: zimbra: Running
Mar 22 10:58:06 mail zimbramon[13794]: 13794:info: 2019-03-22 10:58:01, STATUS: mail.example.com: zimbraAdmin: Running

Hint: you can run /opt/zimbra/libexec/zmstatuslog directly and verify if the messages show up. Normally that is run from cron... Also observe that first line that come back from zmcontrol status... that Host line is the syslog host.

[wauters]

Thank you for all your help!

[wauters]

I ran the grep STATUS command and the services did come back as running. The admin page still shows red x's.

[JDunphy]

Been a few years since I debugged this particular problem so I have forgotten a lot of the failure reasons. A couple of other things to check... permissions for read access for the zimbra user and the date of the file.

Code: Select all

mail:~:41> ls -l /var/log/zimbra-stats.log
-rw-r--r-- 1 zimbra zimbra 23645041 Mar 22 12:08 /var/log/zimbra-stats.log

Also verify the time and pay attention to those entries in the file and the time frame. I seem to remember that you need a few data points here before it recognizes on the gui. Because you said those STATUS lines are present in the file, I am assuming you have loads of those lines and not just one instance because you ran the status command directly and verified that zimbra-stats.log has those lines. In other words, it knows old lines vs new STATUS lines based on time and it needs a few sets of them. A common error I have seen is cron stopped running so verifying those times and the date stamp on the file will identify that as would running tail -f /var/log/zimbra-stats.log and observing.

[JDunphy]

Ok... swatch watches the /var/log/zimbra-stats file and pipes the output to /opt/zimbra/libexec/zmlogger that puts it in the DB for the gui to query and then display as up or down.

Double check you don't have some hosed processes (is that a term)

Code: Select all

% ps auxw |grep swatchrc
zimbra   26002  0.0  0.2  56704 17708 ?        SNs  03:43   0:22 /opt/zimbra/common/bin/swatchdog --config-file=/opt/zimbra/conf/logswatchrc --use-cpan-file-tail --pid-file=/opt/zimbra/log/logswatch.pid --daemon --script-dir=/opt/zimbra/data/tmp --tail-file /var/log/zimbra-stats.log
zimbra   26118  0.0  0.1  38180 10376 ?        SN   03:43   0:00 /usr/bin/perl /opt/zimbra/common/bin/swatchdog --config-file=/opt/zimbra/conf/swatchrc --use-cpan-file-tail --script-dir=/opt/zimbra/data/tmp --tail-file /var/log/zimbra.log

Look inside /opt/zimbra/conf/logswatchrc ... double check you have zmlogger running because without that you can't insert that into the database... zmlogger is just a perl script if you want to see what it is doing.

As I mentioned, if the gui doesn't report running it isn't really that big a deal provided zmcontrol status is reporting what you expect.

[wauters]

It is now working. I followed a rebuilding of the cron job in another post. The cron job was there but wasn't working for some reason and cron was runnning.

Very weird, but I had a power outage due to flooding in our area and I guess after 6 hours the backup batteries die

Thank you for all your help with these issues!!!!

[malko22]

Hello All,

I am facing the same problem. The service is running if see using the command zmcontrol status but when i see on the admin page the service is being shown as not running. I have seen rsyslog service which also running smoothly. The cron is also working but still the services on the admin page is being shown as not working.


HELP NEEDED !!!!!!!!

[JDunphy]

The service is running if see using the command zmcontrol status but when i see on the admin page the service is being shown as not running. I have seen rsyslog service which also running smoothly. The cron is also working but still the services on the admin page is being shown as not working.

Here is a deeper dive.

Most times the problem associated with the admin interface relating to status can be viewed from 2 half's or perspectives. There is the cron job that runs /opt/zimbra/libexec/zmstatuslog that populates /var/log/zimbra-stats.log with STATUS entries of services and then there is a daemon that samples this information from /var/log/zimbra-stats.log and puts it in the database which is described in this thread using swatch to notice those STATUS lines being written and then calls zmlogger to insert it into the DB. The admin interface eventually will issue a soap call to display those results from this DB on user request which doesn't work for you.

I'll label this above part 1 and part 2.

Ref Part1: viewtopic.php?f=13&t=60723&hilit=zmcontrol+status
Ref Part2: viewtopic.php?f=15&t=65863

In Part1, there needs to be an entry in syslog so that the correct entries go to /var/log/zimbra-stats.log
and you need an entry in zimbra's crontab so that these STATUS lines are written to /var/log/zimbra-stats.log periodically.

That syslog config entry is done via this at install:
/opt/zimbra/libexec/zmsyslogsetup
and would create this line required for STATUS to work in syslog.conf/rsyslog.conf, etc... but this is what you need.
local1.* -/var/log/zimbra-stats.log

The crontab entry for zimbra looks like this:
*/2 * * * * /opt/zimbra/libexec/zmstatuslog > /dev/null 2>&1

In the past, some users observed that cron wasn't running or configured or similarly for syslog. The first step is observing /var/log/zimbra-stats.log. In the above cron entry we should see those STATUS lines every 2 minutes.

Code: Select all

% tail -f /var/log/syslog.conf 
or
%  /opt/zimbra/libexec/zmstatuslog 
% grep STATUS /var/log/zimbra-status.log 
Sep  5 08:34:06 mail zimbramon[27690]: 27690:info: 2019-09-05 08:34:01, STATUS: mail.example.com: amavis: Running
Sep  5 08:34:06 mail zimbramon[27690]: 27690:info: 2019-09-05 08:34:01, STATUS: mail.example.com: antispam: Running
Sep  5 08:34:06 mail zimbramon[27690]: 27690:info: 2019-09-05 08:34:01, STATUS: mail.example.com: antivirus: Running
Sep  5 08:34:06 mail zimbramon[27690]: 27690:info: 2019-09-05 08:34:01, STATUS: mail.example.com: convertd: Running
Sep  5 08:34:06 mail zimbramon[27690]: 27690:info: 2019-09-05 08:34:01, STATUS: mail.example.com: ldap: Running
Sep  5 08:34:06 mail zimbramon[27690]: 27690:info: 2019-09-05 08:34:01, STATUS: mail.example.com: logger: Running
Sep  5 08:34:06 mail zimbramon[27690]: 27690:info: 2019-09-05 08:34:01, STATUS: mail.example.com: mailbox: Running
Sep  5 08:34:06 mail zimbramon[27690]: 27690:info: 2019-09-05 08:34:01, STATUS: mail.example.com: memcached: Running
Sep  5 08:34:06 mail zimbramon[27690]: 27690:info: 2019-09-05 08:34:01, STATUS: mail.example.com: mta: Running
Sep  5 08:34:06 mail zimbramon[27690]: 27690:info: 2019-09-05 08:34:01, STATUS: mail.example.com: opendkim: Running
Sep  5 08:34:06 mail zimbramon[27690]: 27690:info: 2019-09-05 08:34:01, STATUS: mail.example.com: proxy: Running
Sep  5 08:34:06 mail zimbramon[27690]: 27690:info: 2019-09-05 08:34:01, STATUS: mail.example.com: service: Running
Sep  5 08:34:06 mail zimbramon[27690]: 27690:info: 2019-09-05 08:34:01, STATUS: mail.example.com: snmp: Running
Sep  5 08:34:06 mail zimbramon[27690]: 27690:info: 2019-09-05 08:34:01, STATUS: mail.example.com: spell: Running
Sep  5 08:34:06 mail zimbramon[27690]: 27690:info: 2019-09-05 08:34:01, STATUS: mail.example.com: stats: Running
Sep  5 08:34:06 mail zimbramon[27690]: 27690:info: 2019-09-05 08:34:01, STATUS: mail.example.com: zimbra: Running
Sep  5 08:34:06 mail zimbramon[27690]: 27690:info: 2019-09-05 08:34:01, STATUS: mail.example.com: zimbraAdmin: Running
Sep  5 08:34:06 mail zimbramon[27690]: 27690:info: 2019-09-05 08:34:01, STATUS: mail.example.com: zimlet: Running
Sep  5 08:34:06 mail zimbramon[27690]: 27690:info: 2019-09-05 08:34:01, STATUS: mail.example.com: zmconfigd: Running

If you don't see new STATUS lines then, investigate around zmstatuslog or part1. Check your cron log to see if it is running zmstatuslog? Do you have a local1.* entry in your syslog so the program can put those entries into /var/log/zimbra-stats.log. Did you reload/restart cron or syslog to pickup any changes to their respective config files. Newer versions can do this automatically but you can force a restart of cron or syslog to rule this out that the daemons didn't pickup your changes. It takes a second to do both daemons.

If you do see new STATUS lines then, look toward this thread for further understanding. Zimbra runs 2 swatch processes... the one you are interested in is the one monitoring the /var/log/zimbra-stats.log file.

Code: Select all

% ps auxw |grep zimbra-stats.log
zimbra   26033  0.0  0.2  56588 17636 ?        SNs  03:35   0:11 /opt/zimbra/common/bin/swatchdog --config-file=/opt/zimbra/conf/logswatchrc --use-cpan-file-tail --pid-file=/opt/zimbra/log/logswatch.pid --daemon --script-dir=/opt/zimbra/data/tmp --tail-file /var/log/zimbra-stats.log

Finally, this needs to work also to display the results to you during the admin interface.

Code: Select all

# su - zimbra
% [zimbra@mail ~]$ zmsoap -z GetServiceStatusRequest
<GetServiceStatusResponse xmlns="urn:zimbraAdmin">
  <timezone displayName="Pacific Standard Time" id="Canada/Pacific"/>
  <status server="mail.example.com" t="1567696680" service="antispam">1</status>
  <status server="mail.example.com" t="1567696680" service="logger">1</status>
  <status server="mail.example.com" t="1567696680" service="ldap">1</status>
  <status server="mail.example.com" t="1567696680" service="convertd">1</status>
  <status server="mail.example.com" t="1567696680" service="zimbra">1</status>
  <status server="mail.example.com" t="1567696680" service="stats">1</status>
  <status server="mail.example.com" t="1567696680" service="opendkim">1</status>
  <status server="mail.example.com" t="1567696680" service="zimlet">1</status>
  <status server="mail.example.com" t="1567696680" service="service">1</status>
  <status server="mail.example.com" t="1567696680" service="memcached">1</status>
  <status server="mail.example.com" t="1567696680" service="mta">1</status>
  <status server="mail.example.com" t="1567696680" service="amavis">1</status>
  <status server="mail.example.com" t="1567696680" service="antivirus">1</status>
  <status server="mail.example.com" t="1567696680" service="zmconfigd">1</status>
  <status server="mail.example.com" t="1567696680" service="snmp">1</status>
  <status server="mail.example.com" t="1567696680" service="proxy">1</status>
  <status server="mail.example.com" t="1567696680" service="spell">1</status>
  <status server="mail.example.com" t="1567696680" service="zimbraAdmin">1</status>
  <status server="mail.example.com" t="1567696680" service="mailbox">1</status>
</GetServiceStatusResponse>

Last bit of advice... wait a few minutes as there is a little bit of delay before the admin side will pick up the changes and display that things are up once you fix the root cause.