Statistics no longer working

Discuss your pilot or production implementation with other Zimbra admins or our engineers.
Bittone
Posts: 21
Joined: Mon Sep 05, 2016 4:30 pm

Statistics no longer working

Post by Bittone »

Hello,
I have a Zimbra 8.7 server on Centos 7 (virtual) and I can no longer access statistics: every graph says "no data available" (or something like that since I'm using Italian locale).
I can see that log files in /opt/zimbra/zmstats are correctly populated.
In "Server statistics" -> <my server name > ->"disk" I can see only the stats for the last 60 days but no data for "last 30 days" and "Last 48 hours".
In "Server statistics" -> <my server name > -> "Message count" I get this error upon accessing teh page: Messaggio: system failure: java.lang.ArrayIndexOutOfBoundsException: 10 Codice di errore: service.FAILURE Method: [unknown] Dettagli:soap:Receiver .
Same error in "Server" statistocs" -> <my server name > -> "Message volume" and "antispma/antivirus".
If I try to produce any "adavanced statistcs " graph, all I get is a "no data availbale ".

Can you please help me in uderstanding what's going on?
Thanks

Alberto
Bittone
Posts: 21
Joined: Mon Sep 05, 2016 4:30 pm

Re: Statistics no longer working

Post by Bittone »

Hi Guys,
ok a brief update : I restated all services after an update pushed through yum, still I can see all services up, logs are beeing populated.
I chacked again /opt/zimbra/zmstat dir and all logs files are continuely updated, the die is already 490MBs big (WOW!).
Also /var/log/zimbra-stats.log is continuely updated .
A new hint might come from /var/log/zmlogswatch.log where , upon trying to consult the stats item that return an error (see message above) I find:

Fri Oct 28 14:32:54 2016: Spawned: 4070 -- Connected from 127.0.0.1:52588
Fri Oct 28 14:32:54 2016: Spawned: 4072 -- Connected from 127.0.0.1:52590
Fri Oct 28 14:32:54 2016: Spawned: 4074 -- Connected from 127.0.0.1:52592
Fri Oct 28 14:32:54 2016: Spawned: 4076 -- Connected from 127.0.0.1:52594
Use of uninitialized value $columns[10] in join or string at /opt/zimbra/libexec/zmrrdfetch line 167.
Use of uninitialized value $columns[10] in join or string at /opt/zimbra/libexec/zmrrdfetch line 167.
Use of uninitialized value $columns[10] in join or string at /opt/zimbra/libexec/zmrrdfetch line 167.
Use of uninitialized value $columns[10] in join or string at /opt/zimbra/libexec/zmrrdfetch line 167.

Can please anyonw help me?
Thanks

Alberto
liverpoolfcfan
Elite member
Elite member
Posts: 1096
Joined: Sat Sep 13, 2014 12:47 am

Re: Statistics no longer working

Post by liverpoolfcfan »

Same thing happened to me after upgrade from 8.5 to 8.6. A year and a half later they still do not work and nobody seems to have any useful ideas. https://bugzilla.zimbra.com/show_bug.cgi?id=101167
User avatar
JDunphy
Outstanding Member
Outstanding Member
Posts: 889
Joined: Fri Sep 12, 2014 11:18 pm
Location: Victoria, BC
ZCS/ZD Version: 9.0.0_P39 NETWORK Edition

Re: Statistics no longer working

Post by JDunphy »

I have a few ideas but need a little more information.

Can you look inside /opt/zimbra/log/mysql_error.log to see if you have any errors listed?

Does this report any errors when you list the hostnames?

Code: Select all

/opt/zimbra/libexec/zmrrdfetch -n
Are the stat programs running?

Code: Select all

zmstatctl status
Watch this for a few minutes to see if stats appear to be updating

Code: Select all

tail -f /var/log/zimbra-stats.log
Does it look like the rrd files are being created.

Code: Select all

cd /opt/zimbra/logger/db/data/rrds
ls -lt
I had a few things go wrong initially on my update but they work now. On one of my machines I initialized the database https://wiki.zimbra.com/wiki/Ajcody-Log ... om_Scratch
liverpoolfcfan
Elite member
Elite member
Posts: 1096
Joined: Sat Sep 13, 2014 12:47 am

Re: Statistics no longer working

Post by liverpoolfcfan »

For my case - all the commands you specified look to be working correctly.
I have one rrd file (5-5.rrd) that is 138GB and several others that are close to 20GB each - is that normal?

Is it safe to do the logger reinitialize?
Does the server have to be brought down - or just the logger stopped.

Thanks for your help/suggestions
liverpoolfcfan
Elite member
Elite member
Posts: 1096
Joined: Sat Sep 13, 2014 12:47 am

Re: Statistics no longer working

Post by liverpoolfcfan »

JDunphy wrote:I have a few ideas but need a little more information.
Forgot to quote you so you would receive a notification ...
User avatar
JDunphy
Outstanding Member
Outstanding Member
Posts: 889
Joined: Fri Sep 12, 2014 11:18 pm
Location: Victoria, BC
ZCS/ZD Version: 9.0.0_P39 NETWORK Edition

Re: Statistics no longer working

Post by JDunphy »

liverpoolfcfan wrote:For my case - all the commands you specified look to be working correctly.
I have one rrd file (5-5.rrd) that is 138GB and several others that are close to 20GB each - is that normal?

Is it safe to do the logger reinitialize?
Does the server have to be brought down - or just the logger stopped.

Thanks for your help/suggestions
From my experience, the rdd directory contains a lot of files and yes some can be very large MB's and perhaps GB's in size depending on how busy the server is. Here is an example after just 24 hours after a re-init for a server.

Code: Select all

testmail:zimbra/logger:49> ls db/data/rrds/
1-0.rrd   1-25.rrd  1-40.rrd  1-56.rrd  1-71.rrd  1-87.rrd  2-15.rrd 
1-10.rrd  1-26.rrd  1-41.rrd  1-57.rrd  1-72.rrd  1-88.rrd  2-16.rrd 
1-11.rrd  1-27.rrd  1-42.rrd  1-58.rrd  1-73.rrd  1-89.rrd  2-17.rrd 
1-12.rrd  1-28.rrd  1-43.rrd  1-59.rrd  1-74.rrd  1-8.rrd   2-18.rrd 
1-13.rrd  1-29.rrd  1-44.rrd  1-5.rrd   1-75.rrd  1-90.rrd  2-19.rrd 
1-14.rrd  1-2.rrd   1-45.rrd  1-60.rrd  1-76.rrd  1-91.rrd  2-1.rrd 
1-15.rrd  1-30.rrd  1-46.rrd  1-61.rrd  1-77.rrd  1-92.rrd  2-2.rrd 
1-16.rrd  1-31.rrd  1-47.rrd  1-62.rrd  1-78.rrd  1-93.rrd  2-3.rrd 
1-17.rrd  1-32.rrd  1-48.rrd  1-63.rrd  1-79.rrd  1-94.rrd  2-4.rrd 
1-18.rrd  1-33.rrd  1-49.rrd  1-64.rrd  1-7.rrd   1-95.rrd  2-5.rrd 
1-19.rrd  1-34.rrd  1-4.rrd   1-65.rrd  1-80.rrd  1-9.rrd   2-6.rrd 
1-1.rrd   1-35.rrd  1-50.rrd  1-66.rrd  1-81.rrd  2-0.rrd   2-7.rrd 
1-20.rrd  1-36.rrd  1-51.rrd  1-67.rrd  1-82.rrd  2-10.rrd  2-8.rrd 
1-21.rrd  1-37.rrd  1-52.rrd  1-68.rrd  1-83.rrd  2-11.rrd  2-9.rrd 
1-22.rrd  1-38.rrd  1-53.rrd  1-69.rrd  1-84.rrd  2-12.rrd  
1-23.rrd  1-39.rrd  1-54.rrd  1-6.rrd   1-85.rrd  2-13.rrd  
1-24.rrd  1-3.rrd   1-55.rrd  1-70.rrd  1-86.rrd  2-14.rrd 
It is safe from my experience... just follow the instructions in the link https://wiki.zimbra.com/wiki/Ajcody-Log ... om_Scratch. I had to do this on one of my machines after an upgrade. I do not bring the server down. It sounds like you are a good candidate for this procedure.

I tend to:

Code: Select all

zmloggerctl stop
cd /opt/zimbra/logger
rm -rf db
/opt/zimbra/libexec/zmloggerinit
zmloggerctl start
zmlogswatchctl start
Because /opt/zimbra/libexec/zmloggerinit will re-create the db directory structure and all the associated directories and files including a new sqlitedb file, my first time I did the move as per the link when I wasn't sure what the zmloggerinit command would do. :-) Removing these files will remove previous statistics however. Because you have a lot of previous data, the move would probably work better so you could go back later.

After, you can verify the .rdd files are being built and you should be good.
liverpoolfcfan
Elite member
Elite member
Posts: 1096
Joined: Sat Sep 13, 2014 12:47 am

Re: Statistics no longer working

Post by liverpoolfcfan »

OK. I pulled the trigger and moved the folder. That was more that 30 minutes ago.

While selecting the counters does not throw errors any more - I am also not getting any data for the MTA. In the Advancved statistics dropdown the zmmtastats option has disappeared completely.

How often do the stats get collated?

Could there be a specific stats counter that has stopped working?

Here is the list I get

Code: Select all

[zimbra@mail rrds]$ zmstatctl status
Running: zmstat-proc
Running: zmstat-ldap
Running: zmstat-mtaqueue
Running: zmstat-mysql
Running: zmstat-nginx
Running: zmstat-allprocs
Running: zmstat-io
Running: zmstat-cpu
Running: zmstat-fd
Running: zmstat-vm
Running: zmstat-io-x
Running: zmstat-df
[zimbra@mail rrds]$
liverpoolfcfan
Elite member
Elite member
Posts: 1096
Joined: Sat Sep 13, 2014 12:47 am

Re: Statistics no longer working

Post by liverpoolfcfan »

Just 2 minutes later zmmtastats has reappeared in the list.

The Message Count/Message Volume/Anti-Spam... Monitor options are still showing No data under every heading. Does this take a while to begin to populate?
User avatar
JDunphy
Outstanding Member
Outstanding Member
Posts: 889
Joined: Fri Sep 12, 2014 11:18 pm
Location: Victoria, BC
ZCS/ZD Version: 9.0.0_P39 NETWORK Edition

Re: Statistics no longer working

Post by JDunphy »

liverpoolfcfan wrote:Just 2 minutes later zmmtastats has reappeared in the list.

The Message Count/Message Volume/Anti-Spam... Monitor options are still showing No data under every heading. Does this take a while to begin to populate?
It does take a bit to get enough data to chart. I will use the Advanced Statistics to see how things are going or do an

Code: Select all

ls -lt /opt/zimbra/logger/db/data/rrds 
to see if things are working.

BTW, I run the NETWORK version and have the same stat programs with the exception of zmstat-convertd. Can I assume you are running the open source version?
Last edited by JDunphy on Sun Nov 06, 2016 3:00 pm, edited 1 time in total.
Post Reply