Statistics no longer working

Discuss your pilot or production implementation with other Zimbra admins or our engineers.
Bittone
Posts: 6
Joined: Mon Sep 05, 2016 4:30 pm

Statistics no longer working

Postby Bittone » Tue Oct 25, 2016 9:51 am

Hello,
I have a Zimbra 8.7 server on Centos 7 (virtual) and I can no longer access statistics: every graph says "no data available" (or something like that since I'm using Italian locale).
I can see that log files in /opt/zimbra/zmstats are correctly populated.
In "Server statistics" -> <my server name > ->"disk" I can see only the stats for the last 60 days but no data for "last 30 days" and "Last 48 hours".
In "Server statistics" -> <my server name > -> "Message count" I get this error upon accessing teh page: Messaggio: system failure: java.lang.ArrayIndexOutOfBoundsException: 10 Codice di errore: service.FAILURE Method: [unknown] Dettagli:soap:Receiver .
Same error in "Server" statistocs" -> <my server name > -> "Message volume" and "antispma/antivirus".
If I try to produce any "adavanced statistcs " graph, all I get is a "no data availbale ".

Can you please help me in uderstanding what's going on?
Thanks

Alberto


Bittone
Posts: 6
Joined: Mon Sep 05, 2016 4:30 pm

Re: Statistics no longer working

Postby Bittone » Fri Oct 28, 2016 12:34 pm

Hi Guys,
ok a brief update : I restated all services after an update pushed through yum, still I can see all services up, logs are beeing populated.
I chacked again /opt/zimbra/zmstat dir and all logs files are continuely updated, the die is already 490MBs big (WOW!).
Also /var/log/zimbra-stats.log is continuely updated .
A new hint might come from /var/log/zmlogswatch.log where , upon trying to consult the stats item that return an error (see message above) I find:

Fri Oct 28 14:32:54 2016: Spawned: 4070 -- Connected from 127.0.0.1:52588
Fri Oct 28 14:32:54 2016: Spawned: 4072 -- Connected from 127.0.0.1:52590
Fri Oct 28 14:32:54 2016: Spawned: 4074 -- Connected from 127.0.0.1:52592
Fri Oct 28 14:32:54 2016: Spawned: 4076 -- Connected from 127.0.0.1:52594
Use of uninitialized value $columns[10] in join or string at /opt/zimbra/libexec/zmrrdfetch line 167.
Use of uninitialized value $columns[10] in join or string at /opt/zimbra/libexec/zmrrdfetch line 167.
Use of uninitialized value $columns[10] in join or string at /opt/zimbra/libexec/zmrrdfetch line 167.
Use of uninitialized value $columns[10] in join or string at /opt/zimbra/libexec/zmrrdfetch line 167.

Can please anyonw help me?
Thanks

Alberto
liverpoolfcfan
Outstanding Member
Outstanding Member
Posts: 907
Joined: Sat Sep 13, 2014 12:47 am

Re: Statistics no longer working

Postby liverpoolfcfan » Thu Nov 03, 2016 2:28 pm

Same thing happened to me after upgrade from 8.5 to 8.6. A year and a half later they still do not work and nobody seems to have any useful ideas. https://bugzilla.zimbra.com/show_bug.cgi?id=101167
User avatar
JDunphy
Advanced member
Advanced member
Posts: 121
Joined: Fri Sep 12, 2014 11:18 pm
Location: Victoria, BC
ZCS/ZD Version: Release 8.7.9_GA_1794.RHEL6_64_2017
Contact:

Re: Statistics no longer working

Postby JDunphy » Thu Nov 03, 2016 5:11 pm

I have a few ideas but need a little more information.

Can you look inside /opt/zimbra/log/mysql_error.log to see if you have any errors listed?

Does this report any errors when you list the hostnames?

Code: Select all

/opt/zimbra/libexec/zmrrdfetch -n


Are the stat programs running?

Code: Select all

zmstatctl status


Watch this for a few minutes to see if stats appear to be updating

Code: Select all

tail -f /var/log/zimbra-stats.log


Does it look like the rrd files are being created.

Code: Select all

cd /opt/zimbra/logger/db/data/rrds
ls -lt


I had a few things go wrong initially on my update but they work now. On one of my machines I initialized the database https://wiki.zimbra.com/wiki/Ajcody-Logger-Issues#Reinitializing_Logger_Database_From_Scratch
liverpoolfcfan
Outstanding Member
Outstanding Member
Posts: 907
Joined: Sat Sep 13, 2014 12:47 am

Re: Statistics no longer working

Postby liverpoolfcfan » Fri Nov 04, 2016 1:03 pm

For my case - all the commands you specified look to be working correctly.
I have one rrd file (5-5.rrd) that is 138GB and several others that are close to 20GB each - is that normal?

Is it safe to do the logger reinitialize?
Does the server have to be brought down - or just the logger stopped.

Thanks for your help/suggestions
liverpoolfcfan
Outstanding Member
Outstanding Member
Posts: 907
Joined: Sat Sep 13, 2014 12:47 am

Re: Statistics no longer working

Postby liverpoolfcfan » Fri Nov 04, 2016 1:05 pm

JDunphy wrote:I have a few ideas but need a little more information.

Forgot to quote you so you would receive a notification ...
User avatar
JDunphy
Advanced member
Advanced member
Posts: 121
Joined: Fri Sep 12, 2014 11:18 pm
Location: Victoria, BC
ZCS/ZD Version: Release 8.7.9_GA_1794.RHEL6_64_2017
Contact:

Re: Statistics no longer working

Postby JDunphy » Fri Nov 04, 2016 1:58 pm

liverpoolfcfan wrote:For my case - all the commands you specified look to be working correctly.
I have one rrd file (5-5.rrd) that is 138GB and several others that are close to 20GB each - is that normal?

Is it safe to do the logger reinitialize?
Does the server have to be brought down - or just the logger stopped.

Thanks for your help/suggestions


From my experience, the rdd directory contains a lot of files and yes some can be very large MB's and perhaps GB's in size depending on how busy the server is. Here is an example after just 24 hours after a re-init for a server.

Code: Select all

testmail:zimbra/logger:49> ls db/data/rrds/
1-0.rrd   1-25.rrd  1-40.rrd  1-56.rrd  1-71.rrd  1-87.rrd  2-15.rrd
1-10.rrd  1-26.rrd  1-41.rrd  1-57.rrd  1-72.rrd  1-88.rrd  2-16.rrd
1-11.rrd  1-27.rrd  1-42.rrd  1-58.rrd  1-73.rrd  1-89.rrd  2-17.rrd
1-12.rrd  1-28.rrd  1-43.rrd  1-59.rrd  1-74.rrd  1-8.rrd   2-18.rrd
1-13.rrd  1-29.rrd  1-44.rrd  1-5.rrd   1-75.rrd  1-90.rrd  2-19.rrd
1-14.rrd  1-2.rrd   1-45.rrd  1-60.rrd  1-76.rrd  1-91.rrd  2-1.rrd
1-15.rrd  1-30.rrd  1-46.rrd  1-61.rrd  1-77.rrd  1-92.rrd  2-2.rrd
1-16.rrd  1-31.rrd  1-47.rrd  1-62.rrd  1-78.rrd  1-93.rrd  2-3.rrd
1-17.rrd  1-32.rrd  1-48.rrd  1-63.rrd  1-79.rrd  1-94.rrd  2-4.rrd
1-18.rrd  1-33.rrd  1-49.rrd  1-64.rrd  1-7.rrd   1-95.rrd  2-5.rrd
1-19.rrd  1-34.rrd  1-4.rrd   1-65.rrd  1-80.rrd  1-9.rrd   2-6.rrd
1-1.rrd   1-35.rrd  1-50.rrd  1-66.rrd  1-81.rrd  2-0.rrd   2-7.rrd
1-20.rrd  1-36.rrd  1-51.rrd  1-67.rrd  1-82.rrd  2-10.rrd  2-8.rrd
1-21.rrd  1-37.rrd  1-52.rrd  1-68.rrd  1-83.rrd  2-11.rrd  2-9.rrd
1-22.rrd  1-38.rrd  1-53.rrd  1-69.rrd  1-84.rrd  2-12.rrd 
1-23.rrd  1-39.rrd  1-54.rrd  1-6.rrd   1-85.rrd  2-13.rrd 
1-24.rrd  1-3.rrd   1-55.rrd  1-70.rrd  1-86.rrd  2-14.rrd


It is safe from my experience... just follow the instructions in the link https://wiki.zimbra.com/wiki/Ajcody-Logger-Issues#Reinitializing_Logger_Database_From_Scratch. I had to do this on one of my machines after an upgrade. I do not bring the server down. It sounds like you are a good candidate for this procedure.

I tend to:

Code: Select all

zmloggerctl stop
cd /opt/zimbra/logger
rm -rf db
/opt/zimbra/libexec/zmloggerinit
zmloggerctl start
zmlogswatchctl start


Because /opt/zimbra/libexec/zmloggerinit will re-create the db directory structure and all the associated directories and files including a new sqlitedb file, my first time I did the move as per the link when I wasn't sure what the zmloggerinit command would do. :-) Removing these files will remove previous statistics however. Because you have a lot of previous data, the move would probably work better so you could go back later.

After, you can verify the .rdd files are being built and you should be good.
liverpoolfcfan
Outstanding Member
Outstanding Member
Posts: 907
Joined: Sat Sep 13, 2014 12:47 am

Re: Statistics no longer working

Postby liverpoolfcfan » Fri Nov 04, 2016 3:26 pm

OK. I pulled the trigger and moved the folder. That was more that 30 minutes ago.

While selecting the counters does not throw errors any more - I am also not getting any data for the MTA. In the Advancved statistics dropdown the zmmtastats option has disappeared completely.

How often do the stats get collated?

Could there be a specific stats counter that has stopped working?

Here is the list I get

Code: Select all

[zimbra@mail rrds]$ zmstatctl status
Running: zmstat-proc
Running: zmstat-ldap
Running: zmstat-mtaqueue
Running: zmstat-mysql
Running: zmstat-nginx
Running: zmstat-allprocs
Running: zmstat-io
Running: zmstat-cpu
Running: zmstat-fd
Running: zmstat-vm
Running: zmstat-io-x
Running: zmstat-df
[zimbra@mail rrds]$
liverpoolfcfan
Outstanding Member
Outstanding Member
Posts: 907
Joined: Sat Sep 13, 2014 12:47 am

Re: Statistics no longer working

Postby liverpoolfcfan » Fri Nov 04, 2016 3:37 pm

Just 2 minutes later zmmtastats has reappeared in the list.

The Message Count/Message Volume/Anti-Spam... Monitor options are still showing No data under every heading. Does this take a while to begin to populate?
User avatar
JDunphy
Advanced member
Advanced member
Posts: 121
Joined: Fri Sep 12, 2014 11:18 pm
Location: Victoria, BC
ZCS/ZD Version: Release 8.7.9_GA_1794.RHEL6_64_2017
Contact:

Re: Statistics no longer working

Postby JDunphy » Fri Nov 04, 2016 3:50 pm

liverpoolfcfan wrote:Just 2 minutes later zmmtastats has reappeared in the list.

The Message Count/Message Volume/Anti-Spam... Monitor options are still showing No data under every heading. Does this take a while to begin to populate?


It does take a bit to get enough data to chart. I will use the Advanced Statistics to see how things are going or do an

Code: Select all

ls -lt /opt/zimbra/logger/db/data/rrds
to see if things are working.

BTW, I run the NETWORK version and have the same stat programs with the exception of zmstat-convertd. Can I assume you are running the open source version?
Last edited by JDunphy on Sun Nov 06, 2016 3:00 pm, edited 1 time in total.

Return to “Administrators”

Who is online

Users browsing this forum: No registered users and 16 guests