Page 1 of 3
Statistics no longer working
Posted: Tue Oct 25, 2016 9:51 am
by Bittone
Hello,
I have a Zimbra 8.7 server on Centos 7 (virtual) and I can no longer access statistics: every graph says "no data available" (or something like that since I'm using Italian locale).
I can see that log files in /opt/zimbra/zmstats are correctly populated.
In "Server statistics" -> <my server name > ->"disk" I can see only the stats for the last 60 days but no data for "last 30 days" and "Last 48 hours".
In "Server statistics" -> <my server name > -> "Message count" I get this error upon accessing teh page: Messaggio: system failure: java.lang.ArrayIndexOutOfBoundsException: 10 Codice di errore: service.FAILURE Method: [unknown] Dettagli:soap:Receiver .
Same error in "Server" statistocs" -> <my server name > -> "Message volume" and "antispma/antivirus".
If I try to produce any "adavanced statistcs " graph, all I get is a "no data availbale ".
Can you please help me in uderstanding what's going on?
Thanks
Alberto
Re: Statistics no longer working
Posted: Fri Oct 28, 2016 12:34 pm
by Bittone
Hi Guys,
ok a brief update : I restated all services after an update pushed through yum, still I can see all services up, logs are beeing populated.
I chacked again /opt/zimbra/zmstat dir and all logs files are continuely updated, the die is already 490MBs big (WOW!).
Also /var/log/zimbra-stats.log is continuely updated .
A new hint might come from /var/log/zmlogswatch.log where , upon trying to consult the stats item that return an error (see message above) I find:
Fri Oct 28 14:32:54 2016: Spawned: 4070 -- Connected from 127.0.0.1:52588
Fri Oct 28 14:32:54 2016: Spawned: 4072 -- Connected from 127.0.0.1:52590
Fri Oct 28 14:32:54 2016: Spawned: 4074 -- Connected from 127.0.0.1:52592
Fri Oct 28 14:32:54 2016: Spawned: 4076 -- Connected from 127.0.0.1:52594
Use of uninitialized value $columns[10] in join or string at /opt/zimbra/libexec/zmrrdfetch line 167.
Use of uninitialized value $columns[10] in join or string at /opt/zimbra/libexec/zmrrdfetch line 167.
Use of uninitialized value $columns[10] in join or string at /opt/zimbra/libexec/zmrrdfetch line 167.
Use of uninitialized value $columns[10] in join or string at /opt/zimbra/libexec/zmrrdfetch line 167.
Can please anyonw help me?
Thanks
Alberto
Re: Statistics no longer working
Posted: Thu Nov 03, 2016 2:28 pm
by liverpoolfcfan
Same thing happened to me after upgrade from 8.5 to 8.6. A year and a half later they still do not work and nobody seems to have any useful ideas.
https://bugzilla.zimbra.com/show_bug.cgi?id=101167
Re: Statistics no longer working
Posted: Thu Nov 03, 2016 5:11 pm
by JDunphy
I have a few ideas but need a little more information.
Can you look inside /opt/zimbra/log/mysql_error.log to see if you have any errors listed?
Does this report any errors when you list the hostnames?
Are the stat programs running?
Watch this for a few minutes to see if stats appear to be updating
Does it look like the rrd files are being created.
Code: Select all
cd /opt/zimbra/logger/db/data/rrds
ls -lt
I had a few things go wrong initially on my update but they work now. On one of my machines I initialized the database
https://wiki.zimbra.com/wiki/Ajcody-Log ... om_Scratch
Re: Statistics no longer working
Posted: Fri Nov 04, 2016 1:03 pm
by liverpoolfcfan
For my case - all the commands you specified look to be working correctly.
I have one rrd file (5-5.rrd) that is 138GB and several others that are close to 20GB each - is that normal?
Is it safe to do the logger reinitialize?
Does the server have to be brought down - or just the logger stopped.
Thanks for your help/suggestions
Re: Statistics no longer working
Posted: Fri Nov 04, 2016 1:05 pm
by liverpoolfcfan
JDunphy wrote:I have a few ideas but need a little more information.
Forgot to quote you so you would receive a notification ...
Re: Statistics no longer working
Posted: Fri Nov 04, 2016 1:58 pm
by JDunphy
liverpoolfcfan wrote:For my case - all the commands you specified look to be working correctly.
I have one rrd file (5-5.rrd) that is 138GB and several others that are close to 20GB each - is that normal?
Is it safe to do the logger reinitialize?
Does the server have to be brought down - or just the logger stopped.
Thanks for your help/suggestions
From my experience, the rdd directory contains a lot of files and yes some can be very large MB's and perhaps GB's in size depending on how busy the server is. Here is an example after just 24 hours after a re-init for a server.
Code: Select all
testmail:zimbra/logger:49> ls db/data/rrds/
1-0.rrd 1-25.rrd 1-40.rrd 1-56.rrd 1-71.rrd 1-87.rrd 2-15.rrd
1-10.rrd 1-26.rrd 1-41.rrd 1-57.rrd 1-72.rrd 1-88.rrd 2-16.rrd
1-11.rrd 1-27.rrd 1-42.rrd 1-58.rrd 1-73.rrd 1-89.rrd 2-17.rrd
1-12.rrd 1-28.rrd 1-43.rrd 1-59.rrd 1-74.rrd 1-8.rrd 2-18.rrd
1-13.rrd 1-29.rrd 1-44.rrd 1-5.rrd 1-75.rrd 1-90.rrd 2-19.rrd
1-14.rrd 1-2.rrd 1-45.rrd 1-60.rrd 1-76.rrd 1-91.rrd 2-1.rrd
1-15.rrd 1-30.rrd 1-46.rrd 1-61.rrd 1-77.rrd 1-92.rrd 2-2.rrd
1-16.rrd 1-31.rrd 1-47.rrd 1-62.rrd 1-78.rrd 1-93.rrd 2-3.rrd
1-17.rrd 1-32.rrd 1-48.rrd 1-63.rrd 1-79.rrd 1-94.rrd 2-4.rrd
1-18.rrd 1-33.rrd 1-49.rrd 1-64.rrd 1-7.rrd 1-95.rrd 2-5.rrd
1-19.rrd 1-34.rrd 1-4.rrd 1-65.rrd 1-80.rrd 1-9.rrd 2-6.rrd
1-1.rrd 1-35.rrd 1-50.rrd 1-66.rrd 1-81.rrd 2-0.rrd 2-7.rrd
1-20.rrd 1-36.rrd 1-51.rrd 1-67.rrd 1-82.rrd 2-10.rrd 2-8.rrd
1-21.rrd 1-37.rrd 1-52.rrd 1-68.rrd 1-83.rrd 2-11.rrd 2-9.rrd
1-22.rrd 1-38.rrd 1-53.rrd 1-69.rrd 1-84.rrd 2-12.rrd
1-23.rrd 1-39.rrd 1-54.rrd 1-6.rrd 1-85.rrd 2-13.rrd
1-24.rrd 1-3.rrd 1-55.rrd 1-70.rrd 1-86.rrd 2-14.rrd
It is safe from my experience... just follow the instructions in the link
https://wiki.zimbra.com/wiki/Ajcody-Log ... om_Scratch. I had to do this on one of my machines after an upgrade. I do not bring the server down. It sounds like you are a good candidate for this procedure.
I tend to:
Code: Select all
zmloggerctl stop
cd /opt/zimbra/logger
rm -rf db
/opt/zimbra/libexec/zmloggerinit
zmloggerctl start
zmlogswatchctl start
Because /opt/zimbra/libexec/zmloggerinit will re-create the db directory structure and all the associated directories and files including a new sqlitedb file, my first time I did the move as per the link when I wasn't sure what the
zmloggerinit command would do.
Removing these files will remove previous statistics however. Because you have a lot of previous data, the move would probably work better so you could go back later.
After, you can verify the .rdd files are being built and you should be good.
Re: Statistics no longer working
Posted: Fri Nov 04, 2016 3:26 pm
by liverpoolfcfan
OK. I pulled the trigger and moved the folder. That was more that 30 minutes ago.
While selecting the counters does not throw errors any more - I am also not getting any data for the MTA. In the Advancved statistics dropdown the zmmtastats option has disappeared completely.
How often do the stats get collated?
Could there be a specific stats counter that has stopped working?
Here is the list I get
Code: Select all
[zimbra@mail rrds]$ zmstatctl status
Running: zmstat-proc
Running: zmstat-ldap
Running: zmstat-mtaqueue
Running: zmstat-mysql
Running: zmstat-nginx
Running: zmstat-allprocs
Running: zmstat-io
Running: zmstat-cpu
Running: zmstat-fd
Running: zmstat-vm
Running: zmstat-io-x
Running: zmstat-df
[zimbra@mail rrds]$
Re: Statistics no longer working
Posted: Fri Nov 04, 2016 3:37 pm
by liverpoolfcfan
Just 2 minutes later zmmtastats has reappeared in the list.
The Message Count/Message Volume/Anti-Spam... Monitor options are still showing No data under every heading. Does this take a while to begin to populate?
Re: Statistics no longer working
Posted: Fri Nov 04, 2016 3:50 pm
by JDunphy
liverpoolfcfan wrote:Just 2 minutes later zmmtastats has reappeared in the list.
The Message Count/Message Volume/Anti-Spam... Monitor options are still showing No data under every heading. Does this take a while to begin to populate?
It does take a bit to get enough data to chart. I will use the Advanced Statistics to see how things are going or do an
Code: Select all
ls -lt /opt/zimbra/logger/db/data/rrds
to see if things are working.
BTW, I run the NETWORK version and have the same stat programs with the exception of zmstat-convertd. Can I assume you are running the open source version?