Statistics no longer working
Statistics no longer working
Hello,
I have a Zimbra 8.7 server on Centos 7 (virtual) and I can no longer access statistics: every graph says "no data available" (or something like that since I'm using Italian locale).
I can see that log files in /opt/zimbra/zmstats are correctly populated.
In "Server statistics" -> <my server name > ->"disk" I can see only the stats for the last 60 days but no data for "last 30 days" and "Last 48 hours".
In "Server statistics" -> <my server name > -> "Message count" I get this error upon accessing teh page: Messaggio: system failure: java.lang.ArrayIndexOutOfBoundsException: 10 Codice di errore: service.FAILURE Method: [unknown] Dettagli:soap:Receiver .
Same error in "Server" statistocs" -> <my server name > -> "Message volume" and "antispma/antivirus".
If I try to produce any "adavanced statistcs " graph, all I get is a "no data availbale ".
Can you please help me in uderstanding what's going on?
Thanks
Alberto
I have a Zimbra 8.7 server on Centos 7 (virtual) and I can no longer access statistics: every graph says "no data available" (or something like that since I'm using Italian locale).
I can see that log files in /opt/zimbra/zmstats are correctly populated.
In "Server statistics" -> <my server name > ->"disk" I can see only the stats for the last 60 days but no data for "last 30 days" and "Last 48 hours".
In "Server statistics" -> <my server name > -> "Message count" I get this error upon accessing teh page: Messaggio: system failure: java.lang.ArrayIndexOutOfBoundsException: 10 Codice di errore: service.FAILURE Method: [unknown] Dettagli:soap:Receiver .
Same error in "Server" statistocs" -> <my server name > -> "Message volume" and "antispma/antivirus".
If I try to produce any "adavanced statistcs " graph, all I get is a "no data availbale ".
Can you please help me in uderstanding what's going on?
Thanks
Alberto
Re: Statistics no longer working
Hi Guys,
ok a brief update : I restated all services after an update pushed through yum, still I can see all services up, logs are beeing populated.
I chacked again /opt/zimbra/zmstat dir and all logs files are continuely updated, the die is already 490MBs big (WOW!).
Also /var/log/zimbra-stats.log is continuely updated .
A new hint might come from /var/log/zmlogswatch.log where , upon trying to consult the stats item that return an error (see message above) I find:
Fri Oct 28 14:32:54 2016: Spawned: 4070 -- Connected from 127.0.0.1:52588
Fri Oct 28 14:32:54 2016: Spawned: 4072 -- Connected from 127.0.0.1:52590
Fri Oct 28 14:32:54 2016: Spawned: 4074 -- Connected from 127.0.0.1:52592
Fri Oct 28 14:32:54 2016: Spawned: 4076 -- Connected from 127.0.0.1:52594
Use of uninitialized value $columns[10] in join or string at /opt/zimbra/libexec/zmrrdfetch line 167.
Use of uninitialized value $columns[10] in join or string at /opt/zimbra/libexec/zmrrdfetch line 167.
Use of uninitialized value $columns[10] in join or string at /opt/zimbra/libexec/zmrrdfetch line 167.
Use of uninitialized value $columns[10] in join or string at /opt/zimbra/libexec/zmrrdfetch line 167.
Can please anyonw help me?
Thanks
Alberto
ok a brief update : I restated all services after an update pushed through yum, still I can see all services up, logs are beeing populated.
I chacked again /opt/zimbra/zmstat dir and all logs files are continuely updated, the die is already 490MBs big (WOW!).
Also /var/log/zimbra-stats.log is continuely updated .
A new hint might come from /var/log/zmlogswatch.log where , upon trying to consult the stats item that return an error (see message above) I find:
Fri Oct 28 14:32:54 2016: Spawned: 4070 -- Connected from 127.0.0.1:52588
Fri Oct 28 14:32:54 2016: Spawned: 4072 -- Connected from 127.0.0.1:52590
Fri Oct 28 14:32:54 2016: Spawned: 4074 -- Connected from 127.0.0.1:52592
Fri Oct 28 14:32:54 2016: Spawned: 4076 -- Connected from 127.0.0.1:52594
Use of uninitialized value $columns[10] in join or string at /opt/zimbra/libexec/zmrrdfetch line 167.
Use of uninitialized value $columns[10] in join or string at /opt/zimbra/libexec/zmrrdfetch line 167.
Use of uninitialized value $columns[10] in join or string at /opt/zimbra/libexec/zmrrdfetch line 167.
Use of uninitialized value $columns[10] in join or string at /opt/zimbra/libexec/zmrrdfetch line 167.
Can please anyonw help me?
Thanks
Alberto
-
- Elite member
- Posts: 1104
- Joined: Sat Sep 13, 2014 12:47 am
Re: Statistics no longer working
Same thing happened to me after upgrade from 8.5 to 8.6. A year and a half later they still do not work and nobody seems to have any useful ideas. https://bugzilla.zimbra.com/show_bug.cgi?id=101167
- JDunphy
- Outstanding Member
- Posts: 897
- Joined: Fri Sep 12, 2014 11:18 pm
- Location: Victoria, BC
- ZCS/ZD Version: 9.0.0_P39 NETWORK Edition
Re: Statistics no longer working
I have a few ideas but need a little more information.
Can you look inside /opt/zimbra/log/mysql_error.log to see if you have any errors listed?
Does this report any errors when you list the hostnames?
Are the stat programs running?
Watch this for a few minutes to see if stats appear to be updating
Does it look like the rrd files are being created.
I had a few things go wrong initially on my update but they work now. On one of my machines I initialized the database https://wiki.zimbra.com/wiki/Ajcody-Log ... om_Scratch
Can you look inside /opt/zimbra/log/mysql_error.log to see if you have any errors listed?
Does this report any errors when you list the hostnames?
Code: Select all
/opt/zimbra/libexec/zmrrdfetch -n
Code: Select all
zmstatctl status
Code: Select all
tail -f /var/log/zimbra-stats.log
Code: Select all
cd /opt/zimbra/logger/db/data/rrds
ls -lt
-
- Elite member
- Posts: 1104
- Joined: Sat Sep 13, 2014 12:47 am
Re: Statistics no longer working
For my case - all the commands you specified look to be working correctly.
I have one rrd file (5-5.rrd) that is 138GB and several others that are close to 20GB each - is that normal?
Is it safe to do the logger reinitialize?
Does the server have to be brought down - or just the logger stopped.
Thanks for your help/suggestions
I have one rrd file (5-5.rrd) that is 138GB and several others that are close to 20GB each - is that normal?
Is it safe to do the logger reinitialize?
Does the server have to be brought down - or just the logger stopped.
Thanks for your help/suggestions
-
- Elite member
- Posts: 1104
- Joined: Sat Sep 13, 2014 12:47 am
Re: Statistics no longer working
Forgot to quote you so you would receive a notification ...JDunphy wrote:I have a few ideas but need a little more information.
- JDunphy
- Outstanding Member
- Posts: 897
- Joined: Fri Sep 12, 2014 11:18 pm
- Location: Victoria, BC
- ZCS/ZD Version: 9.0.0_P39 NETWORK Edition
Re: Statistics no longer working
From my experience, the rdd directory contains a lot of files and yes some can be very large MB's and perhaps GB's in size depending on how busy the server is. Here is an example after just 24 hours after a re-init for a server.liverpoolfcfan wrote:For my case - all the commands you specified look to be working correctly.
I have one rrd file (5-5.rrd) that is 138GB and several others that are close to 20GB each - is that normal?
Is it safe to do the logger reinitialize?
Does the server have to be brought down - or just the logger stopped.
Thanks for your help/suggestions
Code: Select all
testmail:zimbra/logger:49> ls db/data/rrds/
1-0.rrd 1-25.rrd 1-40.rrd 1-56.rrd 1-71.rrd 1-87.rrd 2-15.rrd
1-10.rrd 1-26.rrd 1-41.rrd 1-57.rrd 1-72.rrd 1-88.rrd 2-16.rrd
1-11.rrd 1-27.rrd 1-42.rrd 1-58.rrd 1-73.rrd 1-89.rrd 2-17.rrd
1-12.rrd 1-28.rrd 1-43.rrd 1-59.rrd 1-74.rrd 1-8.rrd 2-18.rrd
1-13.rrd 1-29.rrd 1-44.rrd 1-5.rrd 1-75.rrd 1-90.rrd 2-19.rrd
1-14.rrd 1-2.rrd 1-45.rrd 1-60.rrd 1-76.rrd 1-91.rrd 2-1.rrd
1-15.rrd 1-30.rrd 1-46.rrd 1-61.rrd 1-77.rrd 1-92.rrd 2-2.rrd
1-16.rrd 1-31.rrd 1-47.rrd 1-62.rrd 1-78.rrd 1-93.rrd 2-3.rrd
1-17.rrd 1-32.rrd 1-48.rrd 1-63.rrd 1-79.rrd 1-94.rrd 2-4.rrd
1-18.rrd 1-33.rrd 1-49.rrd 1-64.rrd 1-7.rrd 1-95.rrd 2-5.rrd
1-19.rrd 1-34.rrd 1-4.rrd 1-65.rrd 1-80.rrd 1-9.rrd 2-6.rrd
1-1.rrd 1-35.rrd 1-50.rrd 1-66.rrd 1-81.rrd 2-0.rrd 2-7.rrd
1-20.rrd 1-36.rrd 1-51.rrd 1-67.rrd 1-82.rrd 2-10.rrd 2-8.rrd
1-21.rrd 1-37.rrd 1-52.rrd 1-68.rrd 1-83.rrd 2-11.rrd 2-9.rrd
1-22.rrd 1-38.rrd 1-53.rrd 1-69.rrd 1-84.rrd 2-12.rrd
1-23.rrd 1-39.rrd 1-54.rrd 1-6.rrd 1-85.rrd 2-13.rrd
1-24.rrd 1-3.rrd 1-55.rrd 1-70.rrd 1-86.rrd 2-14.rrd
I tend to:
Code: Select all
zmloggerctl stop
cd /opt/zimbra/logger
rm -rf db
/opt/zimbra/libexec/zmloggerinit
zmloggerctl start
zmlogswatchctl start
After, you can verify the .rdd files are being built and you should be good.
-
- Elite member
- Posts: 1104
- Joined: Sat Sep 13, 2014 12:47 am
Re: Statistics no longer working
OK. I pulled the trigger and moved the folder. That was more that 30 minutes ago.
While selecting the counters does not throw errors any more - I am also not getting any data for the MTA. In the Advancved statistics dropdown the zmmtastats option has disappeared completely.
How often do the stats get collated?
Could there be a specific stats counter that has stopped working?
Here is the list I get
While selecting the counters does not throw errors any more - I am also not getting any data for the MTA. In the Advancved statistics dropdown the zmmtastats option has disappeared completely.
How often do the stats get collated?
Could there be a specific stats counter that has stopped working?
Here is the list I get
Code: Select all
[zimbra@mail rrds]$ zmstatctl status
Running: zmstat-proc
Running: zmstat-ldap
Running: zmstat-mtaqueue
Running: zmstat-mysql
Running: zmstat-nginx
Running: zmstat-allprocs
Running: zmstat-io
Running: zmstat-cpu
Running: zmstat-fd
Running: zmstat-vm
Running: zmstat-io-x
Running: zmstat-df
[zimbra@mail rrds]$
-
- Elite member
- Posts: 1104
- Joined: Sat Sep 13, 2014 12:47 am
Re: Statistics no longer working
Just 2 minutes later zmmtastats has reappeared in the list.
The Message Count/Message Volume/Anti-Spam... Monitor options are still showing No data under every heading. Does this take a while to begin to populate?
The Message Count/Message Volume/Anti-Spam... Monitor options are still showing No data under every heading. Does this take a while to begin to populate?
- JDunphy
- Outstanding Member
- Posts: 897
- Joined: Fri Sep 12, 2014 11:18 pm
- Location: Victoria, BC
- ZCS/ZD Version: 9.0.0_P39 NETWORK Edition
Re: Statistics no longer working
It does take a bit to get enough data to chart. I will use the Advanced Statistics to see how things are going or do anliverpoolfcfan wrote:Just 2 minutes later zmmtastats has reappeared in the list.
The Message Count/Message Volume/Anti-Spam... Monitor options are still showing No data under every heading. Does this take a while to begin to populate?
Code: Select all
ls -lt /opt/zimbra/logger/db/data/rrds
BTW, I run the NETWORK version and have the same stat programs with the exception of zmstat-convertd. Can I assume you are running the open source version?
Last edited by JDunphy on Sun Nov 06, 2016 3:00 pm, edited 1 time in total.