Daily Mail Report - missing data

Discuss your pilot or production implementation with other Zimbra admins or our engineers.
klinet
Posts: 19
Joined: Fri Sep 12, 2014 10:38 pm

Daily Mail Report - missing data

Post by klinet »

My daily mail report seems to be working only in the afternoon.
The per hour traffic summary does not begin until 2PM. :confused:
See attachment
Anyone have any thoughts?
Thanks

Todd
Picture 56.png
User avatar
L. Mark Stone
Ambassador
Ambassador
Posts: 2796
Joined: Wed Oct 09, 2013 11:35 am
Location: Portland, Maine, US
ZCS/ZD Version: 10.0.6 Network Edition
Contact:

Daily Mail Report - missing data

Post by L. Mark Stone »

FWIW we are seeing the exact same thing, except the start-of-data-time for us is different.
We've been busy sorting out more pressing issues from our recent Zimbra upgrade (see ), so haven't gotten to this problem fully just yet since we only got stats back working properly yesterday.
So far, what we see is that the logs /var/log/zimbra*.log seem to be getting rotated around 6:15pm our time. Our server's hardware clocks are set to UTC, and our Time Zone on the servers (and in Zimbra) is set to Eastern, i.e. GMT -4 (daylight savings time).
We'll look at the logrotation scripts and confirm system time later and report back here whatever we find.
Curious what time zone your system is configured for and if the number of hours difference in our respective start-of-data-times in the daily mail report matches exactly with the hours difference between our two time zones.
Hope that helps,

Mark
___________________________________
L. Mark Stone
Mission Critical Email - Zimbra VAR/BSP/Training Partner https://www.missioncriticalemail.com/
AWS Certified Solutions Architect-Associate
klinet
Posts: 19
Joined: Fri Sep 12, 2014 10:38 pm

Daily Mail Report - missing data

Post by klinet »

Thanks for the info, I think you may be onto something with the log rotation times. I see that you are also using SLES 10x64. We are in Central time, so if you are seeing a one hour difference in the start of your hourly info then I think we have a good place to be looking. This is also a low priority for us, but still looking at it as time permits.
Thanks

Todd
User avatar
L. Mark Stone
Ambassador
Ambassador
Posts: 2796
Joined: Wed Oct 09, 2013 11:35 am
Location: Portland, Maine, US
ZCS/ZD Version: 10.0.6 Network Edition
Contact:

Daily Mail Report - missing data

Post by L. Mark Stone »

I think I have identified the root cause of this issue...
I looked at the time stamps for the /var/log/*.bz2 files on about a dozen SLES systems this morning (some Zimbra hosts, some not), and saw that the log files being rotated by scripts in /etc/cron.daily had time stamps of roughly on the hour and fifteen/thirty/forty-five minutes after the hour -- and that the hours were all pretty random. There was no correlation between time zones at all.
I also noticed that the time of day the files were rotated changed by an hour when Daylight Savings Time kicked in.
The new Zimbra logging system as I understand it, including the daily mail report and statistics in the Admin console, relies on data in /var/log/zimbra.log and /var/log/zimbra-stats.log being available to parse.
When these log files are rotated in the middle of the day, the data in the daily mail report and stats in the Admin Console is incomplete.
So why are these log files being rotated by scripts in /etc/cron.daily on such a seemingly random basis?
Because unless the "DAILY_TIME" parameter in /etc/sysconfig/cron is set to a specific time, cron.daily is run by default at the next 15 minute interval past the last boot time hour -- until the server is once again rebooted.
On all the SLES systems I looked at, "DAILY_TIME" was blank.
Consequently, /etc/cron.daily/zimbra is being executed each day based on the last boot time of the server.
This is not what Zimbra I believe intends :rolleyes: so I have filed a bug report here: Bug 43847 – Incorrect Log File Rotation Method Causing Missing Data in Daily Admin Report and Admin Console Stats Pages
Would be interesting if admins using other distros could comment on their setups here as well...
Hope that helps,

Mark
___________________________________
L. Mark Stone
Mission Critical Email - Zimbra VAR/BSP/Training Partner https://www.missioncriticalemail.com/
AWS Certified Solutions Architect-Associate
uxbod
Ambassador
Ambassador
Posts: 7811
Joined: Fri Sep 12, 2014 10:21 pm

Daily Mail Report - missing data

Post by uxbod »

Mark,
on CentOS it is controller by /etc/crontab02 4 * * * root run-parts /etc/cron.daily
User avatar
L. Mark Stone
Ambassador
Ambassador
Posts: 2796
Joined: Wed Oct 09, 2013 11:35 am
Location: Portland, Maine, US
ZCS/ZD Version: 10.0.6 Network Edition
Contact:

Daily Mail Report - missing data

Post by L. Mark Stone »

[quote user="uxbod"]Mark,
on CentOS it is controller by /etc/crontab02 4 * * * root run-parts /etc/cron.daily[/QUOTE]
Thanks for posting that!
So, are there time periods in your daily mail report that are missing data?
All the best,

Mark
___________________________________
L. Mark Stone
Mission Critical Email - Zimbra VAR/BSP/Training Partner https://www.missioncriticalemail.com/
AWS Certified Solutions Architect-Associate
phoenix
Ambassador
Ambassador
Posts: 27272
Joined: Fri Sep 12, 2014 9:56 pm
Location: Liverpool, England

Daily Mail Report - missing data

Post by phoenix »

[quote user="LMStone"]So, are there time periods in your daily mail report that are missing data?[/QUOTE]Not on my CentOS server, there is received mail in all the time slots.
Regards

Bill

Rspamd: A high performance spamassassin replacement

Per ardua ad astra
User avatar
L. Mark Stone
Ambassador
Ambassador
Posts: 2796
Joined: Wed Oct 09, 2013 11:35 am
Location: Portland, Maine, US
ZCS/ZD Version: 10.0.6 Network Edition
Contact:

Daily Mail Report - missing data

Post by L. Mark Stone »

[quote user="10330phoenix"]Not on my CentOS server, there is received mail in all the time slots.[/QUOTE]
Well, we'll see what happens in two days then!
Thanks Bill,

Mark
P.S. Unsupported OS eh? :eek:
___________________________________
L. Mark Stone
Mission Critical Email - Zimbra VAR/BSP/Training Partner https://www.missioncriticalemail.com/
AWS Certified Solutions Architect-Associate
phoenix
Ambassador
Ambassador
Posts: 27272
Joined: Fri Sep 12, 2014 9:56 pm
Location: Liverpool, England

Daily Mail Report - missing data

Post by phoenix »

[quote user="LMStone"]P.S. Unsupported OS eh? :eek:[/QUOTE]Of course but I don't get support for my installation anyway. :)
Regards

Bill

Rspamd: A high performance spamassassin replacement

Per ardua ad astra
jrefl5
Outstanding Member
Outstanding Member
Posts: 203
Joined: Fri Sep 12, 2014 11:12 pm

Daily Mail Report - missing data

Post by jrefl5 »

Like uxbod, and phoenix I don't have any problems, but then I'm also on CENTOS.
you might want to try to set a reasonable time in "DAILY_TIME".
the zmdailyreport attempts to open both the current zimbra.log, and zimbra.log.1 or zimbra.log.1.gz so it should be able to get the prior day.
Post Reply