Hello.
My Zimbra 8.8.10 FOSS server with all roles was up 170 days. I've got a task to optimise zimbra server's resources.
I reduced mysql and java memory usage both to ~1GB and restarted zimbra services. Nothing goes wrong.
Later I stopped zimbra's vm, removed 2 cpu cores and started the vm.
A few minutes later I received an alert that free space is critical. To avoid data loss I stopped zimbra services and free space gradually returned back to normal. At some point I noticed that the size of the /opt/zimbra/zmstat folder was about 24GB, now it's 1.1GB. Seems like the free space has decreased to zero during services shutdown.
Free space, 200GB total:
After the second reboot there was no reduction of free space.
I have some questions for the community:
1. What happened after the 1st reboot? Is this normal or not?
2. Is there a risk of data loss when the free space went down to zero?
3. Is there a risk of data loss after I stopped zimbra's services during strange process ate free space?
Best regards!
UPD:
I don't understand what these peaks means, especially the last one:
Disk full after reboot
Re: Disk full after reboot
I repeated this issue in test environment.
After starting the server, the following processes start writing to disk:
They run till the disk is full:
Approx. after 15 minutes, the csv and gzip file sizes return to normal. It looks like deleting old statistical data. If so, this is a very strange implementation, especially if Zimbra server uses a dynamic virtual disk. It turns out I need to restart services periodically to avoid this issue.
After starting the server, the following processes start writing to disk:
They run till the disk is full:
Approx. after 15 minutes, the csv and gzip file sizes return to normal. It looks like deleting old statistical data. If so, this is a very strange implementation, especially if Zimbra server uses a dynamic virtual disk. It turns out I need to restart services periodically to avoid this issue.
Re: Disk full after reboot
A few simple questions. Why have you reduced the mysql & java memory to 1GB and why have you removed two vCPUs? What is the current configuration of your server, RAM, CPUs etc., etc.? What happens if you revert to your original VM specs? What, if anything, is in the logs that might point to a problem? The two peaks in your graph show a high message count, does that really indicate the level of mail arriving at your server? Around the time of those peaks can you determine from the log files where the emails are coming from?
Re: Disk full after reboot
Hello and sorry for this late reply.
But this is not important for this issue, because my test environment VM have the same problem without mysql, java or VM specs changes. Looks like the only reason is reboot after the long uptime and shrinking a big amount of statistical data. The next reboot will not cause this problem.
BTW I do not see screenshots in this thread. Here are the new ones:
Strange peaks in the current production environment: The reason for lack of space: unzipping (statistical?) data into large csv files, shrinking and zipping them again Could it be that I had to set up automatic cleaning of statistics, but didn’t? I can not find how to do it.
Thanks!
It was a task from my boss. He asked me to reduce resources for this VM because we have only 70 users. Maybe it was not a good idea.phoenix wrote: A few simple questions. Why have you reduced the mysql & java memory to 1GB and why have you removed two vCPUs? What is the current configuration of your server, RAM, CPUs etc., etc.? What happens if you revert to your original VM specs?
But this is not important for this issue, because my test environment VM have the same problem without mysql, java or VM specs changes. Looks like the only reason is reboot after the long uptime and shrinking a big amount of statistical data. The next reboot will not cause this problem.
I did not find anything interesting in the system logs, maybe I can’t search well. But I noticed that inexplicable peaks in the number of messages still occur. Please give me some time for the investigation.phoenix wrote: What, if anything, is in the logs that might point to a problem? The two peaks in your graph show a high message count, does that really indicate the level of mail arriving at your server? Around the time of those peaks can you determine from the log files where the emails are coming from?
BTW I do not see screenshots in this thread. Here are the new ones:
Strange peaks in the current production environment: The reason for lack of space: unzipping (statistical?) data into large csv files, shrinking and zipping them again Could it be that I had to set up automatic cleaning of statistics, but didn’t? I can not find how to do it.
Thanks!
Re: Disk full after reboot
I analyzed a similar peak of email activity from 23:00 to 00:00 in the following way:
There is no noticeable unusual activity. Pflogsumm also shows normal values. After reading the forum, I realized that some other people have a similar problem.
So I guess the only problem is deleting old statistics...
Code: Select all
# cat /var/log/maillog* | grep -E "Aug 26 21:..:.*queue active" | wc -l
51
# cat /var/log/maillog* | grep -E "Aug 26 22:..:.*queue active" | wc -l
66
# cat /var/log/maillog* | grep -E "Aug 26 23:..:.*queue active" | wc -l
78
# cat /var/log/maillog* | grep -E "Aug 27 00:..:.*queue active" | wc -l
63
# cat /var/log/maillog* | grep -E "Aug 27 01:..:.*queue active" | wc -l
81
So I guess the only problem is deleting old statistics...