Disk full after reboot

Discuss your pilot or production implementation with other Zimbra admins or our engineers.
Post Reply
User avatar
SEA80
Posts: 12
Joined: Fri Mar 09, 2018 2:59 pm

Disk full after reboot

Post by SEA80 »

Hello.
My Zimbra 8.8.10 FOSS server with all roles was up 170 days. I've got a task to optimise zimbra server's resources.
I reduced mysql and java memory usage both to ~1GB and restarted zimbra services. Nothing goes wrong.
Later I stopped zimbra's vm, removed 2 cpu cores and started the vm.
A few minutes later I received an alert that free space is critical. To avoid data loss I stopped zimbra services and free space gradually returned back to normal. At some point I noticed that the size of the /opt/zimbra/zmstat folder was about 24GB, now it's 1.1GB. Seems like the free space has decreased to zero during services shutdown.

Free space, 200GB total:
free space.png
free space.png (38.29 KiB) Viewed 3271 times
After the second reboot there was no reduction of free space.

I have some questions for the community:
1. What happened after the 1st reboot? Is this normal or not?
2. Is there a risk of data loss when the free space went down to zero?
3. Is there a risk of data loss after I stopped zimbra's services during strange process ate free space?

Best regards!

UPD:
I don't understand what these peaks means, especially the last one:
Screen Shot 2019-07-07 at 16.16.16.png
Screen Shot 2019-07-07 at 16.16.16.png (26.84 KiB) Viewed 3255 times
User avatar
SEA80
Posts: 12
Joined: Fri Mar 09, 2018 2:59 pm

Re: Disk full after reboot

Post by SEA80 »

I repeated this issue in test environment.
After starting the server, the following processes start writing to disk:
gzip csv.png
gzip csv.png (9.56 KiB) Viewed 3198 times
gzip csv start.png
gzip csv start.png (44.36 KiB) Viewed 3198 times
They run till the disk is full:
gzip csv end.png
gzip csv end.png (45.09 KiB) Viewed 3198 times
gzip csv end df.png
gzip csv end df.png (16.85 KiB) Viewed 3198 times
Approx. after 15 minutes, the csv and gzip file sizes return to normal. It looks like deleting old statistical data. If so, this is a very strange implementation, especially if Zimbra server uses a dynamic virtual disk. It turns out I need to restart services periodically to avoid this issue.
phoenix
Ambassador
Ambassador
Posts: 27272
Joined: Fri Sep 12, 2014 9:56 pm
Location: Liverpool, England

Re: Disk full after reboot

Post by phoenix »

A few simple questions. Why have you reduced the mysql & java memory to 1GB and why have you removed two vCPUs? What is the current configuration of your server, RAM, CPUs etc., etc.? What happens if you revert to your original VM specs? What, if anything, is in the logs that might point to a problem? The two peaks in your graph show a high message count, does that really indicate the level of mail arriving at your server? Around the time of those peaks can you determine from the log files where the emails are coming from?
Regards

Bill

Rspamd: A high performance spamassassin replacement

Per ardua ad astra
User avatar
SEA80
Posts: 12
Joined: Fri Mar 09, 2018 2:59 pm

Re: Disk full after reboot

Post by SEA80 »

Hello and sorry for this late reply.
phoenix wrote: A few simple questions. Why have you reduced the mysql & java memory to 1GB and why have you removed two vCPUs? What is the current configuration of your server, RAM, CPUs etc., etc.? What happens if you revert to your original VM specs?
It was a task from my boss. He asked me to reduce resources for this VM because we have only 70 users. Maybe it was not a good idea.
But this is not important for this issue, because my test environment VM have the same problem without mysql, java or VM specs changes. Looks like the only reason is reboot after the long uptime and shrinking a big amount of statistical data. The next reboot will not cause this problem.
phoenix wrote: What, if anything, is in the logs that might point to a problem? The two peaks in your graph show a high message count, does that really indicate the level of mail arriving at your server? Around the time of those peaks can you determine from the log files where the emails are coming from?
I did not find anything interesting in the system logs, maybe I can’t search well. But I noticed that inexplicable peaks in the number of messages still occur. Please give me some time for the investigation.


BTW I do not see screenshots in this thread. Here are the new ones:

Strange peaks in the current production environment:
Screen Shot 2019-08-22 at 15.01.37.png
Screen Shot 2019-08-22 at 15.01.37.png (17.9 KiB) Viewed 2627 times
The reason for lack of space: unzipping (statistical?) data into large csv files, shrinking and zipping them again
Screen Shot 2019-08-22 at 15.06.56.png
Screen Shot 2019-08-22 at 15.06.56.png (36.2 KiB) Viewed 2627 times
Could it be that I had to set up automatic cleaning of statistics, but didn’t? I can not find how to do it.

Thanks!
User avatar
SEA80
Posts: 12
Joined: Fri Mar 09, 2018 2:59 pm

Re: Disk full after reboot

Post by SEA80 »

I analyzed a similar peak of email activity from 23:00 to 00:00 in the following way:

Code: Select all

# cat /var/log/maillog* | grep -E "Aug 26 21:..:.*queue active" | wc -l
51
# cat /var/log/maillog* | grep -E "Aug 26 22:..:.*queue active" | wc -l
66
# cat /var/log/maillog* | grep -E "Aug 26 23:..:.*queue active" | wc -l
78
# cat /var/log/maillog* | grep -E "Aug 27 00:..:.*queue active" | wc -l
63
# cat /var/log/maillog* | grep -E "Aug 27 01:..:.*queue active" | wc -l
81
There is no noticeable unusual activity. Pflogsumm also shows normal values. After reading the forum, I realized that some other people have a similar problem.

So I guess the only problem is deleting old statistics...
Post Reply