Full Backup with Errors

Discuss your pilot or production implementation with other Zimbra admins or our engineers.
patrickwilson82
Advanced member
Advanced member
Posts: 78
Joined: Tue Mar 13, 2018 5:37 pm

Full Backup with Errors

Postby patrickwilson82 » Thu Sep 27, 2018 12:45 pm

Hello, I'm running 8.7. Every Friday night, my Zimbra server runs a full backup, and usually returns 2-4 accounts with errors. It's almost never the same accounts every time. The errors I get usually have the following message attached:

Cannot remove /mnt/nfs/ZCS_Backup/tmp/full-20180915.050004.691/accounts/8a7/125/8a712571-7ef0-4acd-b2eb-8c6866885009/blobs
java.io.IOException: Cannot remove /mnt/nfs/ZCS_Backup/tmp/full-20180915.050004.691/accounts/8a7/125/8a712571-7ef0-4acd-b2eb-8c6866885009/blobs
at com.zimbra.common.util.FileUtil.deleteDir(FileUtil.java:432)
at com.zimbra.common.util.FileUtil.deleteDirContents(FileUtil.java:448)
at com.zimbra.common.util.FileUtil.deleteDir(FileUtil.java:430)
at com.zimbra.cs.backup.FileBackupTarget$BackupAcctSession.endFullBackup(FileBackupTarget.java:1961)
at com.zimbra.cs.backup.BackupSet.startFullBackup(BackupSet.java:397)
at com.zimbra.cs.backup.FileBackupTarget$FileBackupSet.startFullBackup(FileBackupTarget.java:978)
at com.zimbra.cs.backup.BackupManager.doBackupFull(BackupManager.java:848)
at com.zimbra.cs.backup.BackupManager$FullBackupThread.run(BackupManager.java:236

I found this link, but I'm not sure that it applies to my situation: https://wiki.zimbra.com/wiki/How_to_fix ... ckup_error.

Can anyone provide some insight? Thanks.


phoenix
Ambassador
Ambassador
Posts: 26445
Joined: Fri Sep 12, 2014 9:56 pm
Location: Liverpool, England

Re: Full Backup with Errors

Postby phoenix » Thu Sep 27, 2018 1:27 pm

For how long has this been happening? Did you read the wiki article about not using NFS on this page: https://wiki.zimbra.com/wiki/Performance_Tuning_Guidelines_for_Large_Deployments#Disk - I'd guess you're hitting a problem with that. You also haven't mentioned what type of storage you're using for NFS, I assume you've tested it's performance?
Regards

Bill

Rspamd: A high performance spamassassin replacement

If you'd like to see this implemented in a future version of ZCS then please vote on Bugzilla entries 97706 & 108168
patrickwilson82
Advanced member
Advanced member
Posts: 78
Joined: Tue Mar 13, 2018 5:37 pm

Re: Full Backup with Errors

Postby patrickwilson82 » Thu Sep 27, 2018 2:00 pm

This was set up with NFS by the previous person who was in my position, and we've been getting these errors since before I started here seven months ago. Our backup server uses SATA drives. I'm not aware of any issues with the drives on our backup server. It is also used for Veeam B&R which is not experiencing any problems.
User avatar
pup_seba
Outstanding Member
Outstanding Member
Posts: 687
Joined: Sat Sep 13, 2014 2:43 am
Location: Tarragona - Spain
Contact:

Re: Full Backup with Errors

Postby pup_seba » Thu Sep 27, 2018 2:09 pm

What uid and gid do your zimbra user and group have? Do those blobs have same uid and gid?

As you say that this happens with different users, I'm guessing that uid and gid should be ok, but doublechecking won't hurt.

Can you manually create and remove files with the zimbra user, under /mnt/nfs/ZCS_Backup/tmp/full-20180915.050004.691/accounts/8a7/125/8a712571-7ef0-4acd-b2eb-8c6866885009/

Which options is your exports using?
patrickwilson82
Advanced member
Advanced member
Posts: 78
Joined: Tue Mar 13, 2018 5:37 pm

Re: Full Backup with Errors

Postby patrickwilson82 » Thu Sep 27, 2018 2:47 pm

The UID and GID for the Zimbra user are 999. I'm not that familiar with BLOBs. So I'm not sure what I'm looking for there.

I'm not sure if this has anything to do with the issue or not, but my backups are not going into ZCS_Backups/tmp. They are going into ZCS_Backups/sessions.
User avatar
pup_seba
Outstanding Member
Outstanding Member
Posts: 687
Joined: Sat Sep 13, 2014 2:43 am
Location: Tarragona - Spain
Contact:

Re: Full Backup with Errors

Postby pup_seba » Thu Sep 27, 2018 2:52 pm

Well, the error clearly states this "Cannot remove /mnt/nfs/ZCS_Backup/tmp/full-20180915.050004.691/accounts/8a7/125/8a712571-7ef0-4acd-b2eb-8c6866885009/blobs"

So wherever your backups are going, the task is trying to remove something under the path I tell you to check.

That's why I'm telling you to check if you can create and remove a file there. Use a "touch" or whatever, you don't need to be familiar with blobs...is just the path where the permission error is happening. There you should use the information gathered about the uid and gid to see who is the owner of the files in that path.
patrickwilson82
Advanced member
Advanced member
Posts: 78
Joined: Tue Mar 13, 2018 5:37 pm

Re: Full Backup with Errors

Postby patrickwilson82 » Thu Sep 27, 2018 5:11 pm

The backups don't appear to be going into the /mnt/nfs/ZCS_Backup directory. At least thy're not there currently. I'm wondering if I need to watch this directory when the backups run. There appears to be a process that ius set to run to clean this tmp directory. I wonder if this process is not finding the file and that's what causes the error?

source /opt/zimbra/bin/zmshutil || exit 1
zmsetvars

find ${zimbra_tmp_directory} -maxdepth 1 -type f -mtime +7 -exec rm -f {} \; > /dev/null 2>&1
patrickwilson82
Advanced member
Advanced member
Posts: 78
Joined: Tue Mar 13, 2018 5:37 pm

Re: Full Backup with Errors

Postby patrickwilson82 » Fri Sep 28, 2018 1:27 pm

pup_seba wrote:Well, the error clearly states this "Cannot remove /mnt/nfs/ZCS_Backup/tmp/full-20180915.050004.691/accounts/8a7/125/8a712571-7ef0-4acd-b2eb-8c6866885009/blobs"

So wherever your backups are going, the task is trying to remove something under the path I tell you to check.

That's why I'm telling you to check if you can create and remove a file there. Use a "touch" or whatever, you don't need to be familiar with blobs...is just the path where the permission error is happening. There you should use the information gathered about the uid and gid to see who is the owner of the files in that path.


And I am able to create and delete a file there, so I don't believe it's a permissions issue, from what I can tell.

Return to “Administrators”

Who is online

Users browsing this forum: MSN [Bot] and 8 guests