Full Backup with Errors

[patrickwilson82]

Hello, I'm running 8.7. Every Friday night, my Zimbra server runs a full backup, and usually returns 2-4 accounts with errors. It's almost never the same accounts every time. The errors I get usually have the following message attached:

Cannot remove /mnt/nfs/ZCS_Backup/tmp/full-20180915.050004.691/accounts/8a7/125/8a712571-7ef0-4acd-b2eb-8c6866885009/blobs
java.io.IOException: Cannot remove /mnt/nfs/ZCS_Backup/tmp/full-20180915.050004.691/accounts/8a7/125/8a712571-7ef0-4acd-b2eb-8c6866885009/blobs
at com.zimbra.common.util.FileUtil.deleteDir(FileUtil.java:432)
at com.zimbra.common.util.FileUtil.deleteDirContents(FileUtil.java:448)
at com.zimbra.common.util.FileUtil.deleteDir(FileUtil.java:430)
at com.zimbra.cs.backup.FileBackupTarget$BackupAcctSession.endFullBackup(FileBackupTarget.java:1961)
at com.zimbra.cs.backup.BackupSet.startFullBackup(BackupSet.java:397)
at com.zimbra.cs.backup.FileBackupTarget$FileBackupSet.startFullBackup(FileBackupTarget.java:978)
at com.zimbra.cs.backup.BackupManager.doBackupFull(BackupManager.java:848)
at com.zimbra.cs.backup.BackupManager$FullBackupThread.run(BackupManager.java:236

I found this link, but I'm not sure that it applies to my situation: https://wiki.zimbra.com/wiki/How_to_fix ... ckup_error.

Can anyone provide some insight? Thanks.

[phoenix]

For how long has this been happening? Did you read the wiki article about not using NFS on this page: https://wiki.zimbra.com/wiki/Performanc ... ments#Disk - I'd guess you're hitting a problem with that. You also haven't mentioned what type of storage you're using for NFS, I assume you've tested it's performance?

[patrickwilson82]

This was set up with NFS by the previous person who was in my position, and we've been getting these errors since before I started here seven months ago. Our backup server uses SATA drives. I'm not aware of any issues with the drives on our backup server. It is also used for Veeam B&R which is not experiencing any problems.

[pup_seba]

What uid and gid do your zimbra user and group have? Do those blobs have same uid and gid?

As you say that this happens with different users, I'm guessing that uid and gid should be ok, but doublechecking won't hurt.

Can you manually create and remove files with the zimbra user, under /mnt/nfs/ZCS_Backup/tmp/full-20180915.050004.691/accounts/8a7/125/8a712571-7ef0-4acd-b2eb-8c6866885009/

Which options is your exports using?

[patrickwilson82]

The UID and GID for the Zimbra user are 999. I'm not that familiar with BLOBs. So I'm not sure what I'm looking for there.

I'm not sure if this has anything to do with the issue or not, but my backups are not going into ZCS_Backups/tmp. They are going into ZCS_Backups/sessions.

[pup_seba]

Well, the error clearly states this "Cannot remove /mnt/nfs/ZCS_Backup/tmp/full-20180915.050004.691/accounts/8a7/125/8a712571-7ef0-4acd-b2eb-8c6866885009/blobs"

So wherever your backups are going, the task is trying to remove something under the path I tell you to check.

That's why I'm telling you to check if you can create and remove a file there. Use a "touch" or whatever, you don't need to be familiar with blobs...is just the path where the permission error is happening. There you should use the information gathered about the uid and gid to see who is the owner of the files in that path.

[patrickwilson82]

The backups don't appear to be going into the /mnt/nfs/ZCS_Backup directory. At least thy're not there currently. I'm wondering if I need to watch this directory when the backups run. There appears to be a process that ius set to run to clean this tmp directory. I wonder if this process is not finding the file and that's what causes the error?

source /opt/zimbra/bin/zmshutil || exit 1
zmsetvars

find ${zimbra_tmp_directory} -maxdepth 1 -type f -mtime +7 -exec rm -f {} \; > /dev/null 2>&1

[patrickwilson82]

Well, the error clearly states this "Cannot remove /mnt/nfs/ZCS_Backup/tmp/full-20180915.050004.691/accounts/8a7/125/8a712571-7ef0-4acd-b2eb-8c6866885009/blobs"

So wherever your backups are going, the task is trying to remove something under the path I tell you to check.

That's why I'm telling you to check if you can create and remove a file there. Use a "touch" or whatever, you don't need to be familiar with blobs...is just the path where the permission error is happening. There you should use the information gathered about the uid and gid to see who is the owner of the files in that path.

And I am able to create and delete a file there, so I don't believe it's a permissions issue, from what I can tell.