HSM Failure

pereljon · Post by **pereljon** » Tue Sep 09, 2008 11:42 am

We just tried to run HSM for the first time, however after getting about 1/2 way done the session failed with the following error.
[QUOTE]com.zimbra.common.service.ServiceException: system failure: Unable to copy MailboxBlob: {mailbox=11, message=43559, path=/Volumes/Storage_Drive/store/0/11/msg/10/43559-50835.msg} to volume 3 ExceptionId:BlobMover:1220947696108:00204c89752c5acb Code:service.FAILURE[/QUOTE]

Has anyone seen this error? Any suggestions?
Picture 2.png

stuartg.orion · Post by **stuartg.orion** » Mon Oct 13, 2008 3:33 pm

We've just had pretty much the same error on our very first HSM session.

com.zimbra.common.service.ServiceException: system failure: Unable to copy MailboxBlob: {mailbox=128, message=8560, path=/opt/zimbra/store/0/128/msg/2/8560-33056.msg} to volume 3 ExceptionId:BlobMover:1223928818440:c04111e2fbe74a37 Code:service.FAILURE
Anyone know what this means? Do we have to run a repair tool on a mmailbox?

mmorse · Post by **mmorse** » Mon Oct 13, 2008 11:35 pm

Anything interesting about that blob's contents? (I presume it's there/pretty sure we handle missing blob errors much more gracefully.)
His was halfway through, so we can probably rule out write permissions at the HSM volume (I think we throw another error for that anyways/not going off to check at the moment).

mmorse · Post by **mmorse** » Tue Oct 14, 2008 11:52 am

Could you also post the full stack from the mailbox.log? Thanks.

stuartg.orion · Post by **stuartg.orion** » Tue Oct 14, 2008 2:20 pm

I can't see any thing strange about the BLOB. Is there a tool to get info on a BLOB file?
The stack trace:
2008-10-14 09:13:38,510 INFO [BlobMover] [name=larrys@orionhealth.com;] hsm - Finished moving blobs for all mailboxes.

2008-10-14 09:13:38,510 ERROR [BlobMover] [name=larrys@orionhealth.com;] hsm - An error occurred while running BlobMover

com.zimbra.common.service.ServiceException: system failure: Unable to copy MailboxBlob: {mailbox=128, message=8560, path=/opt/zimbra/store/0/128/msg/2/8560-33056.msg} to volume 3

ExceptionId:BlobMover:1223928818440:c04111e2fbe74a37

Code:service.FAILURE

at com.zimbra.common.service.ServiceException.FAILURE(ServiceException.java:253)

at com.zimbra.cs.hsm.BlobMover.moveBlobs(BlobMover.java:220)

at com.zimbra.cs.hsm.BlobMover.moveBlobs(BlobMover.java:125)

at com.zimbra.cs.hsm.BlobMoverThread.run(BlobMoverThread.java:29)

at java.lang.Thread.run(Thread.java:595)

Caused by: java.io.IOException: link(/opt/zimbra/store_hsm/0/451/msg/0/1497-2891.msg, /opt/zimbra/store_hsm/0/128/msg/2/8560-33056.msg): No such file or directory

at com.zimbra.znative.IO.link0(Native Method)

at com.zimbra.znative.IO.link(IO.java:88)

at com.zimbra.cs.store.FileBlobStore.link(FileBlobStore.java:253)

at com.zimbra.cs.hsm.BlobMover.moveBlobs(BlobMover.java:213)

... 3 more

stuartg.orion · Post by **stuartg.orion** » Tue Oct 14, 2008 2:22 pm

Ok so reading the stacktrace now I see that the blob this blob is a copy of isn't where it should be.

bjared · Post by **bjared** » Wed Nov 26, 2008 12:59 pm

...and the fix for that is... ??? I don't understand what it's doing with the files in the link() statement. In my error, neither parameter of the link() function exist:
ls: /opt/zimbra/hsm01/0/173/msg/0/2396-13956.msg: No such file or directory

ls: /opt/zimbra/hsm01/0/174/msg/0/2523-11289.msg: No such file or directory
And the second one's /opt/zimbra/store/0 equivilant exists, but not the first one. Why would it complain about files not existing on the HSM volume unless it's moving blobs then verifying they've moved. If the first one doesn't exist on the zimbra store, then...where did it go? I don't understand.

Rich Graves · Post by **Rich Graves** » Fri Aug 07, 2009 12:16 pm

Where I'm running into this, the blobs are shared -- copied to more than one user, and single-instance-stored.
Note the link count of 8, second column of ls -l.
An error occurred: com.zimbra.common.service.ServiceException: system failure: Unable to copy MailboxBlob: {mailbox=1001, message=59060, path=/opt/zimbra/store5/0/1001/msg/14/59060-241000.msg} to volume 7 ExceptionId:BlobMover:1249663883484:fabfd081fa2bfc84 Code:service.FAILURE

Moved 400808 blobs dated earlier than Sun May 24 07:45:55 CDT 2009

to volume 7.
[zimbra@mail2 ~]$ ls -l /opt/zimbra/store5/0/1001/msg/14/59060-241000.msg

-rw-r----- 8 zimbra zimbra 13145 May 23 22:18 /opt/zimbra/store5/0/1001/msg/14/59060-241000.msg

Rich Graves · Post by **Rich Graves** » Fri Aug 07, 2009 12:58 pm

...of course, among the 400,808 blobs that were moved successfully, surely some had multiple links.

Klug · Post by **Klug** » Mon Sep 21, 2009 8:32 am

It just happened on a customer's server.
The fix (at least the fix that worked for us) is quite simple:look into /opt/zimbra/log/mailbox.log to get the full error message (not just the one zmhsm prints)

in the full error message, you can see the account that has the problem (while in the zmhsm error message you only see the mailbox number): in stuartg.orion's post above the account with the problem is "larrys@orionhealth.com")

reindex this account (CLI or admin web interface)

restart HSM session (CLi or admin web interface)
CLI command to reindex is:su - zimbra

zmprov rim user@domain.tld start
CLI command to restart HSM session is:su - zimbra

zmhsm -t
CLI command to display HSM status is:su - zimbra

zmhsm -u