HSM Failure

Discuss your pilot or production implementation with other Zimbra admins or our engineers.
pereljon
Posts: 6
Joined: Fri Sep 12, 2014 10:41 pm

HSM Failure

Postby pereljon » Tue Sep 09, 2008 11:42 am

We just tried to run HSM for the first time, however after getting about 1/2 way done the session failed with the following error.
[QUOTE]com.zimbra.common.service.ServiceException: system failure: Unable to copy MailboxBlob: {mailbox=11, message=43559, path=/Volumes/Storage_Drive/store/0/11/msg/10/43559-50835.msg} to volume 3 ExceptionId:BlobMover:1220947696108:00204c89752c5acb Code:service.FAILURE[/QUOTE]


Has anyone seen this error? Any suggestions?

Picture 2.png



stuartg.orion
Advanced member
Advanced member
Posts: 68
Joined: Fri Sep 12, 2014 10:35 pm

HSM Failure

Postby stuartg.orion » Mon Oct 13, 2008 3:33 pm

We've just had pretty much the same error on our very first HSM session.


com.zimbra.common.service.ServiceException: system failure: Unable to copy MailboxBlob: {mailbox=128, message=8560, path=/opt/zimbra/store/0/128/msg/2/8560-33056.msg} to volume 3 ExceptionId:BlobMover:1223928818440:c04111e2fbe74a37 Code:service.FAILURE
Anyone know what this means? Do we have to run a repair tool on a mmailbox?
mmorse
Ambassador
Ambassador
Posts: 6036
Joined: Fri Sep 12, 2014 10:24 pm

HSM Failure

Postby mmorse » Mon Oct 13, 2008 11:35 pm

Anything interesting about that blob's contents? (I presume it's there/pretty sure we handle missing blob errors much more gracefully.)
His was halfway through, so we can probably rule out write permissions at the HSM volume (I think we throw another error for that anyways/not going off to check at the moment).
mmorse
Ambassador
Ambassador
Posts: 6036
Joined: Fri Sep 12, 2014 10:24 pm

HSM Failure

Postby mmorse » Tue Oct 14, 2008 11:52 am

Could you also post the full stack from the mailbox.log? Thanks.
stuartg.orion
Advanced member
Advanced member
Posts: 68
Joined: Fri Sep 12, 2014 10:35 pm

HSM Failure

Postby stuartg.orion » Tue Oct 14, 2008 2:20 pm

I can't see any thing strange about the BLOB. Is there a tool to get info on a BLOB file?
The stack trace:
2008-10-14 09:13:38,510 INFO [BlobMover] [name=larrys@orionhealth.com;] hsm - Finished moving blobs for all mailboxes.

2008-10-14 09:13:38,510 ERROR [BlobMover] [name=larrys@orionhealth.com;] hsm - An error occurred while running BlobMover

com.zimbra.common.service.ServiceException: system failure: Unable to copy MailboxBlob: {mailbox=128, message=8560, path=/opt/zimbra/store/0/128/msg/2/8560-33056.msg} to volume 3

ExceptionId:BlobMover:1223928818440:c04111e2fbe74a37

Code:service.FAILURE

at com.zimbra.common.service.ServiceException.FAILURE(ServiceException.java:253)

at com.zimbra.cs.hsm.BlobMover.moveBlobs(BlobMover.java:220)

at com.zimbra.cs.hsm.BlobMover.moveBlobs(BlobMover.java:125)

at com.zimbra.cs.hsm.BlobMoverThread.run(BlobMoverThread.java:29)

at java.lang.Thread.run(Thread.java:595)

Caused by: java.io.IOException: link(/opt/zimbra/store_hsm/0/451/msg/0/1497-2891.msg, /opt/zimbra/store_hsm/0/128/msg/2/8560-33056.msg): No such file or directory

at com.zimbra.znative.IO.link0(Native Method)

at com.zimbra.znative.IO.link(IO.java:88)

at com.zimbra.cs.store.FileBlobStore.link(FileBlobStore.java:253)

at com.zimbra.cs.hsm.BlobMover.moveBlobs(BlobMover.java:213)

... 3 more
stuartg.orion
Advanced member
Advanced member
Posts: 68
Joined: Fri Sep 12, 2014 10:35 pm

HSM Failure

Postby stuartg.orion » Tue Oct 14, 2008 2:22 pm

Ok so reading the stacktrace now I see that the blob this blob is a copy of isn't where it should be.
bjared
Advanced member
Advanced member
Posts: 51
Joined: Fri Sep 12, 2014 10:12 pm

HSM Failure

Postby bjared » Wed Nov 26, 2008 12:59 pm

...and the fix for that is... ??? I don't understand what it's doing with the files in the link() statement. In my error, neither parameter of the link() function exist:
ls: /opt/zimbra/hsm01/0/173/msg/0/2396-13956.msg: No such file or directory

ls: /opt/zimbra/hsm01/0/174/msg/0/2523-11289.msg: No such file or directory
And the second one's /opt/zimbra/store/0 equivilant exists, but not the first one. Why would it complain about files not existing on the HSM volume unless it's moving blobs then verifying they've moved. If the first one doesn't exist on the zimbra store, then...where did it go? I don't understand.
Rich Graves
Outstanding Member
Outstanding Member
Posts: 687
Joined: Fri Sep 12, 2014 10:24 pm

HSM Failure

Postby Rich Graves » Fri Aug 07, 2009 12:16 pm

Where I'm running into this, the blobs are shared -- copied to more than one user, and single-instance-stored.
Note the link count of 8, second column of ls -l.
An error occurred: com.zimbra.common.service.ServiceException: system failure: Unable to copy MailboxBlob: {mailbox=1001, message=59060, path=/opt/zimbra/store5/0/1001/msg/14/59060-241000.msg} to volume 7 ExceptionId:BlobMover:1249663883484:fabfd081fa2bfc84 Code:service.FAILURE

Moved 400808 blobs dated earlier than Sun May 24 07:45:55 CDT 2009

to volume 7.
[zimbra@mail2 ~]$ ls -l /opt/zimbra/store5/0/1001/msg/14/59060-241000.msg

-rw-r----- 8 zimbra zimbra 13145 May 23 22:18 /opt/zimbra/store5/0/1001/msg/14/59060-241000.msg
Rich Graves
Outstanding Member
Outstanding Member
Posts: 687
Joined: Fri Sep 12, 2014 10:24 pm

HSM Failure

Postby Rich Graves » Fri Aug 07, 2009 12:58 pm

...of course, among the 400,808 blobs that were moved successfully, surely some had multiple links.
Klug
Elite member
Elite member
Posts: 2378
Joined: Mon Dec 16, 2013 11:35 am
Contact:

HSM Failure

Postby Klug » Mon Sep 21, 2009 8:32 am

It just happened on a customer's server.
The fix (at least the fix that worked for us) is quite simple:
  • look into /opt/zimbra/log/mailbox.log to get the full error message (not just the one zmhsm prints)

  • in the full error message, you can see the account that has the problem (while in the zmhsm error message you only see the mailbox number): in stuartg.orion's post above the account with the problem is "larrys@orionhealth.com")

  • reindex this account (CLI or admin web interface)

  • restart HSM session (CLi or admin web interface)

CLI command to reindex is:
su - zimbra

zmprov rim user@domain.tld start

CLI command to restart HSM session is:
su - zimbra

zmhsm -t

CLI command to display HSM status is:
su - zimbra

zmhsm -u

Return to “Administrators”

Who is online

Users browsing this forum: No registered users and 17 guests