a user reported of "missing blob for item" when trying to view emails from 2012
ZCS version
$ zmcontrol -v
Release 8.0.6_GA_5922.RHEL6_64_20131203103753 RHEL6_64 NETWORK edition.
volumes info
$ zmvolume -l
Volume id: 1
name: message1
type: primaryMessage
path: /opt/zimbra/store
compressed: false
current: true
Volume id: 2
name: index1
type: index
path: /opt/zimbra/index
compressed: false
current: true
Volume id: 3
name: hsm-volume
type: secondaryMessage
path: /mnt/hsm
compressed: false
current: false
Volume id: 4
name: hsm-volume2
type: secondaryMessage
path: /mnt/zarchive/hsm
compressed: false
current: true
below is output when I tried running zmblobchk
$ zmblobchk start | grep 'item 2796'
Mailbox 84, item 2796, rev 4835, volume 3, /mnt/zarchive/hsm/0/84/msg/0/2796-4835.msg: file has incorrect revision.
Mailbox 358, item 2796, rev 4703, volume 3, /mnt/zarchive/hsm/0/358/msg/0/2796-4703.msg: file has incorrect revision.
Mailbox 626, item 2796, rev 5151, volume 3, /mnt/zarchive/hsm/0/626/msg/0/2796-5151.msg: file has incorrect revision.
Mailbox 646, item 2796, rev 5274, volume 3, /mnt/zarchive/hsm/0/646/msg/0/2796-5274.msg: file has incorrect revision.
Mailbox 941, item 2796, rev 5022, volume 3, /mnt/zarchive/hsm/0/941/msg/0/2796-5022.msg: file has incorrect revision.
Mailbox 1058, item 2796, rev 5187, volume 3, /mnt/zarchive/hsm/0/1058/msg/0/2796-5187.msg: file has incorrect revision.
Mailbox 1382, item 2796, rev 5212, volume 3, /mnt/zarchive/hsm/0/1382/msg/0/2796-5212.msg: file has incorrect revision.
Mailbox 1487, item 2796, rev 5171, volume 3, /mnt/zarchive/hsm/0/1487/msg/0/2796-5171.msg: file has incorrect revision.
Mailbox 1545, item 2796, rev 5198, volume 3, /mnt/zarchive/hsm/0/1545/msg/0/2796-5198.msg: file has incorrect revision.
Mailbox 1595, item 2796, rev 4748, volume 3, /mnt/zarchive/hsm/0/1595/msg/0/2796-4748.msg: file has incorrect revision.
Mailbox 1638, item 2796, rev 5205, volume 3, /mnt/zarchive/hsm/0/1638/msg/0/2796-5205.msg: file has incorrect revision.
Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.util.Vector.(Vector.java:131)
at org.apache.xerces.dom.AttributeMap.setNamedItem(Unknown Source)
at org.apache.xerces.dom.DeferredElementNSImpl.synchronizeData(Unknown Source)
at org.apache.xerces.dom.ElementNSImpl.getNamespaceURI(Unknown Source)
at com.zimbra.common.soap.W3cDomUtil.needToFlattenElementContent(W3cDomUtil.java:234)
at com.zimbra.common.soap.W3cDomUtil.nodeToElement(W3cDomUtil.java:282)
at com.zimbra.common.soap.W3cDomUtil.toHierarchy(W3cDomUtil.java:328)
at com.zimbra.common.soap.W3cDomUtil.nodeToElement(W3cDomUtil.java:285)
at com.zimbra.common.soap.W3cDomUtil.toHierarchy(W3cDomUtil.java:328)
at com.zimbra.common.soap.W3cDomUtil.nodeToElement(W3cDomUtil.java:285)
at com.zimbra.common.soap.W3cDomUtil.toHierarchy(W3cDomUtil.java:328)
at com.zimbra.common.soap.W3cDomUtil.nodeToElement(W3cDomUtil.java:285)
at com.zimbra.common.soap.W3cDomUtil.toHierarchy(W3cDomUtil.java:328)
at com.zimbra.common.soap.W3cDomUtil.nodeToElement(W3cDomUtil.java:285)
at com.zimbra.common.soap.W3cDomUtil.toHierarchy(W3cDomUtil.java:328)
at com.zimbra.common.soap.W3cDomUtil.nodeToElement(W3cDomUtil.java:285)
at com.zimbra.common.soap.W3cDomUtil.nodeToElement(W3cDomUtil.java:280)
at com.zimbra.common.soap.W3cDomUtil.parseXML(W3cDomUtil.java:219)
at com.zimbra.common.soap.Element.parseXML(Element.java:629)
at com.zimbra.common.soap.SoapTransport.parseSoapResponse(SoapTransport.java:299)
at com.zimbra.common.soap.SoapHttpTransport.invoke(SoapHttpTransport.java:255)
at com.zimbra.common.soap.SoapHttpTransport.invoke(SoapHttpTransport.java:167)
at com.zimbra.common.soap.SoapTransport.invoke(SoapTransport.java:386)
at com.zimbra.common.soap.SoapTransport.invokeWithoutSession(SoapTransport.java:372)
at com.zimbra.cs.account.soap.SoapProvisioning.invokeRequest(SoapProvisioning.java:442)
at com.zimbra.cs.account.soap.SoapProvisioning.invoke(SoapProvisioning.java:450)
at com.zimbra.cs.store.file.BlobConsistencyUtil.checkMailbox(BlobConsistencyUtil.java:244)
at com.zimbra.cs.store.file.BlobConsistencyUtil.run(BlobConsistencyUtil.java:201)
at com.zimbra.cs.store.file.BlobConsistencyUtil.main(BlobConsistencyUtil.java:328)
I also tried to symlink to old /mnt/hsm (former mountpoint of hsm lun) but still same
$ ls -l /mnt/hsm/
total 0
lrwxrwxrwx 1 root root 18 Apr 30 13:02 hsm -> /mnt/zarchive/hsm/
missing blob for item
missing blob for item
Tried copying individual message with same message number as missing blob error for a user - still have missing blob message
I also tried copying synching from hsm to store - still have missing blob message
I also tried copying synching from hsm to store - still have missing blob message
missing blob for item
if anybody is interested, this is how to fix it per mailbox basis
# su - zimbra
Find mailbox_id
$ zmprov gmi user1@domain.com
mailboxId: 692
quotaUsed: 1028382663
Find mboxgroup
$ expr 692 % 100
92
$ zmblobchk -m 692 start > /tmp/blob_user1
$ cat /tmp/blob_user1 | grep -i hsm | grep "incorrect revision" | awk -F '{print $4}' | sed -e 's/,//g' > /tmp/bloblist_user1
In this case I moved from secondary volume 3 to 4. Now to fix it in database:
$ for id in `cat /tmp/bloblist_user1 `; do mysql mboxgroup92 -e "update mail_item set locator=4 where locator=3 and mailbox_id=692 and id=$id" ; done
# su - zimbra
Find mailbox_id
$ zmprov gmi user1@domain.com
mailboxId: 692
quotaUsed: 1028382663
Find mboxgroup
$ expr 692 % 100
92
$ zmblobchk -m 692 start > /tmp/blob_user1
$ cat /tmp/blob_user1 | grep -i hsm | grep "incorrect revision" | awk -F '{print $4}' | sed -e 's/,//g' > /tmp/bloblist_user1
In this case I moved from secondary volume 3 to 4. Now to fix it in database:
$ for id in `cat /tmp/bloblist_user1 `; do mysql mboxgroup92 -e "update mail_item set locator=4 where locator=3 and mailbox_id=692 and id=$id" ; done