[Solved] Zimbra 9 FOSS (Zxtras) e-mail corruption

[jasggomes]

Hi to All,

System::

Code: Select all

OS: Ubuntu 18.04, upgraded from 16.04
RAM: 8GB
CPU: 4 vcpus
Zimbra version: 9.0.0_GA_3969.FOSS

Running on am Hyper-V 2012 VM

Problem::
Well I'm being flooded wit help request from my users regarding several isssues with Outlook(v2013) IMAP Sync:
- Some emails are not being synched;

And now a new issue, when using the WebUI (old), on some emails we are getting an error like this:

Code: Select all

"system failure: MessagingException while creating MimeMessage for item xxx "

This message prevents seeing the related email message on the WebUI, and this seems to be related to a period of time, between the 19th of Oct and the 20th.
Before and after that period we can see the email messages.

Can someone shine some light how to troubleshoot this type of issues?

Regards.
JG

[jasggomes]

Further analysis shows that may be related to a RAM issue yesterday on our Hyper-V server.
VM was running on Dynamic RAM and for a short period of time, there was an overbooking of RAM on the server.
Email messages seem being corrupted or something like that.

I've done a backup of my own mailbox (zmmailbox), that has those issues, deleted it, and recreated it.
This is the error that is thrown during the zmailbox restore process::

Code: Select all

ERROR: zclient.IO_ERROR (Read timed out) (cause: java.net.SocketTimeoutException Read timed out)
com.zimbra.common.zclient.ZClientException: Read timed out
ExceptionId:main:1603276990507:0c8c2a9fefc3a68d
Code:zclient.IO_ERROR
        at com.zimbra.common.zclient.ZClientException.IO_ERROR(ZClientException.java:49)
        at com.zimbra.client.ZMailbox.postRESTResource(ZMailbox.java:3466)
        at com.zimbra.cs.zclient.ZMailboxUtil.doPostRestURL(ZMailboxUtil.java:3036)
        at com.zimbra.cs.zclient.ZMailboxUtil.execute(ZMailboxUtil.java:1313)
        at com.zimbra.cs.zclient.ZMailboxUtil.main(ZMailboxUtil.java:2917)
Caused by: java.net.SocketTimeoutException: Read timed out
        at java.base/sun.nio.ch.NioSocketImpl.timedRead(NioSocketImpl.java:284)
        at java.base/sun.nio.ch.NioSocketImpl.implRead(NioSocketImpl.java:310)
        at java.base/sun.nio.ch.NioSocketImpl.read(NioSocketImpl.java:351)
        at java.base/sun.nio.ch.NioSocketImpl$1.read(NioSocketImpl.java:802)
        at java.base/java.net.Socket$SocketInputStream.read(Socket.java:937)
        at java.base/sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:450)
        at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(SSLSocketInputRecord.java:68)
        at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(SSLSocketImpl.java:1409)
        at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:1022)
        at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
        at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
        at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280)
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
        at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
        at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
        at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157)
        at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
        at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
        at com.zimbra.common.httpclient.HttpClientUtil.executeMethod(HttpClientUtil.java:73)
        at com.zimbra.common.httpclient.HttpClientUtil.executeMethod(HttpClientUtil.java:50)
        at com.zimbra.client.ZMailbox.postRESTResource(ZMailbox.java:3458)
        ... 3 more

Is there a way of recovering from such an issue ??

Regards.
JG

[jasggomes]

Well, thanks for All the replies and suggestions.

The problem has been solved.

Regards and thanks.
JG

[phoenix]

Well, thanks for All the replies and suggestions.

That comment is a bit unnecessary, you're making an assumption that people have seen and read your post. We all post here when we have the time and/or an answer to a problem so maybe you were just unlucky.

The problem has been solved.

For others that may follow, care to share your solution?

[jasggomes]

Well, thanks for All the replies and suggestions.

That comment is a bit unnecessary, you're making an assumption that people have seen and read your post. We all post here when we have the time and/or an answer to a problem so maybe you were just unlucky.

The problem has been solved.

For others that may follow, care to share your solution?

First of all, you are right, it was unnecessary, but it was an urgent issue, and since this is a FOSS setup I couldn't ask for help from Zimbra, and during the 4 days I took to solve it no one replied, but, as soon I post my reply you replied...
But I know, you are only one, and for the few times I came here now, I only see 'you' replying to the requests, or not only, but I see a lot of them from you, and you can't attend us all, but I do know you do your best. For that I thank you.

So here it is my apologies for the post itself and a big thank you all for your work on helping us all.

The solution wasn't a Zimbra issue, it was a disk issue.
Since the emails are stored on the disk, and some of the info, like 'headers', are stored on the BD, when the users access the web GUI they get the error message when hoovering the header of the message, because the system cannot read the message itself from the bad disk sectors where they were stored.
It took such a long time since all backups were corrupted too, and I had to be careful not to lose the whole disk, and only after a full correct backup, I touched the system.
Also, the corrupted messages didn't allow the Outlook IMAP access to synchronize, confusing all, so the answer was to move all that troubling messages to a new folder, to have a visual idea of which they are, and then resynch the Outlook folders. I managed to have an archive email system, thanks to @JorgeDeLaCruz, in 2018, and from there I could resend all messed up emails back to the users. The forwarding of the messages in the Zimbra was not affected, only the ones that were stored on the disk.

I had to copy all its contents to another disk, do file size comparisons, etc etc... lost some of the messages, but those were pretty much recovered by using the archive solution.
In the end, it was a fun week.

Regards.
JG