Page 1 of 2

New java exceptions in logs after upgrade 8.8.9_P2 to 8.8.9_P3

Posted: Fri Aug 24, 2018 2:17 pm
by andrey.ivanov
We have two new types of exceptions in mailboxd.log after upgrade 8.8.9_P2 -> 8.8.9_P3 (NE, CentOS7). One of them is described in the forum thread viewtopic.php?f=15&t=64708 and concerns exceptions during SearchRequests. To find them search the file mailboxd.log for the keyword isSortByReadFlag. We have about 1000 of these per day.

The second new type of exceptions concerns the index code, search for the keyword ZimbraHit to find them:

Code: Select all

grep 'index - Caught ServiceException trying to compare ZimbraHit' /opt/zimbra/log/mailbox.log
We have about 150000 of those per working day. They look like this :

Code: Select all

2018-08-24 10:01:33,636 INFO  [qtp1286783232-154439:https:https://webmail.server.com/service/soap/SearchRequest] [name=some.user@my.zimbra.server;mid=10891;oip=123.45.67.89;port=56774;ua=ZimbraWebClient - FF61 (Win)/8.8.9_GA_3019;soapId=1aa2d45c;] index - Caught ServiceException trying to compare ZimbraHit {"s":2956,"d":1528962759000,"l":"c30c521b-6e3b-479a-8d07-cfedd9ab7198:2","cid":"c30c521b-6e3b-479a-8d07-cfedd9ab7198:-12504","f":"r","rev":55606,"id":"c30c521b-6e3b-479a-8d07-cfedd9ab7198:12504","su":"Date de master M4S","fr":"Ceci est un mail automatique, veuillez ne ...","cm":true,"e":[{"a":"internal_mail@our.domain","d":"toto","p":"toto via toto","t":"f"}],"sf":"0"} to ZimbraHit {"s":9100,"d":1529062749000,"l":"c30c521b-6e3b-479a-8d07-cfedd9ab7198:2","cid":"c30c521b-6e3b-479a-8d07-cfedd9ab7198:12521","f":"r","rev":55706,"id":"c30c521b-6e3b-479a-8d07-cfedd9ab7198:12520","su":"Inquiry about MSc. program","fr":"Dear Sirs: I hope this email finds you well. I am sending this to kindly ask about the masters program course language. I come from an English ...","cm":true,"e":[{"a":"some-email@somewhere.com","d":"John","p":"Jane Doe","t":"f"}],"sf":"0"}
It has appeared because of the changes in the file ./zm-mailbox/store/src/java/com/zimbra/cs/index/ZimbraHit.java at line 340 (diff from 8.8.9_P2 -> 8.8.9_P3) adding the new function protected static final int compareByReadFlag(boolean ascending, ZimbraHit lhs, ZimbraHit rhs). Part of the code generating this exception:

Code: Select all

+        try {
+            long left = getReadStatus(lhs);
+            long right = getReadStatus(rhs);
+            long result = right - left;
+            if (result > 0)
+                retVal = 1;
+            else if (result < 0)
+                retVal = -1;
+            else
+                retVal = 0;
+        } catch (ServiceException e) {
+            ZimbraLog.index.info("Caught ServiceException trying to compare ZimbraHit %s to ZimbraHit %s",
+                lhs, rhs);
+            ZimbraLog.index.debug(e);
+        }
I've opened a Zimbra ticket concerning these two new bugs in Patch 3 of 8.8.9.

Re: New java exceptions in logs after upgrade 8.8.9_P2 to 8.8.9_P3

Posted: Thu Mar 07, 2019 3:10 pm
by gabrieles
Edit, ain't solved anything, the problem is still there.

Had the same identical problem on my personal account. I thought that i can have solved it today, but wasn't.
- The strange behaviour started with ZCS 8.8.9_P2. Now i'm on 8.8.9_P8, the issue stands still
- I've found the issue when investigating slowness at the click on the mountpoint of a shared /Inbox from another account.
- Mine was the only account (out of ~50) experiencing the problem. The suspect /Inbox was shared and mounted in the identical way with other accounts, but not showing the problem.
- At each click mailbox.log produced 7221 lines of index - Caught ServiceException trying to compare ZimbraHit (9MB each time!). In the /Inbox there were exactly 7221 messages.
- Tried to reindex my mailbox and even the sharing mailbox, with no effect.
- I've got another mountpoint to the /Sent of the same account. There's absolutely no problem clicking or working on it.

Deleting and recreating the mountpoint resolved temporarily the problem.

zmmailbox -z -m gabrieles df /MyMountpoint
zmmailbox -z -m gabrieles cm /MyMountpoint sharingaccount@mydomain.com /Inbox

But some hours after, it resurfaced: when clicking that specific folder, it fills 9MB of index - Caught ServiceException trying to compare ZimbraHit . Will keep investigating.

Re: New java exceptions in logs after upgrade 8.8.9_P2 to 8.8.9_P3

Posted: Fri Mar 08, 2019 11:24 am
by gabrieles
Funny thing:
yesterday's mailbox.log was 720MB.
yesterday's mailbox.log without the Caught ServiceException trying to compare ZimbraHit lines was 22MB

Re: New java exceptions in logs after upgrade 8.8.9_P2 to 8.8.9_P3

Posted: Fri Mar 08, 2019 11:34 am
by andrey.ivanov
Yep. it is still not fixed in 8.8.10_P7 (i have not tested 8.8.11) though the support has confirmed the bug in August 2018 :
----
ZimbraHit error has been identified as a bug by the Dev team and it will be fixed in future release. This is an internal bug (ZCS-6022) number for the reference, which you may take a follow up any time with us by raising a case.
----

Re: New java exceptions in logs after upgrade 8.8.9_P2 to 8.8.9_P3

Posted: Wed Jun 12, 2019 10:04 pm
by jgarat
gabrieles wrote:Edit, ain't solved anything, the problem is still there.

Had the same identical problem on my personal account. I thought that i can have solved it today, but wasn't.
- The strange behaviour started with ZCS 8.8.9_P2. Now i'm on 8.8.9_P8, the issue stands still
- I've found the issue when investigating slowness at the click on the mountpoint of a shared /Inbox from another account.
- Mine was the only account (out of ~50) experiencing the problem. The suspect /Inbox was shared and mounted in the identical way with other accounts, but not showing the problem.
- At each click mailbox.log produced 7221 lines of index - Caught ServiceException trying to compare ZimbraHit (9MB each time!). In the /Inbox there were exactly 7221 messages.
- Tried to reindex my mailbox and even the sharing mailbox, with no effect.
- I've got another mountpoint to the /Sent of the same account. There's absolutely no problem clicking or working on it.

Deleting and recreating the mountpoint resolved temporarily the problem.

zmmailbox -z -m gabrieles df /MyMountpoint
zmmailbox -z -m gabrieles cm /MyMountpoint sharingaccount@mydomain.com /Inbox

But some hours after, it resurfaced: when clicking that specific folder, it fills 9MB of index - Caught ServiceException trying to compare ZimbraHit . Will keep investigating.


Hi Gabrieles, yesterday we upgrade from Zimbra Collaboration Open Source 8.8.6 to 8.8.12 Patch 2 and we are experimenting the same problems! We use a lot Shared Inboxes.

You still have this issue? Can you share how do you resolved?

Thanks!

Juan

Re: New java exceptions in logs after upgrade 8.8.9_P2 to 8.8.9_P3

Posted: Thu Jun 13, 2019 8:30 am
by gabrieles
Apparently was caused by the shared folder sorting order.
By default the sorting order of that folder was descending by date.
In some cases, I pressed the blue dot to visualize only the unread messages.
Pressing again the blue dot SEEMS to bring back to the previous state of descending by date, but actually don't.
If you pay careful attention, not all the message are displayed and the order is a little bit strange.
It leaves the folder in a sort of stranded state that s##t a ton of exception in the logs, various exceptions for EACH message in the folder.

The solution is to force back the folder in a well defined order, sorting manually by date, descending.

Re: New java exceptions in logs after upgrade 8.8.9_P2 to 8.8.9_P3

Posted: Mon Jul 08, 2019 10:54 am
by Klug
Hello.

I'm hitting the problem too (8.8.12P3).

Did someone opened a case to Zimbra's support?
I saw nothing in the bugzilla (yet).

It's quite difficult to force the users to use a pre-defined folder order and forbid them to use another one.

Re: New java exceptions in logs after upgrade 8.8.9_P2 to 8.8.9_P3

Posted: Tue Sep 17, 2019 6:17 am
by Gladkovsky
Someone fix this problem?
Open bug https://bugzilla.zimbra.com/show_bug.cgi?id=109227

Re: New java exceptions in logs after upgrade 8.8.9_P2 to 8.8.9_P3

Posted: Mon Mar 02, 2020 10:26 pm
by charneval
Today I tried to upgrade my zimbra server at 8.8.15 p7 version but the problem is always present.
Do you have a solution of this slowly access in the sared folders?

Re: New java exceptions in logs after upgrade 8.8.9_P2 to 8.8.9_P3

Posted: Tue Mar 03, 2020 3:55 pm
by cerri
Hi,
I'm in the same situation.