New java exceptions in logs after upgrade 8.8.9_P2 to 8.8.9_P3

Discuss your pilot or production implementation with other Zimbra admins or our engineers.
andrey.ivanov
Advanced member
Advanced member
Posts: 50
Joined: Wed Aug 08, 2018 8:44 am

New java exceptions in logs after upgrade 8.8.9_P2 to 8.8.9_P3

Post by andrey.ivanov »

We have two new types of exceptions in mailboxd.log after upgrade 8.8.9_P2 -> 8.8.9_P3 (NE, CentOS7). One of them is described in the forum thread viewtopic.php?f=15&t=64708 and concerns exceptions during SearchRequests. To find them search the file mailboxd.log for the keyword isSortByReadFlag. We have about 1000 of these per day.

The second new type of exceptions concerns the index code, search for the keyword ZimbraHit to find them:

Code: Select all

grep 'index - Caught ServiceException trying to compare ZimbraHit' /opt/zimbra/log/mailbox.log
We have about 150000 of those per working day. They look like this :

Code: Select all

2018-08-24 10:01:33,636 INFO  [qtp1286783232-154439:https:https://webmail.server.com/service/soap/SearchRequest] [name=some.user@my.zimbra.server;mid=10891;oip=123.45.67.89;port=56774;ua=ZimbraWebClient - FF61 (Win)/8.8.9_GA_3019;soapId=1aa2d45c;] index - Caught ServiceException trying to compare ZimbraHit {"s":2956,"d":1528962759000,"l":"c30c521b-6e3b-479a-8d07-cfedd9ab7198:2","cid":"c30c521b-6e3b-479a-8d07-cfedd9ab7198:-12504","f":"r","rev":55606,"id":"c30c521b-6e3b-479a-8d07-cfedd9ab7198:12504","su":"Date de master M4S","fr":"Ceci est un mail automatique, veuillez ne ...","cm":true,"e":[{"a":"internal_mail@our.domain","d":"toto","p":"toto via toto","t":"f"}],"sf":"0"} to ZimbraHit {"s":9100,"d":1529062749000,"l":"c30c521b-6e3b-479a-8d07-cfedd9ab7198:2","cid":"c30c521b-6e3b-479a-8d07-cfedd9ab7198:12521","f":"r","rev":55706,"id":"c30c521b-6e3b-479a-8d07-cfedd9ab7198:12520","su":"Inquiry about MSc. program","fr":"Dear Sirs: I hope this email finds you well. I am sending this to kindly ask about the masters program course language. I come from an English ...","cm":true,"e":[{"a":"some-email@somewhere.com","d":"John","p":"Jane Doe","t":"f"}],"sf":"0"}
It has appeared because of the changes in the file ./zm-mailbox/store/src/java/com/zimbra/cs/index/ZimbraHit.java at line 340 (diff from 8.8.9_P2 -> 8.8.9_P3) adding the new function protected static final int compareByReadFlag(boolean ascending, ZimbraHit lhs, ZimbraHit rhs). Part of the code generating this exception:

Code: Select all

+        try {
+            long left = getReadStatus(lhs);
+            long right = getReadStatus(rhs);
+            long result = right - left;
+            if (result > 0)
+                retVal = 1;
+            else if (result < 0)
+                retVal = -1;
+            else
+                retVal = 0;
+        } catch (ServiceException e) {
+            ZimbraLog.index.info("Caught ServiceException trying to compare ZimbraHit %s to ZimbraHit %s",
+                lhs, rhs);
+            ZimbraLog.index.debug(e);
+        }
I've opened a Zimbra ticket concerning these two new bugs in Patch 3 of 8.8.9.
User avatar
gabrieles
Outstanding Member
Outstanding Member
Posts: 233
Joined: Tue Feb 14, 2017 9:40 am

Re: New java exceptions in logs after upgrade 8.8.9_P2 to 8.8.9_P3

Post by gabrieles »

Edit, ain't solved anything, the problem is still there.

Had the same identical problem on my personal account. I thought that i can have solved it today, but wasn't.
- The strange behaviour started with ZCS 8.8.9_P2. Now i'm on 8.8.9_P8, the issue stands still
- I've found the issue when investigating slowness at the click on the mountpoint of a shared /Inbox from another account.
- Mine was the only account (out of ~50) experiencing the problem. The suspect /Inbox was shared and mounted in the identical way with other accounts, but not showing the problem.
- At each click mailbox.log produced 7221 lines of index - Caught ServiceException trying to compare ZimbraHit (9MB each time!). In the /Inbox there were exactly 7221 messages.
- Tried to reindex my mailbox and even the sharing mailbox, with no effect.
- I've got another mountpoint to the /Sent of the same account. There's absolutely no problem clicking or working on it.

Deleting and recreating the mountpoint resolved temporarily the problem.

zmmailbox -z -m gabrieles df /MyMountpoint
zmmailbox -z -m gabrieles cm /MyMountpoint sharingaccount@mydomain.com /Inbox

But some hours after, it resurfaced: when clicking that specific folder, it fills 9MB of index - Caught ServiceException trying to compare ZimbraHit . Will keep investigating.
User avatar
gabrieles
Outstanding Member
Outstanding Member
Posts: 233
Joined: Tue Feb 14, 2017 9:40 am

Re: New java exceptions in logs after upgrade 8.8.9_P2 to 8.8.9_P3

Post by gabrieles »

Funny thing:
yesterday's mailbox.log was 720MB.
yesterday's mailbox.log without the Caught ServiceException trying to compare ZimbraHit lines was 22MB
andrey.ivanov
Advanced member
Advanced member
Posts: 50
Joined: Wed Aug 08, 2018 8:44 am

Re: New java exceptions in logs after upgrade 8.8.9_P2 to 8.8.9_P3

Post by andrey.ivanov »

Yep. it is still not fixed in 8.8.10_P7 (i have not tested 8.8.11) though the support has confirmed the bug in August 2018 :
----
ZimbraHit error has been identified as a bug by the Dev team and it will be fixed in future release. This is an internal bug (ZCS-6022) number for the reference, which you may take a follow up any time with us by raising a case.
----
jgarat
Posts: 1
Joined: Wed Jun 12, 2019 9:50 pm

Re: New java exceptions in logs after upgrade 8.8.9_P2 to 8.8.9_P3

Post by jgarat »

gabrieles wrote:Edit, ain't solved anything, the problem is still there.

Had the same identical problem on my personal account. I thought that i can have solved it today, but wasn't.
- The strange behaviour started with ZCS 8.8.9_P2. Now i'm on 8.8.9_P8, the issue stands still
- I've found the issue when investigating slowness at the click on the mountpoint of a shared /Inbox from another account.
- Mine was the only account (out of ~50) experiencing the problem. The suspect /Inbox was shared and mounted in the identical way with other accounts, but not showing the problem.
- At each click mailbox.log produced 7221 lines of index - Caught ServiceException trying to compare ZimbraHit (9MB each time!). In the /Inbox there were exactly 7221 messages.
- Tried to reindex my mailbox and even the sharing mailbox, with no effect.
- I've got another mountpoint to the /Sent of the same account. There's absolutely no problem clicking or working on it.

Deleting and recreating the mountpoint resolved temporarily the problem.

zmmailbox -z -m gabrieles df /MyMountpoint
zmmailbox -z -m gabrieles cm /MyMountpoint sharingaccount@mydomain.com /Inbox

But some hours after, it resurfaced: when clicking that specific folder, it fills 9MB of index - Caught ServiceException trying to compare ZimbraHit . Will keep investigating.


Hi Gabrieles, yesterday we upgrade from Zimbra Collaboration Open Source 8.8.6 to 8.8.12 Patch 2 and we are experimenting the same problems! We use a lot Shared Inboxes.

You still have this issue? Can you share how do you resolved?

Thanks!

Juan
User avatar
gabrieles
Outstanding Member
Outstanding Member
Posts: 233
Joined: Tue Feb 14, 2017 9:40 am

Re: New java exceptions in logs after upgrade 8.8.9_P2 to 8.8.9_P3

Post by gabrieles »

Apparently was caused by the shared folder sorting order.
By default the sorting order of that folder was descending by date.
In some cases, I pressed the blue dot to visualize only the unread messages.
Pressing again the blue dot SEEMS to bring back to the previous state of descending by date, but actually don't.
If you pay careful attention, not all the message are displayed and the order is a little bit strange.
It leaves the folder in a sort of stranded state that s##t a ton of exception in the logs, various exceptions for EACH message in the folder.

The solution is to force back the folder in a well defined order, sorting manually by date, descending.
Klug
Ambassador
Ambassador
Posts: 2747
Joined: Mon Dec 16, 2013 11:35 am
Location: France - Drôme
ZCS/ZD Version: All of them
Contact:

Re: New java exceptions in logs after upgrade 8.8.9_P2 to 8.8.9_P3

Post by Klug »

Hello.

I'm hitting the problem too (8.8.12P3).

Did someone opened a case to Zimbra's support?
I saw nothing in the bugzilla (yet).

It's quite difficult to force the users to use a pre-defined folder order and forbid them to use another one.
Gladkovsky
Posts: 3
Joined: Wed Apr 03, 2019 11:51 am

Re: New java exceptions in logs after upgrade 8.8.9_P2 to 8.8.9_P3

Post by Gladkovsky »

Someone fix this problem?
Open bug https://bugzilla.zimbra.com/show_bug.cgi?id=109227
charneval
Posts: 11
Joined: Sun Jan 15, 2017 7:16 am

Re: New java exceptions in logs after upgrade 8.8.9_P2 to 8.8.9_P3

Post by charneval »

Today I tried to upgrade my zimbra server at 8.8.15 p7 version but the problem is always present.
Do you have a solution of this slowly access in the sared folders?
cerri
Posts: 27
Joined: Sat Sep 13, 2014 3:03 am

Re: New java exceptions in logs after upgrade 8.8.9_P2 to 8.8.9_P3

Post by cerri »

Hi,
I'm in the same situation.
Post Reply