Page 1 of 2
New warnings after upgrade: CompoundCachingTier - Error overflowing into lower caching tier - The element is too large
Posted: Mon Oct 16, 2017 12:20 pm
by msquadrat
This weekend I upgraded our installation from 8.6 to 8.711 and as always kept an eye on the mailbox.log to see if anything bad happens. Looks good so far, except for one warning which crops up regularly on one of our mailbox servers:
Code: Select all
2017-10-16 03:16:55,831 WARN [ImapServer-2] [name=user1@example.org;ip=...;oip=...;via...;ua=Zimbra/8.7.11_GA_1854;] CompoundCachingTier - Error overflowing '0decc7d9-7df0-4ddb-90b4-7403b1d20c6d:2:997668:1' into lower caching tier org.ehcache.impl.internal.store.offheap.OffHeapStore@75a6bd06
org.ehcache.core.spi.store.StoreAccessException: The element with key '0decc7d9-7df0-4ddb-90b4-7403b1d20c6d:2:997668:1' is too large to be stored in this offheap store.
at org.ehcache.impl.internal.store.offheap.AbstractOffHeapStore.computeWithRetry(AbstractOffHeapStore.java:1090)
at org.ehcache.impl.internal.store.offheap.AbstractOffHeapStore.installMapping(AbstractOffHeapStore.java:1067)
at org.ehcache.impl.internal.store.tiering.CompoundCachingTier$1.onInvalidation(CompoundCachingTier.java:69)
at org.ehcache.impl.internal.store.heap.OnHeapStore$20.onInvalidation(OnHeapStore.java:921)
at org.ehcache.impl.internal.store.heap.OnHeapStore$26.apply(OnHeapStore.java:1575)
at org.ehcache.impl.internal.store.heap.OnHeapStore$26.apply(OnHeapStore.java:1568)
at org.ehcache.impl.internal.concurrent.ConcurrentHashMap.computeIfPresent(ConcurrentHashMap.java:1793)
at org.ehcache.impl.internal.store.heap.SimpleBackend.computeIfPresent(SimpleBackend.java:124)
at org.ehcache.impl.internal.store.heap.OnHeapStore.evict(OnHeapStore.java:1568)
at org.ehcache.impl.internal.store.heap.OnHeapStore.enforceCapacity(OnHeapStore.java:1535)
at org.ehcache.impl.internal.store.heap.OnHeapStore.getOrComputeIfAbsent(OnHeapStore.java:723)
at org.ehcache.impl.internal.store.tiering.CompoundCachingTier.getOrComputeIfAbsent(CompoundCachingTier.java:112)
at org.ehcache.impl.internal.store.tiering.TieredStore.get(TieredStore.java:110)
at org.ehcache.core.Ehcache.get(Ehcache.java:167)
at com.zimbra.cs.imap.EhcacheImapCache.get(EhcacheImapCache.java:134)
at com.zimbra.cs.imap.ImapSessionManager.deserialize(ImapSessionManager.java:647)
at com.zimbra.cs.imap.ImapSession.reload(ImapSession.java:367)
at com.zimbra.cs.imap.ImapSession.getImapFolder(ImapSession.java:97)
at com.zimbra.cs.imap.ImapSessionManager.duplicateExistingSession(ImapSessionManager.java:394)
at com.zimbra.cs.imap.ImapSessionManager.openFolder(ImapSessionManager.java:289)
at com.zimbra.cs.imap.ImapHandler.setSelectedFolder(ImapHandler.java:1028)
at com.zimbra.cs.imap.ImapHandler.selectFolder(ImapHandler.java:1489)
at com.zimbra.cs.imap.ImapHandler.doSELECT(ImapHandler.java:1453)
at com.zimbra.cs.imap.ImapHandler.executeRequest(ImapHandler.java:722)
at com.zimbra.cs.imap.NioImapHandler.processRequest(NioImapHandler.java:126)
at com.zimbra.cs.imap.NioImapHandler.messageReceived(NioImapHandler.java:63)
at com.zimbra.cs.server.NioHandlerDispatcher.messageReceived(NioHandlerDispatcher.java:95)
at org.apache.mina.core.filterchain.DefaultIoFilterChain$TailFilter.messageReceived(DefaultIoFilterChain.java:716)
at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434)
at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1200(DefaultIoFilterChain.java:46)
at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:796)
at org.apache.mina.core.filterchain.IoFilterAdapter.messageReceived(IoFilterAdapter.java:119)
at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434)
at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1200(DefaultIoFilterChain.java:46)
at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:796)
at org.apache.mina.filter.codec.ProtocolCodecFilter$ProtocolDecoderOutputImpl.flush(ProtocolCodecFilter.java:427)
at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:245)
at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434)
at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1200(DefaultIoFilterChain.java:46)
at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:796)
at com.zimbra.cs.server.NioLoggingFilter.messageReceived(NioLoggingFilter.java:68)
at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434)
at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1200(DefaultIoFilterChain.java:46)
at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:796)
at org.apache.mina.core.filterchain.IoFilterEvent.fire(IoFilterEvent.java:75)
at org.apache.mina.core.session.IoEvent.run(IoEvent.java:63)
at org.apache.mina.filter.executor.OrderedThreadPoolExecutor$Worker.runTask(OrderedThreadPoolExecutor.java:780)
at org.apache.mina.filter.executor.OrderedThreadPoolExecutor$Worker.runTasks(OrderedThreadPoolExecutor.java:772)
at org.apache.mina.filter.executor.OrderedThreadPoolExecutor$Worker.run(OrderedThreadPoolExecutor.java:714)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.terracotta.offheapstore.exceptions.OversizeMappingException: Storage Engine and Eviction Failed - Empty Map
Storage Engine : OffHeapBufferStorageEngine allocated=1022.0KB occupied=0B
Storage Area: OffHeapStorageArea
511 2KB pages
Allocator: org.terracotta.offheapstore.storage.allocator.IntegerBestFitAllocator@1e83a621
Page Source: UpfrontAllocatingPageSource
Chunk 1
Size : 1MB
Free Allocator : PowerOfTwoAllocator: Occupied 1022.2KB [Largest Available Area 1KB]
Victim Allocator : PowerOfTwoAllocator: Occupied 272B [Largest Available Area 512KB]
at org.terracotta.offheapstore.AbstractOffHeapClockCache.storageEngineFailure(AbstractOffHeapClockCache.java:84)
at org.terracotta.offheapstore.OffHeapHashMap.writeEntry(OffHeapHashMap.java:689)
at org.terracotta.offheapstore.OffHeapHashMap.computeWithMetadata(OffHeapHashMap.java:1947)
at org.terracotta.offheapstore.AbstractLockedOffHeapHashMap.computeWithMetadata(AbstractLockedOffHeapHashMap.java:582)
at org.terracotta.offheapstore.concurrent.AbstractConcurrentOffHeapMap.computeWithMetadata(AbstractConcurrentOffHeapMap.java:743)
at org.ehcache.impl.internal.store.offheap.EhcacheConcurrentOffHeapClockCache.compute(EhcacheConcurrentOffHeapClockCache.java:152)
at org.ehcache.impl.internal.store.offheap.AbstractOffHeapStore.computeWithRetry(AbstractOffHeapStore.java:1088)
... 49 more
2017-10-16 03:16:55,832 INFO [ImapServer-2] [name=user1@example.org;ip=...;oip=...;via...;ua=Zimbra/8.7.11_GA_1854;] imap - copying message data from existing session: INBOX
All tweaking I did to the EhCache was to reduce the maximum on disk size from the default 100G to 10G to avoid running out of disk (cf. [bug]104846[/bug]). The on disk sizes are way from reaching this limit though:
Code: Select all
zimbra@zm1:~$ zmprov gs $(zmhostname) zimbraImapInactiveSessionCacheMaxDiskSize zimbraImapInactiveSessionEhcacheMaxDiskSize zimbraImapActiveSessionEhcacheMaxDiskSize zimbraImapInactiveSessionEhcacheSize zimbraActiveSyncEhcacheMaxDiskSize zimbraActiveSyncEhcacheHeapSize zimbraActiveSyncEhcacheExpiration
zimbraActiveSyncEhcacheExpiration: 5m
zimbraActiveSyncEhcacheHeapSize: 10485760
zimbraActiveSyncEhcacheMaxDiskSize: 10737418240
zimbraImapActiveSessionEhcacheMaxDiskSize: 10737418240
zimbraImapInactiveSessionCacheMaxDiskSize: 10737418240
zimbraImapInactiveSessionEhcacheMaxDiskSize: 10737418240
zimbraImapInactiveSessionEhcacheSize: 1048576
zimbra@zm1:~$ ls /opt/zimbra/data/mailboxd/*cache*/offheap-disk-store/* -lh
-rw-r----- 1 zimbra zimbra 152M Oct 16 14:15 /opt/zimbra/data/mailboxd/imap-active-session-cache_85be333d34866255dbb2082ee3fb1a79544d0878/offheap-disk-store/ehcache-disk-store.data
-rw-r----- 1 zimbra zimbra 117 Oct 15 15:03 /opt/zimbra/data/mailboxd/imap-active-session-cache_85be333d34866255dbb2082ee3fb1a79544d0878/offheap-disk-store/ehcache-disk-store.meta
-rw-r----- 1 zimbra zimbra 200M Oct 16 14:17 /opt/zimbra/data/mailboxd/imap-inactive-session-cache_fc164b40eddbe2ccbc469be50b2e7494ea03619a/offheap-disk-store/ehcache-disk-store.data
-rw-r----- 1 zimbra zimbra 4.3K Oct 15 15:03 /opt/zimbra/data/mailboxd/imap-inactive-session-cache_fc164b40eddbe2ccbc469be50b2e7494ea03619a/offheap-disk-store/ehcache-disk-store.index
-rw-r----- 1 zimbra zimbra 117 Oct 14 13:44 /opt/zimbra/data/mailboxd/imap-inactive-session-cache_fc164b40eddbe2ccbc469be50b2e7494ea03619a/offheap-disk-store/ehcache-disk-store.meta
-rw-r----- 1 zimbra zimbra 124K Oct 16 14:18 /opt/zimbra/data/mailboxd/sync-state-item-cache_eca9d558d785a39bf537e61b19815681b15d280f/offheap-disk-store/ehcache-disk-store.data
-rw-r----- 1 zimbra zimbra 3.3K Oct 15 15:03 /opt/zimbra/data/mailboxd/sync-state-item-cache_eca9d558d785a39bf537e61b19815681b15d280f/offheap-disk-store/ehcache-disk-store.index
-rw-r----- 1 zimbra zimbra 104 Oct 14 13:44 /opt/zimbra/data/mailboxd/sync-state-item-cache_eca9d558d785a39bf537e61b19815681b15d280f/offheap-disk-store/ehcache-disk-store.meta
Has anybody else seen this as well?
Googling for the exception brought me to a
EhCache change which apparently was fixed in 3.2.1 (ZCS 8.7.11 ships 3.1.2).
Re: New warnings after upgrade: CompoundCachingTier - Error overflowing into lower caching tier - The element is too lar
Posted: Mon Oct 30, 2017 8:18 pm
by markd
I haven't seen these errors buy my disk just went full because of the size of the ehcache files.
Thanks for including the bug reference. I hope we can limit the file size growth.
Re: New warnings after upgrade: CompoundCachingTier - Error overflowing into lower caching tier - The element is too lar
Posted: Mon Feb 12, 2018 3:07 pm
by vladsol
Same problem. ZCS 8.8.6
Code: Select all
Feb 12 17:04:44 zimbra.domain.com mailbox.log: 2018-02-12 17:04:44,628 WARN [ImapSSLServer-13] [name=email@domain.com;mid=60;ip=1.1.1.1;oip=1.1.1.1;via=1.1.1.1(nginx/1.7.1);ua=Zimbra/8.8.6_GA_1906;] CompoundCachingTier - Error overflowing 'ee1336f8-d3b1-4cc3-ad3c-a7912d856511:20813:78625:19506' into lower caching tier org.ehcache.impl.internal.store.offheap.OffHeapStore@35e8316e
Feb 12 17:04:44 zimbra.domain.com mailbox.log: Caused by: org.terracotta.offheapstore.exceptions.OversizeMappingException: Storage Engine and Eviction Failed - Empty Map
Feb 12 17:04:44 zimbra.domain.com mailbox.log: Allocator: org.terracotta.offheapstore.storage.allocator.IntegerBestFitAllocator@4fbe56aa
Feb 12 17:04:44 zimbra.domain.com mailbox.log: at org.terracotta.offheapstore.AbstractOffHeapClockCache.storageEngineFailure(AbstractOffHeapClockCache.java:84)
Feb 12 17:04:44 zimbra.domain.com mailbox.log: at org.terracotta.offheapstore.OffHeapHashMap.writeEntry(OffHeapHashMap.java:689)
Feb 12 17:04:44 zimbra.domain.com mailbox.log: at org.terracotta.offheapstore.OffHeapHashMap.computeWithMetadata(OffHeapHashMap.java:1947)
Feb 12 17:04:44 zimbra.domain.com mailbox.log: at org.terracotta.offheapstore.AbstractLockedOffHeapHashMap.computeWithMetadata(AbstractLockedOffHeapHashMap.java:582)
Feb 12 17:04:44 zimbra.domain.com mailbox.log: at org.terracotta.offheapstore.concurrent.AbstractConcurrentOffHeapMap.computeWithMetadata(AbstractConcurrentOffHeapMap.java:743)
Re: New warnings after upgrade: CompoundCachingTier - Error overflowing into lower caching tier - The element is too lar
Posted: Mon Feb 12, 2018 4:08 pm
by msquadrat
I don't really know if these messages are really an error, I haven't seen any issues yet which I relate to them. I still see them regularly (see below from one of our mailbox servers) and customers see them as well.
Code: Select all
# zgrep -c 'Error overflowing' mailbox.log*.gz
mailbox.log.2018-01-26.gz:164
mailbox.log.2018-01-27.gz:37
mailbox.log.2018-01-28.gz:6
mailbox.log.2018-01-29.gz:81
mailbox.log.2018-01-30.gz:105
mailbox.log.2018-01-31.gz:56
mailbox.log.2018-02-01.gz:272
mailbox.log.2018-02-02.gz:176
mailbox.log.2018-02-03.gz:25
mailbox.log.2018-02-04.gz:26
mailbox.log.2018-02-05.gz:212
mailbox.log.2018-02-06.gz:190
mailbox.log.2018-02-07.gz:268
mailbox.log.2018-02-08.gz:227
mailbox.log.2018-02-09.gz:265
mailbox.log.2018-02-10.gz:62
mailbox.log.2018-02-11.gz:19
Re: New warnings after upgrade: CompoundCachingTier - Error overflowing into lower caching tier - The element is too lar
Posted: Wed Apr 11, 2018 8:22 pm
by Malcolm
vladsol wrote:Same problem. ZCS 8.8.6
Code: Select all
Feb 12 17:04:44 zimbra.domain.com mailbox.log: 2018-02-12 17:04:44,628 WARN [ImapSSLServer-13] [name=email@domain.com;mid=60;ip=1.1.1.1;oip=1.1.1.1;via=1.1.1.1(nginx/1.7.1);ua=Zimbra/8.8.6_GA_1906;] CompoundCachingTier - Error overflowing 'ee1336f8-d3b1-4cc3-ad3c-a7912d856511:20813:78625:19506' into lower caching tier org.ehcache.impl.internal.store.offheap.OffHeapStore@35e8316e
Feb 12 17:04:44 zimbra.domain.com mailbox.log: Caused by: org.terracotta.offheapstore.exceptions.OversizeMappingException: Storage Engine and Eviction Failed - Empty Map
Feb 12 17:04:44 zimbra.domain.com mailbox.log: Allocator: org.terracotta.offheapstore.storage.allocator.IntegerBestFitAllocator@4fbe56aa
Feb 12 17:04:44 zimbra.domain.com mailbox.log: at org.terracotta.offheapstore.AbstractOffHeapClockCache.storageEngineFailure(AbstractOffHeapClockCache.java:84)
Feb 12 17:04:44 zimbra.domain.com mailbox.log: at org.terracotta.offheapstore.OffHeapHashMap.writeEntry(OffHeapHashMap.java:689)
Feb 12 17:04:44 zimbra.domain.com mailbox.log: at org.terracotta.offheapstore.OffHeapHashMap.computeWithMetadata(OffHeapHashMap.java:1947)
Feb 12 17:04:44 zimbra.domain.com mailbox.log: at org.terracotta.offheapstore.AbstractLockedOffHeapHashMap.computeWithMetadata(AbstractLockedOffHeapHashMap.java:582)
Feb 12 17:04:44 zimbra.domain.com mailbox.log: at org.terracotta.offheapstore.concurrent.AbstractConcurrentOffHeapMap.computeWithMetadata(AbstractConcurrentOffHeapMap.java:743)
I have the same problem/error in my 8.8.5 version and as well the java cpu 100% or even more,... any ideas? I have centos 7 64bit and OSE Zimbra, 24GB ram 4 core and 2 thread.
Ideas? thank you
Re: New warnings after upgrade: CompoundCachingTier - Error overflowing into lower caching tier - The element is too lar
Posted: Thu Apr 12, 2018 12:26 pm
by msquadrat
I have filed [bug]108844[/bug] about this issue in the meantime.
The 100% CPU issue might be fixed if you upgrade to 8.8.8. 8.8.5 has some well known issues in that regard.
Re: New warnings after upgrade: CompoundCachingTier - Error overflowing into lower caching tier - The element is too lar
Posted: Sun Apr 15, 2018 10:23 am
by Malcolm
Thank you msquadrat!
I planned to do today to upgrade from 8.8.5 to the last release 8.8.8 wondering it will be fine and after my java CPU troubles will disappear.
If you don't hear from me anything it will be the solution.
msquadrat wrote:I have filed [bug]108844[/bug] about this issue in the meantime.
The 100% CPU issue might be fixed if you upgrade to 8.8.8. 8.8.5 has some well known issues in that regard.
Re: New warnings after upgrade: CompoundCachingTier - Error overflowing into lower caching tier - The element is too lar
Posted: Sun May 06, 2018 4:03 pm
by Malcolm
Hello to everybody,...
after I upgrade to 8.8.8 all is fine since a long time but in a couple of days now is again a java CPU 100% is that normal if I have an 8 core CPU server?
Any advice? Best Regards,
Malcolm
Re: New warnings after upgrade: CompoundCachingTier - Error overflowing into lower caching tier - The element is too lar
Posted: Thu Aug 02, 2018 2:24 pm
by L. Mark Stone
I'm seeing the same error in mailbox.log reported by the OP on 8.8.8 Patch 7.
I've raised a Support Case with Zimbra for this. Will update this thread when I hear back from Support definitively.
Hope that helps,
Mark
Scrubbed mailbox.log entry follows:
Code: Select all
2018-08-02 14:20:09,888 WARN [ImapSSLServer-1] [name=removed_user@removed_domain.com;mid=121;ip=10.7.57.17;oip=removed;via=removed(nginx/1.7.1);ua=Zimbra/8.8.8_GA_3008;cid=1624;] CompoundCachingTier - Error overflowing '8ec5c54f-eb59-4b22-adb6-2f2707617874:5:70072:1' into lower caching tier org.ehcache.impl.internal.store.offheap.OffHeapStore@4d0b8b8b
org.ehcache.core.spi.store.StoreAccessException: The element with key '8ec5c54f-eb59-4b22-adb6-2f2707617874:5:70072:1' is too large to be stored in this offheap store.
at org.ehcache.impl.internal.store.offheap.AbstractOffHeapStore.computeWithRetry(AbstractOffHeapStore.java:1090)
at org.ehcache.impl.internal.store.offheap.AbstractOffHeapStore.installMapping(AbstractOffHeapStore.java:1067)
at org.ehcache.impl.internal.store.tiering.CompoundCachingTier$1.onInvalidation(CompoundCachingTier.java:69)
at org.ehcache.impl.internal.store.heap.OnHeapStore$20.onInvalidation(OnHeapStore.java:921)
at org.ehcache.impl.internal.store.heap.OnHeapStore$26.apply(OnHeapStore.java:1575)
at org.ehcache.impl.internal.store.heap.OnHeapStore$26.apply(OnHeapStore.java:1568)
at org.ehcache.impl.internal.concurrent.ConcurrentHashMap.computeIfPresent(ConcurrentHashMap.java:1793)
at org.ehcache.impl.internal.store.heap.SimpleBackend.computeIfPresent(SimpleBackend.java:124)
at org.ehcache.impl.internal.store.heap.OnHeapStore.evict(OnHeapStore.java:1568)
at org.ehcache.impl.internal.store.heap.OnHeapStore.enforceCapacity(OnHeapStore.java:1535)
at org.ehcache.impl.internal.store.heap.OnHeapStore.getOrComputeIfAbsent(OnHeapStore.java:723)
at org.ehcache.impl.internal.store.tiering.CompoundCachingTier.getOrComputeIfAbsent(CompoundCachingTier.java:112)
at org.ehcache.impl.internal.store.tiering.TieredStore.get(TieredStore.java:110)
at org.ehcache.core.Ehcache.get(Ehcache.java:167)
at com.zimbra.cs.imap.EhcacheImapCache.get(EhcacheImapCache.java:138)
at com.zimbra.cs.imap.EhcacheImapCache.get(EhcacheImapCache.java:40)
at com.zimbra.cs.imap.ImapSessionManager.deserialize(ImapSessionManager.java:799)
at com.zimbra.cs.imap.ImapListener.reload(ImapListener.java:713)
at com.zimbra.cs.imap.ImapListener.getImapFolder(ImapListener.java:399)
at com.zimbra.cs.imap.ImapSessionManager.duplicateExistingSession(ImapSessionManager.java:482)
at com.zimbra.cs.imap.ImapSessionManager.duplicateExistingSession(ImapSessionManager.java:471)
at com.zimbra.cs.imap.ImapSessionManager.openFolder(ImapSessionManager.java:315)
at com.zimbra.cs.imap.ImapHandler.setSelectedFolder(ImapHandler.java:1196)
at com.zimbra.cs.imap.ImapHandler.selectFolder(ImapHandler.java:1784)
at com.zimbra.cs.imap.ImapHandler.doSELECT(ImapHandler.java:1750)
at com.zimbra.cs.imap.ImapHandler.executeRequest(ImapHandler.java:837)
at com.zimbra.cs.imap.NioImapHandler.processRequest(NioImapHandler.java:123)
at com.zimbra.cs.imap.NioImapHandler.messageReceived(NioImapHandler.java:63)
at com.zimbra.cs.server.NioHandlerDispatcher.messageReceived(NioHandlerDispatcher.java:95)
at org.apache.mina.core.filterchain.DefaultIoFilterChain$TailFilter.messageReceived(DefaultIoFilterChain.java:716)
at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434)
at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1200(DefaultIoFilterChain.java:46)
at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:796)
at org.apache.mina.core.filterchain.IoFilterAdapter.messageReceived(IoFilterAdapter.java:119)
at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434)
at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1200(DefaultIoFilterChain.java:46)
at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:796)
at org.apache.mina.filter.codec.ProtocolCodecFilter$ProtocolDecoderOutputImpl.flush(ProtocolCodecFilter.java:427)
at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:245)
at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434)
at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1200(DefaultIoFilterChain.java:46)
at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:796)
at com.zimbra.cs.server.NioLoggingFilter.messageReceived(NioLoggingFilter.java:68)
at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434)
at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1200(DefaultIoFilterChain.java:46)
at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:796)
at org.apache.mina.core.filterchain.IoFilterEvent.fire(IoFilterEvent.java:75)
at org.apache.mina.core.session.IoEvent.run(IoEvent.java:63)
at org.apache.mina.filter.executor.OrderedThreadPoolExecutor$Worker.runTask(OrderedThreadPoolExecutor.java:780)
at org.apache.mina.filter.executor.OrderedThreadPoolExecutor$Worker.runTasks(OrderedThreadPoolExecutor.java:772)
at org.apache.mina.filter.executor.OrderedThreadPoolExecutor$Worker.run(OrderedThreadPoolExecutor.java:714)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.terracotta.offheapstore.exceptions.OversizeMappingException: Storage Engine and Eviction Failed - Empty Map
Storage Engine : OffHeapBufferStorageEngine allocated=1022.0KB occupied=0B
Storage Area: OffHeapStorageArea
511 2KB pages
Allocator: org.terracotta.offheapstore.storage.allocator.IntegerBestFitAllocator@709a4140
Page Source: UpfrontAllocatingPageSource
Chunk 1
Size : 1MB
Free Allocator : PowerOfTwoAllocator: Occupied 1022.3KB [Largest Available Area 1KB]
Victim Allocator : PowerOfTwoAllocator: Occupied 368B [Largest Available Area 512KB]
at org.terracotta.offheapstore.AbstractOffHeapClockCache.storageEngineFailure(AbstractOffHeapClockCache.java:84)
at org.terracotta.offheapstore.OffHeapHashMap.writeEntry(OffHeapHashMap.java:689)
at org.terracotta.offheapstore.OffHeapHashMap.computeWithMetadata(OffHeapHashMap.java:1947)
at org.terracotta.offheapstore.AbstractLockedOffHeapHashMap.computeWithMetadata(AbstractLockedOffHeapHashMap.java:582)
at org.terracotta.offheapstore.concurrent.AbstractConcurrentOffHeapMap.computeWithMetadata(AbstractConcurrentOffHeapMap.java:743)
at org.ehcache.impl.internal.store.offheap.EhcacheConcurrentOffHeapClockCache.compute(EhcacheConcurrentOffHeapClockCache.java:152)
at org.ehcache.impl.internal.store.offheap.AbstractOffHeapStore.computeWithRetry(AbstractOffHeapStore.java:1088)
... 51 more
Re: New warnings after upgrade: CompoundCachingTier - Error overflowing into lower caching tier - The element is too lar
Posted: Thu Aug 09, 2018 7:12 pm
by L. Mark Stone
I committed to updating this thread when I heard back from Zimbra Support definitively...
After reviewing my situation, Zimbra asked me to run:
Code: Select all
zmprov ms `zmhostname` zimbraImapInactiveSessionEhcacheSize 104857600
on all my mailbox servers, and then restart the mailbox service.
The variable is originally set (at least on my system) to 1MB; the above command increases it to 100MB. Zimbra Support also advised that the warnings are essentially benign in that something the system tried to cache wasn't able to be cached.
I'll restart mailboxd during the next maintenance window and report back to confirm that the errors like the ones below have reduced in frequency.
Hope that helps,
Mark
Code: Select all
2018-08-09 18:12:08,006 WARN [ImapSSLServer-106] [name=fake_user@fake_domain.com;mid=109;ip=10.7.57.17;oip=<removed>;via=10.7.57.17(nginx/1.7.1);ua=Zimbra/8.8.8_GA_3008;cid=40159;] CompoundCachingTier - Error overflowing '7d41c83f-c08d-478c-a5be-304b0b1e60af:5:114708:1' into lower caching tier org.ehcache.impl.internal.store.offheap.OffHeapStore@5e5f5ddf
org.ehcache.core.spi.store.StoreAccessException: The element with key '7d41c83f-c08d-478c-a5be-304b0b1e60af:5:114708:1' is too large to be stored in this offheap store.
Caused by: org.terracotta.offheapstore.exceptions.OversizeMappingException: Storage Engine and Eviction Failed - Empty Map
Storage Engine : OffHeapBufferStorageEngine allocated=1022.0KB occupied=0B
Storage Area: OffHeapStorageArea
511 2KB pages
Allocator: org.terracotta.offheapstore.storage.allocator.IntegerBestFitAllocator@331c2171
Page Source: UpfrontAllocatingPageSource
Chunk 1
Size : 1MB
Free Allocator : PowerOfTwoAllocator: Occupied 1022.3KB [Largest Available Area 1KB]
Victim Allocator : PowerOfTwoAllocator: Occupied 368B [Largest Available Area 512KB]
... 51 more