Page 1 of 5

Since Upgrading To Zimbra 8.8.5, Have Been Getting Numerous Lock Errors

Posted: Tue Jan 02, 2018 12:33 am
by mntwinsfan
I recently upgraded to Zimbra 8.8.5 from 8.7.6. Ever since I did, some users (not all) have been having issues getting into their mailbox from time to time. Through the web client, they get the login page, enter their credentials, and then the browser just spins. On their devices, it will eventually time out. If I restart Zimbra, it works fine for everyone for a while but then starts failing again. Never for all users, just a handful. I am seeing the following errors in the mailbox.log:

Code: Select all

2018-01-01 18:49:28,948 ERROR [LmtpServer-1] [name=user@domain.com;mid=16;ip=nn.nn.nn.nn;] mailbox - Failed to lock mailbox
Write Lock Owner - ImapSSLServer-1 prio=5 id=182 state=TERMINATED
Lock Waiter - Timer-Zimbra daemon prio=5 id=37 state=TIMED_WAITING
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireNanos(AbstractQueuedSynchronizer.java:934)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireNanos(AbstractQueuedSynchronizer.java:1247)
        at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.tryLock(ReentrantReadWriteLock.java:1115)
        at com.zimbra.cs.mailbox.MailboxLock.tryLockWithTimeout(MailboxLock.java:114)
        at com.zimbra.cs.mailbox.MailboxLock.lock(MailboxLock.java:194)
        at com.zimbra.cs.mailbox.Mailbox.lock(Mailbox.java:10411)
        at com.zimbra.cs.imap.ImapListener.unload(ImapListener.java:580)
        at com.zimbra.cs.imap.ImapSessionManager$SessionSerializerTask.run(ImapSessionManager.java:195)
        at java.util.TimerThread.mainLoop(Timer.java:555)
        at java.util.TimerThread.run(Timer.java:505)

com.zimbra.cs.mailbox.MailboxLock$LockFailedException: timeout
        at com.zimbra.cs.mailbox.MailboxLock.lock(MailboxLock.java:211)
        at com.zimbra.cs.mailbox.Mailbox.beginTransaction(Mailbox.java:1759)
        at com.zimbra.cs.mailbox.Mailbox.beginReadTransaction(Mailbox.java:1735)
        at com.zimbra.cs.mailbox.Mailbox.getItemById(Mailbox.java:2864)
        at com.zimbra.cs.mailbox.Mailbox.getItemById(Mailbox.java:2856)
        at com.zimbra.cs.mailbox.Mailbox.getFolderById(Mailbox.java:4092)
        at com.zimbra.cs.filter.IncomingMessageHandler.getDefaultFolderPath(IncomingMessageHandler.java:91)
        at com.zimbra.cs.filter.ZimbraMailAdapter.keep(ZimbraMailAdapter.java:526)
        at com.zimbra.cs.filter.ZimbraMailAdapter.executeActionKeepInternal(ZimbraMailAdapter.java:359)
        at com.zimbra.cs.filter.ZimbraMailAdapter.executeAllActions(ZimbraMailAdapter.java:294)
        at com.zimbra.cs.filter.RuleManager.applyRulesToIncomingMessage(RuleManager.java:440)
        at com.zimbra.cs.lmtpserver.ZimbraLmtpBackend.deliverMessageToLocalMailboxes(ZimbraLmtpBackend.java:615)
        at com.zimbra.cs.lmtpserver.ZimbraLmtpBackend.deliver(ZimbraLmtpBackend.java:385)
        at com.zimbra.cs.lmtpserver.LmtpHandler.processMessageData(LmtpHandler.java:445)
        at com.zimbra.cs.lmtpserver.TcpLmtpHandler.continueDATA(TcpLmtpHandler.java:79)
        at com.zimbra.cs.lmtpserver.LmtpHandler.doDATA(LmtpHandler.java:434)
        at com.zimbra.cs.lmtpserver.LmtpHandler.processCommand(LmtpHandler.java:216)
        at com.zimbra.cs.lmtpserver.TcpLmtpHandler.processCommand(TcpLmtpHandler.java:72)
        at com.zimbra.cs.server.ProtocolHandler.processConnection(ProtocolHandler.java:189)
        at com.zimbra.cs.server.ProtocolHandler.run(ProtocolHandler.java:128)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
2018-01-01 18:49:28,948 WARN  [LmtpServer-1] [name=user@domain.com;mid=16;ip=nn.nn.nn.nn;] mailbox - transaction canceled because of lock failure
Searching through these forums, I found some old posts regarding this error message which suggested changing some of the server settings. I did but they had no effect:

Code: Select all

zmlocalconfig -e zimbra_mailbox_lock_timeout=90
zmlocalconfig -e zimbra_mailbox_lock_max_waiting_threads=100
zmlocalconfig -e mailboxd_java_heap_size=4096
zmlocalconfig -e mailboxd_java_heap_memory_percent=25
zmprov ms `zmhostname` zimbraMessageCacheSize 4000
zmprov ms `zmhostname` zimbraImapNumThreads 50
zmprov ms `zmhostname` zimbraHttpNumThreads 600
Does anyone have any further suggestions or know how to fix this problem?

Re: Since Upgrading To Zimbra 8.8.5, Have Been Getting Numerous Lock Errors

Posted: Mon Jan 08, 2018 5:18 pm
by moebius
I am also experiencing mailbox locking issues since 8.8.5 and have to restart the mailboxd service to clear them. zmmailboxctl kill does not work, I have to kill the PID. Seems to pop up every 2 or 3 days.

Re: Since Upgrading To Zimbra 8.8.5, Have Been Getting Numerous Lock Errors

Posted: Mon Jan 08, 2018 5:46 pm
by maxxer
Someone here also suggests increasing http and imap threads.

From other discussions I learnt it could be bond to IMAP usage, especially with Apple clients. Is this your case?

Re: Since Upgrading To Zimbra 8.8.5, Have Been Getting Numerous Lock Errors

Posted: Tue Jan 09, 2018 5:34 pm
by moebius
It's certainly possible, but I don't know how to resolve it.

Re: Since Upgrading To Zimbra 8.8.5, Have Been Getting Numerous Lock Errors

Posted: Tue Jan 09, 2018 8:14 pm
by maxxer
moebius wrote:It's certainly possible, but I don't know how to resolve it.
have you tried this?

Re: Since Upgrading To Zimbra 8.8.5, Have Been Getting Numerous Lock Errors

Posted: Wed Jan 10, 2018 5:50 pm
by moebius
Yes, we already have those values increased unfortunately...

Code: Select all

$ zmprov gcf zimbraLmtpNumThreads
zimbraLmtpNumThreads: 40
$ zmprov gs `zmhostname` zimbraImapNumThreads
zimbraImapNumThreads: 500
$ zmprov gs `zmhostname` zimbraHttpNumThreads    
zimbraHttpNumThreads: 500

Re: Since Upgrading To Zimbra 8.8.5, Have Been Getting Numerous Lock Errors

Posted: Thu Jan 11, 2018 11:39 am
by skurgan
Some problem, help.
mntwinsfan wrote:I recently upgraded to Zimbra 8.8.5 from 8.7.6. Ever since I did, some users (not all) have been having issues getting into their mailbox from time to time. Through the web client, they get the login page, enter their credentials, and then the browser just spins. On their devices, it will eventually time out. If I restart Zimbra, it works fine for everyone for a while but then starts failing again. Never for all users, just a handful. I am seeing the following errors in the mailbox.log:

Code: Select all

2018-01-01 18:49:28,948 ERROR [LmtpServer-1] [name=user@domain.com;mid=16;ip=nn.nn.nn.nn;] mailbox - Failed to lock mailbox
Write Lock Owner - ImapSSLServer-1 prio=5 id=182 state=TERMINATED
Lock Waiter - Timer-Zimbra daemon prio=5 id=37 state=TIMED_WAITING
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireNanos(AbstractQueuedSynchronizer.java:934)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireNanos(AbstractQueuedSynchronizer.java:1247)
        at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.tryLock(ReentrantReadWriteLock.java:1115)
        at com.zimbra.cs.mailbox.MailboxLock.tryLockWithTimeout(MailboxLock.java:114)
        at com.zimbra.cs.mailbox.MailboxLock.lock(MailboxLock.java:194)
        at com.zimbra.cs.mailbox.Mailbox.lock(Mailbox.java:10411)
        at com.zimbra.cs.imap.ImapListener.unload(ImapListener.java:580)
        at com.zimbra.cs.imap.ImapSessionManager$SessionSerializerTask.run(ImapSessionManager.java:195)
        at java.util.TimerThread.mainLoop(Timer.java:555)
        at java.util.TimerThread.run(Timer.java:505)

com.zimbra.cs.mailbox.MailboxLock$LockFailedException: timeout
        at com.zimbra.cs.mailbox.MailboxLock.lock(MailboxLock.java:211)
        at com.zimbra.cs.mailbox.Mailbox.beginTransaction(Mailbox.java:1759)
        at com.zimbra.cs.mailbox.Mailbox.beginReadTransaction(Mailbox.java:1735)
        at com.zimbra.cs.mailbox.Mailbox.getItemById(Mailbox.java:2864)
        at com.zimbra.cs.mailbox.Mailbox.getItemById(Mailbox.java:2856)
        at com.zimbra.cs.mailbox.Mailbox.getFolderById(Mailbox.java:4092)
        at com.zimbra.cs.filter.IncomingMessageHandler.getDefaultFolderPath(IncomingMessageHandler.java:91)
        at com.zimbra.cs.filter.ZimbraMailAdapter.keep(ZimbraMailAdapter.java:526)
        at com.zimbra.cs.filter.ZimbraMailAdapter.executeActionKeepInternal(ZimbraMailAdapter.java:359)
        at com.zimbra.cs.filter.ZimbraMailAdapter.executeAllActions(ZimbraMailAdapter.java:294)
        at com.zimbra.cs.filter.RuleManager.applyRulesToIncomingMessage(RuleManager.java:440)
        at com.zimbra.cs.lmtpserver.ZimbraLmtpBackend.deliverMessageToLocalMailboxes(ZimbraLmtpBackend.java:615)
        at com.zimbra.cs.lmtpserver.ZimbraLmtpBackend.deliver(ZimbraLmtpBackend.java:385)
        at com.zimbra.cs.lmtpserver.LmtpHandler.processMessageData(LmtpHandler.java:445)
        at com.zimbra.cs.lmtpserver.TcpLmtpHandler.continueDATA(TcpLmtpHandler.java:79)
        at com.zimbra.cs.lmtpserver.LmtpHandler.doDATA(LmtpHandler.java:434)
        at com.zimbra.cs.lmtpserver.LmtpHandler.processCommand(LmtpHandler.java:216)
        at com.zimbra.cs.lmtpserver.TcpLmtpHandler.processCommand(TcpLmtpHandler.java:72)
        at com.zimbra.cs.server.ProtocolHandler.processConnection(ProtocolHandler.java:189)
        at com.zimbra.cs.server.ProtocolHandler.run(ProtocolHandler.java:128)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
2018-01-01 18:49:28,948 WARN  [LmtpServer-1] [name=user@domain.com;mid=16;ip=nn.nn.nn.nn;] mailbox - transaction canceled because of lock failure
Searching through these forums, I found some old posts regarding this error message which suggested changing some of the server settings. I did but they had no effect:

Code: Select all

zmlocalconfig -e zimbra_mailbox_lock_timeout=90
zmlocalconfig -e zimbra_mailbox_lock_max_waiting_threads=100
zmlocalconfig -e mailboxd_java_heap_size=4096
zmlocalconfig -e mailboxd_java_heap_memory_percent=25
zmprov ms `zmhostname` zimbraMessageCacheSize 4000
zmprov ms `zmhostname` zimbraImapNumThreads 50
zmprov ms `zmhostname` zimbraHttpNumThreads 600
Does anyone have any further suggestions or know how to fix this problem?

Re: Since Upgrading To Zimbra 8.8.5, Have Been Getting Numerous Lock Errors

Posted: Thu Jan 11, 2018 3:58 pm
by Klug
I'm not sure its a "session number" issue but NIO issue.

Can you check if IMAP NIO is enabled?
See this thread: viewtopic.php?f=15&t=63354

Re: Since Upgrading To Zimbra 8.8.5, Have Been Getting Numerous Lock Errors

Posted: Thu Jan 11, 2018 6:06 pm
by moebius
Enabled for me...

Code: Select all

zimbra@svr02-jls:~$ zmlocalconfig|grep nio_imap_enabled
nio_imap_enabled = true

Re: Since Upgrading To Zimbra 8.8.5, Have Been Getting Numerous Lock Errors

Posted: Wed Jan 24, 2018 12:26 pm
by bgszim
Bump?

What is the status of your lock issues. Have you solved your issues (maybe by upgrading to 8.8.6)?