Since Upgrading To Zimbra 8.8.5, Have Been Getting Numerous Lock Errors

Ask questions about your setup or get help installing ZCS server (ZD section below).
mntwinsfan
Posts: 8
Joined: Tue Jan 02, 2018 12:15 am

Since Upgrading To Zimbra 8.8.5, Have Been Getting Numerous Lock Errors

Post by mntwinsfan »

I recently upgraded to Zimbra 8.8.5 from 8.7.6. Ever since I did, some users (not all) have been having issues getting into their mailbox from time to time. Through the web client, they get the login page, enter their credentials, and then the browser just spins. On their devices, it will eventually time out. If I restart Zimbra, it works fine for everyone for a while but then starts failing again. Never for all users, just a handful. I am seeing the following errors in the mailbox.log:

Code: Select all

2018-01-01 18:49:28,948 ERROR [LmtpServer-1] [name=user@domain.com;mid=16;ip=nn.nn.nn.nn;] mailbox - Failed to lock mailbox
Write Lock Owner - ImapSSLServer-1 prio=5 id=182 state=TERMINATED
Lock Waiter - Timer-Zimbra daemon prio=5 id=37 state=TIMED_WAITING
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireNanos(AbstractQueuedSynchronizer.java:934)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireNanos(AbstractQueuedSynchronizer.java:1247)
        at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.tryLock(ReentrantReadWriteLock.java:1115)
        at com.zimbra.cs.mailbox.MailboxLock.tryLockWithTimeout(MailboxLock.java:114)
        at com.zimbra.cs.mailbox.MailboxLock.lock(MailboxLock.java:194)
        at com.zimbra.cs.mailbox.Mailbox.lock(Mailbox.java:10411)
        at com.zimbra.cs.imap.ImapListener.unload(ImapListener.java:580)
        at com.zimbra.cs.imap.ImapSessionManager$SessionSerializerTask.run(ImapSessionManager.java:195)
        at java.util.TimerThread.mainLoop(Timer.java:555)
        at java.util.TimerThread.run(Timer.java:505)

com.zimbra.cs.mailbox.MailboxLock$LockFailedException: timeout
        at com.zimbra.cs.mailbox.MailboxLock.lock(MailboxLock.java:211)
        at com.zimbra.cs.mailbox.Mailbox.beginTransaction(Mailbox.java:1759)
        at com.zimbra.cs.mailbox.Mailbox.beginReadTransaction(Mailbox.java:1735)
        at com.zimbra.cs.mailbox.Mailbox.getItemById(Mailbox.java:2864)
        at com.zimbra.cs.mailbox.Mailbox.getItemById(Mailbox.java:2856)
        at com.zimbra.cs.mailbox.Mailbox.getFolderById(Mailbox.java:4092)
        at com.zimbra.cs.filter.IncomingMessageHandler.getDefaultFolderPath(IncomingMessageHandler.java:91)
        at com.zimbra.cs.filter.ZimbraMailAdapter.keep(ZimbraMailAdapter.java:526)
        at com.zimbra.cs.filter.ZimbraMailAdapter.executeActionKeepInternal(ZimbraMailAdapter.java:359)
        at com.zimbra.cs.filter.ZimbraMailAdapter.executeAllActions(ZimbraMailAdapter.java:294)
        at com.zimbra.cs.filter.RuleManager.applyRulesToIncomingMessage(RuleManager.java:440)
        at com.zimbra.cs.lmtpserver.ZimbraLmtpBackend.deliverMessageToLocalMailboxes(ZimbraLmtpBackend.java:615)
        at com.zimbra.cs.lmtpserver.ZimbraLmtpBackend.deliver(ZimbraLmtpBackend.java:385)
        at com.zimbra.cs.lmtpserver.LmtpHandler.processMessageData(LmtpHandler.java:445)
        at com.zimbra.cs.lmtpserver.TcpLmtpHandler.continueDATA(TcpLmtpHandler.java:79)
        at com.zimbra.cs.lmtpserver.LmtpHandler.doDATA(LmtpHandler.java:434)
        at com.zimbra.cs.lmtpserver.LmtpHandler.processCommand(LmtpHandler.java:216)
        at com.zimbra.cs.lmtpserver.TcpLmtpHandler.processCommand(TcpLmtpHandler.java:72)
        at com.zimbra.cs.server.ProtocolHandler.processConnection(ProtocolHandler.java:189)
        at com.zimbra.cs.server.ProtocolHandler.run(ProtocolHandler.java:128)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
2018-01-01 18:49:28,948 WARN  [LmtpServer-1] [name=user@domain.com;mid=16;ip=nn.nn.nn.nn;] mailbox - transaction canceled because of lock failure
Searching through these forums, I found some old posts regarding this error message which suggested changing some of the server settings. I did but they had no effect:

Code: Select all

zmlocalconfig -e zimbra_mailbox_lock_timeout=90
zmlocalconfig -e zimbra_mailbox_lock_max_waiting_threads=100
zmlocalconfig -e mailboxd_java_heap_size=4096
zmlocalconfig -e mailboxd_java_heap_memory_percent=25
zmprov ms `zmhostname` zimbraMessageCacheSize 4000
zmprov ms `zmhostname` zimbraImapNumThreads 50
zmprov ms `zmhostname` zimbraHttpNumThreads 600
Does anyone have any further suggestions or know how to fix this problem?
moebius
Posts: 18
Joined: Sat Sep 13, 2014 3:34 am

Re: Since Upgrading To Zimbra 8.8.5, Have Been Getting Numerous Lock Errors

Post by moebius »

I am also experiencing mailbox locking issues since 8.8.5 and have to restart the mailboxd service to clear them. zmmailboxctl kill does not work, I have to kill the PID. Seems to pop up every 2 or 3 days.
User avatar
maxxer
Outstanding Member
Outstanding Member
Posts: 224
Joined: Fri Oct 04, 2013 2:12 am
Contact:

Re: Since Upgrading To Zimbra 8.8.5, Have Been Getting Numerous Lock Errors

Post by maxxer »

Someone here also suggests increasing http and imap threads.

From other discussions I learnt it could be bond to IMAP usage, especially with Apple clients. Is this your case?
moebius
Posts: 18
Joined: Sat Sep 13, 2014 3:34 am

Re: Since Upgrading To Zimbra 8.8.5, Have Been Getting Numerous Lock Errors

Post by moebius »

It's certainly possible, but I don't know how to resolve it.
User avatar
maxxer
Outstanding Member
Outstanding Member
Posts: 224
Joined: Fri Oct 04, 2013 2:12 am
Contact:

Re: Since Upgrading To Zimbra 8.8.5, Have Been Getting Numerous Lock Errors

Post by maxxer »

moebius wrote:It's certainly possible, but I don't know how to resolve it.
have you tried this?
moebius
Posts: 18
Joined: Sat Sep 13, 2014 3:34 am

Re: Since Upgrading To Zimbra 8.8.5, Have Been Getting Numerous Lock Errors

Post by moebius »

Yes, we already have those values increased unfortunately...

Code: Select all

$ zmprov gcf zimbraLmtpNumThreads
zimbraLmtpNumThreads: 40
$ zmprov gs `zmhostname` zimbraImapNumThreads
zimbraImapNumThreads: 500
$ zmprov gs `zmhostname` zimbraHttpNumThreads    
zimbraHttpNumThreads: 500
skurgan
Posts: 1
Joined: Thu Jan 11, 2018 10:50 am

Re: Since Upgrading To Zimbra 8.8.5, Have Been Getting Numerous Lock Errors

Post by skurgan »

Some problem, help.
mntwinsfan wrote:I recently upgraded to Zimbra 8.8.5 from 8.7.6. Ever since I did, some users (not all) have been having issues getting into their mailbox from time to time. Through the web client, they get the login page, enter their credentials, and then the browser just spins. On their devices, it will eventually time out. If I restart Zimbra, it works fine for everyone for a while but then starts failing again. Never for all users, just a handful. I am seeing the following errors in the mailbox.log:

Code: Select all

2018-01-01 18:49:28,948 ERROR [LmtpServer-1] [name=user@domain.com;mid=16;ip=nn.nn.nn.nn;] mailbox - Failed to lock mailbox
Write Lock Owner - ImapSSLServer-1 prio=5 id=182 state=TERMINATED
Lock Waiter - Timer-Zimbra daemon prio=5 id=37 state=TIMED_WAITING
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireNanos(AbstractQueuedSynchronizer.java:934)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireNanos(AbstractQueuedSynchronizer.java:1247)
        at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.tryLock(ReentrantReadWriteLock.java:1115)
        at com.zimbra.cs.mailbox.MailboxLock.tryLockWithTimeout(MailboxLock.java:114)
        at com.zimbra.cs.mailbox.MailboxLock.lock(MailboxLock.java:194)
        at com.zimbra.cs.mailbox.Mailbox.lock(Mailbox.java:10411)
        at com.zimbra.cs.imap.ImapListener.unload(ImapListener.java:580)
        at com.zimbra.cs.imap.ImapSessionManager$SessionSerializerTask.run(ImapSessionManager.java:195)
        at java.util.TimerThread.mainLoop(Timer.java:555)
        at java.util.TimerThread.run(Timer.java:505)

com.zimbra.cs.mailbox.MailboxLock$LockFailedException: timeout
        at com.zimbra.cs.mailbox.MailboxLock.lock(MailboxLock.java:211)
        at com.zimbra.cs.mailbox.Mailbox.beginTransaction(Mailbox.java:1759)
        at com.zimbra.cs.mailbox.Mailbox.beginReadTransaction(Mailbox.java:1735)
        at com.zimbra.cs.mailbox.Mailbox.getItemById(Mailbox.java:2864)
        at com.zimbra.cs.mailbox.Mailbox.getItemById(Mailbox.java:2856)
        at com.zimbra.cs.mailbox.Mailbox.getFolderById(Mailbox.java:4092)
        at com.zimbra.cs.filter.IncomingMessageHandler.getDefaultFolderPath(IncomingMessageHandler.java:91)
        at com.zimbra.cs.filter.ZimbraMailAdapter.keep(ZimbraMailAdapter.java:526)
        at com.zimbra.cs.filter.ZimbraMailAdapter.executeActionKeepInternal(ZimbraMailAdapter.java:359)
        at com.zimbra.cs.filter.ZimbraMailAdapter.executeAllActions(ZimbraMailAdapter.java:294)
        at com.zimbra.cs.filter.RuleManager.applyRulesToIncomingMessage(RuleManager.java:440)
        at com.zimbra.cs.lmtpserver.ZimbraLmtpBackend.deliverMessageToLocalMailboxes(ZimbraLmtpBackend.java:615)
        at com.zimbra.cs.lmtpserver.ZimbraLmtpBackend.deliver(ZimbraLmtpBackend.java:385)
        at com.zimbra.cs.lmtpserver.LmtpHandler.processMessageData(LmtpHandler.java:445)
        at com.zimbra.cs.lmtpserver.TcpLmtpHandler.continueDATA(TcpLmtpHandler.java:79)
        at com.zimbra.cs.lmtpserver.LmtpHandler.doDATA(LmtpHandler.java:434)
        at com.zimbra.cs.lmtpserver.LmtpHandler.processCommand(LmtpHandler.java:216)
        at com.zimbra.cs.lmtpserver.TcpLmtpHandler.processCommand(TcpLmtpHandler.java:72)
        at com.zimbra.cs.server.ProtocolHandler.processConnection(ProtocolHandler.java:189)
        at com.zimbra.cs.server.ProtocolHandler.run(ProtocolHandler.java:128)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
2018-01-01 18:49:28,948 WARN  [LmtpServer-1] [name=user@domain.com;mid=16;ip=nn.nn.nn.nn;] mailbox - transaction canceled because of lock failure
Searching through these forums, I found some old posts regarding this error message which suggested changing some of the server settings. I did but they had no effect:

Code: Select all

zmlocalconfig -e zimbra_mailbox_lock_timeout=90
zmlocalconfig -e zimbra_mailbox_lock_max_waiting_threads=100
zmlocalconfig -e mailboxd_java_heap_size=4096
zmlocalconfig -e mailboxd_java_heap_memory_percent=25
zmprov ms `zmhostname` zimbraMessageCacheSize 4000
zmprov ms `zmhostname` zimbraImapNumThreads 50
zmprov ms `zmhostname` zimbraHttpNumThreads 600
Does anyone have any further suggestions or know how to fix this problem?
Klug
Ambassador
Ambassador
Posts: 2746
Joined: Mon Dec 16, 2013 11:35 am
Location: France - Drôme
ZCS/ZD Version: All of them
Contact:

Re: Since Upgrading To Zimbra 8.8.5, Have Been Getting Numerous Lock Errors

Post by Klug »

I'm not sure its a "session number" issue but NIO issue.

Can you check if IMAP NIO is enabled?
See this thread: viewtopic.php?f=15&t=63354
moebius
Posts: 18
Joined: Sat Sep 13, 2014 3:34 am

Re: Since Upgrading To Zimbra 8.8.5, Have Been Getting Numerous Lock Errors

Post by moebius »

Enabled for me...

Code: Select all

zimbra@svr02-jls:~$ zmlocalconfig|grep nio_imap_enabled
nio_imap_enabled = true
bgszim
Posts: 30
Joined: Sat Sep 13, 2014 1:50 am

Re: Since Upgrading To Zimbra 8.8.5, Have Been Getting Numerous Lock Errors

Post by bgszim »

Bump?

What is the status of your lock issues. Have you solved your issues (maybe by upgrading to 8.8.6)?
Post Reply