Hello friends,
I have few months having issues with my zimbra after upgrading. So far everyday I get a 504 HTTP error and I need to fully restart with zmcontrol restart command.
And today it started showing this message when I try to send an email(so far I'm not able to send emails):
method: [unknown]
msg: system failure: getting database connection
code: service.FAILURE
detail: soap:Receiver
trace: qtp1286783232-199:1550260538015:ad737fce7408a8a5
I had been trying to find something more in the logs, but not luck so far....
I'm using Zimbra 8.8.11 running on Centos7. If more information is needed to help me to debug this problem please let me know
Thanks in advance
system failure: getting database connection
Re: system failure: getting database connection
I got some information from logs:
com.zimbra.common.service.ServiceException: system failure: getting database connection
ExceptionId:ImapSSLServer-0:1550263229460:f7986b543fa93e3e
Code:service.FAILURE
at com.zimbra.common.service.ServiceException.FAILURE(ServiceException.java:288)
at com.zimbra.cs.db.DbPool.getConnection(DbPool.java:360)
at com.zimbra.cs.mailbox.Mailbox$MailboxChange.getConnection(Mailbox.java:445)
at com.zimbra.cs.mailbox.Mailbox.getOperationConnection(Mailbox.java:1687)
at com.zimbra.cs.db.DbMailItem.getFoldersAndTags(DbMailItem.java:2141)
at com.zimbra.cs.mailbox.Mailbox.loadFoldersAndTags(Mailbox.java:2311)
at com.zimbra.cs.mailbox.Mailbox.beginTransaction(Mailbox.java:1851)
at com.zimbra.cs.mailbox.Mailbox.beginReadTransaction(Mailbox.java:1766)
at com.zimbra.cs.mailbox.Mailbox.getItemById(Mailbox.java:2895)
at com.zimbra.cs.mailbox.Mailbox.getItemById(Mailbox.java:2887)
at com.zimbra.cs.mailbox.Mailbox.getFolderById(Mailbox.java:4178)
at com.zimbra.cs.mailbox.Mailbox.getFolderById(Mailbox.java:4226)
at com.zimbra.cs.imap.ImapSessionManager.cacheKey(ImapSessionManager.java:758)
at com.zimbra.cs.imap.ImapListener.serialize(ImapListener.java:808)
at com.zimbra.cs.imap.ImapSession.createPagedFolderData(ImapSession.java:154)
at com.zimbra.cs.imap.ImapListener.unload(ImapListener.java:590)
at com.zimbra.cs.imap.ImapSessionManager.closeFolder(ImapSessionManager.java:671)
at com.zimbra.cs.imap.ImapListener.closeFolder(ImapListener.java:409)
at com.zimbra.cs.imap.ImapHandler.unsetSelectedFolder(ImapHandler.java:1173)
at com.zimbra.cs.imap.NioImapHandler.connectionClosed(NioImapHandler.java:168)
at com.zimbra.cs.server.NioHandlerDispatcher.sessionClosed(NioHandlerDispatcher.java:79)
at org.apache.mina.core.filterchain.DefaultIoFilterChain$TailFilter.sessionClosed(DefaultIoFilterChain.java:665)
at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextSessionClosed(DefaultIoFilterChain.java:395)
at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$900(DefaultIoFilterChain.java:46)
at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.sessionClosed(DefaultIoFilterChain.java:781)
at org.apache.mina.filter.codec.ProtocolCodecFilter.sessionClosed(ProtocolCodecFilter.java:387)
at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextSessionClosed(DefaultIoFilterChain.java:395)
at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$900(DefaultIoFilterChain.java:46)
at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.sessionClosed(DefaultIoFilterChain.java:781)
at org.apache.mina.core.filterchain.IoFilterAdapter.sessionClosed(IoFilterAdapter.java:95)
at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextSessionClosed(DefaultIoFilterChain.java:395)
at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$900(DefaultIoFilterChain.java:46)
at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.sessionClosed(DefaultIoFilterChain.java:781)
at org.apache.mina.core.filterchain.IoFilterEvent.fire(IoFilterEvent.java:110)
at org.apache.mina.core.session.IoEvent.run(IoEvent.java:63)
at org.apache.mina.filter.executor.OrderedThreadPoolExecutor$Worker.runTask(OrderedThreadPoolExecutor.java:780)
at org.apache.mina.filter.executor.OrderedThreadPoolExecutor$Worker.runTasks(OrderedThreadPoolExecutor.java:772)
at org.apache.mina.filter.executor.OrderedThreadPoolExecutor$Worker.run(OrderedThreadPoolExecutor.java:714)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.sql.SQLNonTransientConnectionException: Could not connect to 127.0.0.1:7306 : Connection refused (Connection refused)
at org.mariadb.jdbc.internal.SQLExceptionMapper.get(SQLExceptionMapper.java:136)
at org.mariadb.jdbc.internal.SQLExceptionMapper.throwException(SQLExceptionMapper.java:106)
at org.mariadb.jdbc.Driver.connect(Driver.java:115)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:208)
at org.apache.commons.dbcp.DriverManagerConnectionFactory.createConnection(DriverManagerConnectionFactory.java:78)
at com.zimbra.cs.db.ZimbraConnectionFactory.createConnection(ZimbraConnectionFactory.java:67)
at org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:582)
at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1188)
at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:106)
at com.zimbra.cs.db.DbPool.getConnection(DbPool.java:341)
... 37 more
Caused by: org.mariadb.jdbc.internal.common.QueryException: Could not connect to 127.0.0.1:7306 : Connection refused (Connection refused)
at org.mariadb.jdbc.internal.mysql.MySQLProtocol.connect(MySQLProtocol.java:714)
at org.mariadb.jdbc.internal.mysql.MySQLProtocol.<init>(MySQLProtocol.java:280)
at org.mariadb.jdbc.Driver.connect(Driver.java:111)
... 45 more
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at java.net.Socket.connect(Socket.java:538)
at org.mariadb.jdbc.internal.mysql.MySQLProtocol.connect(MySQLProtocol.java:387)
at org.mariadb.jdbc.internal.mysql.MySQLProtocol.connect(MySQLProtocol.java:709)
... 47 more
com.zimbra.common.service.ServiceException: system failure: getting database connection
ExceptionId:ImapSSLServer-0:1550263229460:f7986b543fa93e3e
Code:service.FAILURE
at com.zimbra.common.service.ServiceException.FAILURE(ServiceException.java:288)
at com.zimbra.cs.db.DbPool.getConnection(DbPool.java:360)
at com.zimbra.cs.mailbox.Mailbox$MailboxChange.getConnection(Mailbox.java:445)
at com.zimbra.cs.mailbox.Mailbox.getOperationConnection(Mailbox.java:1687)
at com.zimbra.cs.db.DbMailItem.getFoldersAndTags(DbMailItem.java:2141)
at com.zimbra.cs.mailbox.Mailbox.loadFoldersAndTags(Mailbox.java:2311)
at com.zimbra.cs.mailbox.Mailbox.beginTransaction(Mailbox.java:1851)
at com.zimbra.cs.mailbox.Mailbox.beginReadTransaction(Mailbox.java:1766)
at com.zimbra.cs.mailbox.Mailbox.getItemById(Mailbox.java:2895)
at com.zimbra.cs.mailbox.Mailbox.getItemById(Mailbox.java:2887)
at com.zimbra.cs.mailbox.Mailbox.getFolderById(Mailbox.java:4178)
at com.zimbra.cs.mailbox.Mailbox.getFolderById(Mailbox.java:4226)
at com.zimbra.cs.imap.ImapSessionManager.cacheKey(ImapSessionManager.java:758)
at com.zimbra.cs.imap.ImapListener.serialize(ImapListener.java:808)
at com.zimbra.cs.imap.ImapSession.createPagedFolderData(ImapSession.java:154)
at com.zimbra.cs.imap.ImapListener.unload(ImapListener.java:590)
at com.zimbra.cs.imap.ImapSessionManager.closeFolder(ImapSessionManager.java:671)
at com.zimbra.cs.imap.ImapListener.closeFolder(ImapListener.java:409)
at com.zimbra.cs.imap.ImapHandler.unsetSelectedFolder(ImapHandler.java:1173)
at com.zimbra.cs.imap.NioImapHandler.connectionClosed(NioImapHandler.java:168)
at com.zimbra.cs.server.NioHandlerDispatcher.sessionClosed(NioHandlerDispatcher.java:79)
at org.apache.mina.core.filterchain.DefaultIoFilterChain$TailFilter.sessionClosed(DefaultIoFilterChain.java:665)
at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextSessionClosed(DefaultIoFilterChain.java:395)
at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$900(DefaultIoFilterChain.java:46)
at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.sessionClosed(DefaultIoFilterChain.java:781)
at org.apache.mina.filter.codec.ProtocolCodecFilter.sessionClosed(ProtocolCodecFilter.java:387)
at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextSessionClosed(DefaultIoFilterChain.java:395)
at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$900(DefaultIoFilterChain.java:46)
at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.sessionClosed(DefaultIoFilterChain.java:781)
at org.apache.mina.core.filterchain.IoFilterAdapter.sessionClosed(IoFilterAdapter.java:95)
at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextSessionClosed(DefaultIoFilterChain.java:395)
at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$900(DefaultIoFilterChain.java:46)
at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.sessionClosed(DefaultIoFilterChain.java:781)
at org.apache.mina.core.filterchain.IoFilterEvent.fire(IoFilterEvent.java:110)
at org.apache.mina.core.session.IoEvent.run(IoEvent.java:63)
at org.apache.mina.filter.executor.OrderedThreadPoolExecutor$Worker.runTask(OrderedThreadPoolExecutor.java:780)
at org.apache.mina.filter.executor.OrderedThreadPoolExecutor$Worker.runTasks(OrderedThreadPoolExecutor.java:772)
at org.apache.mina.filter.executor.OrderedThreadPoolExecutor$Worker.run(OrderedThreadPoolExecutor.java:714)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.sql.SQLNonTransientConnectionException: Could not connect to 127.0.0.1:7306 : Connection refused (Connection refused)
at org.mariadb.jdbc.internal.SQLExceptionMapper.get(SQLExceptionMapper.java:136)
at org.mariadb.jdbc.internal.SQLExceptionMapper.throwException(SQLExceptionMapper.java:106)
at org.mariadb.jdbc.Driver.connect(Driver.java:115)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:208)
at org.apache.commons.dbcp.DriverManagerConnectionFactory.createConnection(DriverManagerConnectionFactory.java:78)
at com.zimbra.cs.db.ZimbraConnectionFactory.createConnection(ZimbraConnectionFactory.java:67)
at org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:582)
at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1188)
at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:106)
at com.zimbra.cs.db.DbPool.getConnection(DbPool.java:341)
... 37 more
Caused by: org.mariadb.jdbc.internal.common.QueryException: Could not connect to 127.0.0.1:7306 : Connection refused (Connection refused)
at org.mariadb.jdbc.internal.mysql.MySQLProtocol.connect(MySQLProtocol.java:714)
at org.mariadb.jdbc.internal.mysql.MySQLProtocol.<init>(MySQLProtocol.java:280)
at org.mariadb.jdbc.Driver.connect(Driver.java:111)
... 45 more
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at java.net.Socket.connect(Socket.java:538)
at org.mariadb.jdbc.internal.mysql.MySQLProtocol.connect(MySQLProtocol.java:387)
at org.mariadb.jdbc.internal.mysql.MySQLProtocol.connect(MySQLProtocol.java:709)
... 47 more
Re: system failure: getting database connection
mores logs mysql_error.log
EFR; InnoDB: End of page dump 2019-01-30 08:13:16 7ff40e1e1b00 InnoDB: uncompressed page, stored checksum in field1 2056531851, calculated checksums for field1: crc32 1623495103, innodb 2510201383, none 3735928559, stored checksum in field2 2264166796, calculated checksums for field2: crc32 1623495103, innodb 4165629484, none 3735928559, page LSN 0 342631507, low 4 bytes of LSN at page end 340084306, page number (if stored to page already) 10, space id (if created with >= MySQL-4.1.1 and stored already) 240 InnoDB: page type 17855 meaning INDEX InnoDB: Page may be an index page where index id is 650 InnoDB: (index "PRIMARY" of table "mboxgroup13"."open_conversation") 2019-01-30 8:13:16 140686185601792 [Note] InnoDB: It is also possible that your operating system has corrupted its own file cache and rebooting your computer removes the error. If the corrupt page is an index page. You can also try to fix the corruption by dumping, dropping, and reimporting the corrupt table. You can use CHECK TABLE to scan your table for corruption. Please refer to http://dev.mysql.com/doc/refman/5.6/en/ ... overy.html for information about forcing recovery. 2019-01-30 8:13:16 140686185601792 [Note] InnoDB: Log sequence number at the start 342631507 and the end 340084306 do not match. 2019-01-30 8:13:16 140686185601792 [ERROR] InnoDB: Database page corruption on disk or a failed file read of tablespace mboxgroup13/open_conversation page [page id: space=240, page number=10]. You may have to recover from a backup. 2019-01-30 08:13:16 7ff40e1e1b00 InnoDB: Page dump in ascii and hex (16384 bytes): len 16384; hex 7a942f8b0000000a0000002e0000000d00000000146c245345bf0000000000000000000000f0001d3f34812609bb1bee15fa0005000000a200000000000000000000000000000000028a0000000000000000000000000000000000000000010002001b696e66696d756d0003000b000073757072656d756d1c0000103c960000000d5a7044706445736276794c66377150656d7270446868387374596f3d00000007da0a220000020c130c0000082a1c0000181f270000000d5a725a364966433648644b63462c52683155424f65384e696267593d00000007cda6df0000018f0139000002f71c0000202af80000000d5a7437334a31794968394379597a6976374e6a736e5149664945493d00000007cba8db0000018c0122000002031c0700281b120000000d612c5950495568556f6f53507564514d6a6c713376446f574434733d00000007ceb4e80000023f014c000003711c00003000370000000d6130446859686b4141466a36786b6930662b55366379774d4c65733d00000007c9aadc00000239014c000001101c000038282d0000000d6131694f705146414b306a693034314850422c6f4c614e396444343d00000007ce269f00000151014c000003321c0000401ef00000000d61325a696f5a3870533757346975464f34423376526f79556b326b3d00000007ce6ac100000179014c000003521c06004835480000000d61363932465833426d397446773635517a6862692b51497064316f3d00000007ce008c0000013e014c000003201c000050074e0000000d6147726b61555062683131556278592c4e54516f384e5839584d383d00000007cea4e0000001ce014c000003691c06005827880000000d614933684b7a4154666e61526f6f55357a54584f6268664c7848413d00000007ca86ca000
EFR; InnoDB: End of page dump 2019-01-30 08:13:16 7ff40e1e1b00 InnoDB: uncompressed page, stored checksum in field1 2056531851, calculated checksums for field1: crc32 1623495103, innodb 2510201383, none 3735928559, stored checksum in field2 2264166796, calculated checksums for field2: crc32 1623495103, innodb 4165629484, none 3735928559, page LSN 0 342631507, low 4 bytes of LSN at page end 340084306, page number (if stored to page already) 10, space id (if created with >= MySQL-4.1.1 and stored already) 240 InnoDB: page type 17855 meaning INDEX InnoDB: Page may be an index page where index id is 650 InnoDB: (index "PRIMARY" of table "mboxgroup13"."open_conversation") 2019-01-30 8:13:16 140686185601792 [Note] InnoDB: It is also possible that your operating system has corrupted its own file cache and rebooting your computer removes the error. If the corrupt page is an index page. You can also try to fix the corruption by dumping, dropping, and reimporting the corrupt table. You can use CHECK TABLE to scan your table for corruption. Please refer to http://dev.mysql.com/doc/refman/5.6/en/ ... overy.html for information about forcing recovery. 2019-01-30 8:13:16 140686185601792 [Note] InnoDB: Log sequence number at the start 342631507 and the end 340084306 do not match. 2019-01-30 8:13:16 140686185601792 [ERROR] InnoDB: Database page corruption on disk or a failed file read of tablespace mboxgroup13/open_conversation page [page id: space=240, page number=10]. You may have to recover from a backup. 2019-01-30 08:13:16 7ff40e1e1b00 InnoDB: Page dump in ascii and hex (16384 bytes): len 16384; hex 7a942f8b0000000a0000002e0000000d00000000146c245345bf0000000000000000000000f0001d3f34812609bb1bee15fa0005000000a200000000000000000000000000000000028a0000000000000000000000000000000000000000010002001b696e66696d756d0003000b000073757072656d756d1c0000103c960000000d5a7044706445736276794c66377150656d7270446868387374596f3d00000007da0a220000020c130c0000082a1c0000181f270000000d5a725a364966433648644b63462c52683155424f65384e696267593d00000007cda6df0000018f0139000002f71c0000202af80000000d5a7437334a31794968394379597a6976374e6a736e5149664945493d00000007cba8db0000018c0122000002031c0700281b120000000d612c5950495568556f6f53507564514d6a6c713376446f574434733d00000007ceb4e80000023f014c000003711c00003000370000000d6130446859686b4141466a36786b6930662b55366379774d4c65733d00000007c9aadc00000239014c000001101c000038282d0000000d6131694f705146414b306a693034314850422c6f4c614e396444343d00000007ce269f00000151014c000003321c0000401ef00000000d61325a696f5a3870533757346975464f34423376526f79556b326b3d00000007ce6ac100000179014c000003521c06004835480000000d61363932465833426d397446773635517a6862692b51497064316f3d00000007ce008c0000013e014c000003201c000050074e0000000d6147726b61555062683131556278592c4e54516f384e5839584d383d00000007cea4e0000001ce014c000003691c06005827880000000d614933684b7a4154666e61526f6f55357a54584f6268664c7848413d00000007ca86ca000
Re: system failure: getting database connection
Hello friends,
Problem solved by following the official guide from zimbra https://wiki.zimbra.com/wiki/Mysql_Crash_Recovery
All the best
Problem solved by following the official guide from zimbra https://wiki.zimbra.com/wiki/Mysql_Crash_Recovery
All the best