system failure: getting database connection

Discuss your pilot or production implementation with other Zimbra admins or our engineers.
Post Reply
User avatar
alebeta
Posts: 12
Joined: Tue Jan 08, 2019 5:22 pm
Contact:

system failure: getting database connection

Post by alebeta »

Hello friends,

I have few months having issues with my zimbra after upgrading. So far everyday I get a 504 HTTP error and I need to fully restart with zmcontrol restart command.

And today it started showing this message when I try to send an email(so far I'm not able to send emails):

method: [unknown]
msg: system failure: getting database connection
code: service.FAILURE
detail: soap:Receiver
trace: qtp1286783232-199:1550260538015:ad737fce7408a8a5

I had been trying to find something more in the logs, but not luck so far....

I'm using Zimbra 8.8.11 running on Centos7. If more information is needed to help me to debug this problem please let me know

Thanks in advance
User avatar
alebeta
Posts: 12
Joined: Tue Jan 08, 2019 5:22 pm
Contact:

Re: system failure: getting database connection

Post by alebeta »

I got some information from logs:


com.zimbra.common.service.ServiceException: system failure: getting database connection
ExceptionId:ImapSSLServer-0:1550263229460:f7986b543fa93e3e
Code:service.FAILURE
at com.zimbra.common.service.ServiceException.FAILURE(ServiceException.java:288)
at com.zimbra.cs.db.DbPool.getConnection(DbPool.java:360)
at com.zimbra.cs.mailbox.Mailbox$MailboxChange.getConnection(Mailbox.java:445)
at com.zimbra.cs.mailbox.Mailbox.getOperationConnection(Mailbox.java:1687)
at com.zimbra.cs.db.DbMailItem.getFoldersAndTags(DbMailItem.java:2141)
at com.zimbra.cs.mailbox.Mailbox.loadFoldersAndTags(Mailbox.java:2311)
at com.zimbra.cs.mailbox.Mailbox.beginTransaction(Mailbox.java:1851)
at com.zimbra.cs.mailbox.Mailbox.beginReadTransaction(Mailbox.java:1766)
at com.zimbra.cs.mailbox.Mailbox.getItemById(Mailbox.java:2895)
at com.zimbra.cs.mailbox.Mailbox.getItemById(Mailbox.java:2887)
at com.zimbra.cs.mailbox.Mailbox.getFolderById(Mailbox.java:4178)
at com.zimbra.cs.mailbox.Mailbox.getFolderById(Mailbox.java:4226)
at com.zimbra.cs.imap.ImapSessionManager.cacheKey(ImapSessionManager.java:758)
at com.zimbra.cs.imap.ImapListener.serialize(ImapListener.java:808)
at com.zimbra.cs.imap.ImapSession.createPagedFolderData(ImapSession.java:154)
at com.zimbra.cs.imap.ImapListener.unload(ImapListener.java:590)
at com.zimbra.cs.imap.ImapSessionManager.closeFolder(ImapSessionManager.java:671)
at com.zimbra.cs.imap.ImapListener.closeFolder(ImapListener.java:409)
at com.zimbra.cs.imap.ImapHandler.unsetSelectedFolder(ImapHandler.java:1173)
at com.zimbra.cs.imap.NioImapHandler.connectionClosed(NioImapHandler.java:168)
at com.zimbra.cs.server.NioHandlerDispatcher.sessionClosed(NioHandlerDispatcher.java:79)
at org.apache.mina.core.filterchain.DefaultIoFilterChain$TailFilter.sessionClosed(DefaultIoFilterChain.java:665)
at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextSessionClosed(DefaultIoFilterChain.java:395)
at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$900(DefaultIoFilterChain.java:46)
at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.sessionClosed(DefaultIoFilterChain.java:781)
at org.apache.mina.filter.codec.ProtocolCodecFilter.sessionClosed(ProtocolCodecFilter.java:387)
at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextSessionClosed(DefaultIoFilterChain.java:395)
at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$900(DefaultIoFilterChain.java:46)
at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.sessionClosed(DefaultIoFilterChain.java:781)
at org.apache.mina.core.filterchain.IoFilterAdapter.sessionClosed(IoFilterAdapter.java:95)
at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextSessionClosed(DefaultIoFilterChain.java:395)
at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$900(DefaultIoFilterChain.java:46)
at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.sessionClosed(DefaultIoFilterChain.java:781)
at org.apache.mina.core.filterchain.IoFilterEvent.fire(IoFilterEvent.java:110)
at org.apache.mina.core.session.IoEvent.run(IoEvent.java:63)
at org.apache.mina.filter.executor.OrderedThreadPoolExecutor$Worker.runTask(OrderedThreadPoolExecutor.java:780)
at org.apache.mina.filter.executor.OrderedThreadPoolExecutor$Worker.runTasks(OrderedThreadPoolExecutor.java:772)
at org.apache.mina.filter.executor.OrderedThreadPoolExecutor$Worker.run(OrderedThreadPoolExecutor.java:714)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.sql.SQLNonTransientConnectionException: Could not connect to 127.0.0.1:7306 : Connection refused (Connection refused)
at org.mariadb.jdbc.internal.SQLExceptionMapper.get(SQLExceptionMapper.java:136)
at org.mariadb.jdbc.internal.SQLExceptionMapper.throwException(SQLExceptionMapper.java:106)
at org.mariadb.jdbc.Driver.connect(Driver.java:115)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:208)
at org.apache.commons.dbcp.DriverManagerConnectionFactory.createConnection(DriverManagerConnectionFactory.java:78)
at com.zimbra.cs.db.ZimbraConnectionFactory.createConnection(ZimbraConnectionFactory.java:67)
at org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:582)
at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1188)
at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:106)
at com.zimbra.cs.db.DbPool.getConnection(DbPool.java:341)
... 37 more
Caused by: org.mariadb.jdbc.internal.common.QueryException: Could not connect to 127.0.0.1:7306 : Connection refused (Connection refused)
at org.mariadb.jdbc.internal.mysql.MySQLProtocol.connect(MySQLProtocol.java:714)
at org.mariadb.jdbc.internal.mysql.MySQLProtocol.<init>(MySQLProtocol.java:280)
at org.mariadb.jdbc.Driver.connect(Driver.java:111)
... 45 more
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at java.net.Socket.connect(Socket.java:538)
at org.mariadb.jdbc.internal.mysql.MySQLProtocol.connect(MySQLProtocol.java:387)
at org.mariadb.jdbc.internal.mysql.MySQLProtocol.connect(MySQLProtocol.java:709)
... 47 more
User avatar
alebeta
Posts: 12
Joined: Tue Jan 08, 2019 5:22 pm
Contact:

Re: system failure: getting database connection

Post by alebeta »

mores logs mysql_error.log


EFR; InnoDB: End of page dump 2019-01-30 08:13:16 7ff40e1e1b00 InnoDB: uncompressed page, stored checksum in field1 2056531851, calculated checksums for field1: crc32 1623495103, innodb 2510201383, none 3735928559, stored checksum in field2 2264166796, calculated checksums for field2: crc32 1623495103, innodb 4165629484, none 3735928559, page LSN 0 342631507, low 4 bytes of LSN at page end 340084306, page number (if stored to page already) 10, space id (if created with >= MySQL-4.1.1 and stored already) 240 InnoDB: page type 17855 meaning INDEX InnoDB: Page may be an index page where index id is 650 InnoDB: (index "PRIMARY" of table "mboxgroup13"."open_conversation") 2019-01-30 8:13:16 140686185601792 [Note] InnoDB: It is also possible that your operating system has corrupted its own file cache and rebooting your computer removes the error. If the corrupt page is an index page. You can also try to fix the corruption by dumping, dropping, and reimporting the corrupt table. You can use CHECK TABLE to scan your table for corruption. Please refer to http://dev.mysql.com/doc/refman/5.6/en/ ... overy.html for information about forcing recovery. 2019-01-30 8:13:16 140686185601792 [Note] InnoDB: Log sequence number at the start 342631507 and the end 340084306 do not match. 2019-01-30 8:13:16 140686185601792 [ERROR] InnoDB: Database page corruption on disk or a failed file read of tablespace mboxgroup13/open_conversation page [page id: space=240, page number=10]. You may have to recover from a backup. 2019-01-30 08:13:16 7ff40e1e1b00 InnoDB: Page dump in ascii and hex (16384 bytes): len 16384; hex 7a942f8b0000000a0000002e0000000d00000000146c245345bf0000000000000000000000f0001d3f34812609bb1bee15fa0005000000a200000000000000000000000000000000028a0000000000000000000000000000000000000000010002001b696e66696d756d0003000b000073757072656d756d1c0000103c960000000d5a7044706445736276794c66377150656d7270446868387374596f3d00000007da0a220000020c130c0000082a1c0000181f270000000d5a725a364966433648644b63462c52683155424f65384e696267593d00000007cda6df0000018f0139000002f71c0000202af80000000d5a7437334a31794968394379597a6976374e6a736e5149664945493d00000007cba8db0000018c0122000002031c0700281b120000000d612c5950495568556f6f53507564514d6a6c713376446f574434733d00000007ceb4e80000023f014c000003711c00003000370000000d6130446859686b4141466a36786b6930662b55366379774d4c65733d00000007c9aadc00000239014c000001101c000038282d0000000d6131694f705146414b306a693034314850422c6f4c614e396444343d00000007ce269f00000151014c000003321c0000401ef00000000d61325a696f5a3870533757346975464f34423376526f79556b326b3d00000007ce6ac100000179014c000003521c06004835480000000d61363932465833426d397446773635517a6862692b51497064316f3d00000007ce008c0000013e014c000003201c000050074e0000000d6147726b61555062683131556278592c4e54516f384e5839584d383d00000007cea4e0000001ce014c000003691c06005827880000000d614933684b7a4154666e61526f6f55357a54584f6268664c7848413d00000007ca86ca000
User avatar
alebeta
Posts: 12
Joined: Tue Jan 08, 2019 5:22 pm
Contact:

Re: system failure: getting database connection

Post by alebeta »

Hello friends,

Problem solved by following the official guide from zimbra https://wiki.zimbra.com/wiki/Mysql_Crash_Recovery

All the best
Post Reply