Backup Failure after LDAP patch. (Resolved)
Posted: Wed Sep 05, 2018 7:20 am
Good day everyone.
First of all, thank you to Zimbra staff's inability to document patch-instructions properly. It Would be nice to get an official PATCH document stating that the SSH KEYS needs to be re-created when you release LDAP patches.
(I am documenting the problem for future reference for anyone to find the fix using any search engine.) - See FIX and NOTE at end of writing.
Details:
On the 21 August 2018 a patch was released that I applied with the YUM command on Centos. After this patch Zimbra backups started failing (I am running WebMin as a front-end on the system)
Patch notification:
Surely this should be documented? It should be PART of the PATCH instructions when you update LDAP to recreate the KEYS, or am I missing something?
FIX:
To fix the error was quite easy, but to find the fix was VERY DIFFICULT!
NOTE: Once this was resolved, all SSH KEYS of remote systems using LDAP had to be destroyed and recreated.
Kind regards
Aubrey Kloppers (aka cyber7) Cape Town, South Africa
First of all, thank you to Zimbra staff's inability to document patch-instructions properly. It Would be nice to get an official PATCH document stating that the SSH KEYS needs to be re-created when you release LDAP patches.
(I am documenting the problem for future reference for anyone to find the fix using any search engine.) - See FIX and NOTE at end of writing.
Details:
On the 21 August 2018 a patch was released that I applied with the YUM command on Centos. After this patch Zimbra backups started failing (I am running WebMin as a front-end on the system)
Patch notification:
Error Message Received:An update to audit from 2.8.1-3.el7 to 2.8.1-3.el7_5.1 is available.
An update to audit-libs from 2.8.1-3.el7 to 2.8.1-3.el7_5.1 is available.
An update to dracut from 033-535.el7 to 033-535.el7_5.1 is available.
An update to dracut-config-rescue from 033-535.el7 to 033-535.el7_5.1 is available.
An update to dracut-network from 033-535.el7 to 033-535.el7_5.1 is available.
An update to initscripts from 9.49.41-1.el7 to 9.49.41-1.el7_5.1 is available.
An update to kpartx from 0.4.9-119.el7 to 0.4.9-119.el7_5.1 is available.
An update to libblkid from 2.23.2-52.el7 to 2.23.2-52.el7_5.1 is available.
An update to libgudev1 from 219-57.el7 to 219-57.el7_5.1 is available.
An update to libmount from 2.23.2-52.el7 to 2.23.2-52.el7_5.1 is available.
An update to libuuid from 2.23.2-52.el7 to 2.23.2-52.el7_5.1 is available.
An update to mariadb-libs from 5.5.56-2.el7 to 5.5.60-1.el7_5 is available.
An update to selinux-policy from 3.13.1-192.el7_5.4 to 3.13.1-192.el7_5.6 is available.
An update to selinux-policy-targeted from 3.13.1-192.el7_5.4 to 3.13.1-192.el7_5.6 is available.
An update to systemd from 219-57.el7 to 219-57.el7_5.1 is available.
An update to systemd-libs from 219-57.el7 to 219-57.el7_5.1 is available.
An update to systemd-sysv from 219-57.el7 to 219-57.el7_5.1 is available.
An update to tuned from 2.9.0-1.el7 to 2.9.0-1.el7_5.2 is available.
An update to util-linux from 2.23.2-52.el7 to 2.23.2-52.el7_5.1 is available.
An update to zimbra-common-core-jar from 1.0.0.1531216364-1.r7 to 2.0.0.1533843772-1.r7 is available.
An update to zimbra-ldap-components from 1.0.0-1zimbra8.7b1.el7 to 1.0.1-1zimbra8.7b1.el7 is available.
An update to zimbra-lmdb from 2.4.44-1zimbra8.7b9.el7 to 2.4.46-1zimbra8.7b2.el7 is available.
An update to zimbra-lmdb-libs from 2.4.44-1zimbra8.7b9.el7 to 2.4.46-1zimbra8.7b2.el7 is available.
An update to zimbra-mbox-webclient-war from 1.0.0.1531295071-1.r7 to 2.0.0.1533844076-1.r7 is available.
An update to zimbra-network-modules-ng from 2.0.2.1532358202-1.r7 to 2.0.3.1533551703-1.r7 is available.
An update to zimbra-openldap-client from 2.4.44-1zimbra8.7b9.el7 to 2.4.46-1zimbra8.7b2.el7 is available.
An update to zimbra-openldap-libs from 2.4.44-1zimbra8.7b9.el7 to 2.4.46-1zimbra8.7b2.el7 is available.
An update to zimbra-openldap-server from 2.4.44-1zimbra8.7b9.el7 to 2.4.46-1zimbra8.7b2.el7 is available.
An update to zimbra-patch from 8.8.9.1531484537.p1-1.r7 to 8.8.9.1533882487.p3-1.r7 is available.
Updates can be installed at http://example.com:10000/package-updates/
After digging for days I finally found a single mail talking about re-creating the SSH KEYS on a LDAP failure.Server: example.com
Label: incr-20180822.140023.748
Type: incremental
Status: completed (with errors)
Started: Wed, 2018/08/22 16:00:23.748 SAST
Ended: Wed, 2018/08/22 16:01:24.699 SAST
Redo log sequence range: 971 .. 972
Number of accounts: 349
Number of errors: 1
ERRORS
system: system failure: LDAP backup failed: system failure: exception executing command: zmbackupldap --outdir /opt/zimbra/backup/tmp/incr-20180822.140023.748/ldap --zip with {RemoteManager: example.com->zimbra@example.com:22}
com.zimbra.common.service.ServiceException: system failure: LDAP backup failed: system failure: exception executing command: zmbackupldap --outdir /opt/zimbra/backup/tmp/incr-20180822.140023.748/ldap --zip with {RemoteManager: example.com->example.com:22}
ExceptionId:qtp1286783232-58103:https:https://localhost:7071/service/admin/soap/BackupRequest:1534946484654:3d70be960262890c
Code:service.FAILURE
at com.zimbra.common.service.ServiceException.FAILURE(ServiceException.java:288)
at com.zimbra.cs.backup.FileBackupTarget$FileBackupSet.backupLdap(FileBackupTarget.java:1474)
at com.zimbra.cs.backup.FileBackupTarget$FileBackupSet.startIncrementalBackup(FileBackupTarget.java:1069)
at com.zimbra.cs.backup.BackupManager.backupIncremental(BackupManager.java:336)
at com.zimbra.cs.service.backup.Backup.handleNetworkRequest(Backup.java:153)
at com.zimbra.cs.service.NetworkDocumentHandler.handle(NetworkDocumentHandler.java:23)
at com.zimbra.soap.SoapEngine.dispatchRequest(SoapEngine.java:643)
at com.zimbra.soap.SoapEngine.dispatch(SoapEngine.java:488)
at com.zimbra.soap.SoapEngine.dispatch(SoapEngine.java:275)
at com.zimbra.soap.SoapServlet.doWork(SoapServlet.java:304)
at com.zimbra.soap.SoapServlet.doPost(SoapServlet.java:214)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at com.zimbra.cs.servlet.ZimbraServlet.service(ZimbraServlet.java:211)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:821)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1685)
at com.zimbra.cs.servlet.CsrfFilter.doFilter(CsrfFilter.java:169)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
at com.zimbra.cs.servlet.RequestStringFilter.doFilter(RequestStringFilter.java:54)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
at com.zimbra.cs.servlet.SetHeaderFilter.doFilter(SetHeaderFilter.java:59)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
at com.zimbra.cs.servlet.ETagHeaderFilter.doFilter(ETagHeaderFilter.java:47)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
at com.zimbra.cs.servlet.ContextPathBasedThreadPoolBalancerFilter.doFilter(ContextPathBasedThreadPoolBalancerFilter.java:107)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
at com.zimbra.cs.servlet.ZimbraQoSFilter.doFilter(ZimbraQoSFilter.java:116)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
at com.zimbra.cs.servlet.ZimbraInvalidLoginFilter.doFilter(ZimbraInvalidLoginFilter.java:117)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
at org.eclipse.jetty.servlets.DoSFilter.doFilterChain(DoSFilter.java:473)
at org.eclipse.jetty.servlets.DoSFilter.doFilter(DoSFilter.java:318)
at org.eclipse.jetty.servlets.DoSFilter.doFilter(DoSFilter.java:288)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1158)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1090)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:109)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:119)
at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:318)
at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:437)
at org.eclipse.jetty.server.handler.DebugHandler.handle(DebugHandler.java:84)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:119)
at org.eclipse.jetty.server.Server.handle(Server.java:517)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:306)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:242)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:261)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:192)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:261)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:75)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:213)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:147)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.zimbra.common.service.ServiceException: system failure: exception executing command: zmbackupldap --outdir /opt/zimbra/backup/tmp/incr-20180822.140023.748/ldap --zip with {RemoteManager: example.com->zimbra@example.com:22}
ExceptionId:qtp1286783232-58103:https:https://localhost:7071/service/admin/soap/BackupRequest:1534946484653:3d70be960262890c
Code:service.FAILURE
at com.zimbra.common.service.ServiceException.FAILURE(ServiceException.java:288)
at com.zimbra.cs.rmgmt.RemoteManager.execute(RemoteManager.java:178)
at com.zimbra.cs.backup.FileBackupTarget$FileBackupSet.backupLdap(FileBackupTarget.java:1471)
... 62 more
Caused by: java.io.IOException: command failed: exit status=1, stdout=STARTCMD: example.com /opt/zimbra/libexec/zmbackupldap --outdir /opt/zimbra/backup/tmp/incr-20180822.140023.748/ldap --zip
ENDCMD: example.com /opt/zimbra/libexec/zmbackupldap --outdir /opt/zimbra/backup/tmp/incr-20180822.140023.748/ldap --zip
, stderr=5b7d6cb4 mdb_db_open: database "": mdb_dbi_open(/opt/zimbra/data/ldap/mdb/db/id2v) failed: MDB_NOTFOUND: No matching key/data pair found (-30798).
5b7d6cb4 backend_startup_one (type=mdb, suffix=""): bi_db_open failed! (-30798)
slap_startup failed
Unable to invoke /opt/zimbra/libexec/zmslapcat /opt/zimbra/backup/tmp/incr-20180822.140023.748/ldap: exit code = 1
at com.zimbra.cs.rmgmt.RemoteManager.execute(RemoteManager.java:170)
... 63 more
Surely this should be documented? It should be PART of the PATCH instructions when you update LDAP to recreate the KEYS, or am I missing something?
FIX:
To fix the error was quite easy, but to find the fix was VERY DIFFICULT!
Code: Select all
zmsshkeygen
zmupdateauthkeys
Kind regards
Aubrey Kloppers (aka cyber7) Cape Town, South Africa