Backup Failure after LDAP patch. (Resolved)

Discuss your pilot or production implementation with other Zimbra admins or our engineers.
Post Reply
User avatar
cyber7
Advanced member
Advanced member
Posts: 192
Joined: Sat Sep 13, 2014 1:14 am
Location: Cape Town
ZCS/ZD Version: Release 9.0.0_GA_3924.RHEL7_64_2020
Contact:

Backup Failure after LDAP patch. (Resolved)

Post by cyber7 »

Good day everyone.
First of all, thank you to Zimbra staff's inability to document patch-instructions properly. It Would be nice to get an official PATCH document stating that the SSH KEYS needs to be re-created when you release LDAP patches.

(I am documenting the problem for future reference for anyone to find the fix using any search engine.) - See FIX and NOTE at end of writing.

Details:
On the 21 August 2018 a patch was released that I applied with the YUM command on Centos. After this patch Zimbra backups started failing (I am running WebMin as a front-end on the system)

Patch notification:
An update to audit from 2.8.1-3.el7 to 2.8.1-3.el7_5.1 is available.
An update to audit-libs from 2.8.1-3.el7 to 2.8.1-3.el7_5.1 is available.
An update to dracut from 033-535.el7 to 033-535.el7_5.1 is available.
An update to dracut-config-rescue from 033-535.el7 to 033-535.el7_5.1 is available.
An update to dracut-network from 033-535.el7 to 033-535.el7_5.1 is available.
An update to initscripts from 9.49.41-1.el7 to 9.49.41-1.el7_5.1 is available.
An update to kpartx from 0.4.9-119.el7 to 0.4.9-119.el7_5.1 is available.
An update to libblkid from 2.23.2-52.el7 to 2.23.2-52.el7_5.1 is available.
An update to libgudev1 from 219-57.el7 to 219-57.el7_5.1 is available.
An update to libmount from 2.23.2-52.el7 to 2.23.2-52.el7_5.1 is available.
An update to libuuid from 2.23.2-52.el7 to 2.23.2-52.el7_5.1 is available.
An update to mariadb-libs from 5.5.56-2.el7 to 5.5.60-1.el7_5 is available.
An update to selinux-policy from 3.13.1-192.el7_5.4 to 3.13.1-192.el7_5.6 is available.
An update to selinux-policy-targeted from 3.13.1-192.el7_5.4 to 3.13.1-192.el7_5.6 is available.
An update to systemd from 219-57.el7 to 219-57.el7_5.1 is available.
An update to systemd-libs from 219-57.el7 to 219-57.el7_5.1 is available.
An update to systemd-sysv from 219-57.el7 to 219-57.el7_5.1 is available.
An update to tuned from 2.9.0-1.el7 to 2.9.0-1.el7_5.2 is available.
An update to util-linux from 2.23.2-52.el7 to 2.23.2-52.el7_5.1 is available.
An update to zimbra-common-core-jar from 1.0.0.1531216364-1.r7 to 2.0.0.1533843772-1.r7 is available.
An update to zimbra-ldap-components from 1.0.0-1zimbra8.7b1.el7 to 1.0.1-1zimbra8.7b1.el7 is available.
An update to zimbra-lmdb from 2.4.44-1zimbra8.7b9.el7 to 2.4.46-1zimbra8.7b2.el7 is available.
An update to zimbra-lmdb-libs from 2.4.44-1zimbra8.7b9.el7 to 2.4.46-1zimbra8.7b2.el7 is available.
An update to zimbra-mbox-webclient-war from 1.0.0.1531295071-1.r7 to 2.0.0.1533844076-1.r7 is available.
An update to zimbra-network-modules-ng from 2.0.2.1532358202-1.r7 to 2.0.3.1533551703-1.r7 is available.
An update to zimbra-openldap-client from 2.4.44-1zimbra8.7b9.el7 to 2.4.46-1zimbra8.7b2.el7 is available.
An update to zimbra-openldap-libs from 2.4.44-1zimbra8.7b9.el7 to 2.4.46-1zimbra8.7b2.el7 is available.
An update to zimbra-openldap-server from 2.4.44-1zimbra8.7b9.el7 to 2.4.46-1zimbra8.7b2.el7 is available.

An update to zimbra-patch from 8.8.9.1531484537.p1-1.r7 to 8.8.9.1533882487.p3-1.r7 is available.
Updates can be installed at http://example.com:10000/package-updates/
Error Message Received:
Server: example.com

Label: incr-20180822.140023.748
Type: incremental
Status: completed (with errors)
Started: Wed, 2018/08/22 16:00:23.748 SAST
Ended: Wed, 2018/08/22 16:01:24.699 SAST
Redo log sequence range: 971 .. 972
Number of accounts: 349
Number of errors: 1


ERRORS

system: system failure: LDAP backup failed: system failure: exception executing command: zmbackupldap --outdir /opt/zimbra/backup/tmp/incr-20180822.140023.748/ldap --zip with {RemoteManager: example.com->zimbra@example.com:22}
com.zimbra.common.service.ServiceException: system failure: LDAP backup failed: system failure: exception executing command: zmbackupldap --outdir /opt/zimbra/backup/tmp/incr-20180822.140023.748/ldap --zip with {RemoteManager: example.com->example.com:22}
ExceptionId:qtp1286783232-58103:https:https://localhost:7071/service/admin/soap/BackupRequest:1534946484654:3d70be960262890c
Code:service.FAILURE
at com.zimbra.common.service.ServiceException.FAILURE(ServiceException.java:288)
at com.zimbra.cs.backup.FileBackupTarget$FileBackupSet.backupLdap(FileBackupTarget.java:1474)
at com.zimbra.cs.backup.FileBackupTarget$FileBackupSet.startIncrementalBackup(FileBackupTarget.java:1069)
at com.zimbra.cs.backup.BackupManager.backupIncremental(BackupManager.java:336)
at com.zimbra.cs.service.backup.Backup.handleNetworkRequest(Backup.java:153)
at com.zimbra.cs.service.NetworkDocumentHandler.handle(NetworkDocumentHandler.java:23)
at com.zimbra.soap.SoapEngine.dispatchRequest(SoapEngine.java:643)
at com.zimbra.soap.SoapEngine.dispatch(SoapEngine.java:488)
at com.zimbra.soap.SoapEngine.dispatch(SoapEngine.java:275)
at com.zimbra.soap.SoapServlet.doWork(SoapServlet.java:304)
at com.zimbra.soap.SoapServlet.doPost(SoapServlet.java:214)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at com.zimbra.cs.servlet.ZimbraServlet.service(ZimbraServlet.java:211)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:821)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1685)
at com.zimbra.cs.servlet.CsrfFilter.doFilter(CsrfFilter.java:169)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
at com.zimbra.cs.servlet.RequestStringFilter.doFilter(RequestStringFilter.java:54)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
at com.zimbra.cs.servlet.SetHeaderFilter.doFilter(SetHeaderFilter.java:59)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
at com.zimbra.cs.servlet.ETagHeaderFilter.doFilter(ETagHeaderFilter.java:47)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
at com.zimbra.cs.servlet.ContextPathBasedThreadPoolBalancerFilter.doFilter(ContextPathBasedThreadPoolBalancerFilter.java:107)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
at com.zimbra.cs.servlet.ZimbraQoSFilter.doFilter(ZimbraQoSFilter.java:116)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
at com.zimbra.cs.servlet.ZimbraInvalidLoginFilter.doFilter(ZimbraInvalidLoginFilter.java:117)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
at org.eclipse.jetty.servlets.DoSFilter.doFilterChain(DoSFilter.java:473)
at org.eclipse.jetty.servlets.DoSFilter.doFilter(DoSFilter.java:318)
at org.eclipse.jetty.servlets.DoSFilter.doFilter(DoSFilter.java:288)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1158)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1090)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:109)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:119)
at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:318)
at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:437)
at org.eclipse.jetty.server.handler.DebugHandler.handle(DebugHandler.java:84)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:119)
at org.eclipse.jetty.server.Server.handle(Server.java:517)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:306)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:242)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:261)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:192)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:261)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:75)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:213)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:147)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.zimbra.common.service.ServiceException: system failure: exception executing command: zmbackupldap --outdir /opt/zimbra/backup/tmp/incr-20180822.140023.748/ldap --zip with {RemoteManager: example.com->zimbra@example.com:22}
ExceptionId:qtp1286783232-58103:https:https://localhost:7071/service/admin/soap/BackupRequest:1534946484653:3d70be960262890c
Code:service.FAILURE
at com.zimbra.common.service.ServiceException.FAILURE(ServiceException.java:288)
at com.zimbra.cs.rmgmt.RemoteManager.execute(RemoteManager.java:178)
at com.zimbra.cs.backup.FileBackupTarget$FileBackupSet.backupLdap(FileBackupTarget.java:1471)
... 62 more
Caused by: java.io.IOException: command failed: exit status=1, stdout=STARTCMD: example.com /opt/zimbra/libexec/zmbackupldap --outdir /opt/zimbra/backup/tmp/incr-20180822.140023.748/ldap --zip
ENDCMD: example.com /opt/zimbra/libexec/zmbackupldap --outdir /opt/zimbra/backup/tmp/incr-20180822.140023.748/ldap --zip
, stderr=5b7d6cb4 mdb_db_open: database "": mdb_dbi_open(/opt/zimbra/data/ldap/mdb/db/id2v) failed: MDB_NOTFOUND: No matching key/data pair found (-30798).
5b7d6cb4 backend_startup_one (type=mdb, suffix=""): bi_db_open failed! (-30798)
slap_startup failed
Unable to invoke /opt/zimbra/libexec/zmslapcat /opt/zimbra/backup/tmp/incr-20180822.140023.748/ldap: exit code = 1
at com.zimbra.cs.rmgmt.RemoteManager.execute(RemoteManager.java:170)
... 63 more
After digging for days I finally found a single mail talking about re-creating the SSH KEYS on a LDAP failure.

Surely this should be documented? It should be PART of the PATCH instructions when you update LDAP to recreate the KEYS, or am I missing something?

FIX:
To fix the error was quite easy, but to find the fix was VERY DIFFICULT!

Code: Select all

zmsshkeygen 
zmupdateauthkeys 
NOTE: Once this was resolved, all SSH KEYS of remote systems using LDAP had to be destroyed and recreated.

Kind regards
Aubrey Kloppers (aka cyber7) Cape Town, South Africa
seidler
Posts: 21
Joined: Fri Jun 30, 2017 8:28 am

Re: Backup Failure after LDAP patch. (Resolved)

Post by seidler »

Thank you, thank you, thank you!

viewtopic.php?f=15&t=64781
User avatar
cyber7
Advanced member
Advanced member
Posts: 192
Joined: Sat Sep 13, 2014 1:14 am
Location: Cape Town
ZCS/ZD Version: Release 9.0.0_GA_3924.RHEL7_64_2020
Contact:

Re: Backup Failure after LDAP patch. (Resolved)

Post by cyber7 »

seidler wrote:Thank you, thank you, thank you!

viewtopic.php?f=15&t=64781
:D
bobbrand
Posts: 2
Joined: Sat Jul 30, 2022 8:40 am

Re: Backup Failure after LDAP patch. (Resolved)

Post by bobbrand »

Hi,

Could someone please explain what is meant by "all SSH KEYS of remote systems using LDAP had to be destroyed and recreated"? For instance, I have a couple of Samba Domain Controllers that use LDAP - do I need to do anything with them?

Thank you,
Bob Brand
Post Reply