ZCS upgrade from 8.6.0 to 8.8.15 failed with wrong keystore password

[joller]

Hi there,

I tried to upgrade ZCS open-source from 8.6.0 to 8.8.15, but some error message showed up and finally the services didn't start.
The message was:

Code: Select all

[] ERROR: could not instantiate Provisioning interface of class 'com.zimbra.cs.account.ldap.LdapProvisioning'; defaulting to LdapProvisioning
java.lang.IllegalStateException: Unable to create CustomTrustManager
        at com.zimbra.common.net.TrustManagers.customTrustManager(TrustManagers.java:58)
        at com.zimbra.cs.ldap.unboundid.LdapSSLUtil.getTrustManager(LdapSSLUtil.java:84)
        at com.zimbra.cs.ldap.unboundid.LdapSSLUtil.createSSLContext(LdapSSLUtil.java:89)
        at com.zimbra.cs.ldap.unboundid.LdapConnectionPool.createConnPool(LdapConnectionPool.java:106)
        at com.zimbra.cs.ldap.unboundid.LdapConnectionPool.createConnectionPool(LdapConnectionPool.java:63)
        at com.zimbra.cs.ldap.unboundid.UBIDLdapContext.init(UBIDLdapContext.java:106)
        at com.zimbra.cs.ldap.unboundid.UBIDLdapClient.init(UBIDLdapClient.java:39)
        at com.zimbra.cs.ldap.LdapClient.getInstanceIfLDAPavailable(LdapClient.java:62)
        at com.zimbra.cs.ldap.LdapClient.getInstance(LdapClient.java:69)
        at com.zimbra.cs.ldap.LdapClient.initialize(LdapClient.java:94)
        at com.zimbra.cs.account.ldap.LdapProv.<init>(LdapProv.java:47)
        at com.zimbra.cs.account.ldap.LdapProvisioning.<init>(LdapProvisioning.java:290)
        at com.zimbra.cs.account.ldap.LdapProvisioning.<init>(LdapProvisioning.java:287)
        at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
        at java.base/java.lang.Class.newInstance(Class.java:584)
        at com.zimbra.cs.account.Provisioning.getInstance(Provisioning.java:354)
        at com.zimbra.cs.account.Provisioning.getInstance(Provisioning.java:310)
        at com.zimbra.cs.account.ProvUtil.initProvisioning(ProvUtil.java:1032)
        at com.zimbra.cs.account.ProvUtil.main(ProvUtil.java:4156)
Caused by: java.security.KeyStoreException: java.io.IOException: Keystore was tampered with, or password was incorrect
        at com.zimbra.common.net.DefaultTrustManager.<init>(DefaultTrustManager.java:51)
        at com.zimbra.common.net.CustomTrustManager.<init>(CustomTrustManager.java:64)
        at com.zimbra.common.net.TrustManagers.customTrustManager(TrustManagers.java:56)
        ... 21 more
Caused by: java.io.IOException: Keystore was tampered with, or password was incorrect
        at java.base/sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:785)
        at java.base/sun.security.util.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:243)
        at java.base/java.security.KeyStore.load(KeyStore.java:1479)
        at com.zimbra.common.net.DefaultTrustManager.<init>(DefaultTrustManager.java:49)
        ... 23 more
Caused by: java.security.UnrecoverableKeyException: Password verification failed
        at java.base/sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:783)
        ... 26 more
[] ERROR: defaulting to com.zimbra.cs.account.ldap.LdapProvisioning
Exception in thread "main" java.lang.NullPointerException
        at com.zimbra.cs.ldap.unboundid.UBIDLdapOperation$GetConnection.execute(UBIDLdapOperation.java:189)
        at com.zimbra.cs.ldap.unboundid.UBIDLdapContext.getConnection(UBIDLdapContext.java:200)
        at com.zimbra.cs.ldap.unboundid.UBIDLdapContext.<init>(UBIDLdapContext.java:169)
        at com.zimbra.cs.ldap.unboundid.UBIDLdapClient.getContextImpl(UBIDLdapClient.java:90)
        at com.zimbra.cs.ldap.LdapClient.getContext(LdapClient.java:135)
        at com.zimbra.cs.account.ldap.ZLdapHelper.getAttributes(ZLdapHelper.java:276)
        at com.zimbra.cs.account.ldap.LdapHelper.getAttributes(LdapHelper.java:201)
        at com.zimbra.cs.account.ldap.LdapProvisioning.getServerByName(LdapProvisioning.java:4199)
        at com.zimbra.cs.account.ldap.LdapProvisioning.getServerByNameInternal(LdapProvisioning.java:4187)
        at com.zimbra.cs.account.ldap.LdapProvisioning.get(LdapProvisioning.java:4165)
        at com.zimbra.cs.account.ProvUtil.lookupServer(ProvUtil.java:3589)
        at com.zimbra.cs.account.ProvUtil.doGetServer(ProvUtil.java:4941)
        at com.zimbra.cs.account.ProvUtil.execute(ProvUtil.java:1264)
        at com.zimbra.cs.account.ProvUtil.main(ProvUtil.java:4160)
Setting defaults...[] ERROR: could not instantiate Provisioning interface of class 'com.zimbra.cs.account.ldap.LdapProvisioning'; defaulting to LdapProvisioning

The same message showed up for several times but didn't stop the upgrade process.
Finally, the following messages emerged and the process stopped:

Code: Select all

*** CONFIGURATION COMPLETE - press 'a' to apply
Select from menu, or press 'a' to apply config (? - help) a
Saving config in /opt/zimbra/config.26119...done.
Operations logged to /tmp/zmsetup.20190802-093921.log
Setting local config values...done.
Initializing core config...Setting up CA...done.
Deploying CA to /opt/zimbra/conf/ca ...failed.

I am using Let's Encrypt certificate for all services except for LDAP, for which the self-signed one is used.
Since the Let's Encrypt CA chain is not included in ZCS-8.6, I have manually added it to the cacerts of Zimbra JRE,
and everything has been working fine for almost one year.

According to the error messages and the source code of DefaultTrustManager.java,
it looks like that the JRE truststore (cacerts) failed to load due to wrong password.
The password is correct, however, after checking with zmlocalcfg (on config mailboxd_truststore and mailboxd_truststore_password) and keytool.

Any idea?

[joller]

I have solved the problem.

For some reason I can't recall, I changed the keystore password of Zimbra JRE's keystore (cacerts) years ago.
Although it matched the setting in the local config (mailboxd_truststore_password),
the upgrade script likely attempted to access the old keystore with the default password "changeit"
(or the new keystore with the old password, which one is the case I'm not sure).

After changing the password back to the default one, the upgrade succeeded.