zmmailboxdctl is not running + java "no valid keystore"

Ask questions about your setup or get help installing ZCS server (ZD section below).
User avatar
SEA80
Posts: 12
Joined: Fri Mar 09, 2018 2:59 pm

zmmailboxdctl is not running + java "no valid keystore"

Post by SEA80 »

I've installed Zimbra 8.8.7 two-servers environment on latest CentOS7 release using manual https://zimbra.github.io/installguides/ ... multi.html.
Installation order:
Step 1 - mailbox server: zimbra-ldap
Step 2 - gate server: zimbra-mta + zimbra-dnscache, zimbra-proxy + zimbra-memcached
Step 3 - mailbox server (from step 1): zimbra-store + zimbra-logger
After step 3 mailbox service won't run because zmmailboxdctl is not running.

On gate server:
$ zmcontrol status

Code: Select all

Host yyy.zzz
        amavis                  Running
        antispam                Running
        antivirus               Running
        dnscache                Running
        memcached               Running
        mta                     Running
        opendkim                Running
        proxy                   Running
        stats                   Running
        zmconfigd               Running
On mailbox server:
$ zmcontrol status

Code: Select all

Host xxx.yyy.zzz
        ldap                    Running
        logger                  Running
        mailbox                 Stopped
                zmmailboxdctl is not running.
        stats                   Running
        zmconfigd               Running
$ cat /var/log/zimbra.log | grep 2858

Code: Select all

Mar 12 12:01:13 xxx zmmailboxdmgr[2858]: stale pid 1977 found in /opt/zimbra/log/zmmailboxd_manager.pid: No such process
Mar 12 12:01:13 xxx zmmailboxdmgr[2858]: assuming no other instance is running
Mar 12 12:01:13 xxx zmmailboxdmgr[2858]: file /opt/zimbra/log/zmmailboxd.pid does not exist
Mar 12 12:01:13 xxx zmmailboxdmgr[2858]: assuming no other instance is running
Mar 12 12:01:13 xxx zmmailboxdmgr[2858]: no manager process is running
$ tail -200 /var/log/zimbra.log
repeating messages from zmmailboxdmgr and zmconfigd

Code: Select all

Mar 12 12:00:18 xxx zimbramon[1108]: 1108:info: Starting mailbox via zmcontrol
Mar 12 12:00:20 xxx zmmailboxdmgr[1965]: stale pid 24599 found in /opt/zimbra/log/zmmailboxd_manager.pid: No such process
Mar 12 12:00:20 xxx zmmailboxdmgr[1965]: assuming no other instance is running
Mar 12 12:00:20 xxx zmmailboxdmgr[1965]: file /opt/zimbra/log/zmmailboxd.pid does not exist
Mar 12 12:00:20 xxx zmmailboxdmgr[1965]: assuming no other instance is running
Mar 12 12:00:20 xxx zmmailboxdmgr[1965]: no manager process is running
Mar 12 12:00:20 xxx zmmailboxdmgr[1976]: start requested
Mar 12 12:00:20 xxx zmmailboxdmgr[1976]: checking if another instance of manager is already running
Mar 12 12:00:20 xxx zmmailboxdmgr[1976]: stale pid 24599 found in /opt/zimbra/log/zmmailboxd_manager.pid: No such process
Mar 12 12:00:20 xxx zmmailboxdmgr[1976]: assuming no other instance is running
Mar 12 12:00:20 xxx zmmailboxdmgr[1976]: file /opt/zimbra/log/zmmailboxd.pid does not exist
Mar 12 12:00:20 xxx zmmailboxdmgr[1976]: assuming no other instance is running
Mar 12 12:00:20 xxx zmmailboxdmgr[1977]: wrote manager pid 1977 to /opt/zimbra/log/zmmailboxd_manager.pid
Mar 12 12:00:20 xxx zmmailboxdmgr[1977]: manager started mailboxd/JVM with pid 1983
Mar 12 12:00:20 xxx zmmailboxdmgr[1983]: wrote java pid 1983 to /opt/zimbra/log/zmmailboxd_java.pid
Mar 12 12:00:25 xxx zmmailboxdmgr[1977]: manager woke up from wait on mailboxd/JVM with pid 1983
Mar 12 12:00:25 xxx zmmailboxdmgr[1977]: manager started mailboxd/JVM with pid 2071
Mar 12 12:00:25 xxx zmmailboxdmgr[2071]: wrote java pid 2071 to /opt/zimbra/log/zmmailboxd_java.pid
Mar 12 12:00:28 xxx zimbramon[1108]: 1108:info: Starting stats via zmcontrol
Mar 12 12:00:29 xxx zmmailboxdmgr[1977]: manager woke up from wait on mailboxd/JVM with pid 2071
Mar 12 12:00:29 xxx zmmailboxdmgr[1977]: mailboxd/JVM exited twice in 4 seconds (tolerance=60)
Mar 12 12:01:13 xxx zmconfigd[1454]: Fetching All configs
Mar 12 12:01:13 xxx zmconfigd[1454]: All configs fetched in 0.13 seconds
Mar 12 12:01:13 xxx zmconfigd[1454]: Tracking service ldap
Mar 12 12:01:13 xxx zmconfigd[1454]: Watchdog: service ldap now available for watchdog.
Mar 12 12:01:13 xxx zmconfigd[1454]: Tracking service logger
Mar 12 12:01:13 xxx zmconfigd[1454]: Watchdog: service logger now available for watchdog.
Mar 12 12:01:13 xxx zmmailboxdmgr[2843]: stale pid 1977 found in /opt/zimbra/log/zmmailboxd_manager.pid: No such process
Mar 12 12:01:13 xxx zmmailboxdmgr[2843]: assuming no other instance is running
Mar 12 12:01:13 xxx zmmailboxdmgr[2843]: file /opt/zimbra/log/zmmailboxd.pid does not exist
Mar 12 12:01:13 xxx zmmailboxdmgr[2843]: assuming no other instance is running
Mar 12 12:01:13 xxx zmmailboxdmgr[2843]: no manager process is running
Mar 12 12:01:13 xxx zmconfigd[1454]: Tracking service mailbox
Mar 12 12:01:13 xxx zmmailboxdmgr[2858]: stale pid 1977 found in /opt/zimbra/log/zmmailboxd_manager.pid: No such process
Mar 12 12:01:13 xxx zmmailboxdmgr[2858]: assuming no other instance is running
Mar 12 12:01:13 xxx zmmailboxdmgr[2858]: file /opt/zimbra/log/zmmailboxd.pid does not exist
Mar 12 12:01:13 xxx zmmailboxdmgr[2858]: assuming no other instance is running
Mar 12 12:01:13 xxx zmmailboxdmgr[2858]: no manager process is running
Mar 12 12:01:13 xxx zmconfigd[1454]: Tracking service mailboxd
Mar 12 12:01:14 xxx zmconfigd[1454]: Tracking service stats
Mar 12 12:01:14 xxx zmconfigd[1454]: Watchdog: service stats now available for watchdog.
Mar 12 12:01:14 xxx zmconfigd[1454]: Watchdog: skipping service antivirus. Service not yet available for restarts.
Mar 12 12:01:14 xxx zmconfigd[1454]: All rewrite threads completed in 0.02 sec
Mar 12 12:01:14 xxx zmconfigd[1454]: All restarts completed in 0.00 sec
$ tail -200 /opt/zimbra/log/zmmailboxd.out
Caused by: MultiException[java.lang.IllegalStateException: no valid keystore, java.lang.IllegalStateException: no valid keystore, java.lang.IllegalStateException: no valid keystore, java.lang.IllegalStateException: no valid keystore]

Code: Select all

OpenJDK 64-Bit Server VM warning: .hotspot_compiler file is present but has been ignored.  Run with -XX:CompileCommandFile=.hotspot_compiler to load the file.
2018-03-12 12:00:26.573:INFO::main: Logging initialized @615ms
JettyMonitor monitoring thread pool qtp998351292{STOPPED,10<=0<=250,i=0,q=0}
Zimbra server reserving server socket port=7110 bindaddr=null ssl=false
Zimbra server reserving server socket port=7995 bindaddr=null ssl=false
Zimbra server reserving server socket port=7143 bindaddr=null ssl=false
Zimbra server reserving server socket port=7993 bindaddr=null ssl=false
Zimbra server reserving server socket port=7025 bindaddr=null ssl=false
2018-03-12 12:00:27.693:INFO:oejs.SetUIDListener:main: Setting umask=027
2018-03-12 12:00:27.694:INFO:oejs.SetUIDListener:main: Current rlimit_nofiles (soft=524288, hard=524288)
2018-03-12 12:00:27.694:INFO:oejs.SetUIDListener:main: Set rlimit_nofiles (soft=65536, hard=65536)
2018-03-12 12:00:27.717:INFO:oejs.SetUIDListener:main: Opened ServerConnector@3439f68d{HTTP/1.1,[http/1.1]}{localhost:80}
2018-03-12 12:00:27.717:INFO:oejs.SetUIDListener:main: Opened ServerConnector@130f889{SSL,[ssl, http/1.1]}{0.0.0.0:443}
2018-03-12 12:00:27.717:INFO:oejs.SetUIDListener:main: Opened ServerConnector@1188e820{SSL,[ssl, http/1.1]}{0.0.0.0:7071}
2018-03-12 12:00:27.718:INFO:oejs.SetUIDListener:main: Opened ServerConnector@2f490758{SSL,[ssl, http/1.1]}{0.0.0.0:7073}
2018-03-12 12:00:27.718:INFO:oejs.SetUIDListener:main: Opened ServerConnector@101df177{SSL,[ssl, http/1.1]}{0.0.0.0:7072}
2018-03-12 12:00:27.718:INFO:oejs.SetUIDListener:main: Setting GID=995
2018-03-12 12:00:27.722:INFO:oejs.SetUIDListener:main: Setting UID=997
2018-03-12 12:00:27.724:INFO:oejs.Server:main: jetty-9.3.5.v20151012
2018-03-12 12:00:28.441:WARN:oejs.SecurityHandler:main: ServletContext@o.e.j.w.WebAppContext@1534f01b{/zimlet,[file:///opt/zimbra/jetty_base/webapps/zimlet/, file:///opt/zimbra/zimlets-deployed/],STARTING}{/zimlet} has uncovered http methods for path: /
2018-03-12 12:00:28.465:INFO:oejsh.ContextHandler:main: Started o.e.j.w.WebAppContext@1534f01b{/zimlet,[file:///opt/zimbra/jetty_base/webapps/zimlet/, file:///opt/zimbra/zimlets-deployed/],AVAILABLE}{/zimlet}
2018-03-12 12:00:28.467:INFO:oejs.AbstractNCSARequestLog:main: Opened /opt/zimbra/log/access_log.2018-03-12
2018-03-12 12:00:28.494:INFO:oejm.ThreadMonitor:main: Thread Monitor started successfully
2018-03-12 12:00:28.502:INFO:oejs.ServerConnector:main: Started ServerConnector@3439f68d{HTTP/1.1,[http/1.1]}{localhost:80}
java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.eclipse.jetty.start.Main.invokeMain(Main.java:214)
        at org.eclipse.jetty.start.Main.start(Main.java:457)
        at org.eclipse.jetty.start.Main.main(Main.java:75)
Caused by: MultiException[java.lang.IllegalStateException: no valid keystore, java.lang.IllegalStateException: no valid keystore, java.lang.IllegalStateException: no valid keystore, java.lang.IllegalStateException: no valid keystore]
        at org.eclipse.jetty.server.Server.doStart(Server.java:347)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
        at org.eclipse.jetty.xml.XmlConfiguration$1.run(XmlConfiguration.java:1510)
        at java.security.AccessController.doPrivileged(Native Method)
        at org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1435)
        ... 7 more
        Suppressed: java.lang.IllegalStateException: no valid keystore
                at org.eclipse.jetty.util.security.CertificateUtils.getKeyStore(CertificateUtils.java:48)
                at org.eclipse.jetty.util.ssl.SslContextFactory.loadKeyStore(SslContextFactory.java:1020)
                at org.eclipse.jetty.util.ssl.SslContextFactory.doStart(SslContextFactory.java:336)
                at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
                at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
                at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
                at org.eclipse.jetty.server.SslConnectionFactory.doStart(SslConnectionFactory.java:64)
                at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
                at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
                at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
                at org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:260)
                at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:81)
                at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:244)
                at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
                at org.eclipse.jetty.server.Server.doStart(Server.java:384)
                ... 11 more
        Suppressed: java.lang.IllegalStateException: no valid keystore
                at org.eclipse.jetty.util.security.CertificateUtils.getKeyStore(CertificateUtils.java:48)
                at org.eclipse.jetty.util.ssl.SslContextFactory.loadKeyStore(SslContextFactory.java:1020)
                at org.eclipse.jetty.util.ssl.SslContextFactory.doStart(SslContextFactory.java:336)
                at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
                at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
                at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
                at org.eclipse.jetty.server.SslConnectionFactory.doStart(SslConnectionFactory.java:64)
                at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
                at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
                at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
                at org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:260)
                at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:81)
                at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:244)
                at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
                at org.eclipse.jetty.server.Server.doStart(Server.java:384)
                ... 11 more
        Suppressed: java.lang.IllegalStateException: no valid keystore
                at org.eclipse.jetty.util.security.CertificateUtils.getKeyStore(CertificateUtils.java:48)
                at org.eclipse.jetty.util.ssl.SslContextFactory.loadKeyStore(SslContextFactory.java:1020)
                at org.eclipse.jetty.util.ssl.SslContextFactory.doStart(SslContextFactory.java:336)
                at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
                at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
                at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
                at org.eclipse.jetty.server.SslConnectionFactory.doStart(SslConnectionFactory.java:64)
                at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
                at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
                at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
                at org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:260)
                at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:81)
                at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:244)
                at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
                at org.eclipse.jetty.server.Server.doStart(Server.java:384)
                ... 11 more
Caused by: java.lang.IllegalStateException: no valid keystore
        at org.eclipse.jetty.util.security.CertificateUtils.getKeyStore(CertificateUtils.java:48)
        at org.eclipse.jetty.util.ssl.SslContextFactory.loadKeyStore(SslContextFactory.java:1020)
        at org.eclipse.jetty.util.ssl.SslContextFactory.doStart(SslContextFactory.java:336)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
        at org.eclipse.jetty.server.SslConnectionFactory.doStart(SslConnectionFactory.java:64)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
        at org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:260)
        at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:81)
        at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:244)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
        at org.eclipse.jetty.server.Server.doStart(Server.java:384)
        ... 11 more

Usage: java -jar start.jar [options] [properties] [configs]
       java -jar start.jar --help  # for more information
All servers' interfaces are in "trusted" zone, it means that firewall allows all.
What I have to dig next? Did I make an error when installing zimbra components?
User avatar
msquadrat
Advanced member
Advanced member
Posts: 183
Joined: Mon Oct 14, 2013 10:09 am

Re: zmmailboxdctl is not running + java "no valid keystore"

Post by msquadrat »

What do you get when you execute the following as user zimbra?

Code: Select all

ls -l "$(zmlocalconfig -m nokey mailboxd_keystore)"
User avatar
SEA80
Posts: 12
Joined: Fri Mar 09, 2018 2:59 pm

Re: zmmailboxdctl is not running + java "no valid keystore"

Post by SEA80 »

msquadrat wrote:What do you get when you execute the following as user zimbra?

Code: Select all

$ ls -l "$(zmlocalconfig -m nokey mailboxd_keystore)"
-rw-r----- 1 zimbra zimbra 2187 Mar  9 16:45 /opt/zimbra/conf/keystore
User avatar
msquadrat
Advanced member
Advanced member
Posts: 183
Joined: Mon Oct 14, 2013 10:09 am

Re: zmmailboxdctl is not running + java "no valid keystore"

Post by msquadrat »

SEA80 wrote:
msquadrat wrote:What do you get when you execute the following as user zimbra?

Code: Select all

$ ls -l "$(zmlocalconfig -m nokey mailboxd_keystore)"
-rw-r----- 1 zimbra zimbra 2187 Mar  9 16:45 /opt/zimbra/conf/keystore
Ok, that's odd. I have a test system which I recently updated to 8.8.7 and the keystore here is located at /opt/zimbra/mailboxd/etc/keystore. Also, the file is about three times as large:

Code: Select all

zimbra@hz26:~$ ls -l "$(zmlocalconfig -m nokey mailboxd_keystore)"
-rw-r----- 1 zimbra zimbra 6273 Mar  9 15:42 /opt/zimbra/mailboxd/etc/keystore
zimbra@hz26:~$ file "$(zmlocalconfig -m nokey mailboxd_keystore)"
/opt/zimbra/mailboxd/etc/keystore: Java KeyStore
What does the file command from above give you?

To me it looks like something changed about the keystore in a recent 8.8.x release which only affects new installations. I faintly remember an issue with the keystore being overridden between updates but I can't find anything related to that in the 8.8.7 release notes.
phoenix
Ambassador
Ambassador
Posts: 27272
Joined: Fri Sep 12, 2014 9:56 pm
Location: Liverpool, England

Re: zmmailboxdctl is not running + java "no valid keystore"

Post by phoenix »

msquadrat wrote:
To me it looks like something changed about the keystore in a recent 8.8.x release which only affects new installations. I faintly remember an issue with the keystore being overridden between updates but I can't find anything related to that in the 8.8.7 release notes.
I've recently done a clean install of ZCS 8.8.7 and it shows the same outrput as you:

Code: Select all

[zimbra@mail03 ~]$ ls -l "$(zmlocalconfig -m nokey mailboxd_keystore)"
-rw-r----- 1 zimbra zimbra 2222 Mar  9 14:50 /opt/zimbra/mailboxd/etc/keystore
[zimbra@mail03 ~]$  file "$(zmlocalconfig -m nokey mailboxd_keystore)"
/opt/zimbra/mailboxd/etc/keystore: Java KeyStore
[zimbra@mail03 ~]$ zmcontrol -v
Release 8.8.7_GA_1964.RHEL7_64_20180223145016 RHEL7_64 FOSS edition.
[zimbra@mail03 ~]$ 
Regards

Bill

Rspamd: A high performance spamassassin replacement

Per ardua ad astra
User avatar
msquadrat
Advanced member
Advanced member
Posts: 183
Joined: Mon Oct 14, 2013 10:09 am

Re: zmmailboxdctl is not running + java "no valid keystore"

Post by msquadrat »

I couldn't find anything in the source of the mailbox service which would make it default to that file either.

But I just noticed the steps which were performed here which are odd
Step 1 - mailbox server: zimbra-ldap
Step 2 - gate server: zimbra-mta + zimbra-dnscache, zimbra-proxy + zimbra-memcached
Step 3 - mailbox server (from step 1): zimbra-store + zimbra-logger
So, first the LDAP was installed on server1, then the edge services on server2 and finally the mailbox-service added to server1. This is uncommon but IIRC I did something similar for certain reasons previously before and ran into issues as well. I think the installer sets some localconfig options when only LDAP is installed which cause issues when the mailbox service is added later on. I think I raised a bugzilla issue for that but a quick search didn't find it but instead I found another issue which is the one we see here: [bug]104170[/bug]. According to that bug report the required steps are

Code: Select all

mv /opt/zimbra/conf/keystore /opt/zimbra/mailboxd/etc
zmlocalconfig -e mailboxd_keystore=/opt/zimbra/mailboxd/etc/keystore
zmmailboxdctl start
I'd first make sure that the first mv command doesn't override an existing good keystore with a broken one (ie. skip it if there is such a file already).
User avatar
SEA80
Posts: 12
Joined: Fri Mar 09, 2018 2:59 pm

Re: zmmailboxdctl is not running + java "no valid keystore"

Post by SEA80 »

Code: Select all

$ file "$(zmlocalconfig -m nokey mailboxd_keystore)"
/opt/zimbra/conf/keystore: Java KeyStore
msquadrat wrote:I faintly remember an issue with the keystore being overridden between updates but I can't find anything related to that in the 8.8.7 release notes.
I've installed 8.8.6 same way before announcing of 8.8.7 and got same error.
I guess the uncommon installation steps was a reason of this issue, but I just following manual https://zimbra.github.io/installguides/ ... er_install, "Order of Installation"
msquadrat wrote:

Code: Select all

mv /opt/zimbra/conf/keystore /opt/zimbra/mailboxd/etc
zmlocalconfig -e mailboxd_keystore=/opt/zimbra/mailboxd/etc/keystore
zmmailboxdctl start
The commands above helped mailbox service to up and run (there were no another keystore file in /opt/zimbra/mailboxd/etc folder).

Code: Select all

$ zmcontrol status
Host xxx.yyy.zzz
        ldap                    Running
        logger                  Running
        mailbox                 Running
        stats                   Running
        zmconfigd               Running
$ ls -l "$(zmlocalconfig -m nokey mailboxd_keystore)"
-rw-r----- 1 zimbra zimbra 2187 Mar  9 16:45 /opt/zimbra/mailboxd/etc/keystore
zmmailboxd.out file looks normal now:

Code: Select all

$ tail -30 /opt/zimbra/log/zmmailboxd.out
OpenJDK 64-Bit Server VM warning: .hotspot_compiler file is present but has been ignored.  Run with -XX:CompileCommandFile=.hotspot_compiler to load the file.
2018-03-13 20:50:24.668:INFO::main: Logging initialized @705ms
JettyMonitor monitoring thread pool qtp998351292{STOPPED,10<=0<=250,i=0,q=0}
Zimbra server reserving server socket port=7110 bindaddr=null ssl=false
Zimbra server reserving server socket port=7995 bindaddr=null ssl=false
Zimbra server reserving server socket port=7143 bindaddr=null ssl=false
Zimbra server reserving server socket port=7993 bindaddr=null ssl=false
Zimbra server reserving server socket port=7025 bindaddr=null ssl=false
2018-03-13 20:50:25.845:INFO:oejs.SetUIDListener:main: Setting umask=027
2018-03-13 20:50:25.845:INFO:oejs.SetUIDListener:main: Current rlimit_nofiles (soft=524288, hard=524288)
2018-03-13 20:50:25.845:INFO:oejs.SetUIDListener:main: Set rlimit_nofiles (soft=65536, hard=65536)
2018-03-13 20:50:25.987:INFO:oejs.SetUIDListener:main: Opened ServerConnector@3439f68d{HTTP/1.1,[http/1.1]}{localhost:80}
2018-03-13 20:50:25.987:INFO:oejs.SetUIDListener:main: Opened ServerConnector@130f889{SSL,[ssl, http/1.1]}{0.0.0.0:443}
2018-03-13 20:50:25.987:INFO:oejs.SetUIDListener:main: Opened ServerConnector@1188e820{SSL,[ssl, http/1.1]}{0.0.0.0:7071}
2018-03-13 20:50:25.987:INFO:oejs.SetUIDListener:main: Opened ServerConnector@2f490758{SSL,[ssl, http/1.1]}{0.0.0.0:7073}
2018-03-13 20:50:25.987:INFO:oejs.SetUIDListener:main: Opened ServerConnector@101df177{SSL,[ssl, http/1.1]}{0.0.0.0:7072}
2018-03-13 20:50:25.987:INFO:oejs.SetUIDListener:main: Setting GID=995
2018-03-13 20:50:26.004:INFO:oejs.SetUIDListener:main: Setting UID=997
2018-03-13 20:50:26.020:INFO:oejs.Server:main: jetty-9.3.5.v20151012
2018-03-13 20:50:26.956:WARN:oejs.SecurityHandler:main: ServletContext@o.e.j.w.WebAppContext@1534f01b{/zimlet,[file:///opt/zimbra/jetty_base/webapps/zimlet/, file:///opt/zimbra/zimlets-deployed/],STARTING}{/zimlet} has uncovered http methods for path: /
2018-03-13 20:50:26.967:INFO:oejsh.ContextHandler:main: Started o.e.j.w.WebAppContext@1534f01b{/zimlet,[file:///opt/zimbra/jetty_base/webapps/zimlet/, file:///opt/zimbra/zimlets-deployed/],AVAILABLE}{/zimlet}
2018-03-13 20:50:26.969:INFO:oejs.AbstractNCSARequestLog:main: Opened /opt/zimbra/log/access_log.2018-03-13
2018-03-13 20:50:26.993:INFO:oejm.ThreadMonitor:main: Thread Monitor started successfully
2018-03-13 20:50:27.002:INFO:oejs.ServerConnector:main: Started ServerConnector@3439f68d{HTTP/1.1,[http/1.1]}{localhost:80}
2018-03-13 20:50:27.056:INFO:oejus.SslContextFactory:main: x509=X509@a1cdc6d(jetty,h=[xxx.yyy.zzz],w=[]) for SslContextFactory@313b2ea6(file:///opt/zimbra/jetty_base/etc/keystore,null)
2018-03-13 20:50:27.247:INFO:oejs.ServerConnector:main: Started ServerConnector@130f889{SSL,[ssl, http/1.1]}{0.0.0.0:443}
2018-03-13 20:50:27.247:INFO:oejs.ServerConnector:main: Started ServerConnector@1188e820{SSL,[ssl, http/1.1]}{0.0.0.0:7071}
2018-03-13 20:50:27.248:INFO:oejs.ServerConnector:main: Started ServerConnector@2f490758{SSL,[ssl, http/1.1]}{0.0.0.0:7073}
2018-03-13 20:50:27.248:INFO:oejs.ServerConnector:main: Started ServerConnector@101df177{SSL,[ssl, http/1.1]}{0.0.0.0:7072}
2018-03-13 20:50:27.249:INFO:oejs.Server:main: Started @3286ms
But some new problems appears:

zimbra.log is full of repeating every 1 min blocks of messages:

Code: Select all

$ cat /var/log/zimbra.log | grep "20:54"
Mar 13 20:54:00 yyy postfix/postqueue[17974]: fatal: Queue report unavailable - mail system is down
Mar 13 20:54:20 xxx zmconfigd[10630]: Fetching All configs
Mar 13 20:54:20 xxx zmconfigd[10630]: All configs fetched in 0.05 seconds
Mar 13 20:54:21 xxx zmconfigd[10630]: Watchdog: skipping service antivirus. Service not yet available for restarts.
Mar 13 20:54:21 xxx zmconfigd[10630]: All rewrite threads completed in 0.00 sec
Mar 13 20:54:21 xxx zmconfigd[10630]: All restarts completed in 0.00 sec
Mar 13 20:54:30 yyy postfix/postqueue[18289]: fatal: Queue report unavailable - mail system is down
Mar 13 20:54:53 yyy zmconfigd[1211]: Fetching All configs
Mar 13 20:54:53 yyy zmconfigd[1211]: All configs fetched in 0.03 seconds
Mar 13 20:54:56 yyy zmconfigd[1211]: Watchdog: service antivirus status is OK.
Mar 13 20:54:56 yyy zmconfigd[1211]: All rewrite threads completed in 0.00 sec
Mar 13 20:54:56 yyy zmconfigd[1211]: All restarts completed in 0.00 sec
Web services at mailbox server (xxx.yyy.zzz) doesn't work:
Error 404 - Not Found.
No context on this server matched or handled this request.
Contexts known to this server are:
/zimlet ---> o.e.j.w.WebAppContext@1534f01b{/zimlet,[file:///opt/zimbra/jetty_base/webapps/zimlet/, file:///opt/zimbra/zimlets-deployed/],AVAILABLE}{/zimlet}
I think the last one happens because no "webapp" services installed at mailbox server. Here's an output from other ldap+mailbox+logger server:

Code: Select all

# sudo -Hu zimbra /opt/zimbra/bin/zmcontrol status | grep webapp
        service webapp          Running
        zimbra webapp           Running
        zimbraAdmin webapp      Running
        zimlet webapp           Running
Last edited by SEA80 on Tue Mar 13, 2018 9:53 pm, edited 1 time in total.
User avatar
DualBoot
Elite member
Elite member
Posts: 1326
Joined: Mon Apr 18, 2016 8:18 pm
Location: France - Earth
ZCS/ZD Version: ZCS FLOSS - 8.8.15 Mutli servers
Contact:

Re: zmmailboxdctl is not running + java "no valid keystore"

Post by DualBoot »

Hello,

sometime just recreate keystore solve the problem.
- stop mailboxd
- delete or rename the keystore
- regenerate certificate
- deploy certificate
- start mailboxd

Regards,
matteo.fracassetti
Posts: 16
Joined: Thu Feb 08, 2018 3:56 pm

Re: zmmailboxdctl is not running + java "no valid keystore"

Post by matteo.fracassetti »

Same issue here, with a new install of 8.8.6_GA FOSS on Oracle Linux 7.
I've followed the "Multi server installation guide": Do you think that those guidelines were "unusual"? Why?

Just upgrading to 8.8.7_GA_1964 then I'll try to fix the keystore location.


EDIT:
I confirm, relocating the keystore solve the issue.
User avatar
DualBoot
Elite member
Elite member
Posts: 1326
Joined: Mon Apr 18, 2016 8:18 pm
Location: France - Earth
ZCS/ZD Version: ZCS FLOSS - 8.8.15 Mutli servers
Contact:

Re: zmmailboxdctl is not running + java "no valid keystore"

Post by DualBoot »

What do you mean relocating ? Change the location to fit zmlocalconfig ?
Thank you for the details.
Post Reply