Zimbra 9 MySql Fail to start

Discuss your pilot or production implementation with other Zimbra admins or our engineers.
User avatar
emnzava
Advanced member
Advanced member
Posts: 53
Joined: Sat Sep 13, 2014 1:28 am
ZCS/ZD Version: 9.0.0.GA_P11 UB 14 Network Edition
Contact:

Zimbra 9 MySql Fail to start

Post by emnzava »

Hi,

Am getting this error when attempting to restart Zimbra.

Zimbra Version Release 9.0.0.GA.3924.UBUNTU14.64

When restarting with zmcontrol restart am getting this error Stopping proxy...failed.

Then i tried to check with MySql Status

Code: Select all

#zmmysqlstatus restart
/opt/zimbra/common/bin/mysql: symbol lookup error: /opt/zimbra/common/bin/mysql: undefined symbol: X509_check_host
While checking Zimbra log i can see these msgs

Code: Select all

Caused by: java.net.ConnectException: Connection refused
	at java.base/sun.nio.ch.Net.pollConnect(Native Method)
	at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:579)
	at java.base/sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:542)
	at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:597)
	at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:339)
	at java.base/java.net.Socket.connect(Socket.java:603)
	at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.connect(AbstractConnectProtocol.java:409)
	at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1128)
	... 75 more
2021-04-04 14:18:19,964 WARN  [main] [] misc - Could not establish a connection to the database.  Retrying in 5 seconds.
com.zimbra.common.service.ServiceException: system failure: getting database connection
ExceptionId:main:1617535099964:127de3dfbc586118
Code:service.FAILURE
	at com.zimbra.common.service.ServiceException.FAILURE(ServiceException.java:288)
	at com.zimbra.cs.db.DbPool.getConnection(DbPool.java:360)
	at com.zimbra.cs.db.DbPool.getConnection(DbPool.java:322)
	at com.zimbra.cs.db.DbPool.waitForDatabase(DbPool.java:243)
	at com.zimbra.cs.db.DbPool.startup(DbPool.java:234)
	at com.zimbra.cs.util.Zimbra.startup(Zimbra.java:218)
	at com.zimbra.cs.util.Zimbra.startup(Zimbra.java:177)
	at com.zimbra.soap.SoapServlet.init(SoapServlet.java:130)
	at javax.servlet.GenericServlet.init(GenericServlet.java:244)
	at org.eclipse.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:671)
	at org.eclipse.jetty.servlet.ServletHolder.initialize(ServletHolder.java:428)
	at org.eclipse.jetty.servlet.ServletHandler.lambda$initialize$0(ServletHandler.java:750)
	at java.base/java.util.stream.SortedOps$SizedRefSortingSink.end(SortedOps.java:357)
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:485)
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
	at java.base/java.util.stream.StreamSpliterators$WrappingSpliterator.forEachRemaining(StreamSpliterators.java:312)
	at java.base/java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:735)
	at java.base/java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:734)
	at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:658)
	at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:744)
	at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:369)
	at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1497)
	at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1459)
	at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:852)
	at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:278)
	at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:545)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:167)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:119)
	at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:167)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:119)
	at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:167)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110)
	at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:167)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110)
	at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
	at org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:406)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:167)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110)
	at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
	at org.eclipse.jetty.server.handler.DebugHandler.doStart(DebugHandler.java:142)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:167)
	at org.eclipse.jetty.server.Server.start(Server.java:418)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110)
	at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
	at org.eclipse.jetty.server.Server.doStart(Server.java:382)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
	at org.eclipse.jetty.xml.XmlConfiguration$1.run(XmlConfiguration.java:1572)
	at org.eclipse.jetty.xml.XmlConfiguration$1.run(XmlConfiguration.java:1512)
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:554)
	at org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1511)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:567)
	at org.eclipse.jetty.start.Main.invokeMain(Main.java:220)
	at org.eclipse.jetty.start.Main.start(Main.java:490)
	at org.eclipse.jetty.start.Main.main(Main.java:77)
Caused by: java.sql.SQLNonTransientConnectionException: Could not connect to address=(host=127.0.0.1)(port=7306)(type=master) : Connection refused
	at org.mariadb.jdbc.internal.util.exceptions.ExceptionMapper.get(ExceptionMapper.java:240)
	at org.mariadb.jdbc.internal.util.exceptions.ExceptionMapper.connException(ExceptionMapper.java:101)
	at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1136)
	at org.mariadb.jdbc.internal.util.Utils.retrieveProxy(Utils.java:620)
	at org.mariadb.jdbc.MariaDbConnection.newConnection(MariaDbConnection.java:175)
	at org.mariadb.jdbc.Driver.connect(Driver.java:92)
	at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:677)
	at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:189)
	at org.apache.commons.dbcp.DriverManagerConnectionFactory.createConnection(DriverManagerConnectionFactory.java:78)
	at com.zimbra.cs.db.ZimbraConnectionFactory.createConnection(ZimbraConnectionFactory.java:67)
	at org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:582)
	at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1188)
	at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:106)
	at com.zimbra.cs.db.DbPool.getConnection(DbPool.java:341)
	... 64 more

Any help on how to solve this?
User avatar
emnzava
Advanced member
Advanced member
Posts: 53
Joined: Sat Sep 13, 2014 1:28 am
ZCS/ZD Version: 9.0.0.GA_P11 UB 14 Network Edition
Contact:

Re: Zimbra 9 MySql Fail to start

Post by emnzava »

Any response? Does this mean MySql Database is corrupt?
User avatar
emnzava
Advanced member
Advanced member
Posts: 53
Joined: Sat Sep 13, 2014 1:28 am
ZCS/ZD Version: 9.0.0.GA_P11 UB 14 Network Edition
Contact:

Re: Zimbra 9 MySql Fail to start

Post by emnzava »

I can't restart MySql to do Database Recovery. I get same error when i try to restart mysql.

Code: Select all

#mysql.server start
Starting mysqld.../opt/zimbra/common/bin/mysqladmin: symbol lookup error: /opt/zimbra/common/bin/mysqladmin: undefined symbol: X509_check_host
/opt/zimbra/common/bin/mysqladmin: symbol lookup error: /opt/zimbra/common/bin/mysqladmin: undefined symbol: X509_check_host
Regards
User avatar
JDunphy
Outstanding Member
Outstanding Member
Posts: 889
Joined: Fri Sep 12, 2014 11:18 pm
Location: Victoria, BC
ZCS/ZD Version: 9.0.0_P39 NETWORK Edition

Re: Zimbra 9 MySql Fail to start

Post by JDunphy »

emnzava wrote: Then i tried to check with MySql Status

Code: Select all

#zmmysqlstatus restart
/opt/zimbra/common/bin/mysql: symbol lookup error: /opt/zimbra/common/bin/mysql: undefined symbol: X509_check_host
?
It means that your mysql binary is missing a library that it was linked against when compiled. If you do something like:

Code: Select all

% ldd /opt/zimbra/common/bin/mysql
It would show you the missing shared library. I don't run Zimbra 9 nor have ubuntu so perhaps repull the update. Nothing wrong with your database. The problem is that mysql isn't running because the binary image isn't complete. Get a working and complete mysql binary and everything should come up for you.

Jim
User avatar
emnzava
Advanced member
Advanced member
Posts: 53
Joined: Sat Sep 13, 2014 1:28 am
ZCS/ZD Version: 9.0.0.GA_P11 UB 14 Network Edition
Contact:

Re: Zimbra 9 MySql Fail to start

Post by emnzava »

Thanks Jim for the response.

Tried to repull the update without success because zimbra is running with latest update.

How do i get binary image for mysql.

Here is the output for command ldd /opt/zimbra/common/bin/mysql

Code: Select all

#:/home/administrator# ldd /opt/zimbra/common/bin/mysql
	linux-vdso.so.1 =>  (0x00007fff86772000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f89cd08f000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f89cce8b000)
	libssl.so.1.0.0 => /lib/x86_64-linux-gnu/libssl.so.1.0.0 (0x00007f89ccc2c000)
	libcrypto.so.1.0.0 => /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 (0x00007f89cc84f000)
	libncurses.so.5 => /lib/x86_64-linux-gnu/libncurses.so.5 (0x00007f89cc62c000)
	libtinfo.so.5 => /lib/x86_64-linux-gnu/libtinfo.so.5 (0x00007f89cc403000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f89cc0fd000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f89cbd34000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f89cd8c7000)
	
Emmanuel
User avatar
JDunphy
Outstanding Member
Outstanding Member
Posts: 889
Joined: Fri Sep 12, 2014 11:18 pm
Location: Victoria, BC
ZCS/ZD Version: 9.0.0_P39 NETWORK Edition

Re: Zimbra 9 MySql Fail to start

Post by JDunphy »

My guess it was missing a shared library wasn't the case but it appears to be built incorrectly. That X509_check_host is part of the openssl library which includes both libcrypto and libssl

Ref: https://www.openssl.org/docs/man1.1.0/m ... _host.html

Here is what the 8.8.15P20 looks like for mysql on my host.

Code: Select all

ldd mysql
	linux-vdso.so.1 =>  (0x00007ffea1b83000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f19b1a4b000)
	librt.so.1 => /lib64/librt.so.1 (0x00007f19b1843000)
	libdl.so.2 => /lib64/libdl.so.2 (0x00007f19b163f000)
	libssl.so.1.0.0 => /opt/zimbra/common/lib/mysql/libssl.so.1.0.0 (0x00007f19b13d0000)
	libcrypto.so.1.0.0 => /opt/zimbra/common/lib/mysql/libcrypto.so.1.0.0 (0x00007f19b0ef4000)
	libncurses.so.5 => /lib64/libncurses.so.5 (0x00007f19b0cd2000)
	libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00007f19b09cc000)
	libm.so.6 => /lib64/libm.so.6 (0x00007f19b0748000)
	libc.so.6 => /lib64/libc.so.6 (0x00007f19b03b4000)
	libtinfo.so.5 => /lib64/libtinfo.so.5 (0x00007f19b0193000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f19b1c68000)
	libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f19aff7d000)
	
The X509_check_host function is located in /opt/zimbra/common/lib/mysql/libcrypto.so.1.0.0 with the 8.8.15P20 version.

Code: Select all

% nm -D /opt/zimbra/common/lib/mysql/libcrypto.so.1.0.0 | grep -i X509_check_host
00000000001e40c0 T X509_check_host
I was expecting your libssl and libcrypt to reference /opt/zimbra/common/lib/mysql but they are

Code: Select all

  libssl.so.1.0.0 => /lib/x86_64-linux-gnu/libssl.so.1.0.0 (0x00007f89ccc2c000)
   libcrypto.so.1.0.0 => /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 (0x00007f89cc84f000)
   
Maybe that is normal for Zimbra 9??? I wonder why X509_check_host is not present in your /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 as I show with nm -D above with 8.8.15P20. Perhaps run nm -D on it as I did for a sanity check. You should expect to see a T (text) entry for it and not U (undefined).I do not know why your mysql binary seems to be built incorrectly. Was this the result from the latest patch P13 for Zimbra 9 announced last week? We had a few problems for older platforms on the 8.8.15P20 patch we are still working through.

Jim
User avatar
emnzava
Advanced member
Advanced member
Posts: 53
Joined: Sat Sep 13, 2014 1:28 am
ZCS/ZD Version: 9.0.0.GA_P11 UB 14 Network Edition
Contact:

Re: Zimbra 9 MySql Fail to start

Post by emnzava »

Am running on Release 9.0.0.GA.3924.UBUNTU14.64 Patch 9.0.0_P11.

Tried to run nm -D and it gives "No such file" error

Code: Select all

# nm -D /opt/zimbra/common/lib/libcrypto.so.1.0.0 | grep -i X509_check_host
nm: '/opt/zimbra/common/lib/libcrypto.so.1.0.0': No such file
If i change to version 1.1 i get expected results.

Code: Select all

# nm -D /opt/zimbra/common/lib/libcrypto.so.1.1 | grep -i X509_check_host
00000000002178c0 T X509_check_host
Below command return empty results

Code: Select all

# nm -D /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 | grep -i X509_check_host
Checking # ls /opt/zimbra/common/lib/ I can see the file with 1.1 seems to be updated file.

Code: Select all

apache2                      libcurl.so.4.4.0                liblber-2.4.so.2.10.12       libroken.so.18.1.0
apr-util-1                   libdb-5.2.so                    libldap-2.4.so.2             librrd.so.2
aspell-0.60                  libevent-2.0.so.5               libldap-2.4.so.2.10.12       librrd.so.2.0.15
engines-1.1                  libevent-2.0.so.5.1.9           libldap_r-2.4.so.2           librrd_th.so.2
jvm                          libevent_core-2.0.so.5          libldap_r-2.4.so.2.10.12     librrd_th.so.2.0.13
jylibs                       libevent_core-2.0.so.5.1.9      liblmdb.so.0                 libsasl2.so.3
libapr-1.so.0                libevent_extra-2.0.so.5         liblmdb.so.0.0.0             libsasl2.so.3.0.0
libapr-1.so.0.5.2            libevent_extra-2.0.so.5.1.9     libltdl.so.7                 libsl.so.0
libaprutil-1.so.0            libevent_pthreads-2.0.so.5      libltdl.so.7.2.1             libsl.so.0.2.1
libaprutil-1.so.0.5.4        libevent_pthreads-2.0.so.5.1.9  libmariadb.so.3              libsodium.so.17
libart_lgpl_2.so.2           libfreetype.so.6                libmysqlclient_r.so.18       libsodium.so.17.0.0
libart_lgpl_2.so.2.3.21      libfreetype.so.6.12.3           libmysqlclient_r.so.18.0.0   libssl.a
libasn1.so.8                 libfreshclam.so.2               libmysqlclient.so.18         libssl.so.1.1
libasn1.so.8.0.0             libfreshclam.so.2.0.0           libmysqlclient.so.18.0.0     libtcmalloc_minimal_debug.so
libaspell.so.15              libgssapi.so.3                  libnetsnmpagent.so.35        libtcmalloc_minimal_debug.so.4
libaspell.so.15.1.5          libgssapi.so.3.0.0              libnetsnmpagent.so.35.0.0    libtcmalloc_minimal_debug.so.4.2.6
libbsd.so.0                  libhcrypto.so.4                 libnetsnmphelpers.so.35      libtcmalloc_minimal.so
libbsd.so.0.7.0              libhcrypto.so.4.1.0             libnetsnmphelpers.so.35.0.0  libtcmalloc_minimal.so.4
libclamav.so.9               libheimbase.so.1                libnetsnmpmibs.so.35         libtcmalloc_minimal.so.4.2.6
libclamav.so.9.0.4           libheimbase.so.1.0.0            libnetsnmpmibs.so.35.0.0     libvbr.so.2
libclammspack.so.0           libheimedit.so.0                libnetsnmp.so.35             libvbr.so.2.0.0
libclammspack.so.0.1.0       libheimedit.so.0.0.36           libnetsnmp.so.35.0.0         libwind.so.0
libclamunrar_iface.so.9      libheimntlm.so.0                libnetsnmptrapd.so.35        libwind.so.0.0.0
libclamunrar_iface.so.9.0.4  libheimntlm.so.0.1.0            libnetsnmptrapd.so.35.0.0    libxml2.so.2
libclamunrar.so.9            libheimsqlite.so.0              libopendkim.so.10            libxml2.so.2.9.4
libclamunrar.so.9.0.4        libheimsqlite.so.0.0.0          libopendkim.so.10.0.3        libzmq.so.5
libcom_err.so.1              libhx509.so.5                   libpng16.so.16               libzmq.so.5.0.0
libcom_err.so.1.1.3          libhx509.so.5.0.0               libpng16.so.16.23.0          perl5
libcrypto.a                  libkrb5.so.26                   libpspell.so.15              plugin
libcrypto.so.1.1             libkrb5.so.26.0.0               libpspell.so.15.1.5          policyd-2.1
libcurl.so.4                 liblber-2.4.so.2                libroken.so.18               sasl2

I have a test server which runs on Release 9.0.0.GA.3924.UBUNTU18.64 Patch 9.0.0_P7. ldd results looks different from production server as follows:-

Code: Select all

#ldd /opt/zimbra/common/bin/mysql
linux-vdso.so.1 (0x00007ffff3b8b000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f2ada0b1000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f2ad9ead000)
        libssl.so.1.0.0 => /opt/zimbra/common/lib/libssl.so.1.0.0 (0x00007f2ad9c3b000)
        libcrypto.so.1.0.0 => /opt/zimbra/common/lib/libcrypto.so.1.0.0 (0x00007f2ad973d000)
        libncurses.so.5 => /lib/x86_64-linux-gnu/libncurses.so.5 (0x00007f2ad951a000)
        libtinfo.so.5 => /lib/x86_64-linux-gnu/libtinfo.so.5 (0x00007f2ad92f0000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f2ad8eff000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f2ada8e6000)
nm -D /opt/zimbra/common/lib/libcrypto.so.1.0.0 retuns result

Code: Select all

# nm -D /opt/zimbra/common/lib/libcrypto.so.1.0.0 | grep -i X509_check_host
00000000001fd390 T X509_check_host
So it sugests that there is updated file in /opt/zimbra/common/lib/ which is libcrypto.so.1.1 and it is not referenced.

How do i go about solving it.

Regards,
User avatar
JDunphy
Outstanding Member
Outstanding Member
Posts: 889
Joined: Fri Sep 12, 2014 11:18 pm
Location: Victoria, BC
ZCS/ZD Version: 9.0.0_P39 NETWORK Edition

Re: Zimbra 9 MySql Fail to start

Post by JDunphy »

You might investigate ldconfig -v to see the search paths and the search order. Didn't exhaustively do a search here but this looks reasonable enough for the answer from what I remember.

Ref: https://unix.stackexchange.com/question ... s-paths-in

If you trust your installation, you could also override the order with LD_LIBRARY_PATH to add the directories of the shared libraries for the zimbra user so that when it starts up mysql if uses a different order. These are wild guesses on my part as I would think this would have been done previously if that is how they pick their libraries/ Probably the safest is to use LD_LIBRARY_PATH and force that with the zimbra script that start mysql like ~zimbra/bin/mysql.server

Wait... I was going to show you how and found this in ~/opt/zimbra/common/bin

Code: Select all

% grep LD_LIBRARY *
envvars:if test "x$LD_LIBRARY_PATH" != "x" ; then
envvars:  LD_LIBRARY_PATH="/opt/zimbra/common/lib:$LD_LIBRARY_PATH"
envvars:  LD_LIBRARY_PATH="/opt/zimbra/common/lib"
envvars:export LD_LIBRARY_PATH
envvars-std:if test "x$LD_LIBRARY_PATH" != "x" ; then
envvars-std:  LD_LIBRARY_PATH="/opt/zimbra/common/lib:$LD_LIBRARY_PATH"
envvars-std:  LD_LIBRARY_PATH="/opt/zimbra/common/lib"
envvars-std:export LD_LIBRARY_PATH
Binary file httpd matches
grep: jhat: No such file or directory
mysqld_safe:# LD_LIBRARY_PATH and stripped from the lib value.
mysqld_safe:      # Must strip path from lib, and add it to LD_LIBRARY_PATH
mysqld_safe:# Returns LD_PRELOAD (and LD_LIBRARY_PATH, if needed) text, quoted to be
mysqld_safe:    [ -n "$LD_LIBRARY_PATH" ] && new_text="$new_text:$LD_LIBRARY_PATH"
mysqld_safe:    text="${text}LD_LIBRARY_PATH="`shell_quote_string "$new_text"`' '
wsrep_sst_rsync:[ "$OS" == "Darwin" ] && export -n LD_LIBRARY_PATH
broken-relay2:~zimbra/common/bin:286> pwd
So probably take a look at mysqld_safe on your two machines to see if that helps.

Jim
User avatar
emnzava
Advanced member
Advanced member
Posts: 53
Joined: Sat Sep 13, 2014 1:28 am
ZCS/ZD Version: 9.0.0.GA_P11 UB 14 Network Edition
Contact:

Re: Zimbra 9 MySql Fail to start

Post by emnzava »

Dear Jim,
Once again thanks for your help.

I did copy libcrypto.so.1.0.0 and libssl.so.1.0.0 from test server running patch 7 into the path mentioned by ldd /opt/zimbra/common/bin/mysql.

Then i managed to start zimbra.

Is it OK to do that?

Here is the output of ldconfig -v. I excluded rest of infos.

Code: Select all

/sbin/ldconfig.real: /lib/x86_64-linux-gnu/libssl.so.1.0.0 is not a symbolic link

	libssl.so.1.0.0 -> libssl.so.1.0.0-Backup

	/sbin/ldconfig.real: /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 is not a symbolic link

	libcrypto.so.1.0.0 -> libcrypto.so.1.0.0-Backup
In /opt/zimbra/common/lib/ i can only see latest Version 1.1 for libcrypto.so.1.1 and libssl.so.1.1.

I think somehow update didn't go as planned for openssl.

Regards,
User avatar
JDunphy
Outstanding Member
Outstanding Member
Posts: 889
Joined: Fri Sep 12, 2014 11:18 pm
Location: Victoria, BC
ZCS/ZD Version: 9.0.0_P39 NETWORK Edition

Re: Zimbra 9 MySql Fail to start

Post by JDunphy »

To be sure I understand what you did.

1) You had a working machine that you believe is the same release and packages as the release on the failed machine
2) You manually updated the 2 shared libraries from the working machine to the failed machine
3) Everything started and is working on the failed machine

If that is the case, you are probably ok. Another method would have been to force the re-installation of the zimbra openssl-lib package to bring that failed machine to the same level. You would need to research ubuntu on how to do that but some methods including purging the library before installing it again might have worked. Seems really odd that this could happen however and I don't have an explanation how that could happen off the top of my head.

I'll add a disclaimer of dangers of switching shared libraries when you don't match exactly the version that the code was compiled against. Those libraries contains data structures and functions that expect the identical arguments and data structures. If you replace a shared library with another shared library with the same name that has a data structure that is smaller/larger or functions where they have added/deleted arguments or new/deleted functions then you might experience core dumps as a result at startup or fail at what feels like random times. In fact you saw this first hand because the binary was expecting a function that wasn't present in the shared library you had on the failed machine. On the other hand, it's rare to see major structural changes when comparing the same major/minor numbers of the shared libraries but it can happen when someone forgets to update those numbers and puts it back out and someone pulls it. In some organizations this is different people and they could move the wrong physical library to the repositories. This should be automated in 2021 with devops so I doubt that is what happened here. In fact, I doubt zimbra does anything other than pull from the opensource version and compiled it before moving it to their repository so that their install script can put it to the expected location. All random guesses by me to as root cause for you.

HTH,

Jim
Post Reply