Error on deploying SSL certificates

Discuss your pilot or production implementation with other Zimbra admins or our engineers.
octet
Posts: 23
Joined: Thu Jan 06, 2022 2:35 pm

Re: Error on deploying SSL certificates

Post by octet »

Slight progress, modified /etc/hosts file to this and ldap now starts:

Code: Select all

[zimbra@mail ~]$ cat /etc/hosts
127.0.0.1       localhost.localdomain   localhost
51.158.xx.xx	mail.XXX.com	mail
51.158.xx.xx	web-mail.XXX.com	web-mail
192.168.168.168	mail.XXX.com mail
10.70.150.43	mail.XXX.com mail
[zimbra@mail ~]$ 
Trying to start zimbra, I get this:

Code: Select all

[zimbra@mail tmp]$ zmcontrol restart
Host mail.XXX.com
	Stopping zmconfigd...Done.
	Stopping zimlet webapp...Done.
	Stopping zimbraAdmin webapp...Done.
	Stopping zimbra webapp...Done.
	Stopping service webapp...Done.
	Stopping stats...Done.
	Stopping mta...Done.
	Stopping spell...Done.
	Stopping snmp...Done.
	Stopping cbpolicyd...Done.
	Stopping archiving...Done.
	Stopping opendkim...Done.
	Stopping amavis...Done.
	Stopping antivirus...Done.
	Stopping antispam...Done.
	Stopping proxy...Done.
	Stopping memcached...Done.
	Stopping mailbox...Done.
	Stopping logger...Done.
	Stopping dnscache...Done.
	Stopping ldap...Done.
Host mail.XXX.com
	Starting ldap...Done.
Search error: Unable to determine enabled services from ldap.
Unable to determine enabled services. Cache is out of date or doesn't exist.
[zimbra@mail tmp]$
Trying to rename the hosts to mail.XXX.com just in case there was anything left with the temp host web-mail.xxx.com I get this:

Code: Select all

[zimbra@mail ~]$ /opt/zimbra/libexec/zmsetservername -f -n mail.XXX.com
Getting local config zimbra_server_hostname=mail.XXX.com
Hostname is already mail.XXX.com.
Getting local config zimbra_ldap_userdn=uid=zimbra,cn=admins,cn=zimbra
Getting local config zimbra_ldap_password=XXXXX
Getting local config ldap_is_master=true
Getting local config ldap_url=ldap://mail.XXX.com:389
Getting local config ldap_master_url=ldap://mail.XXX.com:389
Getting local config ldap_starttls_supported=0
Starting ldap...already running.
Renaming mail.XXXX.com to mail.XXX.com
Shutting down zimbra...done.
Getting local config ldap_host=mail.XXX.com
Getting local config av_notify_user=admin@XXX.com
Getting local config av_notify_domain=XXX.com
Getting local config snmp_trap_host=mail.XXX.com
Getting local config smtp_source=admin@XXX.com
Getting local config smtp_destination=admin@XXX.com
Starting ldap...done.

Failed to get server config for mail.XXX.com. 
ldap bind failed for uid=zimbra,cn=admins,cn=zimbra
ldap bind failed for uid=zimbra,cn=admins,cn=zimbraldap bind failed for uid=zimbra,cn=admins,cn=zimbraServices: 
[zimbra@mail ~]$
I'll try to install / upgrade again, but I now found mysql doesn't start, here is the log:

Code: Select all

220108 15:41:49 mysqld_safe Starting mysqld daemon with databases from /opt/zimbra/db/data
2022-01-08 15:41:49 139717620930432 [Note] /opt/zimbra/common/sbin/mysqld (mysqld 10.1.25-MariaDB) starting as process 197838 ...
2022-01-08 15:41:49 139717620930432 [Note] InnoDB: Using mutexes to ref count buffer pool pages
2022-01-08 15:41:49 139717620930432 [Note] InnoDB: The InnoDB memory heap is disabled
2022-01-08 15:41:49 139717620930432 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2022-01-08 15:41:49 139717620930432 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2022-01-08 15:41:49 139717620930432 [Note] InnoDB: Compressed tables use zlib 1.2.3
2022-01-08 15:41:49 139717620930432 [Note] InnoDB: Using Linux native AIO
2022-01-08 15:41:49 139717620930432 [Note] InnoDB: Using SSE crc32 instructions
2022-01-08 15:41:49 139717620930432 [Note] InnoDB: Initializing buffer pool, size = 2.3G
2022-01-08 15:41:49 139717620930432 [Note] InnoDB: Completed initialization of buffer pool
2022-01-08 15:41:49 139717620930432 [ERROR] InnoDB: Log file ./ib_logfile0 size 524314036 is not a multiple of innodb_page_size
2022-01-08 15:41:49 139717620930432 [ERROR] Plugin 'InnoDB' init function returned error.
2022-01-08 15:41:49 139717620930432 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2022-01-08 15:41:49 139717620930432 [Note] Plugin 'FEEDBACK' is disabled.
2022-01-08 15:41:49 139717620930432 [ERROR] Unknown/unsupported storage engine: InnoDB
2022-01-08 15:41:49 139717620930432 [ERROR] Aborting

220108 15:41:49 mysqld_safe mysqld from pid file /opt/zimbra/log/mysql.pid ended
Moved the ib_log files and tried to restart mysql, crashes completely, so I moved them back.

Code: Select all

[root@mail data]# pwd
/opt/zimbra/db/data
[root@mail data]# rm -rf ib_logfile*

Code: Select all

220108 15:52:26 mysqld_safe Starting mysqld daemon with databases from /opt/zimbra/db/data
2022-01-08 15:52:26 140355787757440 [Note] /opt/zimbra/common/sbin/mysqld (mysqld 10.1.25-MariaDB) starting as process 198834 ...
2022-01-08 15:52:26 140355787757440 [Note] InnoDB: Using mutexes to ref count buffer pool pages
2022-01-08 15:52:26 140355787757440 [Note] InnoDB: The InnoDB memory heap is disabled
2022-01-08 15:52:26 140355787757440 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2022-01-08 15:52:26 140355787757440 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2022-01-08 15:52:26 140355787757440 [Note] InnoDB: Compressed tables use zlib 1.2.3
2022-01-08 15:52:26 140355787757440 [Note] InnoDB: Using Linux native AIO
2022-01-08 15:52:26 140355787757440 [Note] InnoDB: Using SSE crc32 instructions
2022-01-08 15:52:26 140355787757440 [Note] InnoDB: Initializing buffer pool, size = 2.3G
2022-01-08 15:52:26 140355787757440 [Note] InnoDB: Completed initialization of buffer pool
2022-01-08 15:52:26 140355787757440 [Note] InnoDB: Setting log file ./ib_logfile101 size to 500 MB
2022-01-08 15:52:28 140355787757440 [Note] InnoDB: Setting log file ./ib_logfile1 size to 500 MB
2022-01-08 15:52:30 140355787757440 [Note] InnoDB: Renaming log file ./ib_logfile101 to ./ib_logfile0
2022-01-08 15:52:30 140355787757440 [Warning] InnoDB: New log files created, LSN=670990738
2022-01-08 15:52:30 140355787757440 [Note] InnoDB: Highest supported file format is Barracuda.
2022-01-08 15:52:30 140355787757440 [Note] InnoDB: Checksum fields zero but page is not empty.
2022-01-08 15:52:30 140355787757440 [ERROR] InnoDB: Database page corruption on disk or a failed file read of tablespace ./ibdata1 page  [page id: space=0, page number=573]. You may have to recover from a backup.
2022-01-08 15:52:30 7fa720df1780 InnoDB: Page dump in ascii and hex (16384 bytes):
 len 16384; hex 000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000743a0e3951389f379336e1368c35dd352f348533d33327327331c3310d30632faf2eff2e4f2da32cf42c412b822b2d2a142957289e2817279226e025d9252a247f23d22323226f21bf210e205d1fb31eff1e501d9f1cee1c3d1b871a411a73195a189d17eb16df162d15d81529147b13
 InnoDB: End of page dump
2022-01-08 15:52:31 7fa720df1780 InnoDB: uncompressed page, stored checksum in field1 0, calculated checksums for field1: crc32 2697113374, innodb 2698999090, none 3735928559, stored checksum in field2 0, calculated checksums for field2: crc32 2697113374, innodb 1371122432, none 3735928559, page LSN 0 0, low 4 bytes of LSN at page end 0, page number (if stored to page already) 0, space id (if created with >= MySQL-4.1.1 and stored already) 0
InnoDB: page type 0 meaning ALLOCATED
InnoDB: Page may be a freshly allocated page
2022-01-08 15:52:31 140355787757440 [Note] InnoDB: It is also possible that your operating system has corrupted its own file cache and rebooting your computer removes the error. If the corrupt page is an index page. You can also try to fix the corruption by dumping, dropping, and reimporting the corrupt table. You can use CHECK TABLE to scan your table for corruption. Please refer to http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html for information about forcing recovery.
2022-01-08 15:52:31 140355787757440 [ERROR] InnoDB: Ending processing because of a corrupt database page.
2022-01-08 15:52:31 7fa720df1780  InnoDB: Assertion failure in thread 140355787757440 in file ha_innodb.cc line 21945
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
220108 15:52:31 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see https://mariadb.com/kb/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.

Server version: 10.1.25-MariaDB
key_buffer_size=134217728
read_buffer_size=1048576
max_used_connections=0
max_threads=112
thread_count=0
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 362733 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x0 thread_stack 0x48400
/opt/zimbra/common/sbin/mysqld(my_print_stacktrace+0x2e)[0x558b62f2606e]
/opt/zimbra/common/sbin/mysqld(handle_fatal_signal+0x53b)[0x558b62b13a1b]
/lib64/libpthread.so.0(+0x12c20)[0x7fa720775c20]
/lib64/libc.so.6(gsignal+0x10f)[0x7fa71ecc537f]
/lib64/libc.so.6(abort+0x127)[0x7fa71ecafdb5]
/opt/zimbra/common/sbin/mysqld(+0x7718ad)[0x558b62d028ad]
/opt/zimbra/common/sbin/mysqld(+0x8ba1e8)[0x558b62e4b1e8]
/opt/zimbra/common/sbin/mysqld(+0x8d24ab)[0x558b62e634ab]
/opt/zimbra/common/sbin/mysqld(+0x8d3603)[0x558b62e64603]
/opt/zimbra/common/sbin/mysqld(+0x8b1cce)[0x558b62e42cce]
/opt/zimbra/common/sbin/mysqld(+0x8915eb)[0x558b62e225eb]
/opt/zimbra/common/sbin/mysqld(+0x89e06a)[0x558b62e2f06a]
/opt/zimbra/common/sbin/mysqld(+0x8f6182)[0x558b62e87182]
/opt/zimbra/common/sbin/mysqld(+0x8f71e4)[0x558b62e881e4]
/opt/zimbra/common/sbin/mysqld(+0x8d8b4f)[0x558b62e69b4f]
/opt/zimbra/common/sbin/mysqld(+0x84d8ef)[0x558b62dde8ef]
/opt/zimbra/common/sbin/mysqld(+0x763eae)[0x558b62cf4eae]
/opt/zimbra/common/sbin/mysqld(_Z24ha_initialize_handlertonP13st_plugin_int+0x6c)[0x558b62b15b5c]
/opt/zimbra/common/sbin/mysqld(+0x40f695)[0x558b629a0695]
/opt/zimbra/common/sbin/mysqld(_Z11plugin_initPiPPci+0x96a)[0x558b629a17fa]
/opt/zimbra/common/sbin/mysqld(+0x36e410)[0x558b628ff410]
mysys/stacktrace.c:268(my_print_stacktrace)[0x558b62902b93]
sql/signal_handler.cc:168(handle_fatal_signal)[0x7fa71ecb1493]
/opt/zimbra/common/sbin/mysqld(_start+0x2e)[0x558b628f6ade]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
220108 15:52:31 mysqld_safe mysqld from pid file /opt/zimbra/log/mysql.pid ended
Also tried to recover innodb adding this in my.cnf, with the original ib_log files:

Code: Select all

[mysqld]
innodb_force_recovery = 1

Code: Select all

220108 15:54:12 mysqld_safe Starting mysqld daemon with databases from /opt/zimbra/db/data
2022-01-08 15:54:12 139635865540480 [Note] /opt/zimbra/common/sbin/mysqld (mysqld 10.1.25-MariaDB) starting as process 199355 ...
2022-01-08 15:54:12 139635865540480 [Note] InnoDB: Using mutexes to ref count buffer pool pages
2022-01-08 15:54:12 139635865540480 [Note] InnoDB: The InnoDB memory heap is disabled
2022-01-08 15:54:12 139635865540480 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2022-01-08 15:54:12 139635865540480 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2022-01-08 15:54:12 139635865540480 [Note] InnoDB: Compressed tables use zlib 1.2.3
2022-01-08 15:54:12 139635865540480 [Note] InnoDB: Using Linux native AIO
2022-01-08 15:54:12 139635865540480 [Note] InnoDB: Using SSE crc32 instructions
2022-01-08 15:54:12 139635865540480 [Note] InnoDB: Initializing buffer pool, size = 2.3G
2022-01-08 15:54:12 139635865540480 [Note] InnoDB: Completed initialization of buffer pool
2022-01-08 15:54:12 139635865540480 [Note] InnoDB: Highest supported file format is Barracuda.
2022-01-08 15:54:12 139635865540480 [Note] InnoDB: The log sequence number 670990738 in ibdata file do not match the log sequence number 670990860 in the ib_logfiles!
2022-01-08 15:54:13 139635865540480 [Note] InnoDB: Restoring possible half-written data pages from the doublewrite buffer...
2022-01-08 15:54:13 139635865540480 [Note] InnoDB: Checksum fields zero but page is not empty.
2022-01-08 15:54:13 139635865540480 [Note] InnoDB: Trying to recover page 0:524 from the doublewrite buffer.
2022-01-08 15:54:13 139635865540480 [Note] InnoDB: Recovered page 0:524 from the doublewrite buffer.
2022-01-08 15:54:13 139635865540480 [Note] InnoDB: Checksum fields zero but page is not empty.
2022-01-08 15:54:13 139635865540480 [Note] InnoDB: Trying to recover page 0:562 from the doublewrite buffer.
2022-01-08 15:54:13 139635865540480 [Note] InnoDB: Recovered page 0:562 from the doublewrite buffer.
2022-01-08 15:54:13 139635865540480 [Note] InnoDB: Checksum fields zero but page is not empty.
2022-01-08 15:54:13 139635865540480 [Note] InnoDB: Trying to recover page 0:558 from the doublewrite buffer.
2022-01-08 15:54:13 139635865540480 [Note] InnoDB: Recovered page 0:558 from the doublewrite buffer.
2022-01-08 15:54:13 139635865540480 [Note] InnoDB: Log sequence number at the start 1701666665 and the end 0 do not match.
2022-01-08 15:54:13 139635865540480 [Note] InnoDB: Trying to recover page 0:575 from the doublewrite buffer.
2022-01-08 15:54:13 139635865540480 [Note] InnoDB: Recovered page 0:575 from the doublewrite buffer.
2022-01-08 15:54:13 139635865540480 [Note] InnoDB: Checksum fields zero but page is not empty.
2022-01-08 15:54:13 139635865540480 [Note] InnoDB: Trying to recover page 0:534 from the doublewrite buffer.
2022-01-08 15:54:13 139635865540480 [Note] InnoDB: Recovered page 0:534 from the doublewrite buffer.
2022-01-08 15:54:13 139635865540480 [Note] InnoDB: Checksum fields zero but page is not empty.
2022-01-08 15:54:13 139635865540480 [Note] InnoDB: Trying to recover page 0:692 from the doublewrite buffer.
2022-01-08 15:54:13 139635865540480 [Note] InnoDB: Recovered page 0:692 from the doublewrite buffer.
2022-01-08 15:54:13 139635865540480 [Note] InnoDB: Checksum fields zero but page is not empty.
2022-01-08 15:54:13 139635865540480 [Note] InnoDB: Trying to recover page 0:662 from the doublewrite buffer.
2022-01-08 15:54:13 139635865540480 [Note] InnoDB: Recovered page 0:662 from the doublewrite buffer.
2022-01-08 15:54:13 139635865540480 [Note] InnoDB: Checksum fields zero but page is not empty.
2022-01-08 15:54:13 139635865540480 [Note] InnoDB: Trying to recover page 0:704 from the doublewrite buffer.
2022-01-08 15:54:13 139635865540480 [Note] InnoDB: Recovered page 0:704 from the doublewrite buffer.
2022-01-08 15:54:13 139635865540480 [Note] InnoDB: Checksum fields zero but page is not empty.
2022-01-08 15:54:13 139635865540480 [Note] InnoDB: Trying to recover page 0:296 from the doublewrite buffer.
2022-01-08 15:54:13 139635865540480 [Note] InnoDB: Recovered page 0:296 from the doublewrite buffer.
2022-01-08 15:54:13 139635865540480 [Note] InnoDB: Checksum fields zero but page is not empty.
2022-01-08 15:54:13 139635865540480 [Note] InnoDB: Trying to recover page 0:297 from the doublewrite buffer.
2022-01-08 15:54:13 139635865540480 [Note] InnoDB: Recovered page 0:297 from the doublewrite buffer.
2022-01-08 15:54:13 139635865540480 [Note] InnoDB: Checksum fields zero but page is not empty.
2022-01-08 15:54:13 139635865540480 [Note] InnoDB: Trying to recover page 0:817 from the doublewrite buffer.
2022-01-08 15:54:13 139635865540480 [Note] InnoDB: Recovered page 0:817 from the doublewrite buffer.
2022-01-08 15:54:13 139635865540480 [Note] InnoDB: Checksum fields zero but page is not empty.
2022-01-08 15:54:13 139635865540480 [Note] InnoDB: Trying to recover page 0:295 from the doublewrite buffer.
2022-01-08 15:54:13 139635865540480 [Note] InnoDB: Recovered page 0:295 from the doublewrite buffer.
2022-01-08 15:54:13 139635865540480 [Note] InnoDB: Checksum fields zero but page is not empty.
2022-01-08 15:54:13 139635865540480 [Note] InnoDB: Trying to recover page 0:408 from the doublewrite buffer.
2022-01-08 15:54:13 139635865540480 [Note] InnoDB: Recovered page 0:408 from the doublewrite buffer.
2022-01-08 15:54:13 139635865540480 [Note] InnoDB: Checksum fields zero but page is not empty.
2022-01-08 15:54:13 139635865540480 [Note] InnoDB: Trying to recover page 0:415 from the doublewrite buffer.
2022-01-08 15:54:13 139635865540480 [Note] InnoDB: Recovered page 0:415 from the doublewrite buffer.
2022-01-08 15:54:13 139635865540480 [Note] InnoDB: Checksum fields zero but page is not empty.
2022-01-08 15:54:13 139635865540480 [Note] InnoDB: Trying to recover page 0:389 from the doublewrite buffer.
2022-01-08 15:54:13 139635865540480 [Note] InnoDB: Recovered page 0:389 from the doublewrite buffer.
2022-01-08 15:54:13 139635865540480 [Note] InnoDB: Log sequence number at the start 55265091 and the end 0 do not match.
2022-01-08 15:54:13 139635865540480 [Note] InnoDB: Trying to recover page 0:460 from the doublewrite buffer.
2022-01-08 15:54:13 139635865540480 [Note] InnoDB: Recovered page 0:460 from the doublewrite buffer.
2022-01-08 15:54:13 139635865540480 [Note] InnoDB: Log sequence number at the start 4294967295 and the end 0 do not match.
2022-01-08 15:54:13 139635865540480 [Note] InnoDB: Trying to recover page 0:498 from the doublewrite buffer.
2022-01-08 15:54:13 139635865540480 [Note] InnoDB: Recovered page 0:498 from the doublewrite buffer.
2022-01-08 15:54:13 139635865540480 [Note] InnoDB: Checksum fields zero but page is not empty.
2022-01-08 15:54:13 139635865540480 [Note] InnoDB: Trying to recover page 0:479 from the doublewrite buffer.
2022-01-08 15:54:13 139635865540480 [Note] InnoDB: Recovered page 0:479 from the doublewrite buffer.
2022-01-08 15:54:13 139635865540480 [Note] InnoDB: Log sequence number at the start 16783712 and the end 0 do not match.
2022-01-08 15:54:13 139635865540480 [Note] InnoDB: Trying to recover page 35:53 from the doublewrite buffer.
2022-01-08 15:54:13 139635865540480 [Note] InnoDB: Recovered page 35:53 from the doublewrite buffer.
2022-01-08 15:54:13 139635865540480 [Note] InnoDB: Log sequence number at the start 1664314465 and the end 2960261120 do not match.
2022-01-08 15:54:13 139635865540480 [Note] InnoDB: Trying to recover page 35:31 from the doublewrite buffer.
2022-01-08 15:54:13 139635865540480 [Note] InnoDB: Recovered page 35:31 from the doublewrite buffer.
2022-01-08 15:54:13 139635865540480 [Note] InnoDB: Checksum fields zero but page is not empty.
2022-01-08 15:54:13 139635865540480 [Note] InnoDB: Trying to recover page 35:63 from the doublewrite buffer.
2022-01-08 15:54:13 139635865540480 [Note] InnoDB: Recovered page 35:63 from the doublewrite buffer.
2022-01-08 15:54:13 139635865540480 [Note] InnoDB: Log sequence number at the start 1 and the end 1179648 do not match.
2022-01-08 15:54:13 139635865540480 [Note] InnoDB: Trying to recover page 35:41 from the doublewrite buffer.
2022-01-08 15:54:13 139635865540480 [Note] InnoDB: Recovered page 35:41 from the doublewrite buffer.
2022-01-08 15:54:13 139635865540480 [Note] InnoDB: Log sequence number at the start 1480684337 and the end 1664314465 do not match.
2022-01-08 15:54:13 139635865540480 [Note] InnoDB: Trying to recover page 35:80 from the doublewrite buffer.
2022-01-08 15:54:13 139635865540480 [Note] InnoDB: Log sequence number at the start 670962395 and the end 0 do not match.
2022-01-08 15:54:13 139635865540480 [Warning] InnoDB: A doublewrite copy of page 35:80 is corrupted.
2022-01-08 15:54:13 139635865540480 [Note] InnoDB: Checksum fields zero but page is not empty.
2022-01-08 15:54:13 139635865540480 [ERROR] InnoDB: Database page corruption on disk or a failed file read of tablespace ./ibdata1 page  [page id: space=0, page number=573]. You may have to recover from a backup.
2022-01-08 15:54:13 7eff8229d780 InnoDB: Page dump in ascii and hex (16384 bytes):
 len 16384; hex 000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000743a0e3951389f379336e1368c35dd352f348533d33327327331c3310d30632faf2eff2e4f2da32cf42c412b822b2d2a142957289e2817279226e025d9252a247f23d22323226f21bf210e205d1fb31eff1e501d9f1cee1c3d1b871a411a73195a189d17eb16df162d15d81529147b13d1131f127311bf110f10590faf0efb0e4b0d9b0cef0c400b8d0ace0a79096008a307ea076306de062c0525047603cb031e026f01bb010b00656ccd7123024209d9c0172d920000023dffffffff000001750000000000811ef745bf000000000000000000000000001c3b6b00d61e881cf2000000050000006a00000000000000000000000000000000000300000000000000000000000000000000000000000801000003008c696e66696d756d000903000803000073757072656d756d00332f2b27231d1610080000101300ce000000000000000b000000000000000b000000000300800000012d017749445f494e440000000100000003000000000000012e34302c28241d161008000018130111000000000000000b000000000000000c000000000300800000012d01a5464f525f494e440000000100000000000000000000012f34302c28241d161008000020130154000000000000000b000000000000000d000000000300800000012d01d35245465f494e4400000001000000000000000000000130332f2b27231d161008040028130196000000000000000c000000000000000e000000000300800000012d026d49445f494e4400000002000000030000000000000131423e3a36321d1610080000301301e7000000000000000d000000000000000f000000000301810000013201665359535f5441424c455350414345535f535041434500000001000000030000000000000133403c3834301d161008000038130236000000000000000e0000000000000010000000000301810000013201d35359535f4441544146494c45535f53504143450000000100000003000000000000013434302c28241d161008000040130279000000000000000f0000000000000011000000000302820000013501c65052494d4152590000000200000003000000010000000334302c28241d1610080400481302bc00000000000000100000000000000012000000000303830000013601f05052494d4152590000000400000003000000020000000334302c28241d1610080000501302ff00000000000000110000000000000013000000000304840000013701985052494d4152590000000200000003000000030000000334302c28241d16100800005813034200000000000000120000000000000014000000000904840000013702245052494d41525900000001000000030000000400000003332f2b27231d1610080000601303840000000000000012000000000000001500000000090484000001370252695f6e616d6500000001000000020000000400000004332f2b27231d1610080400681303c60000000000000012000000000000001600000000090484000001370280695f70617468000000010000000200000004000000053c3834302c1d16100800007013041100000000000000130000000000000017000000000907870000013a019a47454e5f434c5553545f494e44455800000000000000010000000500000003403c3834301d16100800007813046000000000000000130000000000000018000000000907870000013a01b3695f6d6573736167655f766f6c756d655f6964000000010000000000000005000000044a46423e3a1d1610080000801304b9000000000000001300000000
 InnoDB: End of page dump
2022-01-08 15:54:15 7eff8229d780 InnoDB: uncompressed page, stored checksum in field1 0, calculated checksums for field1: crc32 2697113374, innodb 2698999090, none 3735928559, stored checksum in field2 0, calculated checksums for field2: crc32 2697113374, innodb 1371122432, none 3735928559, page LSN 0 0, low 4 bytes of LSN at page end 0, page number (if stored to page already) 0, space id (if created with >= MySQL-4.1.1 and stored already) 0
InnoDB: page type 0 meaning ALLOCATED
InnoDB: Page may be a freshly allocated page
220108 15:54:15 [ERROR] mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see https://mariadb.com/kb/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.

Server version: 10.1.25-MariaDB
key_buffer_size=134217728
read_buffer_size=1048576
max_used_connections=0
max_threads=112
thread_count=0
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 362733 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x0 thread_stack 0x48400
/opt/zimbra/common/sbin/mysqld(my_print_stacktrace+0x2e)[0x55732ba2a06e]
/opt/zimbra/common/sbin/mysqld(handle_fatal_signal+0x53b)[0x55732b617a1b]
/lib64/libpthread.so.0(+0x12c20)[0x7eff81c21c20]
/opt/zimbra/common/sbin/mysqld(+0x7d6c3d)[0x55732b86bc3d]
/opt/zimbra/common/sbin/mysqld(+0x89150c)[0x55732b92650c]
/opt/zimbra/common/sbin/mysqld(+0x89e06a)[0x55732b93306a]
/opt/zimbra/common/sbin/mysqld(+0x8f6182)[0x55732b98b182]
/opt/zimbra/common/sbin/mysqld(+0x8f71e4)[0x55732b98c1e4]
/opt/zimbra/common/sbin/mysqld(+0x8d8b4f)[0x55732b96db4f]
mysys/stacktrace.c:268(my_print_stacktrace)[0x55732b8e28ef]
include/page0page.ic:679(page_dir_slot_get_rec)[0x55732b7f8eae]
btr/btr0cur.cc:837(btr_cur_search_to_nth_level(dict_index_t*, unsigned long, dtuple_t const*, unsigned long, unsigned long, btr_cur_t*, unsigned long, char const*, unsigned long, mtr_t*))[0x55732b619b5c]
include/ut0byte.ic:133(ut_align_offset)[0x55732b4a4695]
srv/srv0start.cc:2624(innobase_start_or_create_for_mysql())[0x55732b4a57fa]
handler/ha_innodb.cc:4439(innobase_init(void*))[0x55732b403410]
sql/handler.cc:513(ha_initialize_handlerton(st_plugin_int*))[0x55732b406b93]
sql/sql_plugin.cc:1406(plugin_initialize(st_mem_root*, st_plugin_int*, int*, char**, bool))[0x7eff8015d493]
sql/mysqld.cc:5148(init_server_components())[0x55732b3faade]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
220108 15:54:16 mysqld_safe mysqld from pid file /opt/zimbra/log/mysql.pid ended
I also tried to remove the ibdata file + ib_log files

Code: Select all

[root@mail data]# rm -rf ib*
and then tried to start mysql again

Code: Select all

220108 17:17:29 mysqld_safe Starting mysqld daemon with databases from /opt/zimbra/db/data
2022-01-08 17:17:29 140235712141184 [Note] /opt/zimbra/common/sbin/mysqld (mysqld 10.1.25-MariaDB) starting as process 206477 ...
2022-01-08 17:17:30 140235712141184 [Note] InnoDB: Using mutexes to ref count buffer pool pages
2022-01-08 17:17:30 140235712141184 [Note] InnoDB: The InnoDB memory heap is disabled
2022-01-08 17:17:30 140235712141184 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2022-01-08 17:17:30 140235712141184 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2022-01-08 17:17:30 140235712141184 [Note] InnoDB: Compressed tables use zlib 1.2.3
2022-01-08 17:17:30 140235712141184 [Note] InnoDB: Using Linux native AIO
2022-01-08 17:17:30 140235712141184 [Note] InnoDB: Using SSE crc32 instructions
2022-01-08 17:17:30 140235712141184 [Note] InnoDB: Initializing buffer pool, size = 2.3G
2022-01-08 17:17:30 140235712141184 [Note] InnoDB: Completed initialization of buffer pool
2022-01-08 17:17:30 140235712141184 [Note] InnoDB: The first specified data file ./ibdata1 did not exist: a new database to be created!
2022-01-08 17:17:30 140235712141184 [Note] InnoDB: Setting file ./ibdata1 size to 10 MB
2022-01-08 17:17:30 140235712141184 [Note] InnoDB: Setting log file ./ib_logfile101 size to 512 MB
2022-01-08 17:17:31 140235712141184 [Note] InnoDB: Setting log file ./ib_logfile1 size to 512 MB
2022-01-08 17:17:33 140235712141184 [Note] InnoDB: Renaming log file ./ib_logfile101 to ./ib_logfile0
2022-01-08 17:17:33 140235712141184 [Warning] InnoDB: New log files created, LSN=45883
2022-01-08 17:17:33 140235712141184 [Note] InnoDB: Doublewrite buffer not found: creating new
2022-01-08 17:17:33 140235712141184 [Note] InnoDB: Doublewrite buffer created
2022-01-08 17:17:33 140235712141184 [Note] InnoDB: 1 rollback segment(s) are active.
2022-01-08 17:17:33 140235712141184 [ERROR] InnoDB: Database creation was aborted at /home/build/git/88/packages/thirdparty/mariadb/build/RHEL8_64/zimbra-mariadb/rpm/BUILD/mariadb-10.1.25/storage/xtradb/srv/srv0start.cc [2943] with error Generic error. You may need to delete the ibdata1 file before trying to start up again.
2022-01-08 17:17:33 140235712141184 [ERROR] Plugin 'InnoDB' init function returned error.
2022-01-08 17:17:33 140235712141184 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2022-01-08 17:17:33 140235712141184 [Note] Plugin 'FEEDBACK' is disabled.
2022-01-08 17:17:33 140235712141184 [ERROR] Unknown/unsupported storage engine: InnoDB
2022-01-08 17:17:33 140235712141184 [ERROR] Aborting

220108 17:17:33 mysqld_safe mysqld from pid file /opt/zimbra/log/mysql.pid ended
User avatar
JDunphy
Outstanding Member
Outstanding Member
Posts: 889
Joined: Fri Sep 12, 2014 11:18 pm
Location: Victoria, BC
ZCS/ZD Version: 9.0.0_P39 NETWORK Edition

Re: Error on deploying SSL certificates

Post by JDunphy »

octet wrote:Got ldap started manually:

This seems fine, too... I can't understand why it wouldn't let me bind to mail.XXX.com :roll:

Just checked the IP the server has, which is local IP, is this the reason it can't bind on the mail.XXX.com domain, as it is set in /etc/hosts?
You bind to a local ip address on your local machine. You don't bind to some other computers ip address. That means that /etc/hosts needs to be local ip addresses when referencing hostnames that zimbra/ldap is trying to lookup. You have not mentioned a multiple machine configuration nor that you are running zimbra inside a rfc1918 address. You have only shown that you have mail.XXX.com and web-mail.XXXX.com.

Let me try one more way of explaining this.

Both those mail.XXX.com and web-mail.XXXX.com entries should be in your /etc/hosts file and they HAVE to be a local ip address that is configured for this zimbra host. They better not be exterrnal ip addresses off a different computer or fw. If you have done that, comment out those entries. Normally, daemons/services will listen to any interface and all the ip addresses on a computer and is what we did with the manual ldap start method. Zimbra ldap can run on other computers in a multiple machine configuration so they use the ip address explicitly to provide this feature. That is why you see a hostname in the URL with the ldap service. It's also good security practice not to listen on ip addresses that you don't need to provide service on.

Code: Select all

% ps auxw |grep ldap
zimbra   1753749  0.0  0.6 47694828 56564 ?      Ssl   2021   5:03 /opt/zimbra/common/libexec/slapd -l LOCAL0 -u zimbra -h ldap://mail.example.com:389 ldapi:/// -F /opt/zimbra/data/ldap/config
We are telling slapd that we want a listen socket with port 389 on the ip address that is mail.XXX.com ... The program looked it up in /etc/hosts and found ip address of X.Y.X.Y. It would then bind an address and attempt to attach a socket in listen mode on port 389 to the interface on this physical host that has this ip address. Any ldap client making a connection to this ip address on port 389 could than communicate with our daemon. man 2 bind or man 2 listen to read about these system calls.

Does this make sense?
octet wrote:

Code: Select all

[root@mail conf]# ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.70.150.43  netmask 255.255.255.254  broadcast 0.0.0.0
        inet6 2001:bc8:634:2315::1  prefixlen 64  scopeid 0x0<global>
        inet6 fe80::dc1c:a4ff:fe70:a016  prefixlen 64  scopeid 0x20<link>
        ether de:1c:a4:70:a0:16  txqueuelen 1000  (Ethernet)
        RX packets 238528  bytes 71285916 (67.9 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 268041  bytes 365546797 (348.6 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

eth0:0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.168.168  netmask 255.255.255.0  broadcast 192.168.168.255
        ether de:1c:a4:70:a0:16  txqueuelen 1000  (Ethernet)

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 3144  bytes 207417 (202.5 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 3144  bytes 207417 (202.5 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
Ok this is helpful... you have rfc1918 address space from this output. That means that your options are 10.70.150.43 or 192.168.168.168 or 127.0.0.1 on port 398 for possible values for mail.XXX.com in /etc/hosts. When we did it manually, we left off the ip address so it bound to all 3 of those interfaces on port 389. BTW, ipv6 addresses also have the daemon running on port 389 with that manual method since that is also configured here. That would have implications for FW's given there are more pathways to the service. ;-)

Jim
User avatar
JDunphy
Outstanding Member
Outstanding Member
Posts: 889
Joined: Fri Sep 12, 2014 11:18 pm
Location: Victoria, BC
ZCS/ZD Version: 9.0.0_P39 NETWORK Edition

Re: Error on deploying SSL certificates

Post by JDunphy »

octet wrote:Slight progress, modified /etc/hosts file to this and ldap now starts:

Code: Select all

[zimbra@mail ~]$ cat /etc/hosts
127.0.0.1       localhost.localdomain   localhost
51.158.xx.xx	mail.XXX.com	mail
51.158.xx.xx	web-mail.XXX.com	web-mail
192.168.168.168	mail.XXX.com mail
10.70.150.43	mail.XXX.com mail
[zimbra@mail ~]$ 
This is wrong... those external ip entries need to be removed if they are to represent the local machine/service. Get rid of all those 51.158/16 entries. I think you could use 10.70.150.43 for mail.XXX.com also (primary eth0). So I would do something like

Code: Select all

127.0.0.1       localhost.localdomain   localhost
10.70.150.43	mail.XXX.com mail
10.70.150.43    web-mail.XXX.com web-mail
octet
Posts: 23
Joined: Thu Jan 06, 2022 2:35 pm

Re: Error on deploying SSL certificates

Post by octet »

mysql starting now, with all ib files removed

Code: Select all

220108 17:31:32 mysqld_safe Starting mysqld daemon with databases from /opt/zimbra/db/data
2022-01-08 17:31:32 140492844128128 [Note] /opt/zimbra/common/sbin/mysqld (mysqld 10.1.25-MariaDB) starting as process 210327 ...
2022-01-08 17:31:32 140492844128128 [Note] InnoDB: Using mutexes to ref count buffer pool pages
2022-01-08 17:31:32 140492844128128 [Note] InnoDB: The InnoDB memory heap is disabled
2022-01-08 17:31:32 140492844128128 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2022-01-08 17:31:32 140492844128128 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2022-01-08 17:31:32 140492844128128 [Note] InnoDB: Compressed tables use zlib 1.2.3
2022-01-08 17:31:32 140492844128128 [Note] InnoDB: Using Linux native AIO
2022-01-08 17:31:32 140492844128128 [Note] InnoDB: Using SSE crc32 instructions
2022-01-08 17:31:32 140492844128128 [Note] InnoDB: Initializing buffer pool, size = 2.3G
2022-01-08 17:31:32 140492844128128 [Note] InnoDB: Completed initialization of buffer pool
2022-01-08 17:31:32 140492844128128 [Note] InnoDB: The first specified data file ./ibdata1 did not exist: a new database to be created!
2022-01-08 17:31:32 140492844128128 [Note] InnoDB: Setting file ./ibdata1 size to 10 MB
2022-01-08 17:31:32 140492844128128 [Note] InnoDB: Setting log file ./ib_logfile101 size to 512 MB
2022-01-08 17:31:34 140492844128128 [Note] InnoDB: Setting log file ./ib_logfile1 size to 512 MB
2022-01-08 17:31:35 140492844128128 [Note] InnoDB: Renaming log file ./ib_logfile101 to ./ib_logfile0
2022-01-08 17:31:35 140492844128128 [Warning] InnoDB: New log files created, LSN=45883
2022-01-08 17:31:35 140492844128128 [Note] InnoDB: Doublewrite buffer not found: creating new
2022-01-08 17:31:35 140492844128128 [Note] InnoDB: Doublewrite buffer created
2022-01-08 17:31:36 140492844128128 [Note] InnoDB: 128 rollback segment(s) are active.
2022-01-08 17:31:36 140492844128128 [Warning] InnoDB: Creating foreign key constraint system tables.
2022-01-08 17:31:36 140492844128128 [Note] InnoDB: Foreign key constraint system tables created
2022-01-08 17:31:36 140492844128128 [Note] InnoDB: Creating tablespace and datafile system tables.
2022-01-08 17:31:36 140492844128128 [Note] InnoDB: Tablespace and datafile system tables created.
2022-01-08 17:31:36 140492844128128 [Note] InnoDB: Waiting for purge to start
2022-01-08 17:31:36 140492844128128 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.36-82.0 started; log sequence number 0
2022-01-08 17:31:36 140490055509760 [Note] InnoDB: Dumping buffer pool(s) not yet started
2022-01-08 17:31:36 140492844128128 [Note] Plugin 'FEEDBACK' is disabled.
2022-01-08 17:31:36 140492844128128 [Note] Server socket created on IP: '127.0.0.1'.
2022-01-08 17:31:36 140492843584256 [Warning] InnoDB: Cannot open table mysql/gtid_slave_pos from the internal data dictionary of InnoDB though the .frm file for the table exists. See http://dev.mysql.com/doc/refman/5.6/en/innodb-troubleshooting.html for how you can resolve the problem.
2022-01-08 17:31:36 140492843584256 [Warning] Failed to load slave replication state from table mysql.gtid_slave_pos: 1932: Table 'mysql.gtid_slave_pos' doesn't exist in engine
2022-01-08 17:31:36 140492844128128 [Note] /opt/zimbra/common/sbin/mysqld: ready for connections.
Version: '10.1.25-MariaDB'  socket: '/opt/zimbra/data/tmp/mysql/mysql.sock'  port: 7306  Zimbra MariaDB binary distribution
Tried running the install script again:

Code: Select all

License Terms for this Zimbra Collaboration Suite Software:
https://www.zimbra.com/license/zimbra-public-eula-2-6.html
----------------------------------------------------------------------



Do you agree with the terms of the software license agreement? [N] y



Checking current number of databases...

Do you want to verify message store database integrity? [Y] y
Verifying integrity of message store databases.  This may take a while.
Starting mysqld...done.
/opt/zimbra/common/bin/mysqlcheck: Got error: 1045: Access denied for user 'root'@'localhost' (using password: YES) when trying to connect
No errors found
command failed 
Stopping mysqld... done.
[root@mail zcs-8.8.15_GA_3953.RHEL8_64.20200629025823]#
At least we made some progress! :lol:
octet
Posts: 23
Joined: Thu Jan 06, 2022 2:35 pm

Re: Error on deploying SSL certificates

Post by octet »

JDunphy wrote: This is wrong... those external ip entries need to be removed if they are to represent the local machine/service. Get rid of all those 51.158/16 entries. I think you could use 10.70.150.43 for mail.XXX.com also (primary eth0). So I would do something like

Code: Select all

127.0.0.1       localhost.localdomain   localhost
10.70.150.43	mail.XXX.com mail
10.70.150.43    web-mail.XXX.com web-mail
Done, ldap starts fine, thanks.

Some further progress in running the install.sh script, without checking the integrity, here are some of the errors encountered:

Code: Select all

Saving existing configuration file to /opt/zimbra/.saveconfig

Shutting down zimbra mail

Backing up the ldap database...failed.

Code: Select all

Restoring existing configuration file from /opt/zimbra/.saveconfig/localconfig.xml...done
Operations logged to /tmp/zmsetup.20220108-180309.log
Adding /opt/zimbra/conf/ca/ca.pem to cacerts
Checking ldap status....not running.
Starting ldap...done.
[] INFO: master is down, falling back to replica...
[] FATAL: failed to initialize LDAP client
com.zimbra.cs.ldap.LdapException: LDAP error: : invalid credentials
ExceptionId:main:1641665000642:63065acb80f8465d
Code:ldap.LDAP_ERROR
	at com.zimbra.cs.ldap.LdapException.LDAP_ERROR(LdapException.java:90)
	at com.zimbra.cs.ldap.unboundid.UBIDLdapException.mapToLdapException(UBIDLdapException.java:74)
	at com.zimbra.cs.ldap.unboundid.UBIDLdapException.mapToLdapException(UBIDLdapException.java:40)
	at com.zimbra.cs.ldap.unboundid.LdapConnectionPool.createConnPool(LdapConnectionPool.java:117)
	at com.zimbra.cs.ldap.unboundid.LdapConnectionPool.createConnectionPool(LdapConnectionPool.java:63)
	at com.zimbra.cs.ldap.unboundid.UBIDLdapContext.init(UBIDLdapContext.java:111)
	at com.zimbra.cs.ldap.unboundid.UBIDLdapClient.init(UBIDLdapClient.java:39)
	at com.zimbra.cs.ldap.LdapClient.getInstanceIfLDAPavailable(LdapClient.java:62)
	at com.zimbra.cs.ldap.LdapClient.getInstance(LdapClient.java:69)
	at com.zimbra.cs.ldap.LdapClient.initialize(LdapClient.java:94)
	at com.zimbra.cs.account.ldap.LdapProv.<init>(LdapProv.java:47)
	at com.zimbra.cs.account.ldap.LdapProvisioning.<init>(LdapProvisioning.java:292)
	at com.zimbra.cs.account.ldap.LdapProvisioning.<init>(LdapProvisioning.java:289)
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500)
	at java.base/java.lang.reflect.ReflectAccess.newInstance(ReflectAccess.java:166)
	at java.base/jdk.internal.reflect.ReflectionFactory.newInstance(ReflectionFactory.java:404)
	at java.base/java.lang.Class.newInstance(Class.java:591)
	at com.zimbra.cs.account.Provisioning.getInstance(Provisioning.java:354)
	at com.zimbra.cs.account.Provisioning.getInstance(Provisioning.java:310)
	at com.zimbra.cs.account.ProvUtil.initProvisioning(ProvUtil.java:1032)
	at com.zimbra.cs.account.ProvUtil.main(ProvUtil.java:4156)
Caused by: LDAPException(resultCode=49 (invalid credentials), errorMessage='invalid credentials')
	at com.unboundid.ldap.sdk.LDAPConnection.bind(LDAPConnection.java:1894)
	at com.unboundid.ldap.sdk.LDAPConnectionPool.createConnection(LDAPConnectionPool.java:988)
	at com.unboundid.ldap.sdk.LDAPConnectionPool.<init>(LDAPConnectionPool.java:876)
	at com.unboundid.ldap.sdk.LDAPConnectionPool.<init>(LDAPConnectionPool.java:779)
	at com.unboundid.ldap.sdk.LDAPConnectionPool.<init>(LDAPConnectionPool.java:726)
	at com.zimbra.cs.ldap.unboundid.LdapConnectionPool.createConnPool(LdapConnectionPool.java:114)
	... 20 more
[] INFO: master is down, falling back to replica...
[] FATAL: failed to initialize LDAP client
com.zimbra.cs.ldap.LdapException: LDAP error: : invalid credentials
ExceptionId:main:1641665003962:14f46bed7b85a1a7
Code:ldap.LDAP_ERROR
	at com.zimbra.cs.ldap.LdapException.LDAP_ERROR(LdapException.java:90)
	at com.zimbra.cs.ldap.unboundid.UBIDLdapException.mapToLdapException(UBIDLdapException.java:74)
	at com.zimbra.cs.ldap.unboundid.UBIDLdapException.mapToLdapException(UBIDLdapException.java:40)
	at com.zimbra.cs.ldap.unboundid.LdapConnectionPool.createConnPool(LdapConnectionPool.java:117)
	at com.zimbra.cs.ldap.unboundid.LdapConnectionPool.createConnectionPool(LdapConnectionPool.java:63)
	at com.zimbra.cs.ldap.unboundid.UBIDLdapContext.init(UBIDLdapContext.java:111)
	at com.zimbra.cs.ldap.unboundid.UBIDLdapClient.init(UBIDLdapClient.java:39)
	at com.zimbra.cs.ldap.LdapClient.getInstanceIfLDAPavailable(LdapClient.java:62)
	at com.zimbra.cs.ldap.LdapClient.getInstance(LdapClient.java:69)
	at com.zimbra.cs.ldap.LdapClient.initialize(LdapClient.java:94)
	at com.zimbra.cs.account.ldap.LdapProv.<init>(LdapProv.java:47)
	at com.zimbra.cs.account.ldap.LdapProvisioning.<init>(LdapProvisioning.java:292)
	at com.zimbra.cs.account.ldap.LdapProvisioning.<init>(LdapProvisioning.java:289)
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500)
	at java.base/java.lang.reflect.ReflectAccess.newInstance(ReflectAccess.java:166)
	at java.base/jdk.internal.reflect.ReflectionFactory.newInstance(ReflectionFactory.java:404)
	at java.base/java.lang.Class.newInstance(Class.java:591)
	at com.zimbra.cs.account.Provisioning.getInstance(Provisioning.java:354)
	at com.zimbra.cs.account.Provisioning.getInstance(Provisioning.java:310)
	at com.zimbra.cs.account.ProvUtil.initProvisioning(ProvUtil.java:1032)
	at com.zimbra.cs.account.ProvUtil.main(ProvUtil.java:4156)
Caused by: LDAPException(resultCode=49 (invalid credentials), errorMessage='invalid credentials')
	at com.unboundid.ldap.sdk.LDAPConnection.bind(LDAPConnection.java:1894)
	at com.unboundid.ldap.sdk.LDAPConnectionPool.createConnection(LDAPConnectionPool.java:988)
	at com.unboundid.ldap.sdk.LDAPConnectionPool.<init>(LDAPConnectionPool.java:876)
	at com.unboundid.ldap.sdk.LDAPConnectionPool.<init>(LDAPConnectionPool.java:779)
	at com.unboundid.ldap.sdk.LDAPConnectionPool.<init>(LDAPConnectionPool.java:726)
	at com.zimbra.cs.ldap.unboundid.LdapConnectionPool.createConnPool(LdapConnectionPool.java:114)
	... 20 more
Setting defaults...done.
Setting defaults from existing config...done.
Initializing core config...Setting up CA...done.
Deploying CA to /opt/zimbra/conf/ca ...done.
Installing mailboxd SSL certificates...failed.
[root@mail zcs-8.8.15_GA_3953.RHEL8_64.20200629025823]# 
 
Checking ldap credentials:

Code: Select all

[zimbra@mail ~]$ zmlocalconfig -s | grep ldap.*_pass
ldap_amavis_password = i1yYlsdll4
ldap_bes_searcher_password = i1yYlsdll4
ldap_nginx_password = i1yYlsdll4
ldap_postfix_password = i1yYlsdll4
ldap_replication_password = i1yYlsdll4
ldap_root_password = i1yYlsdll4
zimbra_ldap_password = i1yYlsdll4
[zimbra@mail bin]$ zmldappasswd i1yYlsdll4
Updating local config and LDAP
[zimbra@mail bin]$ zmlocalconfig -s zimbra_ldap_password
zimbra_ldap_password = i1yYlsdll4
[zimbra@mail bin]$ ldapsearch -x -LLL -H ldapi:/// -D uid=zimbra,cn=admins,cn=zimbra -w `zmlocalconfig -m nokey -s zimbra_ldap_password` -s base -b ""
ldap_bind: Invalid credentials (49)
[zimbra@mail bin]$ 
octet
Posts: 23
Joined: Thu Jan 06, 2022 2:35 pm

Re: Error on deploying SSL certificates

Post by octet »

Tried fiddling more with ldap, restored an old version of the data.mdb

Code: Select all

[zimbra@mail ldap]$ zmcontrol status
Unable to start TLS: SSL connect attempt failed error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed when connecting to ldap master.
Cannot determine services - exiting
[zimbra@mail ldap]$ zmlocalconfig -e ldap_starttls_required=false
[zimbra@mail ldap]$ zmlocalconfig -e ldap_starttls_supported=0
[zimbra@mail ldap]$ ldap restart
Killing slapd with pid 342818. done.
Started slapd: pid 343311
[zimbra@mail ldap]$ zmcontrol status
Search error: Unable to determine enabled services from ldap.
Unable to determine enabled services. Cache is out of date or doesn’t exist.
[zimbra@mail ldap]$
[zimbra@mail ldap]$ ldapsearch -x -LLL -H ldapi:/// -D uid=zimbra,cn=admins,cn=zimbra -w `zmlocalconfig -m nokey -s zimbra_ldap_password` -s base -b “”
ldap_bind: Invalid credentials (49)
[zimbra@mail ldap]$ zmlocalconfig -s | grep ldap.*_pass
ldap_amavis_password = i1yYlsdll4
ldap_bes_searcher_password = i1yYlsdll4
ldap_nginx_password = i1yYlsdll4
ldap_postfix_password = i1yYlsdll4
ldap_replication_password = i1yYlsdll4
ldap_root_password = i1yYlsdll4
zimbra_ldap_password = i1yYlsdll4
[zimbra@mail ldap]$
[zimbra@mail db]$ ll
total 36
-rw------- 1 zimbra zimbra 85899345920 Jan 3 11:09 data.mdb
-rw------- 1 zimbra zimbra    8192 Jan 3 11:24 lock.mdb
Tried re-running the install/update script:

Code: Select all

[root@mail zcs-8.8.15_GA_3953.RHEL8_64.20200629025823]# ./install.sh 

Operations logged to /tmp/install.log.EW6xRTwQ
Checking for existing installation...
    zimbra-drive...FOUND zimbra-drive-1.0.13.1576152256-1
    zimbra-imapd...FOUND zimbra-imapd-8.8.15_GA_3953
    zimbra-patch...FOUND zimbra-patch-8.8.15.1639579168
    zimbra-mta-patch...FOUND zimbra-mta-patch-8.8.15.1638533801
    zimbra-proxy-patch...FOUND zimbra-proxy-patch-8.8.15.1634196512
    zimbra-license-tools...NOT FOUND
    zimbra-license-extension...NOT FOUND
    zimbra-network-store...NOT FOUND
    zimbra-network-modules-ng...NOT FOUND
    zimbra-chat...FOUND zimbra-chat-3.0.1.1594306000-1
    zimbra-talk...NOT FOUND
    zimbra-ldap...FOUND zimbra-ldap-8.8.15_GA_3953
    zimbra-logger...FOUND zimbra-logger-8.8.15_GA_3953
    zimbra-mta...FOUND zimbra-mta-8.8.15_GA_3953
    zimbra-dnscache...NOT FOUND
    zimbra-snmp...FOUND zimbra-snmp-8.8.15_GA_3953
    zimbra-store...FOUND zimbra-store-8.8.15_GA_3953
    zimbra-apache...FOUND zimbra-apache-8.8.15_GA_3953
    zimbra-spell...FOUND zimbra-spell-8.8.15_GA_3953
    zimbra-convertd...NOT FOUND
    zimbra-memcached...FOUND zimbra-memcached-1.6.5-1zimbra8.7b1
    zimbra-proxy...FOUND zimbra-proxy-8.8.15_GA_3953
    zimbra-archiving...NOT FOUND
    zimbra-core...FOUND zimbra-core-8.8.15_GA_3953
ZCS upgrade from 8.8.15 to 8.8.15 will be performed.
Validating ldap configuration
Error: Unable to bind to the LDAP server as the zimbra LDAP user.
       This is required to upgrade.
Followed this link to try and fix it:
https://wiki.zimbra.com/wiki/Upgrade_Sc ... figuration

Code: Select all

[zimbra@mail ldap]$ /opt/zimbra/common/sbin/slappasswd -s i1yYlsdll4
{SSHA}gHeGB2kpPQihEIfNtVKJC66l4MjKqccS
[zimbra@mail cn=config]$ cat olcDatabase={0}config.ldif
# AUTO-GENERATED FILE - DO NOT EDIT!! Use ldapmodify.
# CRC32 5a16166c
dn: olcDatabase={0}config
objectClass: olcDatabaseConfig
olcDatabase: {0}config
olcAccess: {0}to * by * none
olcLastMod: TRUE
olcMaxDerefDepth: 15
olcReadOnly: FALSE
olcRootDN: cn=config
olcRootPW:: e1NTSEE1MTJ9QlJCZlhDOEVwY3VzbDFFeFJScEhSYXpuQ0hpcTkrNDNKQzFhemww
 R2ZtdGVsM3M2Z2ZOYmdSSklaTVJPb0QvWFpQdDd0ditycEhlUGJmYTExcW5kKzRQQmhQdzlUK0p
 y
olcMonitoring: FALSE
structuralObjectClass: olcDatabaseConfig
entryUUID: 152a93d4-333e-102d-86fe-d562901af228
creatorsName: cn=config
createTimestamp: 20081020215916Z
entryCSN: 20081020215916.275992Z#000000#000#000000
modifiersName: cn=config
modifyTimestamp: 20081020215916Z
[zimbra@mail cn=config]$

[root@mail zcs-8.8.15_GA_3953.RHEL8_64.20200629025823]# cat /opt/zimbra/data/ldap/config/cn=config/olcDatabase={0}config.ldif
# AUTO-GENERATED FILE - DO NOT EDIT!! Use ldapmodify.
# CRC32 5a16166c
dn: olcDatabase={0}config
objectClass: olcDatabaseConfig
olcDatabase: {0}config
olcAccess: {0}to *  by * none
olcLastMod: TRUE
olcMaxDerefDepth: 15
olcReadOnly: FALSE
olcRootDN: cn=config
olcRootPW: {SSHA}gHeGB2kpPQihEIfNtVKJC66l4MjKqccS
olcMonitoring: FALSE
structuralObjectClass: olcDatabaseConfig
entryUUID: 152a93d4-333e-102d-86fe-d562901af228
creatorsName: cn=config
createTimestamp: 20081020215916Z
entryCSN: 20081020215916.275992Z#000000#000#000000
modifiersName: cn=config
modifyTimestamp: 20081020215916Z
[root@mail zcs-8.8.15_GA_3953.RHEL8_64.20200629025823]# 
Also tried creating some self-signed certificates see if I can get zimbra to start:

Code: Select all

[zimbra@mail ~]$ /opt/zimbra/bin/zmcertmgr  viewdeployedcrt all |grep notAfter
notAfter=Apr  6 17:17:37 2022 GMT
notAfter=Dec 30 16:50:49 2026 GMT
notAfter=Dec 27 03:28:24 2021 GMT
notAfter=Dec 30 16:50:49 2026 GMT
notAfter=Dec 30 16:50:49 2026 GMT
[zimbra@mail ~]$ /opt/zimbra/bin/zmcertmgr createca -new
** Recreating /opt/zimbra/ssl/zimbra/ca/zmssl.cnf
** Using CA private key in '/opt/zimbra/ssl/zimbra/ca/ca.key'
** Creating CA with existing private key /opt/zimbra/ssl/zimbra/ca/ca.key
[zimbra@mail ~]$ /opt/zimbra/bin/zmcertmgr createcrt -new -days 1095
** Backup /opt/zimbra/ssl/zimbra to /opt/zimbra/ssl/zimbra.20220109104248
** Recreating /opt/zimbra/conf/zmssl.cnf
** Generating a server CSR of type 'self' for download
** Using CA cert in '/opt/zimbra/ssl/zimbra/ca/ca.pem'
** Using CA private key in '/opt/zimbra/ssl/zimbra/ca/ca.key'
** Using Commercial CA cert in '/opt/zimbra/ssl/zimbra/commercial/commercial_ca.crt'
** Creating server cert request /opt/zimbra/ssl/zimbra/server/server.csr with keysize=2048 digest=sha256
** Saving config key 'zimbraSSLPrivateKey' via zmprov modifyServer mail.mc3lab.com...failed (rc=1)
** Signing cert request /opt/zimbra/ssl/zimbra/server/server.csr
[zimbra@mail ~]$ /opt/zimbra/bin/zmcertmgr deploycrt self
** Saving config key 'zimbraSSLCertificate' via zmprov modifyServer mail.mc3lab.com...failed (rc=1)
** Installing imapd certificate '/opt/zimbra/conf/imapd.crt' and key '/opt/zimbra/conf/imapd.key'
** Copying '/opt/zimbra/ssl/zimbra/server/server.crt' to '/opt/zimbra/conf/imapd.crt'
** Copying '/opt/zimbra/ssl/zimbra/server/server.key' to '/opt/zimbra/conf/imapd.key'
ERROR: imapd keytool(-delete -alias jetty) returned non-zero(1):
keytool error: java.security.cert.CertificateParsingException: signed overrun, bytes = 88
** Creating file '/opt/zimbra/ssl/zimbra/jetty.pkcs12'
** Creating keystore '/opt/zimbra/conf/imapd.keystore'
ERROR: com.zimbra.cert.MyPKCS12Import to '/opt/zimbra/ssl/zimbra/jetty.pkcs12' returned non-zero(1):
Exception in thread "main" java.security.cert.CertificateParsingException: signed overrun, bytes = 88
	at java.base/sun.security.x509.X509CertImpl.parse(X509CertImpl.java:1820)
	at java.base/sun.security.x509.X509CertImpl.<init>(X509CertImpl.java:188)
	at java.base/sun.security.provider.X509Factory.engineGenerateCertificate(X509Factory.java:105)
	at java.base/java.security.cert.CertificateFactory.generateCertificate(CertificateFactory.java:355)
	at java.base/sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:725)
	at java.base/sun.security.util.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:222)
	at java.base/java.security.KeyStore.load(KeyStore.java:1472)
	at com.zimbra.cert.MyPKCS12Import.main(MyPKCS12Import.java:104)
[zimbra@mail ~]$ /opt/zimbra/bin/zmcertmgr deployca
** Saving config key 'zimbraCertAuthorityCertSelfSigned' via zmprov modifyConfig...failed (rc=1)
[zimbra@mail ~]$ 
Still no luck, any other ideas?
User avatar
JDunphy
Outstanding Member
Outstanding Member
Posts: 889
Joined: Fri Sep 12, 2014 11:18 pm
Location: Victoria, BC
ZCS/ZD Version: 9.0.0_P39 NETWORK Edition

Re: Error on deploying SSL certificates

Post by JDunphy »

octet wrote: Still no luck, any other ideas?
Not really.

Initially I thought it was a permission problem because it failed on openssl pkcs12 export but now wondering if there is something wrong with openssl or its supporting files. I am looking at the initial link I provided in this thread right now where I show the relevant section of zmcertmgr for that error on certificate installation. I noticed they grab mailboxd_keystore_password and use that with the openssl command they are running that fails. I think they are pulling that password from /opt/zimbra/conf/localconfig.xml so I doubt they would have a problem with grabbing that unless there is a problem with the password or that file that is causing a parsing error. A casual look at that file would tell you if it appears corrupted. I think they are just exporting the certificates into a pkcs12 archive format or keystore that can be used by mailboxd/java. Ref: https://dev.to/manukam/pkcs12-b2m

Code: Select all

...
        my $pkcsf = $self->sslFiles("pkcs");
        my $kpass = $self->lc->get("mailboxd_keystore_password");
        print("** Creating file '$pkcsf'\n");
        @out = $self->run(
            $self->Openssl, "pkcs12", "-inkey", $keyf,
            "-in",          $crtf,    "-name",  $server,
            "-export",      "-out",   $pkcsf,   "-passout",
            "pass:$kpass",  "2>&1"
        );
        if ( $? != 0 or !-s $pkcsf ) {
            $self->error( "openssl pkcs12 export to '$pkcsf' failed(",
                $? >> 8, "):\n", @out );
                
Maybe modify this snippet from zmcertmgr and try to install a certificate a few times but add some additional debug statements to print out the exact syntax and values from $self->run they are attempting. I believe they are running openssl and passing any arguments to a shell to execute. Once you have that syntax, you should be able to run it directly as the zimbra user from the shell until you narrow down the root cause.

Hopefully someone else can offer better suggestions.

Jim
octet
Posts: 23
Joined: Thu Jan 06, 2022 2:35 pm

Re: Error on deploying SSL certificates

Post by octet »

Good progress, thanks Jim!

I found an old back from aug 2021 and tried to restore the ldap db from that, although I'm missing 2 mailboxes. Stopped ldap then:

Code: Select all

[root@mail opt]# mv /opt/zimbra/data/ldap/config/  /opt/zimbra/data/ldap/OLDconfig/
[root@mail opt]# mv /opt/zimbra/data/ldap/mdb/db  /opt/zimbra/data/ldap/mdb/OLD_db/
[root@mail opt]# mkdir -p /opt/zimbra/data/ldap/mdb/db /opt/zimbra/data/ldap/mdb/logs
[root@mail opt]# chown -R zimbra:zimbra /opt/zimbra/data/ldap
[root@mail opt]# /opt/zimbra/libexec/zmfixperms -verbose
Fixing ownership and permissions on /opt/zimbra/conf
Fixing permissions on /opt/zimbra/conf/ca
Fixing permissions on /opt/zimbra/conf/ca/commercial_ca_1.crt
Fixing permissions on /opt/zimbra/conf/ca/commercial_ca_2.crt
Fixing permissions on /opt/zimbra/conf/ca/commercial_ca_3.crt
Fixing permissions on /opt/zimbra/conf/ca/ca.pem
Fixing ownership and permissions on /opt/zimbra/conf/ldap-canonical.cf
Fixing ownership and permissions on /opt/zimbra/conf/ldap-slm.cf
Fixing ownership and permissions on /opt/zimbra/conf/ldap-transport.cf
Fixing ownership and permissions on /opt/zimbra/conf/ldap-vad.cf
Fixing ownership and permissions on /opt/zimbra/conf/ldap-vam.cf
Fixing ownership and permissions on /opt/zimbra/conf/ldap-vmd.cf
Fixing ownership and permissions on /opt/zimbra/conf/ldap-vmm.cf
Fixing permissions and ownership on /opt/zimbra/conf/imapd.crt
Fixing permissions and ownership on /opt/zimbra/conf/nginx.crt
Fixing permissions and ownership on /opt/zimbra/conf/slapd.crt
Fixing permissions and ownership on /opt/zimbra/conf/smtpd.crt
Fixing permissions and ownership on /opt/zimbra/conf/imapd.key
Fixing permissions and ownership on /opt/zimbra/conf/nginx.key
Fixing permissions and ownership on /opt/zimbra/conf/slapd.key
Fixing permissions and ownership on /opt/zimbra/conf/smtpd.key
Fixing permissions and ownership on /opt/zimbra/conf/zmssl.cnf
Fixing ownership and permissions on /var/log/zimbra.log
Fixing ownership and permissions on /opt/zimbra/conf/crontabs
Fixing ownership and permissions on /opt/zimbra/common/lib/jylibs
Fixing ownership and permissions on /opt/zimbra/lib
Fixing ownership and permissions on /opt/zimbra/db
Fixing ownership and permissions on /opt/zimbra/data/sasl2/state
Fixing ownership and permissions on /opt/zimbra/data/amavisd
Fixing ownership and permissions on /opt/zimbra/jetty
Fixing ownership and permissions on /opt/zimbra/ssl
Fixing ownership and permissions on /opt/zimbra/data/ldap
Fixing ownership and permissions on /opt/zimbra/logger/db
Fixing ownership and permissions on /opt/zimbra/zmstat
Fixing postfix related permissions
Fixing ownership and permissions on /opt/zimbra/data/postfix
[root@mail opt]# 
Then restored the ldap from the backup:

Code: Select all

slapadd -q -n 0 -F /opt/zimbra/data/ldap/config -cv -l /opt/zimbra/backup/ldap-config.bak
slapadd -q -b "" -F /opt/zimbra/data/ldap/config -cv -l /opt/zimbra/backup/ldap.bak
With ldap restored:

Code: Select all

[zimbra@mail ~]$ zmcontrol start
Host mail.XXX.com
	Starting zmconfigd...Done.
	Starting logger...Done.
	Starting mailbox...Done.
	Starting memcached...Done.
	Starting proxy...Done.
	Starting amavis...Done.
	Starting antispam...Done.
	Starting antivirus...Done.
	Starting opendkim...Done.
	Starting snmp...Done.
	Starting spell...Done.
	Starting mta...Done.
	Starting stats...Done.
	Starting service webapp...Done.
	Starting zimbra webapp...Done.
	Starting zimbraAdmin webapp...Done.
	Starting zimlet webapp...Done.
[zimbra@mail ~]$
So Zimbra seems to be up and running.

I then tried to deploy the self signed certificate, crashes in the same place:

Code: Select all

[zimbra@mail tmp]$ /opt/zimbra/bin/zmcertmgr deploycrt comm mail.XXX.com.cer fullchain.cer
** Verifying 'mail.XXX.com.cer' against '/opt/zimbra/ssl/zimbra/commercial/commercial.key'
Certificate 'mail.XXX.com.cer' and private key '/opt/zimbra/ssl/zimbra/commercial/commercial.key' match.
** Verifying 'mail.XXX.com.cer' against 'fullchain.cer'
Valid certificate chain: mail.XXX.com.cer: OK
** Copying 'mail.XXX.com.cer' to '/opt/zimbra/ssl/zimbra/commercial/commercial.crt'
** Copying 'fullchain.cer' to '/opt/zimbra/ssl/zimbra/commercial/commercial_ca.crt'
** Appending ca chain 'fullchain.cer' to '/opt/zimbra/ssl/zimbra/commercial/commercial.crt'
** Importing cert '/opt/zimbra/ssl/zimbra/commercial/commercial_ca.crt' as 'zcs-user-commercial_ca' into cacerts '/opt/zimbra/common/lib/jvm/java/lib/security/cacerts'
** NOTE: restart mailboxd to use the imported certificate.
** Saving config key 'zimbraSSLCertificate' via zmprov modifyServer mail.XXX.com...ok
** Saving config key 'zimbraSSLPrivateKey' via zmprov modifyServer mail.XXX.com...ok
** Installing imapd certificate '/opt/zimbra/conf/imapd.crt' and key '/opt/zimbra/conf/imapd.key'
** Copying '/opt/zimbra/ssl/zimbra/commercial/commercial.crt' to '/opt/zimbra/conf/imapd.crt'
** Copying '/opt/zimbra/ssl/zimbra/commercial/commercial.key' to '/opt/zimbra/conf/imapd.key'
ERROR: imapd keytool(-delete -alias jetty) returned non-zero(1):
keytool error: java.security.cert.CertificateParsingException: signed overrun, bytes = 88
** Creating file '/opt/zimbra/ssl/zimbra/jetty.pkcs12'
** Creating keystore '/opt/zimbra/conf/imapd.keystore'
ERROR: com.zimbra.cert.MyPKCS12Import to '/opt/zimbra/ssl/zimbra/jetty.pkcs12' returned non-zero(1):
Exception in thread "main" java.security.cert.CertificateParsingException: signed overrun, bytes = 88
	at java.base/sun.security.x509.X509CertImpl.parse(X509CertImpl.java:1820)
	at java.base/sun.security.x509.X509CertImpl.<init>(X509CertImpl.java:188)
	at java.base/sun.security.provider.X509Factory.engineGenerateCertificate(X509Factory.java:105)
	at java.base/java.security.cert.CertificateFactory.generateCertificate(CertificateFactory.java:355)
	at java.base/sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:725)
	at java.base/sun.security.util.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:222)
	at java.base/java.security.KeyStore.load(KeyStore.java:1472)
	at com.zimbra.cert.MyPKCS12Import.main(MyPKCS12Import.java:104)
I then tried to run the install.sh again, but crashes here:

Code: Select all

Restoring existing configuration file from /opt/zimbra/.saveconfig/localconfig.xml...done
Operations logged to /tmp/zmsetup.20220109-213615.log
Adding /opt/zimbra/conf/ca/ca.pem to cacerts
Checking ldap status....not running.
Starting ldap...done.
Setting defaults...done.
Setting defaults from existing config...done.
Checking for port conflicts
Setting defaults from ldap...done.
Saving config in /opt/zimbra/config.412295...done.
Operations logged to /tmp/zmsetup.20220109-213615.log
Setting local config values...done.
Initializing core config...Setting up CA...done.
Deploying CA to /opt/zimbra/conf/ca ...done.
Installing mailboxd SSL certificates...failed.
[root@mail zcs-8.8.15_GA_3953.RHEL8_64.20200629025823]#
User avatar
JDunphy
Outstanding Member
Outstanding Member
Posts: 889
Joined: Fri Sep 12, 2014 11:18 pm
Location: Victoria, BC
ZCS/ZD Version: 9.0.0_P39 NETWORK Edition

Re: Error on deploying SSL certificates

Post by JDunphy »

I don't remember if you showed this already... I am wondering if there is a corrupt keystore/permission/file caused by a manual update, etc. Can you see if you have sane values (non zero) for keystore?

Code: Select all

# ls -l /opt/zimbra/common/etc/java/cacerts /opt/zimbra/jetty_base/etc/keystore /opt/zimbra/jetty_base/etc/keystore
-rw-r--r-- 1 zimbra zimbra 231088 Jan  8 12:52 /opt/zimbra/common/etc/java/cacerts
-rw-r----- 1 zimbra zimbra   5432 Jan  8 12:52 /opt/zimbra/jetty_base/etc/keystore
-rw-r----- 1 zimbra zimbra   5432 Jan  8 12:52 /opt/zimbra/jetty_base/etc/keystore
You should be able to view the contents by providing your password (see /opt/zimbra/conf/localconfig.xml ... So something like this:

Code: Select all

# su - zimbra
% grep -A 2 mailboxd_keystore_password /opt/zimbra/conf/localconfig.xml
% keytool -list -v  -keystore  /opt/zimbra/jetty_base/etc/keystore -storepass yourpassword_from_grep_above
Then look at the results... you should find SubjectAlternativeName (DNSName) and they should represent what you zmhostname is and any other names you have.

One of my machines auto-renewed last night and this was in the tripwire reports which might be more complete that what I showed with the wiki page. Check for zimbra permission just in case.

Code: Select all


f   ...    .C    : /opt/zimbra/common/etc/java/cacerts
f   ...    .C    : /opt/zimbra/conf/ca/commercial_ca_1.crt
f   ...    .C    : /opt/zimbra/conf/imapd.crt
f   ...    .C    : /opt/zimbra/conf/imapd.keystore
f   ...    .C    : /opt/zimbra/conf/nginx.crt
f   ...    .C    : /opt/zimbra/conf/slapd.crt
f   ...    .C    : /opt/zimbra/conf/smtpd.crt
f   ...    .C    : /opt/zimbra/conf/zextras/cluster_config/zextras.json
f   ...    .C    : /opt/zimbra/conf/zextras/cluster_config/zextras.json.backup
f   ...    .C    : /opt/zimbra/jetty_base/etc/keystore
d   ...   in     : /opt/zimbra/jetty_base/work/zimbraAdmin/jsp
f   ...    .C    : /opt/zimbra/ssl/zimbra/commercial/commercial.crt
f   ...    .C    : /opt/zimbra/ssl/zimbra/commercial/commercial_ca.crt
f   ...    .C    : /opt/zimbra/ssl/zimbra/jetty.pkcs12
I also found this reference... ignore most of it but look at the section of "Generate a Java keystore to hold the certificates"... While the syntax is close, I believe this is what they are trying to do. They were getting similar errors like you are getting but for a different application. Ref: https://support.inductiveautomation.com ... =answerbot and this was the fix for this application by support. I guess we need to figure out the command and issue the command from zmcertmgr. I'll see what I can do tomorrow to get that exact openssl command syntax and understand that section of zmcertmgr. I have always tried not to understand more than I have had to with that zmcertmgr perl script. ;-)
octet
Posts: 23
Joined: Thu Jan 06, 2022 2:35 pm

Re: Error on deploying SSL certificates

Post by octet »

That keystore file doesn't get created, probably that's where the script is ending:

Code: Select all

[root@mail tmp]# ls -l /opt/zimbra/common/etc/java/cacerts /opt/zimbra/jetty_base/etc/keystore /opt/zimbra/jetty_base/etc/keystore
ls: cannot access '/opt/zimbra/jetty_base/etc/keystore': No such file or directory
ls: cannot access '/opt/zimbra/jetty_base/etc/keystore': No such file or directory
-rw-r--r-- 1 zimbra zimbra 178635 Jan  9 22:10 /opt/zimbra/common/etc/java/cacerts
[root@mail tmp]# 

Code: Select all

[root@mail tmp]# su - zimbra
Last login: Mon Jan 10 03:50:32 UTC 2022
[zimbra@mail ~]$ grep -A 2 mailboxd_keystore_password /opt/zimbra/conf/localconfig.xml
  <key name="mailboxd_keystore_password">
    <value>ecMJv6Cn6L</value>
  </key>
[zimbra@mail ~]$ keytool -list -v  -keystore  /opt/zimbra/jetty_base/etc/keystore -storepass ecMJv6Cn6L
keytool error: java.lang.Exception: Keystore file does not exist: /opt/zimbra/jetty_base/etc/keystore
java.lang.Exception: Keystore file does not exist: /opt/zimbra/jetty_base/etc/keystore
	at java.base/sun.security.tools.keytool.Main.doCommands(Main.java:910)
	at java.base/sun.security.tools.keytool.Main.run(Main.java:416)
	at java.base/sun.security.tools.keytool.Main.main(Main.java:409)
[zimbra@mail ~]$ 

Code: Select all

[zimbra@mail ~]$ ll /opt/zimbra/jetty_base/etc/
total 244
-rw-rw-r-- 1 zimbra zimbra  5825 Jun 29  2020 jetty-setuid.xml
-r--r----- 1 zimbra zimbra 37163 Jan 10 03:50 jetty.xml
-rw-r----- 1 zimbra zimbra 40204 Dec 15 14:40 jetty.xml.in
-rw-rw-r-- 1 zimbra zimbra   546 Jun 29  2020 jettyrc
-r--r----- 1 zimbra zimbra   330 Jan 10 03:50 krb5.ini
-rw-rw-r-- 1 zimbra zimbra   430 Jun 29  2020 krb5.ini.in
-rw-r----- 1 zimbra zimbra  1328 Jan  3 10:40 mailboxd.der
-rw-r----- 1 zimbra zimbra  1842 Jan  3 10:40 mailboxd.pem
-rw-r----- 1 zimbra zimbra 26648 Apr 16  2021 service.web.xml.in
-r--r----- 1 zimbra zimbra   511 Jan 10 03:50 spnego.conf
-rw-rw-r-- 1 zimbra zimbra   571 Jun 29  2020 spnego.conf.in
-r--r----- 1 zimbra zimbra   169 Jan 10 03:50 spnego.properties
-rw-rw-r-- 1 zimbra zimbra   199 Jun 29  2020 spnego.properties.in
-rw-rw-r-- 1 zimbra zimbra 27924 Jun 29  2020 webdefault.xml
-rw-rw-r-- 1 zimbra zimbra  1795 Nov 12 07:46 zimbra-jetty-env.xml.in
-rw-rw-r-- 1 zimbra zimbra  3187 Jun 29  2020 zimbra.policy.example
-rw-r----- 1 zimbra zimbra 23963 Nov 12 07:46 zimbra.web.xml.in
-rw-rw-r-- 1 zimbra zimbra  1795 Dec  3 12:14 zimbraAdmin-jetty-env.xml.in
-rw-r----- 1 zimbra zimbra 22919 Dec  3 12:14 zimbraAdmin.web.xml.in
-rw-r----- 1 zimbra zimbra  1787 Jun 29  2020 zimlet.web.xml.in
Post Reply