Maldua's Zimbra FOSS Builds - Share your feedback

[adrian.gibanel.btactic]

And RHEL9 is supported since 10.1.2. But Maldua build for RHEL9 is still marked as beta.
https://wiki.zimbra.com/wiki/Zimbra_Releases/10.1.2

When Maldua was built (one week before 10.1.2 NE release) it was the first time that we built RHEL9. That's a good reason to label it as beta.

As I said I should think about adding an explanation as why I label stuff as BETA instead of just writing BETA there and that's it.

[adrian.gibanel.btactic]

Hello,

Maldua's Zimbra 10.0.10 FOSS Builds have just been released.

Please share your experience when upgrading or installing from scratch.

Versions:

• 9.0.0.p41
• 9.0.0.p42
• 10.0.10

Platforms:

• RHEL7 (Red Hat Enterprise Linux 7, Oracle Linux 7, CentOS 7)
• RHEL8 (Red Hat Enterprise Linux 8, Oracle Linux 8, CentOS 8, Rocky Linux 8)
• Ubuntu 18.04
• Ubuntu 20.04

Thank you very much!

Notes:

• These are recent releases. They will become stable releases when after 15 days of being public there has not been any major negative feedback.
• This is not an official Zimbra/Synacor build.
• These are some tags that should have been built some weeks ago instead of right now.

[adrian.gibanel.btactic]

As per Zimbra OSS - what are you using to install/upgrade? thread feedback I have revamped the downloads page.

The main downloads page only features the very latest stable download from 10.1.x, 10.0.x and 9.0.0.pxx.
It has also been made clear that the different versions are equivalent to the NE versions regarding security fixes and patch levels.

The main idea behind this Downloads page is not to scare newcomers to the page.

The traditional download page has been moved/renamed to Advanced Downloads page.

As always feedback is welcome!

[lytledd]

Tried upgrading from Release 10.1.0.GA.4200000.UBUNTU22.64 UBUNTU22_64 FOSS edition to

zcs-10.1.2_GA_4200000.UBUNTU22_64.20240924075512.tgz

Running on Ubuntu 22.04.5 LTS with 4 Cores & 16GB memory. It failed in the middle of processing.

Both the install logs and the zmsetup logs show there were no errors, but the console output showed

Code: Select all

ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/opt/zimbra/data/tmp/mysql/mysql.sock' (111 "Connection refused")
Sat Nov  9 10:31:45 2024: Error while running '/opt/zimbra/bin/mysql --user=zimbra --password=REDACTED --database=zimbra --batch --skip-column-names'.

Reverted the snapshop and then cloned the VM for futher testing.

Should this be a new thread?

Zimbra install logs
https://pastebin.com/hzAiWxTU

zmsetup logs

https://pastebin.com/rdSRNiZD

Thanks!

Doug

[adrian.gibanel.btactic]

Tried upgrading from Release 10.1.0.GA.4200000.UBUNTU22.64 UBUNTU22_64 FOSS edition to

zcs-10.1.2_GA_4200000.UBUNTU22_64.20240924075512.tgz

Running on Ubuntu 22.04.5 LTS with 4 Cores & 16GB memory. It failed in the middle of processing.

Both the install logs and the zmsetup logs show there were no errors, but the console output showed

Code: Select all

ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/opt/zimbra/data/tmp/mysql/mysql.sock' (111 "Connection refused")
Sat Nov  9 10:31:45 2024: Error while running '/opt/zimbra/bin/mysql --user=zimbra --password=REDACTED --database=zimbra --batch --skip-column-names'.

So you updated from ZCS 10.1.0 to ZCS 10.1.2.

1) Do you mean that you saw this error and the installation finished ok?
2) Or did you see this error and the installation halted?
3) Did you start the installation with Zimbra services started or stopped? (They should have been started in the first place.)

Reverted the snapshop and then cloned the VM for futher testing.

Well, it could be nice if you use that VM snapshot (Before the installation) and try again to see if it always happens.

Should this be a new thread?

4) If you manage to reproduce again this bug please open an issue at zimbra-foss-builder issues page so that I can reproduce it at my side and ask Zimbra to fix it.

Zimbra install logs
https://pastebin.com/hzAiWxTU

zmsetup logs

https://pastebin.com/rdSRNiZD

As you say there is nothing relevant on those logs. Maybe you could take a look in your VM snapshot (done after the failed upgrade) and check the /opt/zimbra/data/tmp/mysql_upgrade.out contents just in case it gives any clues.

5) Also a copy-and-paste or an screenshot of when/where those lines appear could be helpful.

Thank you.

[lytledd]

So you updated from ZCS 10.1.0 to ZCS 10.1.2.

1) Do you mean that you saw this error and the installation finished ok?
2) Or did you see this error and the installation halted?
3) Did you start the installation with Zimbra services started or stopped? (They should have been started in the first place.)

I got this error and the installation halted.

4) If you manage to reproduce again this bug please open an issue at zimbra-foss-builder issues page so that I can reproduce it at my side and ask Zimbra to fix it.

I'll try again this weekend and open a github ticket

5) Also a copy-and-paste or an screenshot of when/where those lines appear could be helpful.

I'll get a screen shot.

Thank you!

Doug

[lytledd]

Okay,

Doing several tests this weekend and this morning before work. I appear to have come up with a workaround.

Working from a clone of the production server, on initial try I get a failure like the other times that I've tried (I'll include the requested screen shot and mysql.out log file below)

The workaround is to run the failing mysql upgrade command as the zimbra user a second time and then re-run the install script. I've tried it twice and both times it worked. The command is

Code: Select all

/opt/zimbra/common/bin/mysql_upgrade --defaults-file=/opt/zimbra/conf/my.cnf -S /opt/zimbra/data/tmp/mysql/mysql.sock --user=root --password=REDACTED > /opt/zimbra/data/tmp/mysql_upgrade.out 2>&1

When run as the Zimbra user manually, it is successful. Then I ran the install script again and the upgrade completed.

I'll be trying the upgrade on the production server tonight after work.

Mysql upgrade logs

https://pastebin.com/bQpvd36M

Doug

[adrian.gibanel.btactic]

Mysql upgrade logs

https://pastebin.com/bQpvd36M

Given your mysql upgrade logs I would say that this has nothing to do with a wrong ZCS build but with a faulty database.

Take a look at your

Code: Select all

/opt/zimbra/log/mysql_error.log

file which should point you to dumping the database and recreating it properly before starting your upgrade.

If you think this is a problem from the build after trying what I have described above please open an issue in the github page.

[lytledd]

It shouldn't be a database problem; it's a fresh version 10.1.0 install. I used imap sync to move the email from a version 9 Zimbra and manually recreated the setup. It's a very small home mail server.

Reviewing the mysql_error.log shows that it isn't complaining about a bad database, it's an actual crash with debug info as soon as the install script tries to run the update.

https://pastebin.com/uYVYkeU4

Since the logs indicated that it could be hardware related, I moved it to a different host with the same results. The VM is running under ESXI 7.03 on a HPE Proliant DL380 GEN9

It's really got me scratching my head.

Doug

[adrian.gibanel.btactic]

It shouldn't be a database problem; it's a fresh version 10.1.0 install. I used imap sync to move the email from a version 9 Zimbra and manually recreated the setup. It's a very small home mail server.

Reviewing the mysql_error.log shows that it isn't complaining about a bad database, it's an actual crash with debug info as soon as the install script tries to run the update.

1) Again. Despite what the mysql_error says I insist that you should dump all of your databases to prove that your database is sane prior to the upgrade. You know, before even trying to run the upgrade script.

2) Also check /var/log/syslog or journalctl -r to see if there is an Out Of Memory error there that kills somehow mysqld.

3) Make sure to upgrade your system thanks to

Code: Select all

apt update && apt dist-upgrade

and reboot it before starting the upgrade.

4) Regarding you saying that it's a fresh version 10.1.0 install that doesn't prove anything. E.g. you can have a fresh version 10.1.0 install and make your VM snapshot with your VM running which doesn't ensure a proper mysql state.

5) Open issue on Github. Thank you.

https://pastebin.com/uYVYkeU4

6) I extract relevant log from your link below:

Code: Select all

2024-11-18 12:17:56 139794969538240 [Note] /opt/zimbra/common/sbin/mysqld: ready for connections.
Version: '10.1.25-MariaDB'  socket: '/opt/zimbra/data/tmp/mysql/mysql.sock'  port: 7306  Zimbra MariaDB binary distribution
241118 12:18:25 [ERROR] mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
 
Server version: 10.1.25-MariaDB
key_buffer_size=134217728
read_buffer_size=1048576
max_used_connections=1
max_threads=112
thread_count=1
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 362733 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
 
Thread pointer: 0x562eea5ec008
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7f23487dd138 thread_stack 0x48400
/opt/zimbra/common/sbin/mysqld(my_print_stacktrace+0x32)[0x562ed2663282]
/opt/zimbra/common/sbin/mysqld(handle_fatal_signal+0x566)[0x562ed227f086]
/lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7f248d868520]
/opt/zimbra/common/sbin/mysqld(_ZN7sp_headD0Ev+0x31)[0x562ed23b2a31]
/opt/zimbra/common/sbin/mysqld(_ZN3THD13end_statementEv+0x45)[0x562ed2095cc5]
/opt/zimbra/common/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x1f3)[0x562ed20cd9b3]
/opt/zimbra/common/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x216d)[0x562ed20c032d]
/opt/zimbra/common/sbin/mysqld(_Z10do_commandP3THD+0x1d2)[0x562ed20c0b52]
/opt/zimbra/common/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x5a9)[0x562ed21b9819]
/opt/zimbra/common/sbin/mysqld(handle_one_connection+0x42)[0x562ed21b2fe2]
/lib/x86_64-linux-gnu/libc.so.6(+0x94ac3)[0x7f248d8baac3]
/lib/x86_64-linux-gnu/libc.so.6(+0x126850)[0x7f248d94c850]
 
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x562ee4260020): CREATE PROCEDURE mysql.count_duplicate_host_names() BEGIN   SET @duplicate_hosts=(SELECT count(*) FROM mysql.user GROUP BY user, lower(host) HAVING count(*) > 1 LIMIT 1);   IF @duplicate_hosts > 1 THEN     SIGNAL SQLSTATE '45000'  SET MESSAGE_TEXT = 'Multiple accounts exist for @user_name, @host_name that differ only in Host lettercase; remove all except one of them';   END IF; END
Connection ID (thread ID): 17
Status: NOT_KILLED
 
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=off