Mysql crash

Discuss your pilot or production implementation with other Zimbra admins or our engineers.
Post Reply
pokrakos
Posts: 2
Joined: Fri Oct 27, 2017 6:12 am

Mysql crash

Post by pokrakos »

Hello. I migrating Mails from groupwise tn zimbre. Some 500 users and 700 GB of content. Everything went well until the content was over 500 GB. Then mysql crash. I do not want to run every time.
I have an error:

Code: Select all

Starting mysqld ... *** buffer overflow detected ***: / opt / zimbra / common / sbin / mysqld terminated
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x777e5)[0x7fbec735b7e5]
/lib/x86_64-linux-gnu/libc.so.6(__fortify_fail+0x5c)[0x7fbec73fd11c]
/lib/x86_64-linux-gnu/libc.so.6(+0x117120)[0x7fbec73fb120]
/lib/x86_64-linux-gnu/libc.so.6(+0x119067)[0x7fbec73fd067]
/ Opt / Zimbra / common / sbin / Mysqld (my_addr_resolve + 0x48) [0x56283b808ca8]
/ Opt / Zimbra / common / sbin / Mysqld (my_print_stacktrace 0x1e2 +) [0x56283b7f2e12]
/ Opt / Zimbra / common / sbin / Mysqld (handle_fatal_signal 0x2f5 +) [0x56283b3cc7d5]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x11390)[0x7fbec7d4a390]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x38)[0x7fbec7319428]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x16a)[0x7fbec731b02a]
/ Opt / Zimbra / common / sbin / Mysqld (+ 0x84c6d7) [0x56283b6846d7]
/ Opt / Zimbra / common / sbin / Mysqld (+ 0x84d89a) [0x56283b68589a]
/ Opt / Zimbra / common / sbin / Mysqld (+ 0x7e8305) [0x56283b620305]
/ Opt / Zimbra / common / sbin / Mysqld (+ 0x91059d) [0x56283b74859d]
/ Opt / Zimbra / common / sbin / Mysqld (+ 0x9117a4) [0x56283b7497a4]
/ Opt / Zimbra / common / sbin / Mysqld (+ 0x7f0a74) [0x56283b628a74]
/ Opt / Zimbra / common / sbin / Mysqld (+ 0x7adcfe) [0x56283b5e5cfe]
/ Opt / Zimbra / common / sbin / Mysqld (+ 0x81c80c) [0x56283b65480c]
/ Opt / Zimbra / common / sbin / Mysqld (+ 0x81cf41) [0x56283b654f41]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba)[0x7fbec7d406ba]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7fbec73eb3dd]
======= Memory map: ========
...
7f0107f2e000-7f0107f2f000 rw-p 0006f000 08:01 4063811                    /lib/x86_64-linux-gnu/libpcre.so.3.13.2
7f0107f2f000-7f0107f30000 r-xp 00000000 08:01 4063717                    /lib/x86_64-linux-gnu/libaio.so.1.0.1
7f0107f30000-7f010812f000 ---p 00001000 08:01 4063717                    /lib/x86_64-linux-gnu/libaio.so.1.0.1
7f010812f000-7f0108130000 r--p 00000000 08:01 4063717                    /lib/x86_64-linux-gnu/libaio.so.1.0.1
7f0108130000-7f0108131000 rw-p 00001000 08:01 4063717                    /lib/x86_64-linux-gnu/libaio.so.1.0.1
7f0108131000-7f0108140000 r-xp 00000000 08:01 4063730                    /lib/x86_64-linux-gnu/libbz2.so.1.0.4
7f0108140000-7f010833f000 ---p 0000f000 08:01 4063730                    /lib/x86_64-linux-gnu/libbz2.so.1.0.4
7f010833f000-7f0108340000 r--p 0000e000 08:01 4063730                    /lib/x86_64-linux-gnu/libbz2.so.1.0.4
7f0108340000-7f0108341000 rw-p 0000f000 08:01 4063730                    /lib/x86_64-linux-gnu/libbz2.so.1.0.4
7f0108341000-7f0108368000 r-xp 00000000 08:11 7340353                    /opt/zimbra/common/lib/libtcmalloc_minimal.so.4.2.6
7f0108368000-7f0108567000 ---p 00027000 08:11 7340353                    /opt/zimbra/common/lib/libtcmalloc_minimal.so.4.2.6
7f0108567000-7f0108568000 r--p 00026000 08:11 7340353                    /opt/zimbra/common/lib/libtcmalloc_minimal.so.4.2.6
7f0108568000-7f0108569000 rw-p 00027000 08:11 7340353                    /opt/zimbra/common/lib/libtcmalloc_minimal.so.4.2.6
7f0108569000-7f010858e000 rw-p 00000000 00:00 0
7f010858e000-7f01085b4000 r-xp 00000000 08:01 4063707                    /lib/x86_64-linux-gnu/ld-2.23.so
7f010876b000-7f0108770000 rw-s 00000000 00:0d 666307                     /[aio] (deleted)
7f0108770000-7f0108775000 rw-s 00000000 00:0d 666306                     /[aio] (deleted)
7f0108775000-7f010877a000 rw-s 00000000 00:0d 666305                     /[aio] (deleted)
7f010877a000-7f010877f000 rw-s 00000000 00:0d 666304                     /[aio] (deleted)
7f010877f000-7f0108784000 rw-s 00000000 00:0d 666303                     /[aio] (deleted)
7f0108784000-7f0108789000 rw-s 00000000 00:0d 666302                     /[aio] (deleted)
7f0108789000-7f010878e000 rw-s 00000000 00:0d 666301                     /[aio] (deleted)
7f010878e000-7f0108793000 rw-s 00000000 00:0d 666300                     /[aio] (deleted)
7f0108793000-7f0108798000 rw-s 00000000 00:0d 666299                     /[aio] (deleted)
7f0108798000-7f010879d000 rw-s 00000000 00:0d 666298                     /[aio] (deleted)
7f010879d000-7f01087a2000 rw-s 00000000 00:0d 666297                     /[aio] (deleted)
7f01087a2000-7f01087ac000 rw-p 00000000 00:00 0
7f01087ac000-7f01087b1000 rw-s 00000000 00:0d 666296                     /[aio] (deleted)
7f01087b1000-7f01087b3000 rw-p 00000000 00:00 0
7f01087b3000-7f01087b4000 r--p 00025000 08:01 4063707                    /lib/x86_64-linux-gnu/ld-2.23.so
7f01087b4000-7f01087b5000 rw-p 00026000 08:01 4063707                    /lib/x86_64-linux-gnu/ld-2.23.so
...
in mysql_error.log:

Code: Select all

2017-10-27 08:16:23 7fbe73281700  InnoDB: Assertion failure in thread 140455952520960 in file btr0cur.cc line 294
InnoDB: Failing assertion: sibling_mode == RW_NO_LATCH || btr_page_get_next(get_block->frame, mtr) == page_get_page_no(page)
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
171027  8:16:23 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see https://mariadb.com/kb/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

Server version: 10.1.25-MariaDB
key_buffer_size=134217728
read_buffer_size=1048576
max_used_connections=0
max_threads=112
thread_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 362733 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x0 thread_stack 0x48400
171027 08:16:23 mysqld_safe mysqld from pid file /opt/zimbra/log/mysql.pid ended
my my.cnf:

Code: Select all

[mysqld]

basedir        = /opt/zimbra/common
datadir        = /opt/zimbra/db/data
socket         = /opt/zimbra/data/tmp/mysql/mysql.sock
pid-file       = /opt/zimbra/log/mysql.pid
bind-address   = 127.0.0.1
port           = 7306
user           = zimbra
tmpdir         = /opt/zimbra/data/tmp

external-locking
slow_query_log = 1
slow_query_log_file = /opt/zimbra/log/myslow.log

general_log_file = /opt/zimbra/log/mysql-mailboxd.log

long_query_time  = 1
log_queries_not_using_indexes

thread_cache_size = 110
max_connections   = 110

# We do a lot of writes, query cache turns out to be not useful.
query_cache_type = 0

key_buffer_size=134217728
read_buffer_size=1048576
max_used_connections=0
max_threads=112
thread_count=0


sort_buffer_size = 1048576
#read_buffer_size = 1048576

# (Num mailbox groups * Num tables in each group) + padding
table_open_cache = 1200

innodb_data_file_path          = ibdata1:10M:autoextend
innodb_buffer_pool_size        = 1240328601
innodb_log_file_size           = 524288000
innodb_log_buffer_size         = 8388608
#
innodb_file_per_table=1

# Value is: 200 + max_connections + 2 * table_open_cache
innodb_open_files              = 2710

innodb_max_dirty_pages_pct     = 30
innodb_flush_method            = O_DIRECT
innodb_flush_log_at_trx_commit = 0
max_allowed_packet             = 16777216

# sysctl vm.swappiness.
vm.swappiness = 60

# Skip reverse DNS lookup of clients
skip-name-resolve

[mysqld_safe]

log-error      = /opt/zimbra/log/mysqld.log
pid-file     = /opt/zimbra/log/mysql.pid
Any suggestion what is wrong?
phoenix
Ambassador
Ambassador
Posts: 27272
Joined: Fri Sep 12, 2014 9:56 pm
Location: Liverpool, England

Re: Mysql crash

Post by phoenix »

You should always post the details of your ZCS version by giving the full output of the following command:

Code: Select all

zmcontrol -v
In cases such as this you should also give full details of the specifications of your server, how much RAM, Storage (which type, NAS, ISCSI etc), processors, users, whether this is a VM and which hypervisor and etc., etc. You should also tell us which of the wiki articles (or forum threads) you've followed about MySQL recovery and what results you got from the recovery.
Regards

Bill

Rspamd: A high performance spamassassin replacement

Per ardua ad astra
pokrakos
Posts: 2
Joined: Fri Oct 27, 2017 6:12 am

Re: Mysql crash

Post by pokrakos »

Code: Select all

Release 8.8.3.GA.1872.UBUNTU16.64 UBUNTU16_64 FOSS edition.
Zimbra server - hiper-v vm
12 GB RAM
4 virtual cores
/opt/zimbra on iscsi

i tired change values
key_buffer_size
innodb_buffer_pool_size

no success
netvirus
Posts: 2
Joined: Thu Jun 18, 2020 4:43 am

Re: Mysql crash

Post by netvirus »

The same error

zimbra@mail:~$ zmcontrol -v
Release 8.8.15.GA.3869.UBUNTU18.64 UBUNTU18_64 FOSS edition, Patch 8.8.15_P10.


zimbra@mail:~$ * buffer overflow detected *: /opt/zimbra/common/sbin/mysqld terminated
* buffer overflow detected *: /opt/zimbra/common/sbin/mysqld terminated
* buffer overflow detected *: /opt/zimbra/common/sbin/mysqld terminated
netvirus
Posts: 2
Joined: Thu Jun 18, 2020 4:43 am

Re: Mysql crash

Post by netvirus »

I have damaged tables now.
MySQL crashes and starts at a short interval

** buffer overflow detected ***: /opt/zimbra/common/sbin/mysqld terminated
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7f439a52d218 thread_stack 0x48400
200618 15:00:32 mysqld_safe Number of processes running now: 0
200618 15:00:32 mysqld_safe mysqld restarted

2020-06-18 15:00:32 140137432889216 [Note] /opt/zimbra/common/sbin/mysqld (mysqld 10.1.25-MariaDB) starting as process 9772 ...
2020-06-18 15:00:32 140137432889216 [Note] InnoDB: Using mutexes to ref count buffer pool pages
2020-06-18 15:00:32 140137432889216 [Note] InnoDB: The InnoDB memory heap is disabled
2020-06-18 15:00:32 140137432889216 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2020-06-18 15:00:32 140137432889216 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2020-06-18 15:00:32 140137432889216 [Note] InnoDB: Compressed tables use zlib 1.2.3
2020-06-18 15:00:32 140137432889216 [Note] InnoDB: Using Linux native AIO
2020-06-18 15:00:32 140137432889216 [Note] InnoDB: Using SSE crc32 instructions
2020-06-18 15:00:32 140137432889216 [Note] InnoDB: Initializing buffer pool, size = 2.3G
2020-06-18 15:00:33 140137432889216 [Note] InnoDB: Completed initialization of buffer pool
2020-06-18 15:00:33 140137432889216 [Note] InnoDB: Highest supported file format is Barracuda.
2020-06-18 15:00:33 140137432889216 [Note] InnoDB: The log sequence number 315021959 in ibdata file do not match the log sequence number 317930730 in the ib_
logfiles!
2020-06-18 15:00:34 140137432889216 [Note] InnoDB: Restoring possible half-written data pages from the doublewrite buffer...
2020-06-18 15:00:36 140137432889216 [Note] InnoDB: 128 rollback segment(s) are active.
2020-06-18 15:00:36 140137432889216 [Note] InnoDB: Waiting for purge to start
2020-06-18 15:00:36 140137432889216 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.36-82.0 started; log sequence number 317930730
2020-06-18 15:00:36 140134587320064 [Note] InnoDB: Dumping buffer pool(s) not yet started
2020-06-18 15:00:36 140137432889216 [Note] Plugin 'FEEDBACK' is disabled.
2020-06-18 15:00:36 140137432889216 [Note] Recovering after a crash using tc.log
2020-06-18 15:00:36 140137432889216 [Note] Starting crash recovery...
2020-06-18 15:00:36 140137432889216 [Note] Crash recovery finished.
2020-06-18 15:00:36 140137432889216 [Note] Server socket created on IP: '127.0.0.1'.
2020-06-18 15:00:36 140137432889216 [Note] /opt/zimbra/common/sbin/mysqld: ready for connections.
Version: '10.1.25-MariaDB' socket: '/opt/zimbra/data/tmp/mysql/mysql.sock' port: 7306 Zimbra MariaDB binary distribution
2020-06-18 15:02:55 7f7449d6bb00 InnoDB: Assertion failure in thread 140137431743232 in file btr0cur.cc line 294
InnoDB: Failing assertion: sibling_mode == RW_NO_LATCH || btr_page_get_next(get_block->frame, mtr) == page_get_page_no(page)
InnoDB: We intentionally generate a memory trap.

InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/ ... overy.html
InnoDB: about forcing recovery.
200618 15:02:55 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.


How to fix it ?
User avatar
L. Mark Stone
Ambassador
Ambassador
Posts: 2796
Joined: Wed Oct 09, 2013 11:35 am
Location: Portland, Maine, US
ZCS/ZD Version: 10.0.6 Network Edition
Contact:

Re: Mysql crash

Post by L. Mark Stone »

___________________________________
L. Mark Stone
Mission Critical Email - Zimbra VAR/BSP/Training Partner https://www.missioncriticalemail.com/
AWS Certified Solutions Architect-Associate
Post Reply