ZCS 8.8.15 redolog on Ubuntu 18.04LTS

Discuss your pilot or production implementation with other Zimbra admins or our engineers.
Post Reply
oneuser
Posts: 7
Joined: Mon Aug 15, 2022 9:33 pm

ZCS 8.8.15 redolog on Ubuntu 18.04LTS

Post by oneuser »

There were 2 ZCS nodes with Ubuntu 16.04LTS and 8.8.15.GA FOSS P29 (don't remender exactly what patch level it was).
One node as master, second as live_sync. Worked as expected for 2 years.

Then OS were upgraded to 18.04LTS on both.

Code: Select all

root@zimbra:/opt/zimbra/redolog# uname -a
Linux zimbra 4.15.0-189-generic #200-Ubuntu SMP Wed Jun 22 19:53:37 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
root@zimbra:/opt/zimbra/redolog# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 18.04.6 LTS
Release:        18.04
Codename:       bionic
Patch level also was updated.
So for now:

Code: Select all

zimbra@zimbra:~$ zmcontrol -v
Release 8.8.15.GA.3869.UBUNTU18.64 UBUNTU18_64 FOSS edition, Patch 8.8.15_P33.
But I think this problem is not dependent on patch... it is broken on OS level - 18.04LTS.
Everything works fine except redolog.

Lib version:

Code: Select all

zimbra@zimbra:~$ zmjava com.zimbra.cs.redolog.util.GetVersion
1.42
I have checked another installation of 8.8.15_P29 on 16.04LTS. It has the same lib version.

Even if there is nothing to redo it stucks on process exiting:

Code: Select all

zimbra@zimbra:~$ zmjava com.zimbra.cs.redolog.util.PlaybackUtil
Replaying operations for all mailboxes
Using 50 redo player threads
Using 100 as queue capacity for each redo player thread
0 redolog files to play back
It just stucks here not returning to CLI.
And this makes /opt/zimbra/live_sync/live_syncd unusable.

Please help me to get it working.
oneuser
Posts: 7
Joined: Mon Aug 15, 2022 9:33 pm

Re: ZCS 8.8.15 redolog on Ubuntu 18.04LTS

Post by oneuser »

Update. I reverted twise to VM snapshots on second node.
1. It's 18.04.6 LTS.

Code: Select all

zimbra@zimbra:~$ zmcontrol -v
Release 8.8.15.GA.3869.UBUNTU18.64 UBUNTU18_64 FOSS edition, Patch  8.8.15_P32.
Redolog is not working.

2. It's 16.04.6 LTS.

Code: Select all

zimbra@zimbra:~$ zmcontrol -v
Release 8.8.15.GA.3829.UBUNTU16.64 UBUNTU16_64 FOSS edition, Patch 8.8.15_P31.
Redolog works as expected.
Anyone have the same problem?
oneuser
Posts: 7
Joined: Mon Aug 15, 2022 9:33 pm

Re: ZCS 8.8.15 redolog on Ubuntu 18.04LTS

Post by oneuser »

Update2.
1. Before

Code: Select all

zimbra@zimbra:~$ zmcontrol -v
Release 8.8.15.GA.3829.UBUNTU16.64 UBUNTU16_64 FOSS edition, Patch 8.8.15_P31.
zmplayredo works fine

2. After

Code: Select all

zimbra@zimbra:~$ zmcontrol -v
Release 8.8.15.GA.3829.UBUNTU16.64 UBUNTU16_64 FOSS edition, Patch 8.8.15_P33.
zmplayredo is not processing files and not exiting to shell.
So looks like it's broken between P31 and P33.
saket.patel
Zimbra Employee
Zimbra Employee
Posts: 137
Joined: Mon Apr 11, 2022 8:39 pm

Re: ZCS 8.8.15 redolog on Ubuntu 18.04LTS

Post by saket.patel »

Can you check mailbox.log and zmmailboxd.out files when that command gets stuck? do we have any useful information there?
oneuser
Posts: 7
Joined: Mon Aug 15, 2022 9:33 pm

Re: ZCS 8.8.15 redolog on Ubuntu 18.04LTS

Post by oneuser »

saket.patel wrote:Can you check mailbox.log and zmmailboxd.out files when that command gets stuck? do we have any useful information there?
Thank you for your answer.

I'm now on 16.04.6LTS and Patch 8.8.15_P33.

Code: Select all

zimbra@zimbra:~$ zmplayredo
mailboxd is not running.
Replaying operations for all mailboxes
Using 50 redo player threads
Using 100 as queue capacity for each redo player thread
332 redolog files to play back
Then it stops.
No any information about "Processing file ..." as usual.
No updates in /opt/zimbra/log/zmmailboxd.out.
No updates in /opt/zimbra/log/mailbox.log.
saket.patel
Zimbra Employee
Zimbra Employee
Posts: 137
Joined: Mon Apr 11, 2022 8:39 pm

Re: ZCS 8.8.15 redolog on Ubuntu 18.04LTS

Post by saket.patel »

Ok, we have identified issue in latest patch and working on fixing it.
oneuser
Posts: 7
Joined: Mon Aug 15, 2022 9:33 pm

Re: ZCS 8.8.15 redolog on Ubuntu 18.04LTS

Post by oneuser »

saket.patel wrote:Ok, we have identified issue in latest patch and working on fixing it.
Thank you!
oneuser
Posts: 7
Joined: Mon Aug 15, 2022 9:33 pm

Re: ZCS 8.8.15 redolog on Ubuntu 18.04LTS

Post by oneuser »

saket.patel wrote:Ok, we have identified issue in latest patch and working on fixing it.
Hello,
Can you please provide info... is this issue fixed in P34? Or not?
Cause I don't see any info in P34 Release Notes.
saket.patel
Zimbra Employee
Zimbra Employee
Posts: 137
Joined: Mon Apr 11, 2022 8:39 pm

Re: ZCS 8.8.15 redolog on Ubuntu 18.04LTS

Post by saket.patel »

It's fixed in Patch 34, let us know if it's not working for you
Screenshot 2022-10-18 at 6.45.04 PM.png
Screenshot 2022-10-18 at 6.45.04 PM.png (47.97 KiB) Viewed 2107 times
oneuser
Posts: 7
Joined: Mon Aug 15, 2022 9:33 pm

Re: ZCS 8.8.15 redolog on Ubuntu 18.04LTS

Post by oneuser »

saket.patel wrote:It's fixed in Patch 34, let us know if it's not working for you
Works as expected. Thanks for fixing it!
Post Reply