Clean and empty ZCS 8.8.9 Server and Ubuntu 16.04 issues

Discuss your pilot or production implementation with other Zimbra admins or our engineers.
realsparticle
Posts: 41
Joined: Sat Sep 13, 2014 3:29 am

Clean and empty ZCS 8.8.9 Server and Ubuntu 16.04 issues

Post by realsparticle »

I am struggling with a new installation it is going to be the target system for a ZCS migration from an old 12.04 system running 8.0.8.

I have a new Ubuntu 16.04.5 install minimal server then zcs 8.8.9 installed.

All installs fine but the system is very slow as an empty server compared to even the 12.04 8.0.8 server with many users and 20+GB of mail.

Even accessing the admin console is a chore as it can take minutes to respond. I have checked the DDos setting and make adjustments to allow my local ips etc.

The hardware spec is identical to the 12.04 system 4 cpuś and 8GB of memory and they seem to be maxed out all the time. There really is nothing running on the server apart from sshd and whatever zimbra installed. Before zimbra install the server was operating exactly as expected. NO issues networking or otherwise.It seemed to be the perfect empty target for a new zimbra install.

Also noticed that the admin account is spammed massively as the services are continually stopped and started at an alarming rate. Why are these services being stopped and started so frequently. I thought that the watchdog service would only restart the services if it found one stalled or failed. Looking at the logs I see many of the following entries:
All rewrite threads completed in 0.01 sec
Oct 9 16:21:26 newzimbra zmconfigd[794]: All restarts completed in 0.00 sec

This is the third iteration of installing on a clean server just to check I had not incorrectly configured something or other. It is pretty minimal empty 16.04.5 server and a vanilla zimbra install.
I was hoping to use Zextras backup to restore a ZxMIg export and switch off the old server, but am now having serious doubts as to the performance and operation of the new combination.
Doeas anyone have any advice on this or known issues with this combination. I could not see anything detrimental on searching and this looked a good move forward for our mailserver.

Version Zimbra version: 8.8.9_GA_3019 20180809160254 20180823-0614 FOSS
Ubuntu 16.04.5 Server

Cheers
Spart
User avatar
axslingr
Outstanding Member
Outstanding Member
Posts: 256
Joined: Sat Sep 13, 2014 2:20 am
ZCS/ZD Version: 8.8.15.GA.3869.UBUNTU18.64 UBUNTU18

Re: Clean and empty ZCS 8.8.9 Server and Ubuntu 16.04 issues

Post by axslingr »

How is your disk subsystem setup? Hopefully not RAID 5. I'd also recommend disabling apparmor.

Lance
User avatar
L. Mark Stone
Ambassador
Ambassador
Posts: 2796
Joined: Wed Oct 09, 2013 11:35 am
Location: Portland, Maine, US
ZCS/ZD Version: 10.0.6 Network Edition
Contact:

Re: Clean and empty ZCS 8.8.9 Server and Ubuntu 16.04 issues

Post by L. Mark Stone »

A few comments:

"Also noticed that the admin account is spammed massively as the services are continually stopped and started at an alarming rate." Not sure your conclusion is correct here. zmconfigd watches for changes in configuration files and then restarts the necessary services. It looks at the config files every few minutes, and writes those entries to the log files even if there are no changes. The log file entry could be better composed, but what it is really saying is that "All restarts (if any) completed in xx seconds." I don't understand why you think this is being caused by the Admin account being spammed?

8GB RAM is supported, but a little light in my experience (16GB will support up to thousands of users). If you haven't set vm.swappiness to 0 I expect you will find the system is swapping a good deal. Ubuntu 16 seems to be more aggressive at swapping than earlier Ubuntu versions too. This swapping overhead consumes CPU cycles.

As the previous poster mentioned, you want to make sure that apparmor is turned off, profiles unloaded and apparmor removed.

Further, even with a lot of RAM, the first load of the Admin Console and the end user web UI takes several seconds for the cacheable web page portions to be built. Subsequent logins are faster, even from totally different browsers. The Admin Console also runs a lot of scripts on initial load, so if you system is already swapping this will take extra time as well.

Also, 8.8 comes with DoSFilter already configured with what are IMHO too aggressive settings. The DosFilter wiki will show you how to relax these settings, and also help you to make sure that ancillary settings (like zimbraHttpThrottleSafeIPs) are configured correctly. https://wiki.zimbra.com/wiki/DoSFilter

Lastly, 8.8.10 is now out and although I doubt the fixes in 8.8.10 will solve your problems, I would recommend getting on the latest version before doing your production cutover.

Feel free to post back with specific log file entries and there will be no shortage of helpful comments I am sure.

Hope that helps,
Mark
___________________________________
L. Mark Stone
Mission Critical Email - Zimbra VAR/BSP/Training Partner https://www.missioncriticalemail.com/
AWS Certified Solutions Architect-Associate
realsparticle
Posts: 41
Joined: Sat Sep 13, 2014 3:29 am

Re: Clean and empty ZCS 8.8.9 Server and Ubuntu 16.04 issues

Post by realsparticle »

Mark,

Firstly thank you for taking the time to respond.

I pressed ahead with this system on 8.8.9 earlier today it did look as though all was well. All users migrated, system responsive ish...definately slower than the ubuntu 12.04 8.0.7 system it replaced.

After a reboot earlier in fact at this point 50 mins uptime and I cannot even query the status of running servers. The server is slow to respond as all 8 cores are maxed at 100% Between reboots I doubled the number of cores to 8 and increased memory to 12GB.

$ zmcontrol status
Host xxxxxxxxxxxxxxxxx
amavis Running
Timeout after 180 seconds

This simply times out. there is no admin gui or mailbox web access either 502 or timeout errors.
mailbox.log shows the last entries from when it was rebooted nothing new since!

My sense is it is stuck doing something, but I do not know what. At this time trying to do a zmcontrol restart. I see freshclam and java at the top of the htop list
I have done the usual network checking and all seems well dns resolves fine etc.

At a loss to understand how it can be working then on a reboot with double the resources it is essentially unresponsive.

I had set ddos settings and safe ips before migrating.

Any ideas appreciated.

Cheers
Spart

UPDATE

The zmcontrol restart took over 2 hours:

$ zmcontrol restart
Host xxxxxxxxxxxxxxxxxx
Stopping zmconfigd...Done.
Stopping zimlet webapp...Done.
Stopping zimbraAdmin webapp...Done.
Stopping zimbra webapp...Done.
Stopping service webapp...Done.
Stopping stats...Done.
Stopping mta...Done.
Stopping spell...Done.
Stopping snmp...Done.
Stopping cbpolicyd...Done.
Stopping archiving...Done.
Stopping opendkim...Done.
Stopping amavis...Done.
Stopping antivirus...Done.
Stopping antispam...Done.
Stopping proxy...Done.
Stopping memcached...Done.
Stopping mailbox...Done.
Stopping logger...Done.
Stopping dnscache...Done.
Stopping ldap...Done.
Host xxxxxxxxxxxxxxxxxxxxxx
Starting ldap...Done.
Starting zmconfigd...Failed.
Starting zmconfigd...failed.
Starting dnscache...Done.
Starting logger...Done.
Starting mailbox...Done.
Starting memcached...Done.
Starting proxy...Done.
Starting amavis...Done.
Starting antispam...Done.
Starting antivirus...Done.
Starting opendkim...Done.
Starting snmp...Done.
Starting spell...Done.
Starting mta...Done.
Starting stats...Done.
Starting service webapp...Done.
Starting zimbra webapp...Done.
Starting zimbraAdmin webapp...Done.
Starting zimlet webapp...Done.
$ zmconfigdctl start
Starting zmconfigd...zmconfigd is already running.

During that time the CPU;s x 8 were all at 100% afterwards still no access to GUI or Mailbox. At this point I cannot see the wood for the trees!

Cheers
Spart
User avatar
L. Mark Stone
Ambassador
Ambassador
Posts: 2796
Joined: Wed Oct 09, 2013 11:35 am
Location: Portland, Maine, US
ZCS/ZD Version: 10.0.6 Network Edition
Contact:

Re: Clean and empty ZCS 8.8.9 Server and Ubuntu 16.04 issues

Post by L. Mark Stone »

Freshclam uses a lot of cpu, but that zmconfigd did not start cleanly is a good sign that something is not quite right.

Let’s start with the basics: Please post the contents of /etc/resolv.conf, /etc/hosts, and please confirm apparmor is disabled.

Mark
___________________________________
L. Mark Stone
Mission Critical Email - Zimbra VAR/BSP/Training Partner https://www.missioncriticalemail.com/
AWS Certified Solutions Architect-Associate
realsparticle
Posts: 41
Joined: Sat Sep 13, 2014 3:29 am

Re: Clean and empty ZCS 8.8.9 Server and Ubuntu 16.04 issues

Post by realsparticle »

Mark,

There are no network issues:

$ host newzimbra.xxx.xxx
newzimbra.xxx.xxx has address 192.168.0.4
newzimbra.xxx.xxx mail is handled by 30 newzimbra.xxxx.xxx
$ ping newzimbra
PING newzimbra.xxx.xxx (192.168.0.4) 56(84) bytes of data.
64 bytes from newzimbra.xxx.xxx (192.168.0.4): icmp_seq=1 ttl=64 time=0.278 ms
64 bytes from newzimbra.xxx.xxx (192.168.0.4): icmp_seq=2 ttl=64 time=0.254 ms
64 bytes from newzimbra.xxx.xxx (192.168.0.4): icmp_seq=3 ttl=64 time=0.227 ms

Apparmor has been purged from the system.

$ apt policy apparmor
apparmor:
Installed: (none)
Candidate: 2.10.95-0ubuntu2.10
Version table:
2.10.95-0ubuntu2.10 500
500 http://gb.archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages
500 http://security.ubuntu.com/ubuntu xenial-security/main amd64 Packages
2.10.95-0ubuntu2 500
500 http://gb.archive.ubuntu.com/ubuntu xenial/main amd64 Packages

Is there a way to hack to startup config to stop av and spam services being started. CPU looks ok until it hits mailbox 2nd time around zmconfigd started. Took forever.

Cheers
Spart
User avatar
L. Mark Stone
Ambassador
Ambassador
Posts: 2796
Joined: Wed Oct 09, 2013 11:35 am
Location: Portland, Maine, US
ZCS/ZD Version: 10.0.6 Network Edition
Contact:

Re: Clean and empty ZCS 8.8.9 Server and Ubuntu 16.04 issues

Post by L. Mark Stone »

What’s in /etc/resolv.conf please and what’s the output of:

host 192.168.0.4

and

Ifconfig

Thanks,
Mark
___________________________________
L. Mark Stone
Mission Critical Email - Zimbra VAR/BSP/Training Partner https://www.missioncriticalemail.com/
AWS Certified Solutions Architect-Associate
realsparticle
Posts: 41
Joined: Sat Sep 13, 2014 3:29 am

Re: Clean and empty ZCS 8.8.9 Server and Ubuntu 16.04 issues

Post by realsparticle »

L. Mark Stone wrote:What’s in /etc/resolv.conf please and what’s the output of:
$ cat /etc/resolv.conf
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
# DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
nameserver 127.0.0.1
search xxx.xxx

host 192.168.0.4
$ host newzimbra.xxx.xxx
newzimbra.xxx.xxx has address 192.168.0.4
newzimbra.xxx.xxx mail is handled by 30 newzimbra.xxxx.xxx
$ ping newzimbra
PING newzimbra.xxx.xxx (192.168.0.4) 56(84) bytes of data.
64 bytes from newzimbra.xxx.xxx (192.168.0.4): icmp_seq=1 ttl=64 time=0.278 ms
64 bytes from newzimbra.xxx.xxx (192.168.0.4): icmp_seq=2 ttl=64 time=0.254 ms
64 bytes from newzimbra.xxx.xxx (192.168.0.4): icmp_seq=3 ttl=64 time=0.227 ms
and

Ifconfig

$ ifconfig
enp0s17 Link encap:Ethernet HWaddr xx:xx:xx:xx:xx:xx
inet addr:192.168.0.4 Bcast:192.168.0.255 Mask:255.255.255.0
inet6 addr: fe80::a00:27ff:fe44:6e74/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:782 errors:0 dropped:2 overruns:0 frame:0
TX packets:211 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:58575 (58.5 KB) TX bytes:34267 (34.2 KB)

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:2159 errors:0 dropped:0 overruns:0 frame:0
TX packets:2159 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1
RX bytes:5179640 (5.1 MB) TX bytes:5179640 (5.1 MB)


Thanks,
Mark
User avatar
L. Mark Stone
Ambassador
Ambassador
Posts: 2796
Joined: Wed Oct 09, 2013 11:35 am
Location: Portland, Maine, US
ZCS/ZD Version: 10.0.6 Network Edition
Contact:

Re: Clean and empty ZCS 8.8.9 Server and Ubuntu 16.04 issues

Post by L. Mark Stone »

Please remove IPv6 and reboot, and confirm that when you ran the Zimbra installer, you configured Zimbra for just IPv4.

Please also post /etc/hosts. Should be just two lines:

127.0.0.1 localhost.localdomain localhost
192.168.0.4 theServerFqdn theServerHostname

Mark
___________________________________
L. Mark Stone
Mission Critical Email - Zimbra VAR/BSP/Training Partner https://www.missioncriticalemail.com/
AWS Certified Solutions Architect-Associate
realsparticle
Posts: 41
Joined: Sat Sep 13, 2014 3:29 am

Re: Clean and empty ZCS 8.8.9 Server and Ubuntu 16.04 issues

Post by realsparticle »

I have just shutdown the server and put it back to 4 CPU and 8 GB all starts fine and the server is back up and running and responsive.

This makes no sense!

Any ideas?

Cheers
Spart
Post Reply