Zimbra webmail SLOW

Discuss your pilot or production implementation with other Zimbra admins or our engineers.
mafiabusiness
Advanced member
Advanced member
Posts: 53
Joined: Sat Sep 13, 2014 3:28 am

Zimbra webmail SLOW

Post by mafiabusiness »

Hi there,

I am relatively new to this world so I apologise if I have not submitted anything that you need to help me.
I have a HP server with a RAID-5 (and yes yes, I can hear you, but bear with me please) and somewhat 100GB of RAM.
I have 700+ users on the server.

I have less than 5 people using webmail.
I have employed a company to take a look at it, and they shake their head.

My issue is this:

Webmail is like syrop.
It feels like a Ferarri on first get, revving like mad, but going nowhere anytime soon.
Click on the Sent Folder with 0 emails in there and it can take 20 seconds for it to finish loading the page.
The "connection to server is slow..." comes up very often. Imagine now if you are using it all the time.
Click on the inbox, 20 seconds. Click on the Calendar, 20 seconds. Click back to inbox, 20 seconds.
The 20 seconds is the worst I have seen, but it is not uncommon.
"Connection to server is slow" happens very often. If I use the server for 10 minutes, I will have seen it at least 5 times.. :-(

Apple Mail works a charm, as does all mobile devices. Sure, a bit more speed could be appreciated, but in general, it feels ok.

IOwait is around 10% constantly.
Total RAM: 98GB
Free RAM: 1.5GB (at the moment)
HP CPU: 2 x 4 core (avg 45% load)
zmcontrol-v: 8.7.11_GA_1854 FOSS Edition (It has been this slow since the day I started with Zimbra (7.something))

It is slow if I run it off Firefox local on the server also.
The above support company, myself and two other "consultants" have looked at the Zimbra Tune Server for Large Deployments (a link I have seen all over the internet).
I have made some changes there, and it at times makes a change, but, I feel that is more because I restart zimbra than anything else.
See, If I restart Zimbra, it tends to speed up a bit, nothing like google's webmail, but at least "reasonably" workable.
There is a constant lag, and the lag is so bad that people refuse to use it and set up their Apple Mail clients instead.
That is fine, but I have a large client that wants desperately to use it.

What else do you need to know to assist?

(you already have my eternal gratitude for having taken yourself the time to read this)

Miguel de Sousa Pires
User avatar
gabrieles
Outstanding Member
Outstanding Member
Posts: 233
Joined: Tue Feb 14, 2017 9:40 am

Re: Zimbra webmail SLOW

Post by gabrieles »

IOwait is around 10% constantly.
there's a high probability that you have a disk issue. That means slow working mysql and mailboxd. That means general poor performances.
phoenix
Ambassador
Ambassador
Posts: 27272
Joined: Fri Sep 12, 2014 9:56 pm
Location: Liverpool, England

Re: Zimbra webmail SLOW

Post by phoenix »

Have you read the Performance Tuning article from the wiki? https://wiki.zimbra.com/wiki/Performanc ... eployments

Do note th at it mentions "no raid" and really you should not be doing it and you'll be more likely to have continuing problems.
Regards

Bill

Rspamd: A high performance spamassassin replacement

Per ardua ad astra
User avatar
pup_seba
Outstanding Member
Outstanding Member
Posts: 687
Joined: Sat Sep 13, 2014 2:43 am
Location: Tarragona - Spain
Contact:

Re: Zimbra webmail SLOW

Post by pup_seba »

Kind of difficult to "guess". It could be lots of things.

Does your Zimbra run on a VM or directly over your HP server? RAID-5...ok...but what disk technology do you use and how is that raid configured (like 10x512GB SSD or 3x8TB SATA 5200RPM?).

I personally don't like such big machines. All my Zimbra servers have no more than 4 CPUs and 16GB of RAM. If I need more "power", I deploy a new server to work along with the first one.

Try to read the guide Phoenix mention, although is quite outdated (lots of those things are already implemented or do not apply anymore as for changes in Java version), it is a good guide and a good starting point to understand the main things to consider.

Also, when you say "free ram", are you sure what you mean by free ram? Just asking this because lots of times people confuses what free ram really is by not understanding what cached memory is. Maybe an output of "free -m" could help.

If you and that company are already watching system parameters, things I can come up are:
- How big is your mariadb database and how much memory did you assigned to the innodb buffer?
- Which parameters do your java process execute with?
- Chech for abnormal garbage collection times for both new gen and long term memory areas.
- Can you see any deny of service issue like jetty DDoS filters being applied with a delay for new petitions?

You will need to dig around to see where the problem is. Maybe trying to see the troubleshooting guide? https://wiki.zimbra.com/wiki/Troubleshooting
mafiabusiness
Advanced member
Advanced member
Posts: 53
Joined: Sat Sep 13, 2014 3:28 am

Re: Zimbra webmail SLOW

Post by mafiabusiness »

Thank you friends for quick and probably, sadly, accurate answers. I will have to re-think the raid-5 solution pretty ___ quick. Would y'all say that a 4 drive 7200rpm sata on raid 10 to be a better (read: quicker) solution for me?

Yes, the mariadb is quick big, has been allocated a fair whack of the ram installed.

Everything points to the raid.
I have just been trying to ignore it, as it means change of server.
The way that hosting is going, i am not sure i will want to continue with it.. #office365 #google.. :-(

Miguel
User avatar
pup_seba
Outstanding Member
Outstanding Member
Posts: 687
Joined: Sat Sep 13, 2014 2:43 am
Location: Tarragona - Spain
Contact:

Re: Zimbra webmail SLOW

Post by pup_seba »

Hi,

"Would y'all say that a 4 drive 7200rpm sata on raid 10 to be a better (read: quicker) solution for me?"
-> You do not provide enough information to answer that question. And yet again, you fail to understand the already referenced "tunning guides" provided by Zimbra where beside the "NO RAID5" they also say "No SATA".

"The way that hosting is going, i am not sure i will want to continue with it.. #office365 #google.. "
-> Just for your information, I usually have ~1000 users per mailstore, each of them having no more than 16GB of RAM...and no performance problem whatsoever. I even have 1 customer with ~3000 users on each of their 4 mailstores and also no performance problem. Again, I never configure systems with more than 4 vCPUs and 16GB of RAM.

office365 or gSuite, will be around 40€ per user/year (unless you can subscribe to gob or academic licensing). 700 users * 40€ per year = 28.000€ per year and you will still need to deal with administration and figure out a way to do backups of that information to fit your company backup policies. Not to talk about vendor lock-ins, confidentiality and that you will be subject of any of their espontaneus policy price change if they want to...(usually people forgets about asking how to move away from those services just to realize that it is not an easy job to do).

I would say that it should be easier and cheaper to plan a good Zimbra deployment, than running to someone else computer (msft or google) and pay them that amount of money for something you could have by 1/5 of their price. But that's just a recommendation.

Anyways, hope you solve your problem!!! gl!
mafiabusiness
Advanced member
Advanced member
Posts: 53
Joined: Sat Sep 13, 2014 3:28 am

Re: Zimbra webmail SLOW

Post by mafiabusiness »

Hi there,

apologies for slow reply on this subject. I will re-format my backup server and make it a RAID-10 solution instead to ensure enhanced performance.
I am afraid I do not have the funds for SAS drives, so need to stick with my SATA drives.

That said, I have found that when I restarted the server, out of the 100GB I have available, I ended up with 80GB free.
At the end of the day, I had 40GB free. At the end of two days, I had 700MB free.

I have attached a screenshot of what is taking up CPU and RAM.

Knowing I am not very skilled in this subject, is there anybody here that can from this spot what I might have done wrong (apart from not read that famous website about how to deploy a large scale zimra solution + put the zimbra on a RAID-5). Accepting that, might there be something else not quite right, that can be read from this?

Moving it all over to a RAID-10 solution will take quite somet time, and I doubt it will resolve my second issue of RAM-consuming applications / processes.

Why does MySQL take up so much RAM?
Why does Java consume so much CPU?

Miguel
Attachments
Screen Shot 2018-09-16 at 13.30.12.png
Screen Shot 2018-09-16 at 13.30.12.png (746.4 KiB) Viewed 7688 times
User avatar
pup_seba
Outstanding Member
Outstanding Member
Posts: 687
Joined: Sat Sep 13, 2014 2:43 am
Location: Tarragona - Spain
Contact:

Re: Zimbra webmail SLOW

Post by pup_seba »

Hi!

"That said, I have found that when I restarted the server, out of the 100GB I have available, I ended up with 80GB free.
At the end of the day, I had 40GB free. At the end of two days, I had 700MB free.

I have attached a screenshot of what is taking up CPU and RAM."
I'll copy/paste what I a fragment from my previous comment which you still did not answer. """Also, when you say "free ram", are you sure what you mean by free ram? Just asking this because lots of times people confuses what free ram really is by not understanding what cached memory is. Maybe an output of "free -m" could help."""

"Knowing I am not very skilled in this subject, is there anybody here that can from this spot what I might have done wrong (apart from not read that famous website about how to deploy a large scale zimra solution + put the zimbra on a RAID-5). Accepting that, might there be something else not quite right, that can be read from this?"
Hard to tell. Did you read the installation guide and the administration guide? Did you read wiki guides about best practices for spam and others? Cause if you didn't, I would be pretty sure that there going to be many things to improve in your installation. I've seen plenty of "next next next" kind of deployments and then people being frustrated because "it only gives me problems". Zimbra is pretty robust, but it needs a carefull deployment. I've been working with it for over 6 years and still to come to the forums to learn (sometimes by asking, sometimes by trying to help or sometimes just by reading others solutions), read articles, read wikis, try things in my lab.

"Moving it all over to a RAID-10 solution will take quite somet time, and I doubt it will resolve my second issue of RAM-consuming applications / processes."
The change itself could be challenging. I don't know which strategy you are going to use (adding a secondary store, rsync, adding a new server to the deployment and moving the mailboxes to later decomission the old server, zextras backup/restore, others...), but yeah, I guess that moving away from raid5 should help...but so far, we really don't know if that is the thing making things go wrong. It is a bet, and we would still have sata drives. So, I can only cross my fingers and hope the work done to move to a new store, really fix the problem!!!

"Why does MySQL take up so much RAM?"
Zimbra default installation configures the innodb_buffer for the database to use 30% of your ram. So mysql tries to copy its whole db to that buffer on ram. Problems come when the db size is bigger than the innodb_buffer size, making that 30% not enough to hold the db. That's when that process will start making a lot of read/writes to your hdd and things would start to go bad. Sending that "free -m" I asked you would really help to understand things. The tunning guide explains how % should be according to different systems.

Why does Java consume so much CPU?
Why you say it is consuming "much cpu"? Correct me if I'm wrong but you said that you had 2 processors with 4 cores each. That would take a total of 8 cpus? (cat /proc/cpuinfo | wc -l to find out) Meaning that you could go up to 800% in a top. So your java, which holds your entire mailboxd process is consimung 1/8 of your cpu power. But again, I may be missinterpreting what you sent...is that a top command?

Also, I asked you some other things in the previous comment that you still not answered...
"
If you and that company are already watching system parameters, things I can come up are:
- How big is your mariadb database and how much memory did you assigned to the innodb buffer?
- Which parameters do your java process execute with?
- Chech for abnormal garbage collection times for both new gen and long term memory areas.
- Can you see any deny of service issue like jetty DDoS filters being applied with a delay for new petitions?

You will need to dig around to see where the problem is. Maybe trying to see the troubleshooting guide? https://wiki.zimbra.com/wiki/Troubleshooting
"

Also, by any chance did you deployed zimbra with certain amount of memory installed, and later on you added memory to the server. Let's say you deployed zimbra with the server having 24GB and with time you added more memory until you reached 100GB? What's the output of "ps -ef | grep -i java"

Anyways, good luck!!!
mafiabusiness
Advanced member
Advanced member
Posts: 53
Joined: Sat Sep 13, 2014 3:28 am

Re: Zimbra webmail SLOW

Post by mafiabusiness »

Hi pup_seba,

I have added the screenshot as you wanted it.

Yes, I had the system installed with 32Gb or RAM, but when i thought i found that I needed more, so I added another 64GB of RAM.
98Gb in total.

I have read that wiki on how to optimise the installation, over and over again.
There are things I have not done (raid-5, sata), and there are things I have done (turned off system junk, optimised heap, changed simultanious imap connections, looked at nginx ram allocation).

Another possible bottle-neck is that I run the /opt/logs on the RAID-5 volume, something I have at long considered moving elsewhere, out of the /opt folder, to ensure is elsewhere thus not causing unecessary writes to an already highly critisised storage unit.

I would have done this differently today, knowing what I know, about CentOS and Zimbra.
Your help is much appreciated.

Miguel
Attachments
Screen Shot 2018-09-17 at 08.21.11.png
Screen Shot 2018-09-17 at 08.21.11.png (300.16 KiB) Viewed 7541 times
mafiabusiness
Advanced member
Advanced member
Posts: 53
Joined: Sat Sep 13, 2014 3:28 am

Re: Zimbra webmail SLOW

Post by mafiabusiness »

Here is iostat, before you ask… ;-)

system: sda (2 x 500GB mirror raid)
/opt: drbd0 (3 x 3TB RAID-5)
backup: sdc (1 x 6TB USB drive)

I had this issue with slow Zimbra webmail ever since I set the server up.
I am considering turning the browser to HTML by detafault, I have noticed it a bit quicker..

What I get is that spinning blue ball often + also the grey box appearing saying "system is slow or not responding, please wait…"

many thanks,

Miguel
Attachments
Screen Shot 2018-09-17 at 08.27.47.png
Screen Shot 2018-09-17 at 08.27.47.png (86.32 KiB) Viewed 7540 times
Post Reply