Increase Server Connections/ Threads for Large Deployment

[ckcheong79]

I am running Zimbra 8.8.6 FOSS on Centos 7 Clound Platform with 500Mbps Fix IP, single server environment.
CPU 8 Cores
RAM 32GB
HDD 800GB (SSD Storage)

Currently is configure for 40 domains with the total of 800+ mailboxes. It contains multiple types of email clients such as Outlook, Mobile, and Zimbra Desktop Clients.

I have followed the article https://wiki.zimbra.com/wiki/Performanc ... eployments and modified some settings. My current server is running at below configurations.

[zimbra@mail ~]$ zmprov gs mail.domain.com | grep Threads
zimbraAdminImapImportNumThreads: 20
zimbraClamAVMaxThreads: 10
zimbraHttpNumThreads: 2000
zimbraHttpSSLNumThreads: 500
zimbraImapNumThreads: 2000
zimbraLmtpNumThreads: 100
zimbraMilterNumThreads: 100
zimbraPop3NumThreads: 500
zimbraScheduledTaskNumThreads: 20

After implemented these settings, POP and IMAP services seem not to have any problem, however Zimbra Desktop and SMTP (Submission Port 587) still stuck during Peak Hour.

Example:-
Users are using Zimbra Desktop (Version 7.2.7 - 7.3.1), sometime finding difficulty to sync.
Users are using Outlook / Windows Mail, email stuck at Outbox and it need to wait for more than 30 mins to be send out.

I am looking for any setting in Proxy that limit the connections but couldn't find any. Here, need all your help. Appreciate and Thank you.

Code: Select all

zimbraAdminProxyPort: 9071
zimbraImapProxyBindPort: 143
zimbraImapSSLProxyBindPort: 993
zimbraMailProxyMaxFails: 0
zimbraMailProxyPort: 80
zimbraMailProxyReconnectTimeout: 0
zimbraMailSSLProxyClientCertPort: 3443
zimbraMailSSLProxyPort: 443
zimbraMtaSmtpdProxyTimeout: 100s
zimbraPop3ProxyBindPort: 110
zimbraPop3SSLProxyBindPort: 995
zimbraReverseProxyAcceptMutex: on
zimbraReverseProxyAdminEnabled: FALSE
zimbraReverseProxyAvailableLookupTargets: mail.domain.com
zimbraReverseProxyClientCertMode: off
zimbraReverseProxyConnectTimeout: 120000ms
zimbraReverseProxyDnsLookupInServerEnabled: TRUE
zimbraReverseProxyExactServerVersionCheck: on
zimbraReverseProxyGenConfigPerVirtualHostname: TRUE
zimbraReverseProxyHttpEnabled: TRUE
zimbraReverseProxyIPThrottleWhitelistTime: 300s
zimbraReverseProxyImapEnabledCapability: ACL
zimbraReverseProxyImapEnabledCapability: BINARY
zimbraReverseProxyImapEnabledCapability: CATENATE
zimbraReverseProxyImapEnabledCapability: CHILDREN
zimbraReverseProxyImapEnabledCapability: CONDSTORE
zimbraReverseProxyImapEnabledCapability: ENABLE
zimbraReverseProxyImapEnabledCapability: ESEARCH
zimbraReverseProxyImapEnabledCapability: ESORT
zimbraReverseProxyImapEnabledCapability: I18NLEVEL=1
zimbraReverseProxyImapEnabledCapability: ID
zimbraReverseProxyImapEnabledCapability: IDLE
zimbraReverseProxyImapEnabledCapability: IMAP4rev1
zimbraReverseProxyImapEnabledCapability: LIST-EXTENDED
zimbraReverseProxyImapEnabledCapability: LIST-STATUS
zimbraReverseProxyImapEnabledCapability: LITERAL+
zimbraReverseProxyImapEnabledCapability: MULTIAPPEND
zimbraReverseProxyImapEnabledCapability: NAMESPACE
zimbraReverseProxyImapEnabledCapability: QRESYNC
zimbraReverseProxyImapEnabledCapability: QUOTA
zimbraReverseProxyImapEnabledCapability: RIGHTS=ektx
zimbraReverseProxyImapEnabledCapability: SASL-IR
zimbraReverseProxyImapEnabledCapability: SEARCHRES
zimbraReverseProxyImapEnabledCapability: SORT
zimbraReverseProxyImapEnabledCapability: THREAD=ORDEREDSUBJECT
zimbraReverseProxyImapEnabledCapability: UIDPLUS
zimbraReverseProxyImapEnabledCapability: UNSELECT
zimbraReverseProxyImapEnabledCapability: WITHIN
zimbraReverseProxyImapEnabledCapability: XLIST
zimbraReverseProxyImapExposeVersionOnBanner: FALSE
zimbraReverseProxyImapSaslGssapiEnabled: FALSE
zimbraReverseProxyImapSaslPlainEnabled: TRUE
zimbraReverseProxyImapStartTlsMode: on
zimbraReverseProxyInactivityTimeout: 1h
zimbraReverseProxyLogLevel: info
zimbraReverseProxyLookupTarget: TRUE
zimbraReverseProxyMailEnabled: TRUE
zimbraReverseProxyMailImapEnabled: TRUE
zimbraReverseProxyMailImapsEnabled: TRUE
zimbraReverseProxyMailMode: https
zimbraReverseProxyMailPop3Enabled: TRUE
zimbraReverseProxyMailPop3sEnabled: TRUE
zimbraReverseProxyPassErrors: TRUE
zimbraReverseProxyPop3EnabledCapability: EXPIRE 31 USER
zimbraReverseProxyPop3EnabledCapability: TOP
zimbraReverseProxyPop3EnabledCapability: UIDL
zimbraReverseProxyPop3EnabledCapability: USER
zimbraReverseProxyPop3EnabledCapability: XOIP
zimbraReverseProxyPop3ExposeVersionOnBanner: FALSE
zimbraReverseProxyPop3SaslGssapiEnabled: FALSE
zimbraReverseProxyPop3SaslPlainEnabled: TRUE
zimbraReverseProxyPop3StartTlsMode: on
zimbraReverseProxyRouteLookupTimeout: 15s
zimbraReverseProxyRouteLookupTimeoutCache: 60s
zimbraReverseProxySNIEnabled: FALSE
zimbraReverseProxySSLCiphers: ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128:AES256:HIGH:!aNULL:!eNULL:!EXPORT:!DES:!MD5:!PSK:!RC4
zimbraReverseProxySSLProtocols: TLSv1
zimbraReverseProxySSLProtocols: TLSv1.1
zimbraReverseProxySSLProtocols: TLSv1.2
zimbraReverseProxySSLSessionCacheSize: 10m
zimbraReverseProxySSLSessionTimeout: 10m
zimbraReverseProxySSLToUpstreamEnabled: TRUE
zimbraReverseProxyUpstreamConnectTimeout: 25
zimbraReverseProxyUpstreamFairShmSize: 32
zimbraReverseProxyUpstreamLoginServers: mail.domain.com
zimbraReverseProxyUpstreamPollingTimeout: 1h
zimbraReverseProxyUpstreamReadTimeout: 60s
zimbraReverseProxyUpstreamSendTimeout: 60s
zimbraReverseProxyWorkerConnections: 10240
zimbraReverseProxyWorkerProcesses: 8
zimbraReverseProxyXmppBoshEnabled: FALSE
zimbraReverseProxyXmppBoshLocalHttpBindURL: /http-bind
zimbraReverseProxyXmppBoshSSL: FALSE
zimbraReverseProxyXmppBoshTimeout: 90s
zimbraReverseProxyZmlookupCachingEnabled: TRUE

[L. Mark Stone]

I'd speculate this is a disk I/O limitation, not a resource limitation.

Suggest configuring a RAM disk for Amavis's temp directory. If Ubuntu 16.04, you can do something like:

As Zimbra user:
zmamavisdctl stop

As root user:
cat /etc/passwd | grep zimbra
and confirm uid:gid for zimbra are 999 (make note of whatever they are for the next step)

Add the following line to /etc/fstab to create a 1GB RAM disk for Amavis under 8.8.8:
tmpfs /opt/zimbra/data/amavisd/tmp tmpfs defaults,noexec,nodev,nosuid,size=1024m,mode=750,uid=999,gid=999 0 0 . <<== Change uid/gid values to whatever is in /etc/passwd!

Then:
mount /opt/zimbra/data/amavisd/tmp
df -h <<== to make sure your mount succeeded, then...

Become the zimbra user:
zmamavisdctl start; sleep 15; zmamavisdctl status

Amavis can take as long as ten seconds to process an email with a big attachment. Using a RAM disk cuts that down to under a second in my experience.

Hope that helps,
Mark

[ckcheong79]

Hi Mark,

Suggestion very much appreciated. Below is my result.

[root@mail ~]# cat /etc/passwd | grep zimbra
zimbra996:994::/opt/zimbra:/bin/bash
[root@mail ~]# vi /etc/fstab
[root@mail ~]# mount /opt/zimbra/data/amavisd/tmp
[root@mail ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/vda1 200G 23G 178G 12% /
devtmpfs 16G 0 16G 0% /dev
tmpfs 16G 0 16G 0% /dev/shm
tmpfs 16G 1.7G 14G 11% /run
tmpfs 16G 0 16G 0% /sys/fs/cgroup
/dev/vdb1 600G 481G 120G 81% /opt
tmpfs 3.2G 0 3.2G 0% /run/user/0
tmpfs 1.0G 0 1.0G 0% /opt/zimbra/data/amavisd/tmp
[root@mail ~]# su - zimbra
Last login: Sat Jun 2 20:19:14 +08 2018 on pts/0
[zimbra@mail ~]$ zmamavisdctl start; sleep 15; zmamavisdctl status
Starting amavisd-mc...done.
Starting amavisd...done.
amavisd is running.
amavisd-mc is running.

Will monitor how does it work out, Thanks!

[vavai]

Hi,

Just another suggestion. For such deployment, I would prefer to use multi server scenario on top of virtualization platform, so you can separate each server function on different server. Based on my experience, even using similar server specification has an increasingly performance.