MemcachedConnection makes my zimbra server slowly

Discuss your pilot or production implementation with other Zimbra admins or our engineers.
hoanglv
Posts: 3
Joined: Thu May 17, 2018 3:20 am

MemcachedConnection makes my zimbra server slowly

Postby hoanglv » Thu May 17, 2018 3:33 am

Hi,

after i blocked memcache by this guide

https://wiki.zimbra.com/wiki/Blocking_Memcached_Attack

i entered 2 commands:

su - zimbra
/opt/zimbra/bin/zmprov ms `zmhostname` zimbraMemcachedBindAddress 127.0.0.1
/opt/zimbra/bin/zmprov ms `zmhostname` zimbraMemcachedClientServerList 127.0.0.1

after that my zimbra server log said:

2018-05-17 10:06:40,568 WARN [Memcached IO over {MemcachedConnection to /127.0.0.1:11211}] [] MemcachedConnection - Closing, and reopening {QA sa=/127.0.0.1:11211, #Rops=0, #Wops=43949, #iq=0, topRop=null, topWop=net.spy.memcached.protocol.ascii.StoreOperationImpl@619a3efd, toWrite=0, interested=0}, attempt 8551.

2018-05-17 10:06:53,065 WARN [ImapServer-200] [name=customerservice@mydomain.com;mid=58;ip=xxx.xxx.xxx.xxx;ua=Microsoft Outlook/16.0.4266.1001;] MemcachedConnection - Could not redistribute to another node, retrying primary node for zmImap:3d34c5d7-2711-4545-8440-a264a3f880aa:10674:64533:51742.

my zimbra server is too slow after that

Image

any one has this problem ?

have any solutions ?

thanks so much.


kleversoft
Posts: 5
Joined: Fri Sep 29, 2017 11:55 am

Re: MemcachedConnection makes my zimbra server slowly

Postby kleversoft » Thu May 31, 2018 8:09 am

Hello same problem, can someone help us with some solution?

Thank you
User avatar
L. Mark Stone
Elite member
Elite member
Posts: 1898
Joined: Wed Oct 09, 2013 11:35 am
Location: Portland, Maine
ZCS/ZD Version: 8.8.10 Network Edition
Contact:

Re: MemcachedConnection makes my zimbra server slowly

Postby L. Mark Stone » Thu May 31, 2018 6:40 pm

https://wiki.zimbra.com/wiki/Ports

https://wiki.zimbra.com/wiki/Blocking_Memcached_Attack

Essentially, it sounds like you need to configure your firewall so that only Zimbra's intended public-facing ports are available to the public Internet. memcached's port is not one of those ports.

Hope that helps,
Mark
___________________________________
L. Mark Stone
Mission Critical Email - Zimbra VAR/BSP/Training Partner https://www.missioncriticalemail.com/
Zeta Alliance http://www.zetalliance.org/
kleversoft
Posts: 5
Joined: Fri Sep 29, 2017 11:55 am

Re: MemcachedConnection makes my zimbra server slowly

Postby kleversoft » Fri Jun 01, 2018 11:45 am

Hello, someone can help us with some more info, I can't solve the problem.... thank you very much.


[name=main@xxxx.com;ip=163.172.201.150;oip=80.156.160.86;via=162.156.134.150(nginx/1.2.0-zimbra);ua=iPhone Mail/15E216;] misc - memcached asyncGet timed out after 10000ms
net.spy.memcached.internal.CheckedOperationTimeoutException: Timed out waiting for operation - failing node: /127.0.0.1:11211
at net.spy.memcached.internal.OperationFuture.get(OperationFuture.java:65)
at net.spy.memcached.internal.GetFuture.get(GetFuture.java:37)
at com.zimbra.common.util.memcached.ZimbraMemcachedClient.get(ZimbraMemcachedClient.java:281)
at com.zimbra.common.util.memcached.ZimbraMemcachedClient.get(ZimbraMemcachedClient.java:261)
at com.zimbra.common.util.memcached.MemcachedMap.get(MemcachedMap.java:98)
at com.zimbra.cs.imap.MemcachedImapCache.get(MemcachedImapCache.java:58)
at com.zimbra.cs.imap.ImapSessionManager.deserialize(ImapSessionManager.java:634)
at com.zimbra.cs.imap.ImapSession.reload(ImapSession.java:355)
at com.zimbra.cs.imap.ImapSession.getImapFolder(ImapSession.java:96)
at com.zimbra.cs.imap.ImapSessionManager.duplicateExistingSession(ImapSessionManager.java:390)
at com.zimbra.cs.imap.ImapSessionManager.openFolder(ImapSessionManager.java:285)
at com.zimbra.cs.imap.ImapHandler.setSelectedFolder(ImapHandler.java:1028)
at com.zimbra.cs.imap.ImapHandler.selectFolder(ImapHandler.java:1489)
at com.zimbra.cs.imap.ImapHandler.doSELECT(ImapHandler.java:1453)
at com.zimbra.cs.imap.ImapHandler.executeRequest(ImapHandler.java:722)
at com.zimbra.cs.imap.NioImapHandler.processRequest(NioImapHandler.java:126)
at com.zimbra.cs.imap.NioImapHandler.messageReceived(NioImapHandler.java:63)
at com.zimbra.cs.server.NioHandlerDispatcher.messageReceived(NioHandlerDispatcher.java:95)
at org.apache.mina.core.filterchain.DefaultIoFilterChain$TailFilter.messageReceived(DefaultIoFilterChain.java:716)
at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434)
at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1200(DefaultIoFilterChain.java:46)
at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:796)
at org.apache.mina.filter.codec.ProtocolCodecFilter$ProtocolDecoderOutputImpl.flush(ProtocolCodecFilter.java:427)
at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:245)
at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434)
at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1200(DefaultIoFilterChain.java:46)
at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:796)
at com.zimbra.cs.server.NioLoggingFilter.messageReceived(NioLoggingFilter.java:62)
at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434)
at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1200(DefaultIoFilterChain.java:46)
at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:796)
at org.apache.mina.core.filterchain.IoFilterEvent.fire(IoFilterEvent.java:75)
at org.apache.mina.core.session.IoEvent.run(IoEvent.java:63)
at org.apache.mina.filter.executor.OrderedThreadPoolExecutor$Worker.runTask(OrderedThreadPoolExecutor.java:780)
at org.apache.mina.filter.executor.OrderedThreadPoolExecutor$Worker.runTasks(OrderedThreadPoolExecutor.java:772)
at org.apache.mina.filter.executor.OrderedThreadPoolExecutor$Worker.run(OrderedThreadPoolExecutor.java:714)
at java.lang.Thread.run(Thread.java:745)
phoenix
Ambassador
Ambassador
Posts: 25816
Joined: Fri Sep 12, 2014 9:56 pm

Re: MemcachedConnection makes my zimbra server slowly

Postby phoenix » Fri Jun 01, 2018 12:29 pm

Single server or a multi-server set-up? If it's a multi server then you've used the wrong commands, read the article again for the multi-server config and revert the changes you've made.
Regards

Bill

Rspamd: A high performance spamassassin replacement

If you'd like to see this implemented in a future version of ZCS then please vote on Bugzilla entries 97706 & 108168
kleversoft
Posts: 5
Joined: Fri Sep 29, 2017 11:55 am

Re: MemcachedConnection makes my zimbra server slowly

Postby kleversoft » Fri Jun 01, 2018 11:24 pm

is single server, not memcached installed.
phoenix
Ambassador
Ambassador
Posts: 25816
Joined: Fri Sep 12, 2014 9:56 pm

Re: MemcachedConnection makes my zimbra server slowly

Postby phoenix » Sat Jun 02, 2018 5:02 am

kleversoft wrote:is single server, not memcached installed.
You've posted this in a thread about memcached causing a problem on a server. You tell us that you've got 'the same problem', now you tell us that it's not installed and you seem surprised that your ZCS server can't connect to it - that's what's causing your problem. You problem is not the same as the original post. Why is memcached not installed and why do you think it benefits you to not have it installed? I'd suggest you install it, are there any other components that you've not installed (or that have been removed)?
Regards

Bill

Rspamd: A high performance spamassassin replacement

If you'd like to see this implemented in a future version of ZCS then please vote on Bugzilla entries 97706 & 108168
kleversoft
Posts: 5
Joined: Fri Sep 29, 2017 11:55 am

Re: MemcachedConnection makes my zimbra server slowly

Postby kleversoft » Sat Jun 02, 2018 7:36 am

Hi, I have the same problem, execute this command by mistake

su - zimbra
/opt/zimbra/zimbra/bin/zmprov ms `zmhostname` zimbraMemcachedBindAddress 127.0.0.1
/opt/zimbra/zimbra/bin/zmprov ms `zmhostname` zimbraMemcachedClientServerList 127.0.0.1

in this zimbra 8.6 I had no memcached installed.

from that moment on zimbra looks for the memcached, and the problem is that I don't know how to deactivate it.
phoenix
Ambassador
Ambassador
Posts: 25816
Joined: Fri Sep 12, 2014 9:56 pm

Re: MemcachedConnection makes my zimbra server slowly

Postby phoenix » Sat Jun 02, 2018 8:21 am

kleversoft wrote:Hi, I have the same problem,
You do not have the same problem as the o/p, he actually had memcached installed.

kleversoft wrote:execute this command by mistake
Why did you make those changes when you don't have memcached installed, that wasn't a mistake it was a deliberate act ion on your part - what were you trying to achieve?

kleversoft wrote: in this zimbra 8.6 I had no memcached installed.
You should always post details of your current ZCS version by posing the full output of the following command:

Code: Select all

zmcontrol -v
In addition, you should actually give a full description of the problem and not a 'me too' post, it would help us to help you when we know the full facts in advance - we're not mind readers.

kleversoft wrote:from that moment on zimbra looks for the memcached, and the problem is that I don't know how to deactivate it.
My additional question was why didn't you install memcached when you installed ZCS?

The version of ZCS that you're using will be EOL later this year you will need upgrade to the most recent version. In the current version of ZCS the proxy & memcached are required to be installed and I'd suggest you install those now in preparation for an upgrade, see the following article: https://wiki.zimbra.com/wiki/Enabling_Z ... _memcached

It's worth noting that you should not blindly make changes to the ZCS LDAP objects without knowing what (or why) you're doing it. Before making any changes to ZCS you should look at the current contents of an object, make a note of it just in case you have a problem so you can revert to the original settings then you won't be caught in this situation again.
Regards

Bill

Rspamd: A high performance spamassassin replacement

If you'd like to see this implemented in a future version of ZCS then please vote on Bugzilla entries 97706 & 108168
User avatar
vavai
Advanced member
Advanced member
Posts: 154
Joined: Thu Nov 14, 2013 2:41 pm
Location: Indonesia
Contact:

Re: MemcachedConnection makes my zimbra server slowly

Postby vavai » Sat Jun 02, 2018 10:03 pm

kleversoft wrote:Hi, I have the same problem, execute this command by mistake

su - zimbra
/opt/zimbra/zimbra/bin/zmprov ms `zmhostname` zimbraMemcachedBindAddress 127.0.0.1
/opt/zimbra/zimbra/bin/zmprov ms `zmhostname` zimbraMemcachedClientServerList 127.0.0.1

in this zimbra 8.6 I had no memcached installed.

from that moment on zimbra looks for the memcached, and the problem is that I don't know how to deactivate it.


You can check and also disabling memcached port with following command :

Code: Select all

su - zimbra
zmprov gs `zmhostname` zimbraMemcachedBindAddress
zmprov ms `zmhostname` zimbraMemcachedBindAddress ''
zmprov gs `zmhostname` zimbraMemcachedClientServerList
zmprov ms `zmhostname` zimbraMemcachedClientServerList ''

Return to “Administrators”

Who is online

Users browsing this forum: No registered users and 18 guests