memcached amplification attack

Discuss your pilot or production implementation with other Zimbra admins or our engineers.
davidkillingsworth
Outstanding Member
Outstanding Member
Posts: 251
Joined: Sat Sep 13, 2014 2:26 am
ZCS/ZD Version: 8.8.15.GA.3869.UBUNTU14.64-Patch 24

Re: memcached amplification attack

Post by davidkillingsworth »

GlooM wrote:Hello!

Release 8.5.1.GA.3056.UBUNTU14.64 UBUNTU14_64 FOSS edition. (Single server installation)

For me this fix from article:

su - zimbra
/opt/zimbra/bin/zmprov ms `zmhostname` zimbraMemcachedBindAddress 127.0.0.1
/opt/zimbra/bin/zmprov ms `zmhostname` zimbraMemcachedClientServerList 127.0.0.1

DONT WORK!!!

Iptables rules works fine!

After 8 hours after turning on the firewall - it drop 61 megabyte UDP traffic to this port!
I had the same problem, and I noticed that there was a whitespace at the end of the first line. Not sure if that made a difference.

I also rebooted my sever fully, not just restarted memchaced and that did the trick.
GlooM
Advanced member
Advanced member
Posts: 127
Joined: Sat Sep 13, 2014 12:50 am

Re: memcached amplification attack

Post by GlooM »

davidkillingsworth wrote:
I also rebooted my sever fully, not just restarted memchaced and that did the trick.
I rebooted the server completely (Operation system reboot), not only zimbra memacached. Fix didnt work, only firewall.
davidkillingsworth
Outstanding Member
Outstanding Member
Posts: 251
Joined: Sat Sep 13, 2014 2:26 am
ZCS/ZD Version: 8.8.15.GA.3869.UBUNTU14.64-Patch 24

Re: memcached amplification attack

Post by davidkillingsworth »

GlooM wrote:
davidkillingsworth wrote:
I also rebooted my sever fully, not just restarted memchaced and that did the trick.
I rebooted the server completely (Operation system reboot), not only zimbra memacached. Fix didnt work, only firewall.
I had the issue "come back" too, but as mentioned, once my ISP unsuspended my server, I logged in and re-typed the two commands from the WIKI making sure not to have any whitespace at the end, then I restarted zimbra fully.

After that I did a test by telneting to ports 22, 587, and 11211 to make sure that the changes took place and I was able to telnet to 22 and 587, but 11211 was now blocked.
GlooM
Advanced member
Advanced member
Posts: 127
Joined: Sat Sep 13, 2014 12:50 am

Re: memcached amplification attack

Post by GlooM »

davidkillingsworth wrote:
I had the issue "come back" too, but as mentioned, once my ISP unsuspended my server, I logged in and re-typed the two commands from the WIKI making sure not to have any whitespace at the end, then I restarted zimbra fully.

After that I did a test by telneting to ports 22, 587, and 11211 to make sure that the changes took place and I was able to telnet to 22 and 587, but 11211 was now blocked.

Very interesting. But now I will not disable the firewall.
PiJToo
Posts: 1
Joined: Wed Jul 25, 2018 3:40 pm

Re: memcached amplification attack

Post by PiJToo »

Hello, I have some issues after one of those attacks.
Release : zcs-8.8.8_GA_2009.UBUNTU16_64 (single server)

Since I've used the commands bellow, my service memcached isn't starting anymore.

su - zimbra
/opt/zimbra/bin/zmprov ms `zmhostname` zimbraMemcachedBindAddress 127.0.0.1
/opt/zimbra/bin/zmprov ms `zmhostname` zimbraMemcachedClientServerList 127.0.0.1

The operations seems to be successfull since I can tellnet on port 587, or 22 for example and not on the port 11211.

BUT each time I try to restart the memcashed service, the operation failed.

zimbra@xxx:/root$ zmmemcachedctl restart
Stopping memcached...memcached is not running.
Starting memcached...failed.

Since there is no error code, i can't properly identify the reason.

I've also tried those modifications on a test mail server, the same issues happend too.

Many thanks for you help.
Pierre.
User avatar
jholder
Ambassador
Ambassador
Posts: 4824
Joined: Fri Sep 12, 2014 10:00 pm

Re: memcached amplification attack

Post by jholder »

PiJToo wrote:Hello, I have some issues after one of those attacks.
Release : zcs-8.8.8_GA_2009.UBUNTU16_64 (single server)

Since I've used the commands bellow, my service memcached isn't starting anymore.

su - zimbra
/opt/zimbra/bin/zmprov ms `zmhostname` zimbraMemcachedBindAddress 127.0.0.1
/opt/zimbra/bin/zmprov ms `zmhostname` zimbraMemcachedClientServerList 127.0.0.1

The operations seems to be successfull since I can tellnet on port 587, or 22 for example and not on the port 11211.

BUT each time I try to restart the memcashed service, the operation failed.

zimbra@xxx:/root$ zmmemcachedctl restart
Stopping memcached...memcached is not running.
Starting memcached...failed.

Since there is no error code, i can't properly identify the reason.

I've also tried those modifications on a test mail server, the same issues happend too.

Many thanks for you help.
Pierre.

Do a
ps aux | grep memcache

and make sure it's not running. Zm scripts do a terrible job of making sure it stopped-- so if it doesn't stop (but thinks it did) it'll try to start it while it's running.
If it's running kill it, then start it.
davidkillingsworth
Outstanding Member
Outstanding Member
Posts: 251
Joined: Sat Sep 13, 2014 2:26 am
ZCS/ZD Version: 8.8.15.GA.3869.UBUNTU14.64-Patch 24

Re: memcached amplification attack

Post by davidkillingsworth »

@jholder, I am really happy to see that Zimbra employers are looking at this forum. It's refreshing.

However, it has been a year since people were having issues with the memcached amplification attack.

I would love to see more participation by Zimbra staff here so don't be offended, but it just struck me as odd that no one from Zimbra/Synacor weighed in with this was a huge issue for many of us.
Post Reply