Possible Bug - nginx proxy not seeing memcache when DoS workaround applied

[guitardood]

Hello All,
I just found a problem while troubleshooting another issue. For some reason, in the nginx.log file, I found it littered with the error "no memcache server available". I remembered modifying the memcached IP address to 127.0.0.1 to mitigate DoS against memcached (per https://wiki.zimbra.com/wiki/Blocking_Memcached_Attack). While looking through the config files for nginx, I found the line containing the "servers" entry still had the zmhostname's IP address rather than the IP from the zimbraMemcachedBindAddress setting.

This is on ZCS Version - Release 8.8.10.GA.3039.UBUNTU16.64 UBUNTU16_64 FOSS edition, Patch 8.8.10_P5.

As a temporary workaround, I found the servers entry in the template (/opt/zimbra/conf/nginx/templates/nginx.conf.memcache.template), "${memcache.:servers}", and commented it out. I then added a new line with the entry "servers 127.0.0.1:11211;".

The workaround works, but I thought I'd mention it here for anyone else who might be experiencing this issue.

Best,
Guitardood

[smclinden]

Can somebody please address a fix for this.

Making the changes in https://wiki.zimbra.com/wiki/Blocking_Memcached_Attack does NOT appear to affect the value as defined in

/opt/zimbra/conf/nginx/templates/nginx.conf.memcache.template

specifically:

Code: Select all

${memcache.:servers}

The suggested "workaround", to hard code the localhost in the template is not acceptable and potentially not persistent across updates.

[smclinden]

So, apparently, making the changes as described in the above link DOES NOT change the value which is returned by:

Code: Select all

$zmprov gs server.domain.tld zimbaMemcachedBindAddress
# name server.domain.tld 
zimbraMemcachedBindAddress: 127.0.0.1

$ zmprov gamcs
server.domain.tld:11211

This strikes me as being a bug. What value is there to setting zimbraMemcachedBindAddress if it is ignored when nginx is configured?

[guitardood]

This strikes me as being a bug.

It is.

What value is there to setting zimbraMemcachedBindAddress if it is ignored when nginx is configured?

It does set the bind address on which memcached runs, just doesn't effect the memcached address which nginx uses.

I'm not familiar enough with the innards of Zimbra to know if memcached is used for other purposes besides nginx, I just happened to find this one while troubleshooting another issue.

The workaround I posted works, but definitely should only be a temporary workaround until this bug gets resolved.

Best,
guitardood

[smclinden]

Another workaround (as someone posted) would be to enable the memcached service on the public IP, then use IP tables to block access from other than the local host.

To me this is, at best, a less efficient solution than simply not listening, but it would work with the existing distribution.

But, in general, I favor not listening on any IPs that are not necessary for the proper operation of the system (slapd comes to mind).

Sean

[guitardood]

That would definitely work, as would adding a deny rule on a gateway server, if installed. Though, on a busy Zimbra server, adding iptable filtering could be the straw to break the camel's back. Once turned on, whether the rule is for that connection or not, it will have to check every packet going in/out of the server. Seems it would be a bit inefficient.

Best,
guitardood