zimbra behind haproxy

Discuss your pilot or production implementation with other Zimbra admins or our engineers.
Post Reply
rainer_d
Advanced member
Advanced member
Posts: 97
Joined: Fri Sep 12, 2014 11:40 pm

zimbra behind haproxy

Post by rainer_d »

Hi,

since quite some time, Zimbra's nginx supports the PROXY protocol to enable it to work behind haproxy.

When I enable this (you need to edit the templates in /opt/zimbra/conf/templates), I still don't get the original source IP of the request in the logs.

For this to work, it seems you need another module that Zimbra hasn't included (yet) - or rather two modules:
- http_real_ip
- mail_real_ip

https://docs.nginx.com/nginx/admin-guid ... -protocol/

I assume, short of Zimbra including these my only chance is to do a build of the OSS version and modify it so these two are built?
ghen
Outstanding Member
Outstanding Member
Posts: 273
Joined: Thu May 12, 2016 1:56 pm
Location: Belgium
ZCS/ZD Version: 9.0.0

Re: zimbra behind haproxy

Post by ghen »

Why do you want another haproxy in front of Zimbra nginx? If you want to apply additional restrictions, you can probably do that with custom nginx config as well?
Last edited by ghen on Thu May 16, 2024 12:03 pm, edited 1 time in total.
rainer_d
Advanced member
Advanced member
Posts: 97
Joined: Fri Sep 12, 2014 11:40 pm

Re: zimbra behind haproxy

Post by rainer_d »

I want to do load-balancing.

I cannot use a transparent proxy.

Thus proxy_protocol.
ghen
Outstanding Member
Outstanding Member
Posts: 273
Joined: Thu May 12, 2016 1:56 pm
Location: Belgium
ZCS/ZD Version: 9.0.0

Re: zimbra behind haproxy

Post by ghen »

Load balancing over multiple nginx reverse proxies? If you put multiple IP's on your public DNS hostname, clients will spread automatically?
rainer_d
Advanced member
Advanced member
Posts: 97
Joined: Fri Sep 12, 2014 11:40 pm

Re: zimbra behind haproxy

Post by rainer_d »

Yes. When they are all up.
rainer_d
Advanced member
Advanced member
Posts: 97
Joined: Fri Sep 12, 2014 11:40 pm

Re: zimbra behind haproxy

Post by rainer_d »

So, my nginx config template looks like this:

Code: Select all

server
{
    ${core.ipboth.enabled}listen                  [::]:${web.https.port} ipv6only=off ssl http2  proxy_protocol;
    ${core.ipv4only.enabled}listen                ${web.https.port} ssl http2  proxy_protocol;
    ${core.ipv6only.enabled}listen                [::]:${web.https.port} ssl http2  proxy_protocol;

    ${web.add.headers.default}
    server_name             ${web.server_name.default}; # add aliases and perhaps public
    client_max_body_size    0;
	set_real_ip_from 192.168.185.195;
	real_ip_header proxy_protocol;
Haproxy config is like this:

Code: Select all

backend https-webmail-backend-ssl
  balance leastconn
  mode tcp
  server pm1 192.168.185.206:443 maxconn 3000 inter 10s fastinter 2s downinter 2s check send-proxy-v2-ssl verify none
  server pm2 192.168.185.207:443 maxconn 3000 inter 10s fastinter 2s downinter 2s check send-proxy-v2-ssl verify none

However, I still do not see the original IP in the logs on the backends.

I've changed the log_format on the frontend and it then does show the client IP.

Code: Select all

    log_format upstream '$proxy_protocol_addr $remote_addr:$remote_port - $remote_user [$time_local]  '
      '"$request_method $scheme://$host$request_uri $server_protocol" $status $bytes_sent '
      '"$http_referer" "$http_user_agent" "$upstream_addr" "$server_addr:$server_port"';
    access_log ${web.logfile} upstream;
User avatar
JDunphy
Outstanding Member
Outstanding Member
Posts: 910
Joined: Fri Sep 12, 2014 11:18 pm
Location: Victoria, BC
ZCS/ZD Version: 9.0.0_P40 NETWORK Edition

Re: zimbra behind haproxy

Post by JDunphy »

When I have done it with lots of proxies in front... I'll tend to do something like this because X-Forwarded-For: client_ip, proxy1_ip, proxy2_ip, proxy3_ip. I have an include file that has all my set_real_ip_from cidr's to remove my trusted proxies vs someone attempting to add another fake ip address in front of their ip hoping to cover their tracks.

Code: Select all

real_ip_header X-Forwarded-For;   # Use the X-Forwarded-For header
real_ip_recursive on;             # Recursively find the real 
For cloudflare, they prefer a different header:

Code: Select all

# --- Cloudflare likes this
real_ip_header CF-Connecting-IP;
real_ip_recursive on;             # Recursively find the real 
I ran nginx+lua+modsecurity for 9 months without issues on a single server Network install. Works well but it's was a pain to redo a lot of my modules whenever the version changed. Still a fun exercise to intercept attacks in real-time by putting them in an ipset. Documented in the developers forum here.

Tip: tools like burp suite are super handy to test/retest and verify it works like you expect

Jim
rainer_d
Advanced member
Advanced member
Posts: 97
Joined: Fri Sep 12, 2014 11:40 pm

Re: zimbra behind haproxy

Post by rainer_d »

I've added real_ip_recursive but no difference.

This is a multi-server install (2* proxy, 2x mailbox, 2x directory server).

It works for POP and IMAP, that's the weird thing.
User avatar
JDunphy
Outstanding Member
Outstanding Member
Posts: 910
Joined: Fri Sep 12, 2014 11:18 pm
Location: Victoria, BC
ZCS/ZD Version: 9.0.0_P40 NETWORK Edition

Re: zimbra behind haproxy

Post by JDunphy »

rainer_d wrote: Thu May 16, 2024 6:30 pm I've added real_ip_recursive but no difference.

This is a multi-server install (2* proxy, 2x mailbox, 2x directory server).

It works for POP and IMAP, that's the weird thing.
Strange... maybe give this a try since you are using the realip module:

Code: Select all

 proxy_set_header X-Forwarded-For "$http_x_forwarded_for, $realip_remote_addr";
I am guessing now so maybe someone else has an idea of what the backend is doing.

Curious if you attempted to do something like this first:

Code: Select all

% zmlocalconfig zimbra_http_originating_ip_header
zimbra_http_originating_ip_header = X-Forwarded-For

% zmprov mcf +zimbraMailTrustedIP 127.0.0.1
% zmprov mcf +zimbraMailTrustedIP <proxy ip here>
% zmprov mcf +zimbraMailTrustedIP <more proxy here>
% zmmailboxdctl restart
Ref: https://wiki.zimbra.com/wiki/Log_Files# ... inating_IP

Jim
Post Reply