WAF integration

Introduction

In a number of customer scenarios, we will receive requests to use existing customer solutions for SSL termination/client authentication in place of HAProxy. There are a number of reasons for this but primarily two:

As SSL termination/client authentication typically happens in an untrusted network zone - often referred to generically as the "DMZ" - any solution that is already in place has gone through extensive scrutiny already and supports what the customer needs in terms of security (such as HSM support for private keys, for example). Therefore, to deploy HAProxy would extend our design/implementation stages (and ultimately sales cycles) until it met the customers needs.
HAProxy cannot rival the capabilities of already hardened solutions that exist in this place - Citrix Netscaler, F5 APM, CA API Gateway etc. It is highly likely that it would never "pass" the aforementioned scrutiny that customers driving this "need" would expect and we would and up looking for an alternative anyway.

In this example it is used a Citrix Netscaler for this purpose however the concepts here can easily be adapted for other solutions.

Scope

This design focusses on two approaches.

Using different ports for services (SAML, DMZ and WEBSEC).
Using a single FQDN but filters to a specific resource based upon the path (/idp, /websec /dmzwebsec). On Netscaler this is called "content switching".

Objectives and Sample Architecture

In simplistic terms, the proxy must do exactly what HAProxy does in our "vanilla" deployment:

SSL Termination
SSL Client Authentication - When access to the API requires authentication, the client device must present a client certificate. This client certificate is issued by the VeridiumID CA so this should be "trusted" as a CA on the appliance.
Pass Headers - Following client authentication, so that the API can differentiate one client from another, it must be passed information from the client certificate in HTTP headers.

In this example, the Veridium server is referred to by two FQDN's.

External - vext.domain.com - this is the address internet facing devices use to connect to VeridiumID (SAML clients, mobile applications).
Internal - vint.domain.local - internal FQDN used by clients connecting internally to the VeridiumID servers (SAML clients, credential providers, etc).

Each FQDN has a corresponding certificate. Here I assume the customer has two namespaces (example above) however if the customer has split DNS (single namespace, then only one certificate would be used).

External - Must be publicly trusted by iOS, Android and common browsers. This certificate is the one that corresponds to the licence SHA1 value we use for certificate pinning the mobile SDK. This is installed on the third party proxy.
Internal - This can be publicly or internally trusted (customers own CA). This is configured inside HAProxy for the listeners there.

Sample Architecture/Flow

A) Different Ports

B) Content Switching

FlowByPath.pdf

General Configuration Steps

As basic rule, the third party should

A) Perform SSL Termination

B) Validate the certificate against the VeridiumID CA cert (you will provide this - /etc/veridiumid/haproxy/client-ca.pem is a fast way to get it).

C) Extract the Distinguished Name from the client certificate and forward as a header to the listener on the VeridiumID server. This header MUST** be called X-SSL-Client-DN otherwise websec will not recognise the DN and the authentication will fail.

D) Create a custom header called x-ssl-termination-proxy-secret with the value of the secret in haproxy.cfg. If the customer cannot set this on their proxy, it can be set on the HAProxy listener (see below).

CODE

frontend frontend-https-waf
        log-format %ci:%cp\ [%t]\ %ft\ %b/%s\ %Tq/%Tw/%Tc/%Tr/%Tt\ %ST\ %B\ %CC\ %CS\ %tsc\ %ac/%fc/%bc/%sc/%rc\ %sq/%bq\ %hr\ %hs\ %{+Q}[ssl_c_s_dn]\ %{+Q}r\ %sslc\ %sslv
        option tcplog
        bind *:8444 ssl crt /etc/veridiumid/haproxy/server.pem
        
###Set the mode to HTTP ###
		mode http

        option socket-stats
        maxconn 300

###Persist the value of the header passed by the customers proxy (x-tls-clientcert-dn - in this example)###
        http-request set-var(txn.request_host_header) req.hdr(x-tls-clientcert-dn)
###

		tcp-request inspect-delay 5s
        tcp-request content accept if { req_ssl_hello_type 1 }
        default_backend backend-bops-https-waf

backend backend-bops-https-waf
        balance roundrobin
        mode http

###If the customers proxy cannot add custom headers, add the secret here too###
	  	http-request set-header x-ssl-termination-proxy-secret fGQazLDRzYBfiKQ
###

###Map the DN value to the header that websec expects X-SSL-Client-DN###
		http-request set-header X-SSL-Client-DN %[var(txn.request_host_header)]
###

        rspadd Strict-Transport-Security:\ max-age=31536000;\ includeSubDomains
        rspirep ^(set-cookie:.*)  \1;\ Secure
                server server-127.0.0.1 127.0.0.1:8083 check

NGINX

For NGINX, please see this sample NGINX.cfg file (located under /etc/nginx/). This file assumes you have one external DNS name (vid.rdemo.co, for example) and you direct traffic to the various webapps based on paths. The benefit here is you only need one SSL certificate and do not need any custom internet facing ports.

NGINX.conf

Instructions are inline (so suggest reading down the file where config changes are obvious) however the only mandatory changes are:

Change server_name (modify the <<ENVIRONMENT-NAME>> value)
Define SSL certificate by amending paths to the cert and key files (the key should be unencrypted, so just split server.pem in HAProxy if you use the same certificate there).
Add your VeridiumID server CA certificate
Modify the Proxy Pass variable with the one used by the Veridium Server, to obtain the correct value run the following command on a Veridium Webapp Node:

CODE

cat /etc/veridiumid/haproxy/haproxy.cfg | grep x-ssl-termination-proxy-secret | rev | cut -d" " -f1 | rev | uniq

Add the IP addresses of the VeridiumID server

Note, a listener has been created for websecadmin which should not be the case in production deployments (where admin will not be exposed to the internet). However, for lab environments, it is much easier to access like this.

CODE

events {
  worker_connections 1000;
}

http {
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile            on;
    tcp_nopush          on;
    tcp_nodelay         on;
    keepalive_timeout   65;
    types_hash_max_size 2048;

    include             /etc/nginx/mime.types;
    default_type        application/octet-stream;

    # Load modular configuration files from the /etc/nginx/conf.d directory.
    # See http://nginx.org/en/docs/ngx_core_module.html#include
    # for more information.
    include /etc/nginx/conf.d/*.conf;

## HTTPS Public Listener with MTLS configuration
  server {
    listen *:443 ssl;

	# Add your public DNS name
    server_name <<ENVIRONMENT-NAME>>;

    ssl_protocols TLSv1.1 TLSv1.2;
    # Your SSL Certificate
    ssl_certificate /etc/nginx/cert.crt;
    ssl_certificate_key /etc/nginx/key.key;

    # Client Certificate Authentication - Add the VeridiumID CA  (/etc/veridiumid/haproxy/client-ca.pem on VID server if you need it)
    ssl_client_certificate /etc/nginx/clientca.pem;
    ssl_verify_client optional;

## Add VeridiumID (HAProxy Listeners)

## Websec

    location /websec {
      # If client auth fails, return an error
      if ($ssl_client_verify != SUCCESS) {
        return 403;
      }

      proxy_set_header        X-SSL-Client-DN $ssl_client_s_dn;
      proxy_set_header        x-ssl-termination-proxy-secret <<PROXY-PASS>>;
	  # If Nginx will be accessed directly from the Outside network, X-Forwarded-for must be set to $remote_addr
	  proxy_set_header        X-Forwarded-For "$http_x_forwarded_for";
      # Add the VeridiumID IP and Proxy Listener Port for Websec
      proxy_pass          https://<<VERIDIUMID-IP>>:8444/websec;
      proxy_set_header Host $host;
      proxy_read_timeout  90;

    }

## Websecadmin (Optional - Typically Used for Labs/Non Prod where admin could be internet exposed)

    location /websecadmin {
      # If client auth fails, return an error
      if ($ssl_client_verify != SUCCESS) {
        return 403;
      }

      proxy_set_header        X-SSL-Client-DN $ssl_client_s_dn;
      proxy_set_header        x-ssl-termination-proxy-secret <<PROXY-PASS>>;
	  # If Nginx will be accessed directly from the Outside network, X-Forwarded-for must be set to $remote_addr
	  proxy_set_header        X-Forwarded-For "$http_x_forwarded_for";
      # Add the VeridiumID IP and Proxy Listener Port for Websecadmin
      proxy_pass          https://<<VERIDIUMID-IP>>:8449/websecadmin;
      proxy_set_header Host $host;
      proxy_read_timeout  90;

    }

    
    location /veridium-manager {

       limit_req zone=mylimit burst=12;
       # If client auth fails, return an error
	   if ($ssl_client_verify != SUCCESS){
		 return 403;
	   }

	   proxy_set_header      X-SSL-Client-DN $ssl_client_s_dn;
	   proxy_set_header      x-ssl-termination-proxy-secret <<PROXY-PASS>>;
       # If Nginx will be accessed directly from the Outside network, X-Forwarded-for must be set to $remote_addr
	   proxy_set_header      X-Forwarded-For "$http_x_forwarded_for";

	   # Add the VeridiumID IP and Proxy Listener Port for Websec
	   proxy_pass            https://<<VERIDIUMID-IP>>:8449/veridium-manager;
	   proxy_set_header Host $host;
	   proxy_read_timeout 90;

	}
  }
  # A second server entry without MTLS configuration for DMZ and Shibboleth IDP   
  server {
    listen *:8944 ssl;
    server_name <<ENVIRONMENT-NAME>>;

    ssl_protocols TLSv1.2;
    ssl_protocols TLSv1.1 TLSv1.2; 
    # Your SSL Certificate
    ssl_certificate /etc/nginx/cert.crt;
    ssl_certificate_key /etc/nginx/key.key;

## DMZWebsec

    location /dmzwebsec {

      # Add the VeridiumID IP and Proxy Listener Port for DMZWebsec
      # If Nginx will be accessed directly from the Outside network, X-Forwarded-for must be set to $remote_addr
	  proxy_set_header    X-Forwarded-For "$http_x_forwarded_for";
      proxy_pass          https://<<VERIDIUMID-IP>>:8448;
      proxy_set_header Host $host;
      proxy_read_timeout  90;
    }

## Shibboleth

    location /idp {
      # Add the VeridiumID IP and Proxy Listener Port for Shibboleth
      # If Nginx will be accessed directly from the Outside network, X-Forwarded-for must be set to $remote_addr
	  proxy_set_header    X-Forwarded-For "$http_x_forwarded_for";
      proxy_pass          https://<<VERIDIUMID-IP>>:8944;
      proxy_set_header Host $host;
      proxy_read_timeout  90;
    }

  }
}

NGINX as a load balancer

To use NGINX as a load balancer you must add upstreams for each location configured in the http section of the configuration, similar to:

CODE

http {

    upstream websec {
		server <<VERIDIUMID-IP-1>>:8444;
		server <<VERIDIUMID-IP-2>>:8444;
    }

    upstream websecadmin {
		server <<VERIDIUM-IP-1>>:8949;
		server <<VERIDIUM-IP-2>>:8949;
    }

	upstream dmz {
		server <<VERIDIUM-IP-1>>:8448;
		server <<VERIDIUM-IP-1>>:8448;
	}

	upstream shibboleth {
		# To maintain stickyness
		ip_hash;
		server <<VERIDIUM-IP-1>>:8944;
		server <<VERIDIUM-IP-1>>:8944;
	}

}

While inside the location instead of the IP address you will use the upstream name configured for the location:

CODE

location /websec {
	....
	proxy_pass https://websec/websec;
}

location /websecadmin {
	....
	proxy_pass https://websecadmin/websecadmin;
}

location /veridium-manager {
	....
	proxy_pass https://websecadmin/veridium-manager;
}


location /dmz {
	....
	proxy_pass https://dmz/dmz;
}


location /idp {
	....
	proxy_pass https://shibboleth/idp;
}

NGINX connection and rate limiting

In order to limit the number of connections/request generated from the same source IP the following must be configured in the nginx.conf file:

CODE

http {
	# Example: Set a total number of 100 requests/second from the same source IP
	limit_req_zone $http_x_forwarded_for zone=reqlimit:10m rate=100r/s;
	# Example: Creating a connect limit rule
	limit_conn_zone $http_x_forwarded_for zone=conlimit:10m;

	# If Nginx will be accessed directly from the Outside network, $http_x_forwarded_for must be replaced with $remote_addr

	server {
		...
		location / {
			# To apply the rate limiting rule allowing bursts of up to 12 requests/second
			limit_req zone=reqlimit burst=12;
		}

		location / {
			# To apply the connection limiting rule and allow a maximum of 10 connections from the same source IP
			limit_conn conlimit 10;
		}
	}
}

NGINX HAProxy Configuration

As you can see above, the "default" haproxy.cfg will not have the appropriate listeners by default. For example, a listener for 8448 or 8444 does not exist. Therefore, after making the NGINX configuration, you need to edit the HAProxy configuration as follows. You should be able to add these to the bottom of the existing file, save changes and restart haproxy. Note - not all of these have backends defined, as they use the ones that exist in haproxy.cfg by default, so this is intentional.

CODE

frontend frontend-https-proxy
        log-format %ci:%cp\ [%t]\ %ft\ %b/%s\ %Tq/%Tw/%Tc/%Tr/%Tt\ %ST\ %B\ %CC\ %CS\ %tsc\ %ac/%fc/%bc/%sc/%rc\ %sq/%bq\ %hr\ %hs\ %{+Q}[ssl_c_s_dn]\ %{+Q}r\ %sslc\ %sslv
        option tcplog
        bind *:8444 ssl crt /etc/veridiumid/haproxy/server.pem
        mode tcp
        option socket-stats
        maxconn 300
        tcp-request inspect-delay 5s
        tcp-request content accept if { req_ssl_hello_type 1 }
        default_backend backend-bops-https-proxy

backend backend-bops-https-proxy
        balance roundrobin
        mode http
        http-response add-header Strict-Transport-Security "max-age=31536000; includeSubDomains"
		http-response replace-header Set-Cookie (.*) \1;\ SameSite=Strict;\ Secure
                server server-127.0.0.1 127.0.0.1:8083 check

frontend frontend-dmz-https-proxy
        log-format %ci:%cp\ [%t]\ %ft\ %b/%s\ %Tq/%Tw/%Tc/%Tr/%Tt\ %ST\ %B\ %CC\ %CS\ %tsc\ %ac/%fc/%bc/%sc/%rc\ %sq/%bq\ %hr\ %hs\ %{+Q}[ssl_c_s_dn]\ %{+Q}r\ %sslc\ %sslv
        bind *:8448 ssl crt /etc/veridiumid/haproxy/server.pem
        option forwardfor
        option httplog
        option dontlognull
        http-response add-header Strict-Transport-Security "max-age=31536000; includeSubDomains"
        http-request del-header Origin
        http-request set-header X-Forwarded-Proto https
        http-request set-header X-SSL %[ssl_fc]
        default_backend backend-dmz-http
        http-response set-header X-Frame-Options DENY


frontend frontend-websec-admin-https-proxy
        log-format %ci:%cp\ [%t]\ %ft\ %b/%s\ %Tq/%Tw/%Tc/%Tr/%Tt\ %ST\ %B\ %CC\ %CS\ %tsc\ %ac/%fc/%bc/%sc/%rc\ %sq/%bq\ %hr\ %hs\ %{+Q}[ssl_c_s_dn]\ %{+Q}r\ %sslc\ %sslv
        bind *:8449 ssl crt /etc/veridiumid/haproxy/server.pem ca-file
        option forwardfor
        option httplog
        option dontlognull
        http-response add-header Strict-Transport-Security "max-age=31536000; includeSubDomains"
        default_backend backend-websec-admin-http-proxy
        http-response set-header X-Frame-Options DENY


backend backend-websec-admin-http-proxy
        balance roundrobin
        http-response replace-header Set-Cookie (.*) \1;\ SameSite=Strict;\ Secure
        server server-127.0.0.1 127.0.0.1:9090 check

frontend frontend-ssp-https-proxy
        mode http
        log-format %ci:%cp\ [%t]\ %ft\ %b/%s\ %Tq/%Tw/%Tc/%Tr/%Tt\ %ST\ %B\ %CC\ %CS\ %tsc\ %ac/%fc/%bc/%sc/%rc\ %sq/%bq\ %hr\ %hs\ %{+Q}[ssl_c_s_dn]\ %{+Q}r\ %sslc\ %sslv
        bind *:9988 ssl crt /etc/veridiumid/haproxy/server.pem
        option httplog
	option forwardfor
        option dontlognull
        http-response add-header Strict-Transport-Security "max-age=31536000; includeSubDomains"
        http-request del-header Origin
        http-request set-header X-Forwarded-Proto https
        http-request set-header X-SSL %[ssl_fc]
        http-response replace-header Set-Cookie (.*) \1;\ SameSite=Strict;\ Secure
                default_backend backend-ssp-http-proxy
        http-response set-header X-Frame-Options DENY

backend backend-ssp-http-proxy
        mode http
        balance roundrobin
        http-response replace-header Set-Cookie (.*) \1;\ SameSite=Strict;\ Secure
        http-response set-header X-Content-Type-Options nosniff
        	server server-127.0.0.1 127.0.0.1:9986 check

NEVIS

CODE

. Infrastructure:

 

On the SSL connector (port 443) set the Client CA Truststore to the veridium client CA

 

. Configuration:

 

The veridium application needs the following mappings:

/dmzwebsec/

/idp/

/dmzweb/

 

Assign application servers:

/dmzwebsec/ - Veridium app server:port 8448

/idp/       - Veridium app server:port 8644

/dmzweb/    - Veridium app server:port 8444

 

On /websec/ add two extra filters:

 

- a delegation filter for the headers

  Parameter Delegate Settings:

  ENV:SSL_CLIENT_S_DN:X-SSL-Client-DN

  ENV:SSL_CLIENT_S_DN_CN:X-SSL-Client-CN

  ENV:SSL_CLIENT_I_DN: X-SSL-Issuer

  ENV:SSL_CLIENT_V_START:X-SSL-Client-Not-Before

  ENV:SSL_CLIENT_V_END:X-SSL-Client-Not-After

  CONST:@VER_PROXY_SECRET@:x-ssl-termination-proxy-secret

 

- a Lua filter to check the client certificate

  Script.InputHeaderFunctionName = InputHeader

  Script:

      function InputHeader(req, resp)

            -- Check Client DN

            -- Check Issuer

            -- Check Validity

      end

F5 APM

The APM is the application firewall module of the F5 BIG-IP appliance. It is comparible in terms of features and functions to Citrix Netscaler. From a HAProxy perspective, please use the one for Netscaler above.

1. APM has the concept of iRules that allow headers to be created and passed to VeridiumID. Create this iRule and map it to the websec & websecadmin listeners.

CODE

###Retrieve the client certificate DN

when CLIENTSSL_CLIENTHELLO {

    set dist_name ""

}

when CLIENTSSL_CLIENTCERT {

    if {[SSL::cert count] > 0} {

        set dist_name [X509::subject [SSL::cert 0]]

        log local0. "$dist_name"

    }

}

###Create the Headers to insert the proxy secret and the client certificate DN. 

when HTTP_REQUEST {

    log local0. "$dist_name"

    HTTP::header insert "x-ssl-termination-proxy-secret" "fGQazLDRzYBfiKQ"

    if { $dist_name != ""} {

        HTTP::header insert "X-SSL-Client-DN" $dist_name  

    }

}

Airlock WAF

The Airlock WAF has one primary difference in that once client certificate authentication is configured, assuming you set "Send Environment Variables" in the virtualID configuration, all client certificate properties are sent to the VeridiumID (configured as a backend). However these are sent as cookies NOT as custom headers. Therefore, what you have to do is convert the cookie into a header in HAProxy. It looks something like this:

CODE

frontend frontend-bops-http 
        log-format %ci:%cp\ [%t]\ %ft\ %b/%s\ %Tq/%Tw/%Tc/%Tr/%Tt\ %ST\ %B\ %CC\ %CS\ %tsc\ %ac/%fc/%bc/%sc/%rc\ %sq/%bq\ %hr\ %hs\ %{+Q}[ssl_c_s_dn]\ %{+Q}r\ %sslc\ %sslv
        mode http
        bind 127.0.0.1:4141 accept-proxy ssl crt /etc/veridiumid/haproxy/server.pem
        rspadd Strict-Transport-Security:\ max-age=31536000;\ includeSubDomains
##Capture the Cookie##
		capture cookie AL_ENV_SSL_CLIENT_S_DN len 256
        http-request del-header Origin
        http-request set-header X-Forwarded-Proto https
        default_backend backend-bops
        ###RASPADD not supported anymore since HAPROXY 2.1######rspadd X-Frame-Options:\ DENY

backend backend-bops
        balance roundrobin
        mode http
##Use the Cookie Value to Create a Header##
        http-request set-header  X-SSL-Client-DN  %[req.cook(AL_ENV_SSL_CLIENT_S_DN)]
        server server-127.0.0.1 127.0.0.1:8083 check

Adding the proxy secret is straightforward in Airlock as with other proxies:

Airlock UI - Expert Settings - Security Gate / Apache

{code:java} EnvVarCookiePlainChars "!#$&*-./0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ^_`abcdefghijklmnopqrstuvwxyz|~=" {code}

[data-colorid=ny7qkst3nn]{color:#242424} html[data-color-mode=dark] [data-colorid=ny7qkst3nn]{color:#dbdbdb}Introduction

Scope

Objectives and Sample Architecture

General Configuration Steps

NGINX

NGINX.conf

NGINX as a load balancer

NGINX connection and rate limiting

NGINX HAProxy Configuration

NEVIS

F5 APM

Airlock WAF

Introduction