WAF integration
Introduction
In a number of customer scenarios, we will receive requests to use existing customer solutions for SSL termination/client authentication in place of HAProxy. There are a number of reasons for this but primarily two:
- As SSL termination/client authentication typically happens in an untrusted network zone - often referred to generically as the "DMZ" - any solution that is already in place has gone through extensive scrutiny already and supports what the customer needs in terms of security (such as HSM support for private keys, for example). Therefore, to deploy HAProxy would extend our design/implementation stages (and ultimately sales cycles) until it met the customers needs.
- HAProxy cannot rival the capabilities of already hardened solutions that exist in this place - Citrix Netscaler, F5 APM, CA API Gateway etc. It is highly likely that it would never "pass" the aforementioned scrutiny that customers driving this "need" would expect and we would and up looking for an alternative anyway.
In this example it is used a Citrix Netscaler for this purpose however the concepts here can easily be adapted for other solutions.
Scope
This design focusses on two approaches.
- Using different ports for services (SAML, DMZ and WEBSEC).
- Using a single FQDN but filters to a specific resource based upon the path (/idp, /websec /dmzwebsec). On Netscaler this is called "content switching".
Objectives and Sample Architecture
In simplistic terms, the proxy must do exactly what HAProxy does in our "vanilla" deployment:
- SSL Termination
- SSL Client Authentication - When access to the API requires authentication, the client device must present a client certificate. This client certificate is issued by the VeridiumID CA so this should be "trusted" as a CA on the appliance.
- Pass Headers - Following client authentication, so that the API can differentiate one client from another, it must be passed information from the client certificate in HTTP headers.
In this example, the Veridium server is referred to by two FQDN's.
- External - vext.domain.com - this is the address internet facing devices use to connect to VeridiumID (SAML clients, mobile applications).
- Internal - vint.domain.local - internal FQDN used by clients connecting internally to the VeridiumID servers (SAML clients, credential providers, etc).
Each FQDN has a corresponding certificate. Here I assume the customer has two namespaces (example above) however if the customer has split DNS (single namespace, then only one certificate would be used).
- External - Must be publicly trusted by iOS, Android and common browsers. This certificate is the one that corresponds to the licence SHA1 value we use for certificate pinning the mobile SDK. This is installed on the third party proxy.
- Internal - This can be publicly or internally trusted (customers own CA). This is configured inside HAProxy for the listeners there.
Sample Architecture/Flow
A) Different Ports
B) Content Switching
General Configuration Steps
As basic rule, the third party should
A) Perform SSL Termination
B) Validate the certificate against the VeridiumID CA cert (you will provide this - /etc/veridiumid/haproxy/client-ca.pem is a fast way to get it).
C) Extract the Distinguished Name from the client certificate and forward as a header to the listener on the VeridiumID server. This header MUST** be called X-SSL-Client-DN otherwise websec will not recognise the DN and the authentication will fail.
D) Create a custom header called x-ssl-termination-proxy-secret with the value of the secret in haproxy.cfg. If the customer cannot set this on their proxy, it can be set on the HAProxy listener (see below).
frontend frontend-https-waf
log-format %ci:%cp\ [%t]\ %ft\ %b/%s\ %Tq/%Tw/%Tc/%Tr/%Tt\ %ST\ %B\ %CC\ %CS\ %tsc\ %ac/%fc/%bc/%sc/%rc\ %sq/%bq\ %hr\ %hs\ %{+Q}[ssl_c_s_dn]\ %{+Q}r\ %sslc\ %sslv
option tcplog
bind *:8444 ssl crt /etc/veridiumid/haproxy/server.pem
###Set the mode to HTTP ###
mode http
option socket-stats
maxconn 300
###Persist the value of the header passed by the customers proxy (x-tls-clientcert-dn - in this example)###
http-request set-var(txn.request_host_header) req.hdr(x-tls-clientcert-dn)
###
tcp-request inspect-delay 5s
tcp-request content accept if { req_ssl_hello_type 1 }
default_backend backend-bops-https-waf
backend backend-bops-https-waf
balance roundrobin
mode http
###If the customers proxy cannot add custom headers, add the secret here too###
http-request set-header x-ssl-termination-proxy-secret fGQazLDRzYBfiKQ
###
###Map the DN value to the header that websec expects X-SSL-Client-DN###
http-request set-header X-SSL-Client-DN %[var(txn.request_host_header)]
###
rspadd Strict-Transport-Security:\ max-age=31536000;\ includeSubDomains
rspirep ^(set-cookie:.*) \1;\ Secure
server server-127.0.0.1 127.0.0.1:8083 check
NGINX
For NGINX, please see this sample NGINX.cfg file (located under /etc/nginx/). This file assumes you have one external DNS name (vid.rdemo.co, for example) and you direct traffic to the various webapps based on paths. The benefit here is you only need one SSL certificate and do not need any custom internet facing ports.
NGINX.conf
Instructions are inline (so suggest reading down the file where config changes are obvious) however the only mandatory changes are:
- Change server_name (modify the <<ENVIRONMENT-NAME>> value)
- Define SSL certificate by amending paths to the cert and key files (the key should be unencrypted, so just split server.pem in HAProxy if you use the same certificate there).
- Add your VeridiumID server CA certificate
- Modify the Proxy Pass variable with the one used by the Veridium Server, to obtain the correct value run the following command on a Veridium Webapp Node:
cat /etc/veridiumid/haproxy/haproxy.cfg | grep x-ssl-termination-proxy-secret | rev | cut -d" " -f1 | rev | uniq
- Add the IP addresses of the VeridiumID server
Note, a listener has been created for websecadmin which should not be the case in production deployments (where admin will not be exposed to the internet). However, for lab environments, it is much easier to access like this.
events {
worker_connections 1000;
}
http {
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;
include /etc/nginx/mime.types;
default_type application/octet-stream;
# Load modular configuration files from the /etc/nginx/conf.d directory.
# See http://nginx.org/en/docs/ngx_core_module.html#include
# for more information.
include /etc/nginx/conf.d/*.conf;
## HTTPS Public Listener with MTLS configuration
server {
listen *:443 ssl;
# Add your public DNS name
server_name <<ENVIRONMENT-NAME>>;
ssl_protocols TLSv1.1 TLSv1.2;
# Your SSL Certificate
ssl_certificate /etc/nginx/cert.crt;
ssl_certificate_key /etc/nginx/key.key;
# Client Certificate Authentication - Add the VeridiumID CA (/etc/veridiumid/haproxy/client-ca.pem on VID server if you need it)
ssl_client_certificate /etc/nginx/clientca.pem;
ssl_verify_client optional;
## Add VeridiumID (HAProxy Listeners)
## Websec
location /websec {
# If client auth fails, return an error
if ($ssl_client_verify != SUCCESS) {
return 403;
}
proxy_set_header X-SSL-Client-DN $ssl_client_s_dn;
proxy_set_header x-ssl-termination-proxy-secret <<PROXY-PASS>>;
# If Nginx will be accessed directly from the Outside network, X-Forwarded-for must be set to $remote_addr
proxy_set_header X-Forwarded-For "$http_x_forwarded_for";
# Add the VeridiumID IP and Proxy Listener Port for Websec
proxy_pass https://<<VERIDIUMID-IP>>:8444/websec;
proxy_set_header Host $host;
proxy_read_timeout 90;
}
## Websecadmin (Optional - Typically Used for Labs/Non Prod where admin could be internet exposed)
location /websecadmin {
# If client auth fails, return an error
if ($ssl_client_verify != SUCCESS) {
return 403;
}
proxy_set_header X-SSL-Client-DN $ssl_client_s_dn;
proxy_set_header x-ssl-termination-proxy-secret <<PROXY-PASS>>;
# If Nginx will be accessed directly from the Outside network, X-Forwarded-for must be set to $remote_addr
proxy_set_header X-Forwarded-For "$http_x_forwarded_for";
# Add the VeridiumID IP and Proxy Listener Port for Websecadmin
proxy_pass https://<<VERIDIUMID-IP>>:8449/websecadmin;
proxy_set_header Host $host;
proxy_read_timeout 90;
}
location /veridium-manager {
limit_req zone=mylimit burst=12;
# If client auth fails, return an error
if ($ssl_client_verify != SUCCESS){
return 403;
}
proxy_set_header X-SSL-Client-DN $ssl_client_s_dn;
proxy_set_header x-ssl-termination-proxy-secret <<PROXY-PASS>>;
# If Nginx will be accessed directly from the Outside network, X-Forwarded-for must be set to $remote_addr
proxy_set_header X-Forwarded-For "$http_x_forwarded_for";
# Add the VeridiumID IP and Proxy Listener Port for Websec
proxy_pass https://<<VERIDIUMID-IP>>:8449/veridium-manager;
proxy_set_header Host $host;
proxy_read_timeout 90;
}
}
# A second server entry without MTLS configuration for DMZ and Shibboleth IDP
server {
listen *:8944 ssl;
server_name <<ENVIRONMENT-NAME>>;
ssl_protocols TLSv1.2;
ssl_protocols TLSv1.1 TLSv1.2;
# Your SSL Certificate
ssl_certificate /etc/nginx/cert.crt;
ssl_certificate_key /etc/nginx/key.key;
## DMZWebsec
location /dmzwebsec {
# Add the VeridiumID IP and Proxy Listener Port for DMZWebsec
# If Nginx will be accessed directly from the Outside network, X-Forwarded-for must be set to $remote_addr
proxy_set_header X-Forwarded-For "$http_x_forwarded_for";
proxy_pass https://<<VERIDIUMID-IP>>:8448;
proxy_set_header Host $host;
proxy_read_timeout 90;
}
## Shibboleth
location /idp {
# Add the VeridiumID IP and Proxy Listener Port for Shibboleth
# If Nginx will be accessed directly from the Outside network, X-Forwarded-for must be set to $remote_addr
proxy_set_header X-Forwarded-For "$http_x_forwarded_for";
proxy_pass https://<<VERIDIUMID-IP>>:8944;
proxy_set_header Host $host;
proxy_read_timeout 90;
}
}
}
NGINX as a load balancer
To use NGINX as a load balancer you must add upstreams for each location configured in the http section of the configuration, similar to:
http {
upstream websec {
server <<VERIDIUMID-IP-1>>:8444;
server <<VERIDIUMID-IP-2>>:8444;
}
upstream websecadmin {
server <<VERIDIUM-IP-1>>:8949;
server <<VERIDIUM-IP-2>>:8949;
}
upstream dmz {
server <<VERIDIUM-IP-1>>:8448;
server <<VERIDIUM-IP-1>>:8448;
}
upstream shibboleth {
# To maintain stickyness
ip_hash;
server <<VERIDIUM-IP-1>>:8944;
server <<VERIDIUM-IP-1>>:8944;
}
}
While inside the location instead of the IP address you will use the upstream name configured for the location:
location /websec {
....
proxy_pass https://websec/websec;
}
location /websecadmin {
....
proxy_pass https://websecadmin/websecadmin;
}
location /veridium-manager {
....
proxy_pass https://websecadmin/veridium-manager;
}
location /dmz {
....
proxy_pass https://dmz/dmz;
}
location /idp {
....
proxy_pass https://shibboleth/idp;
}
NGINX connection and rate limiting
In order to limit the number of connections/request generated from the same source IP the following must be configured in the nginx.conf file:
http {
# Example: Set a total number of 100 requests/second from the same source IP
limit_req_zone $http_x_forwarded_for zone=reqlimit:10m rate=100r/s;
# Example: Creating a connect limit rule
limit_conn_zone $http_x_forwarded_for zone=conlimit:10m;
# If Nginx will be accessed directly from the Outside network, $http_x_forwarded_for must be replaced with $remote_addr
server {
...
location / {
# To apply the rate limiting rule allowing bursts of up to 12 requests/second
limit_req zone=reqlimit burst=12;
}
location / {
# To apply the connection limiting rule and allow a maximum of 10 connections from the same source IP
limit_conn conlimit 10;
}
}
}
NGINX HAProxy Configuration
As you can see above, the "default" haproxy.cfg will not have the appropriate listeners by default. For example, a listener for 8448 or 8444 does not exist. Therefore, after making the NGINX configuration, you need to edit the HAProxy configuration as follows. You should be able to add these to the bottom of the existing file, save changes and restart haproxy. Note - not all of these have backends defined, as they use the ones that exist in haproxy.cfg by default, so this is intentional.
frontend frontend-https-proxy
log-format %ci:%cp\ [%t]\ %ft\ %b/%s\ %Tq/%Tw/%Tc/%Tr/%Tt\ %ST\ %B\ %CC\ %CS\ %tsc\ %ac/%fc/%bc/%sc/%rc\ %sq/%bq\ %hr\ %hs\ %{+Q}[ssl_c_s_dn]\ %{+Q}r\ %sslc\ %sslv
option tcplog
bind *:8444 ssl crt /etc/veridiumid/haproxy/server.pem
mode tcp
option socket-stats
maxconn 300
tcp-request inspect-delay 5s
tcp-request content accept if { req_ssl_hello_type 1 }
default_backend backend-bops-https-proxy
backend backend-bops-https-proxy
balance roundrobin
mode http
http-response add-header Strict-Transport-Security "max-age=31536000; includeSubDomains"
http-response replace-header Set-Cookie (.*) \1;\ SameSite=Strict;\ Secure
server server-127.0.0.1 127.0.0.1:8083 check
frontend frontend-dmz-https-proxy
log-format %ci:%cp\ [%t]\ %ft\ %b/%s\ %Tq/%Tw/%Tc/%Tr/%Tt\ %ST\ %B\ %CC\ %CS\ %tsc\ %ac/%fc/%bc/%sc/%rc\ %sq/%bq\ %hr\ %hs\ %{+Q}[ssl_c_s_dn]\ %{+Q}r\ %sslc\ %sslv
bind *:8448 ssl crt /etc/veridiumid/haproxy/server.pem
option forwardfor
option httplog
option dontlognull
http-response add-header Strict-Transport-Security "max-age=31536000; includeSubDomains"
http-request del-header Origin
http-request set-header X-Forwarded-Proto https
http-request set-header X-SSL %[ssl_fc]
default_backend backend-dmz-http
http-response set-header X-Frame-Options DENY
frontend frontend-websec-admin-https-proxy
log-format %ci:%cp\ [%t]\ %ft\ %b/%s\ %Tq/%Tw/%Tc/%Tr/%Tt\ %ST\ %B\ %CC\ %CS\ %tsc\ %ac/%fc/%bc/%sc/%rc\ %sq/%bq\ %hr\ %hs\ %{+Q}[ssl_c_s_dn]\ %{+Q}r\ %sslc\ %sslv
bind *:8449 ssl crt /etc/veridiumid/haproxy/server.pem ca-file
option forwardfor
option httplog
option dontlognull
http-response add-header Strict-Transport-Security "max-age=31536000; includeSubDomains"
default_backend backend-websec-admin-http-proxy
http-response set-header X-Frame-Options DENY
backend backend-websec-admin-http-proxy
balance roundrobin
http-response replace-header Set-Cookie (.*) \1;\ SameSite=Strict;\ Secure
server server-127.0.0.1 127.0.0.1:9090 check
frontend frontend-ssp-https-proxy
mode http
log-format %ci:%cp\ [%t]\ %ft\ %b/%s\ %Tq/%Tw/%Tc/%Tr/%Tt\ %ST\ %B\ %CC\ %CS\ %tsc\ %ac/%fc/%bc/%sc/%rc\ %sq/%bq\ %hr\ %hs\ %{+Q}[ssl_c_s_dn]\ %{+Q}r\ %sslc\ %sslv
bind *:9988 ssl crt /etc/veridiumid/haproxy/server.pem
option httplog
option forwardfor
option dontlognull
http-response add-header Strict-Transport-Security "max-age=31536000; includeSubDomains"
http-request del-header Origin
http-request set-header X-Forwarded-Proto https
http-request set-header X-SSL %[ssl_fc]
http-response replace-header Set-Cookie (.*) \1;\ SameSite=Strict;\ Secure
default_backend backend-ssp-http-proxy
http-response set-header X-Frame-Options DENY
backend backend-ssp-http-proxy
mode http
balance roundrobin
http-response replace-header Set-Cookie (.*) \1;\ SameSite=Strict;\ Secure
http-response set-header X-Content-Type-Options nosniff
server server-127.0.0.1 127.0.0.1:9986 check
NEVIS
. Infrastructure:
On the SSL connector (port 443) set the Client CA Truststore to the veridium client CA
. Configuration:
The veridium application needs the following mappings:
/dmzwebsec/
/idp/
/dmzweb/
Assign application servers:
/dmzwebsec/ - Veridium app server:port 8448
/idp/ - Veridium app server:port 8644
/dmzweb/ - Veridium app server:port 8444
On /websec/ add two extra filters:
- a delegation filter for the headers
Parameter Delegate Settings:
ENV:SSL_CLIENT_S_DN:X-SSL-Client-DN
ENV:SSL_CLIENT_S_DN_CN:X-SSL-Client-CN
ENV:SSL_CLIENT_I_DN: X-SSL-Issuer
ENV:SSL_CLIENT_V_START:X-SSL-Client-Not-Before
ENV:SSL_CLIENT_V_END:X-SSL-Client-Not-After
CONST:@VER_PROXY_SECRET@:x-ssl-termination-proxy-secret
- a Lua filter to check the client certificate
Script.InputHeaderFunctionName = InputHeader
Script:
function InputHeader(req, resp)
-- Check Client DN
-- Check Issuer
-- Check Validity
end
F5 APM
The APM is the application firewall module of the F5 BIG-IP appliance. It is comparible in terms of features and functions to Citrix Netscaler. From a HAProxy perspective, please use the one for Netscaler above.
1. APM has the concept of iRules that allow headers to be created and passed to VeridiumID. Create this iRule and map it to the websec & websecadmin listeners.
###Retrieve the client certificate DN
when CLIENTSSL_CLIENTHELLO {
set dist_name ""
}
when CLIENTSSL_CLIENTCERT {
if {[SSL::cert count] > 0} {
set dist_name [X509::subject [SSL::cert 0]]
log local0. "$dist_name"
}
}
###Create the Headers to insert the proxy secret and the client certificate DN.
when HTTP_REQUEST {
log local0. "$dist_name"
HTTP::header insert "x-ssl-termination-proxy-secret" "fGQazLDRzYBfiKQ"
if { $dist_name != ""} {
HTTP::header insert "X-SSL-Client-DN" $dist_name
}
}
Airlock WAF
The Airlock WAF has one primary difference in that once client certificate authentication is configured, assuming you set "Send Environment Variables" in the virtualID configuration, all client certificate properties are sent to the VeridiumID (configured as a backend). However these are sent as cookies NOT as custom headers. Therefore, what you have to do is convert the cookie into a header in HAProxy. It looks something like this:
frontend frontend-bops-http
log-format %ci:%cp\ [%t]\ %ft\ %b/%s\ %Tq/%Tw/%Tc/%Tr/%Tt\ %ST\ %B\ %CC\ %CS\ %tsc\ %ac/%fc/%bc/%sc/%rc\ %sq/%bq\ %hr\ %hs\ %{+Q}[ssl_c_s_dn]\ %{+Q}r\ %sslc\ %sslv
mode http
bind 127.0.0.1:4141 accept-proxy ssl crt /etc/veridiumid/haproxy/server.pem
rspadd Strict-Transport-Security:\ max-age=31536000;\ includeSubDomains
##Capture the Cookie##
capture cookie AL_ENV_SSL_CLIENT_S_DN len 256
http-request del-header Origin
http-request set-header X-Forwarded-Proto https
default_backend backend-bops
###RASPADD not supported anymore since HAPROXY 2.1######rspadd X-Frame-Options:\ DENY
backend backend-bops
balance roundrobin
mode http
##Use the Cookie Value to Create a Header##
http-request set-header X-SSL-Client-DN %[req.cook(AL_ENV_SSL_CLIENT_S_DN)]
server server-127.0.0.1 127.0.0.1:8083 check
Adding the proxy secret is straightforward in Airlock as with other proxies:
Airlock UI - Expert Settings - Security Gate / Apache
{code:java} EnvVarCookiePlainChars "!#$&*-./0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ^_`abcdefghijklmnopqrstuvwxyz|~=" {code}