Knox HA / Loadbalancing using Haproxy

[#1]

Hi All,

Below are the steps to setup load balancer on top of multiple Knox gateways. I will assume that you have KnoxSSO already set and services are going through it. 

1.  Install knox gateway on another node Ambari/CM -> Hosts -> Select any host(where you want to install knox) and add knox gateway.
2. Installation of HAproxy and Rsyslog:

yum install -y haproxy rsyslog

• Step 3: Update the Rsyslog for checking the haproxy logs /etc/rsyslog.conf:

## Uncomment below line
$ModLoad imudp
$UDPServerRun 514

##Add following directive in the same file.

# HAProxy Logging
local2.*        /var/log/haproxy.log

• Create SSL certificate for haproxy:

openssl genrsa -out /etc/haproxy/haproxy.key 2048
openssl req -new -key /etc/haproxy/haproxy.key -out /etc/haproxy/haproxy.csr -subj "/C=US/ST=North Carolina/L=Raleigh/O=HWX/OU=Support/CN=$(hostname -f)"
openssl x509 -req -days 365 -in /etc/haproxy/haproxy.csr -signkey /etc/haproxy/haproxy.key -out /etc/haproxy/haproxy.crt 
cat /etc/haproxy/haproxy.key /etc/haproxy/haproxy.crt > /etc/haproxy/haproxy.pem

• Update haproxy configuration file /etc/haproxy/haproxy.cfg with both Knox hosts and LB server, In my case c2110-node1 and c2110-node2 are Knox gateway host and c2110-node3 is the LB node:

#---------------------------------------------------------------------
# main frontend which proxys to the backends
#---------------------------------------------------------------------
frontend knox-frontend-443
    mode http
    option httplog
    bind *:443 ssl crt /etc/haproxy/haproxy.pem
    option forwardfor
    http-request redirect location https://%[req.hdr(Host)]/gateway/homepage/home/ if { path / }
    default_backend knox-backend-8443

backend knox-backend-8443
    mode http
    option httplog
    option forwardfor
    balance roundrobin
    stick-table type ip size 1m expire 24h
    stick on src
    option httpchk HEAD /gateway/knoxsso/knoxauth/login.html HTTP/1.1\r\nHost:\ c1110-node1:8443
    http-check expect status 200
    server knox1 c1110-node2:8443 check ssl verify none
    server knox2 c1110-node3:8443 check ssl verify none

• Start the services:

systemctl start haproxy
systemctl enable haproxy
systemctl start rsyslog.service

• set LB url using ambari-server setup-sso command, Only change the url as PEM is already setup for knox1 hosts:

[root@c2110-node1 ~]# ambari-server setup-sso
Using python  /usr/bin/python
Setting up SSO authentication properties...
Enter Ambari Admin login: admin
Enter Ambari Admin password: 
SSO is currently enabled
Do you want to disable SSO authentication [y/n] (n)? 
Provider URL (https://c2110-node3.squadron.support.hortonworks.com:8443/gateway/knoxsso/api/v1/websso): https://c2110-node3.squadron.support.hortonworks.com:443/gateway/knoxsso/api/v1/websso
The SSO provider's public certificate has already set. Do you want to change it [y/n] (n)? 
Use SSO for Ambari [y/n] (n)? 
Manage SSO configurations for eligible services [y/n] (y)? 
 Use SSO for all services [y/n] (y)? 
JWT Cookie name (hadoop-jwt): 
JWT audiences list (comma-separated), empty for any (): 
Ambari Server 'setup-sso' completed successfully.

• Generate the gateway-identity file and copy on both nodes:

# will create gateway.jks in keystore folder
$GATEWAY_HOME/bin/knoxcli.sh create-cert --hostname localhost  

# to check the CN=localhost
keytool -v -list -keystore gateway.jks 

# create the gateway-identity.pem certificate from gateway.jks file
keytool -exportcert -alias gateway-identity -rfc -file $certificate_path -keystore $gateway_home/data/security/keystores/gateway.jks

# to check the CN=localhost
openssl x509 -in gateway-identity.pem -text  

# scp these two files from one node to another
scp /usr/hdp/3.1.0.0-78/knox/data/security/keystores/gateway* c2110-node2:/usr/hdp/3.1.0.0-78/knox/data/security/keystores/

Thanks for reading.