How to setup HAProxy for Secured (Kerberos) WebHDFS HA

[#1]

Servers used in this document:

node1.localdomain = HAProxy server
node2.localdomain = NameNode1
node3.localdomain = NameNode2

1. Install HAProxy

[root@node1 ~]# yum install -y haproxy

2. Set up HAProxy with minimum configuration

[root@node1 ~]# cp -p /etc/haproxy/haproxy.cfg /etc/haproxy/haproxy.cfg.orig
[root@node1 ~]# vim /etc/haproxy/haproxy.cfg
... (snip) ...
#---------------------------------------------------------------------
# main frontend which proxys to the backends
#---------------------------------------------------------------------
frontend main *:50070
    default_backend app
#---------------------------------------------------------------------
# round robin balancing between the various backends
#---------------------------------------------------------------------
backend app
    balance roundrobin
    option httpchk GET /webhdfs/v1/?op=CHECKACCESS
    http-check expect rstatus ([23][0-9][0-9]|401)
    server node2 node2.localdomain:50070 check
    server node3 node3.localdomain:50070 check

3. Create a new service principal for HAProxy from your KDC server

kadmin.local -q "addprinc -randkey HTTP/node1.localdomain@HO-UBU02"

4. Create a keytab. If same file already exists, just in case, taking a backup

[root@node1 ~]# mv /etc/security/keytabs/spnego.service.keytab /etc/security/keytabs/spnego.service.keytab.old
[root@node1 ~]# kadmin -p ambari/admin -q "ktadd -k /etc/security/keytabs/spnego.service.keytab HTTP/node1.localdomain@HO-UBU02"
Authenticating as principal ambari/admin with password.
Password for ambari/admin@HO-UBU02:
...

5. Copy this keytab into NameNode servers:

[root@node1 ~]# scp /etc/security/keytabs/spnego.service.keytab node2.localdomain:/tmp/node1.spnego.service.keytab
spnego.service.keytab 100% 306 0.3KB/s 00:00
[root@node1 ~]# scp /etc/security/keytabs/spnego.service.keytab node3.localdomain:/tmp/node1.spnego.service.keytab
spnego.service.keytab 100% 306 0.3KB/s 00:00

6. Merge keytabs:

First, confirm the keytabl path:

[root@node2 ~]# grep 'dfs.web.authentication.kerberos.keytab' -A1 /etc/hadoop/conf/hdfs-site.xml
<name>dfs.web.authentication.kerberos.keytab</name>
<value>/etc/security/keytabs/spnego.service.keytab</value>

Make a backup:

[root@node2 ~]# mv /etc/security/keytabs/spnego.service.keytab /etc/security/keytabs/spnego.service.keytab.orig

Merge keytabs with ktutil:

[root@node2 ~]# ktutil
ktutil: rkt /etc/security/keytabs/spnego.service.keytab.orig
ktutil: rkt /tmp/node1.spnego.service.keytab
ktutil: wkt /etc/security/keytabs/spnego.service.keytab
ktutil: quit 

Make sure owner and permission:

[root@node2 ~]# chown root:hadoop /etc/security/keytabs/spnego.service.keytab
[root@node2 ~]# chmod 440 /etc/security/keytabs/spnego.service.keytab

Confirm:

[root@node2 ~]# klist -kte /etc/security/keytabs/spnego.service.keytab
Keytab name: FILE:/etc/security/keytabs/spnego.service.keytab
KVNO Timestamp Principal
---- ----------------- --------------------------------------------------------
2 03/22/17 08:45:02 HTTP/node2.localdomain@HO-UBU02 (aes256-cts-hmac-sha1-96)
2 03/22/17 08:45:02 HTTP/node2.localdomain@HO-UBU02 (aes128-cts-hmac-sha1-96)
2 03/22/17 08:45:02 HTTP/node2.localdomain@HO-UBU02 (des3-cbc-sha1)
2 03/22/17 08:45:02 HTTP/node2.localdomain@HO-UBU02 (arcfour-hmac)
5 03/22/17 08:45:02 HTTP/node1.localdomain@HO-UBU02 (aes256-cts-hmac-sha1-96)
5 03/22/17 08:45:02 HTTP/node1.localdomain@HO-UBU02 (aes128-cts-hmac-sha1-96)
5 03/22/17 08:45:02 HTTP/node1.localdomain@HO-UBU02 (des3-cbc-sha1)
5 03/22/17 08:45:02 HTTP/node1.localdomain@HO-UBU02 (arcfour-hmac)

7. Do above steps on Node3 as well

8. From Ambari, search and change dfs.web.authentication.kerberos.principal to "*"

NOTE: Newer Ambari does not allow to change this. In that case, please use configs.sh (deprecated) or configs.py

9. (Optional but strongly recommended) Just in case, stop ambari-server and take a database backup, for example:

ambari-server stop
pg_dump -Uambari -Z 9 -f ./ambari_$(date +"%Y%m%d%H%M%S").sql.gz

10. Login to Ambari database, for example: psql -Uambari ambari then run UPDATE statement below:

update alert_definition set alert_source = replace(alert_source, '{hdfs-site/dfs.web.authentication.kerberos.principal}', '{hdfs-site/dfs.namenode.kerberos.internal.spnego.principal}') where alert_source like '%{hdfs-site/dfs.web.authentication.kerberos.principal}%' and component_name in ('NAMENODE', 'JOURNALNODE', 'DATANODE');

11. Login to Ambari-server with SSH as *root*, then type the following commands:

cd /var/lib/ambari-server/resources/common-services/HDFS/2.1.0.2.0/package/alerts
sed -i_$(date +"%Y%m%d%H%M%S").bak 's/dfs.web.authentication.kerberos.principal/dfs.namenode.kerberos.internal.spnego.principal/' *.py
ambari-server restart

12. From Ambari UI, Disable and Enable each Alerts which are failing due to ".../spnego.service.keytab * > ..." error.

When you disable an Alert, please wait until all red alerts are disappeared.

13. Restart HDFS components from Ambari

14. Test ("HTTP/1.1 200 OK" means good)

[root@node1 ~]# curl --negotiate -u : -X GET 'http://node3.localdomain:50070/webhdfs/v1/?op=CHECKACCESS'
[root@node1 ~]# curl --negotiate -u : -X GET 'http://node2.localdomain:50070/webhdfs/v1/?op=CHECKACCESS'
{"RemoteException":{"exception":"StandbyException","javaClassName":"org.apache.hadoop.ipc.StandbyException","message":"Operation category READ is not supported in state standby"}}[root@node1 ~]#
[root@node1 ~]# curl -s -I --negotiate -u : 'http://node1.localdomain:50070/webhdfs/v1/?op=CHECKACCESS' | grep ^HTTP
HTTP/1.1 401 Authentication required
HTTP/1.1 200 OK

[#2]

Many thanks @Hajime San for writing this one. Now haproxy can't frighten me any more 🙂

Cheers !

[#3]

You can find HAProxy configs for the Big Data ecosystem and other open source technologies on GitHub:

https://github.com/HariSekhon/haproxy-configs

[#4]

With these haproxy parameters, HA doesn't actually work. It shows the active nn, then the standby