Servers used in this document: node1.localdomain = HAProxy server node2.localdomain = NameNode1 node3.localdomain = NameNode2 1. Install HAProxy [root@node1 ~]# yum install -y haproxy 2. Set up HAProxy with minimum configuration [root@node1 ~]# cp -p /etc/haproxy/haproxy.cfg /etc/haproxy/haproxy.cfg.orig [root@node1 ~]# vim /etc/haproxy/haproxy.cfg ... (snip) ... #--------------------------------------------------------------------- # main frontend which proxys to the backends #--------------------------------------------------------------------- frontend main *:50070 default_backend app #--------------------------------------------------------------------- # round robin balancing between the various backends #--------------------------------------------------------------------- backend app balance roundrobin option httpchk GET /webhdfs/v1/?op=CHECKACCESS http-check expect rstatus ([23][0-9][0-9]|401) server node2 node2.localdomain:50070 check server node3 node3.localdomain:50070 check 3. Create a new service principal for HAProxy from your KDC server kadmin.local -q "addprinc -randkey HTTP/node1.localdomain@HO-UBU02" 4. Create a keytab. If same file already exists, just in case, taking a backup [root@node1 ~]# mv /etc/security/keytabs/spnego.service.keytab /etc/security/keytabs/spnego.service.keytab.old [root@node1 ~]# kadmin -p ambari/admin -q "ktadd -k /etc/security/keytabs/spnego.service.keytab HTTP/node1.localdomain@HO-UBU02" Authenticating as principal ambari/admin with password. Password for ambari/admin@HO-UBU02: ... 5. Copy this keytab into NameNode servers: [root@node1 ~]# scp /etc/security/keytabs/spnego.service.keytab node2.localdomain:/tmp/node1.spnego.service.keytab spnego.service.keytab 100% 306 0.3KB/s 00:00 [root@node1 ~]# scp /etc/security/keytabs/spnego.service.keytab node3.localdomain:/tmp/node1.spnego.service.keytab spnego.service.keytab 100% 306 0.3KB/s 00:00 6. Merge keytabs: First, confirm the keytabl path: [root@node2 ~]# grep 'dfs.web.authentication.kerberos.keytab' -A1 /etc/hadoop/conf/hdfs-site.xml <name>dfs.web.authentication.kerberos.keytab</name> <value>/etc/security/keytabs/spnego.service.keytab</value> Make a backup: [root@node2 ~]# mv /etc/security/keytabs/spnego.service.keytab /etc/security/keytabs/spnego.service.keytab.orig Merge keytabs with ktutil: [root@node2 ~]# ktutil ktutil: rkt /etc/security/keytabs/spnego.service.keytab.orig ktutil: rkt /tmp/node1.spnego.service.keytab ktutil: wkt /etc/security/keytabs/spnego.service.keytab ktutil: quit Make sure owner and permission: [root@node2 ~]# chown root:hadoop /etc/security/keytabs/spnego.service.keytab [root@node2 ~]# chmod 440 /etc/security/keytabs/spnego.service.keytab Confirm: [root@node2 ~]# klist -kte /etc/security/keytabs/spnego.service.keytab Keytab name: FILE:/etc/security/keytabs/spnego.service.keytab KVNO Timestamp Principal ---- ----------------- -------------------------------------------------------- 2 03/22/17 08:45:02 HTTP/node2.localdomain@HO-UBU02 (aes256-cts-hmac-sha1-96) 2 03/22/17 08:45:02 HTTP/node2.localdomain@HO-UBU02 (aes128-cts-hmac-sha1-96) 2 03/22/17 08:45:02 HTTP/node2.localdomain@HO-UBU02 (des3-cbc-sha1) 2 03/22/17 08:45:02 HTTP/node2.localdomain@HO-UBU02 (arcfour-hmac) 5 03/22/17 08:45:02 HTTP/node1.localdomain@HO-UBU02 (aes256-cts-hmac-sha1-96) 5 03/22/17 08:45:02 HTTP/node1.localdomain@HO-UBU02 (aes128-cts-hmac-sha1-96) 5 03/22/17 08:45:02 HTTP/node1.localdomain@HO-UBU02 (des3-cbc-sha1) 5 03/22/17 08:45:02 HTTP/node1.localdomain@HO-UBU02 (arcfour-hmac) 7. Do above steps on Node3 as well 8. From Ambari, search and change dfs.web.authentication.kerberos.principal to "*" NOTE: Newer Ambari does not allow to change this. In that case, please use configs.sh (deprecated) or configs.py 9. (Optional but strongly recommended) Just in case, stop ambari-server and take a database backup, for example: ambari-server stop pg_dump -Uambari -Z 9 -f ./ambari_$(date +"%Y%m%d%H%M%S").sql.gz 10. Login to Ambari database, for example: psql -Uambari ambari then run UPDATE statement below: update alert_definition set alert_source = replace(alert_source, '{hdfs-site/dfs.web.authentication.kerberos.principal}', '{hdfs-site/dfs.namenode.kerberos.internal.spnego.principal}') where alert_source like '%{hdfs-site/dfs.web.authentication.kerberos.principal}%' and component_name in ('NAMENODE', 'JOURNALNODE', 'DATANODE'); 11. Login to Ambari-server with SSH as *root*, then type the following commands: cd /var/lib/ambari-server/resources/common-services/HDFS/2.1.0.2.0/package/alerts sed -i_$(date +"%Y%m%d%H%M%S").bak 's/dfs.web.authentication.kerberos.principal/dfs.namenode.kerberos.internal.spnego.principal/' *.py ambari-server restart 12. From Ambari UI, Disable and Enable each Alerts which are failing due to ".../spnego.service.keytab * > ..." error. When you disable an Alert, please wait until all red alerts are disappeared. 13. Restart HDFS components from Ambari 14. Test ("HTTP/1.1 200 OK" means good) [root@node1 ~]# curl --negotiate -u : -X GET 'http://node3.localdomain:50070/webhdfs/v1/?op=CHECKACCESS' [root@node1 ~]# curl --negotiate -u : -X GET 'http://node2.localdomain:50070/webhdfs/v1/?op=CHECKACCESS' {"RemoteException":{"exception":"StandbyException","javaClassName":"org.apache.hadoop.ipc.StandbyException","message":"Operation category READ is not supported in state standby"}}[root@node1 ~]# [root@node1 ~]# curl -s -I --negotiate -u : 'http://node1.localdomain:50070/webhdfs/v1/?op=CHECKACCESS' | grep ^HTTP HTTP/1.1 401 Authentication required HTTP/1.1 200 OK ... View more