Hortonworks Secure Cluster with Isilon OneFS

[#1]

Prerequisites:

• HDP 2.x
• OneFS 7.2
• Ambari should report all green for all hosts in cluster
• Forward and reverse hostnames or smart connect should be configured

Kerberos Requirements:

• All KDC’s need to have different realm name
• One KDC per zone
• Disable AES encryption in client krb5.conf
• Deleting principals from Isilon doesn’t remove them from kdc
• Don’t use the isi auth krb5 spn fix command

Overview:

Following these steps in the order below will accomplish these tasks:

• KDC Setup: install and configure
• Hadoop Client Setup: Kerberos configured and tested
• Secure Isilon Setup: configure, create principals and set proxyusers
• Finish Hadoop Client Setup: Create all necessary principals, place keytabs on the correct hosts and start services
• Finish Hadoop Client Setup: kerberos_only configuration

KDC Setup:

Configure KDC: http://docs.hortonworks.com/HDPDocuments/HDP1/HDP-1.3.1/bk_installing_manually_book/content/rpm-chap... below is an overview of the steps taken in the link provided

• Follow section 13: 1.2 through 13: 1.4
• Modify kdc.conf for supported encryption type
  • • supported_enctypes = RC4-HMAC:normal DES-CBC-MD5:normal DES-CBC-CRC:normal
• Modify kdc.conf to have corrected Realm Name.
• Update kadm5.acl to reflect the new Realm Name
• Create kdc
  • /usr/sbin/kdb5 util_create –s
  • /etc/rc.d/init.d/krb5kdc start
  • /etc/rc.d/init.d/kadmin start

Hadoop Client Setup:

• Run this on all hadoop clients yum install krb5-workstation krb5-libs
• Modify and copy the krb5.conf from the the kdc to all the clients
  • Update Realm Name, kdc and admin server
    • Make sure to update the default realm
  • Update encryption to not include AES
    • default_tgs_enctypes = RC4-HMAC DES-CBC-MD5 DES-CBC-CRC
    • default_tkt_enctypes = RC4-HMAC DES-CBC-MD5 DES-CBC-CRC
    • permitted_enctypes = RC4-HMAC DES-CBC-MD5 DES-CBC-CRC
    • preferred_enctypes = RC4-HMAC DES-CBC-MD5 DES-CBC-CRC

• Now you can test using kinit from the clients and it should work
• kinit kadmin/admin

Secure Isilon Setup:

• To prevent auto spn generation in the system zone you need to set ‘All Auth Providers’ setting on the system zone to ‘No’.
  • isi zone zones modify --zone=system --all-auth-providers=No
• Add the KDC to the Isilon cluster and each KDC needs a unique name
  • isi auth krb5 create --realm=EXAMPLE.COM --admin-server=kdc.example.com --kdc=kdc.example.com --user=kadmin/admin --password=isi
• To verify the join and list all the auth providers for the cluster
  • isi auth status
• Modify zone to use authenticaion provider
  • isi zone zones modify --zone=zone-example --add-auth-provider=krb5:EXAMPLE.COM
• Verify
  • isi zone zones view --zone=zone-example
• Create the Isilon spn’s for the zone. The format needs to be hdfs/<cluster hostname/SC name>@REALM and HTTP/<cluster hostname/SC name>@REALM
  • isi auth krb5 spn create --provider-name=EXAMPLE.COM --spn=hdfs/cluster.example.com@EXAMPLE.COM --user=kadmin/admin --password=isi
  • isi auth krb5 spn create --provider-name=EXAMPLE.COM --spn=HTTP/cluster.example.com@EXAMPLE.COM --user=kadmin/admin --password=isi
• Verify spn creation
  • isi auth krb5 spn list --provider-name=EXAMPLE.COM
• Lastly create proxy users
  • isi hdfs proxyusers create oozie --zone=zone-example --add-user=ambari-qa
  • isi hdfs proxyusers create hive --zone=zone-example --add-user=ambari-qa

Finish Hadoop Client Setup:

• Enter the Ambari secure setup wizard
  • Admin -> security -> enable security
• Click through the wizard untill you get to the screen that configures the principals. Note: Isilon does not convert principal names to short names using rules so don’t use aliases(e.g. rm instead of yarn)
  • Realm name
  • Hdfs -> namenode hdfs/cluster.example.com@EXAMPLE.COM
  • Hdfs -> secondarynamenode hdfs/cluster.example.com@EXAMPLE.COM
  • Hdfs -> datanode hdfs/cluster.example.com@EXAMPLE.COM
  • Falcon -> namenode hdfs/cluster.example.com@EXAMPLE.COM
  • Yarn -> resourceManager yarn/_HOST
  • Yarn -> nodemanager yarn/_HOST
  • Mapreduce2 -> history server principal -> mapred/_HOST
  • DFS Web Principal -> HTTP/cluster.example.com@EXAMPLE.COM
• Now download the csv and copy it to the ambari server
  • On the server put the file in /var/lib/ambari-server/resources/scripts/
• On the ambari server go to that path and run the keytabs.sh script
  • ./keytabs.sh host-principal-keytab-list.csv > generate_keytabs.sh
• In the generate_keytabs.sh script that was just generated you need to comment out all the lines that have to do with principals being created for hdfs or for the Isilon Cluster. This script is going to generate all the principals for the hadoop services but Isilon principals are already created by the Isilon cluster so there is no need to create them again. Doing so will cause secure cluster to not authenticate properly.
• Finally execute the ./generate_keytabs.sh and this will create all the principals for the hadoop services and export a keytab for every host in the cluster.
• Copy the keytab tar files created to the clients and extract them in the proper location.
• Finish the wizard install

Finish Secure Isilon Setup:

• After everything has finished installing you need to configure the Isilon zone to only allow secure connections.
  • isi zone zones modify --zone=zone-example --hdfs-authentication=kerberos_only

[#2]

It's not clear which version of Ambari this is for. Maybe 2.0?

Current steps (Ambari 2.1/2.2, HDP 2.3/2.4 and OneFS 7.2.1.2/8.0.0.1) for most easily completing Kerberos configuration are linked in answer to this question: https://community.hortonworks.com/questions/38583/configure-kerberos-using-ambari-with-emc-isilon.ht...