In the previous articles, we showed how to deploy an HDF 2.x/3.0 cluster, enable SSL for Nifi and setup the Ranger Nifi plugin. Here we will build on the same cluster and show how to enable kerberos using Active Directory. Summary To achieve this, the high level steps we will follow are: Setup certificate trust for HDF nodes Run Ambari security wizard Create Ranger policy for nifiadmin user Delete certificate Login to Nifi using AD principal credentials Pre-requisites You have correctly setup AD as described here Active Directory setup with domain: CLOUD.HORTONWORKS.COM AD already preconfigured with LDAPS Certificate (.crt) used to enable LDAPS is available OU created where HDF principals will be created hadoop user has permission to write principals to above OU nifiadmin user created in AD (optionally synced over to Ranger) Test to ensure you can access AD over LDAPS using hadoopadmin user succeeds: ldapsearch -H ldaps://sme-security-ad03.cloud.hortonworks.com:636 -D hadoopadmin@cloud.hortonworks.com -w BadPass#1 Steps 1. Setup trust for all HDF nodes using the AD certificate #run on all HDF nodes before running security wizard using AD ad_ip=xx.xx.xx.xx ##replace with IP of your AD cert_url=http://someurl/mycertificate.crt ## replace with location of exported AD certificate echo "${ad_ip} ad01.lab.hortonworks.net ad01" | sudo tee -a /etc/hosts sudo yum -y install openldap-clients ca-certificates #instead of downloading the cert, you could also manually transfer the .cert file to below location sudo curl -sSL "${cert_url}" -o /etc/pki/ca-trust/source/anchors/hortonworks-net.crt sudo update-ca-trust force-enable sudo update-ca-trust extract sudo update-ca-trust check # edit /etc/openldap/ldap.conf to include LDAP url and base sudo tee -a /etc/openldap/ldap.conf > /dev/null << EOF TLS_CACERT /etc/pki/tls/cert.pem URI ldaps://ad01.lab.hortonworks.net ldap://ad01.lab.hortonworks.net BASE dc=cloud,dc=hortonworks,dc=com EOF #test using openssl - should return 0 openssl s_client -connect ad01:636 </dev/null #test using ldapsearch ldapsearch -H ldaps://sme-security-ad03.cloud.hortonworks.com:636 -D nifiadmin@cloud.hortonworks.com -w BadPass#1 2. Run Ambari Security Wizard Launch security wizard via Ambari (under Admin > Kerberos) and enter below: The ‘Configure Kerberos’ page is the only one you will need to update. Enter the below then click Next on all remaining screens. KDC host: FQDN of AD Realm name: CLOUD.HORTONWORKS.COM Kadmin host: FQDN of AD node Admin principal: hadoopadmin@cloud.hortonworks.com Password: BadPass#1 On ‘Configure Identities’ page, users will be shown the option to customize the keytabs/principals for all components: The Nifi ones are under Advanced tab: Click Next to proceed using the default keytab/principal names Click Next to proceed through all remaining steps of the wizard. What’s happening to Nifi under the covers when security wizard runs? a) NiFi principal and keytabs will be automatically be created/distributed across the cluster where needed by Ambari b) Kerberos-related nifi.properties fields will automatically be updated: NiFi.kerberos.service.principal NiFi.kerberos.keytab.location NiFi.kerberos.krb5.file NiFi.kerberos.authentication.expiration c) Login provider will also be switched to kerberos under the covers d) As part of the process, other HDF components were also kerberized including ‘Ambari Infra’ service. This mean that Ranger audits are now being written to kerberized Solr After security wizard completes, NiFi’s kerberos details will appear alongside other components (under Admin > Kerberos). At this point, Kerberos security will be enabled for all components running on the cluster: On a node running Nifi, you can verify the keytab was generated and list its principal # klist -kt /etc/security/keytabs/nifi.service.keytab Keytab name: FILE:/etc/security/keytabs/nifi.service.keytab KVNO Timestamp Principal ---- ------------------- ------------------------------------------------------ 1 09/28/2016 04:55:08 nifi/abajwa-hdf-qe-hdfsecured-1.openstacklocal@CLOUD.HORTONWORKS.COM 1 09/28/2016 04:55:08 nifi/abajwa-hdf-qe-hdfsecured-1.openstacklocal@CLOUD.HORTONWORKS.COM 1 09/28/2016 04:55:08 nifi/abajwa-hdf-qe-hdfsecured-1.openstacklocal@CLOUD.HORTONWORKS.COM 1 09/28/2016 04:55:08 nifi/abajwa-hdf-qe-hdfsecured-1.openstacklocal@CLOUD.HORTONWORKS.COM 1 09/28/2016 04:55:08 nifi/abajwa-hdf-qe-hdfsecured-1.openstacklocal@CLOUD.HORTONWORKS.COM You can also verify the nifi configs for kerberos were automatically populated: # cat /etc/nifi/conf/nifi.properties | grep kerberos nifi.kerberos.krb5.file=/etc/krb5.conf nifi.kerberos.service.keytab.location=/etc/security/keytabs/nifi.service.keytab nifi.kerberos.service.principal=nifi/abajwa-hdf-qe-hdfsecured-1.openstacklocal@CLOUD.HORTONWORKS.COM nifi.kerberos.spnego.authentication.expiration=12 hours nifi.kerberos.spnego.keytab.location=/etc/security/keytabs/spnego.service.keytab nifi.kerberos.spnego.principal=HTTP/abajwa-hdf-qe-hdfsecured-1.openstacklocal@CLOUD.HORTONWORKS.COM nifi.security.user.login.identity.provider=kerberos-provider You can also verify that the login-identity-provider or Nifi has now been switched to kerberos # tail /etc/nifi/conf/login-identity-providers.xml <provider> <identifier>kerberos-provider</identifier> <class>org.apache.nifi.kerberos.KerberosProvider</class> <property name="Default Realm">HORTONWORKS.COM</property> <property name="Authentication Expiration">12 hours</property> </provider> 3. Login to Nifi UI without certificate Now that kerberos is enabled, lets try to login without using certificate Make sure nifiadmin user exists in Ranger (if you ran Ranger sync earlier this should have been imported already). If not, create the user in Ranger by navigating to below url and entering below http://<Ranger_node>:6080/index.html#!/user/create Create Ranger policy for new user In Ranger, under ‘Access Manager, click ‘HDF-nifi’ Click Edit button on the /* policy we previously added nifiadmin@CLOUD.HORTONWORKS.COM to Add the newly created nifiadmin user to the policy, and click Save Delete previously imported .p12 certificates from your browser e.g. if using Chrome on OSX you can delete previously imported certificates using ‘Keychain Access’ application Restart Chrome and open Nifi UI. It should now display a login page If not, try opening “Incognito Window” Enter username as nifiadmin and the password you set The Nifi UI should open now and you will be logged in as that user You can see who you are logged in as by checking top-right corner of Nifi UI This completes the tutorial. If you made it this far in the series, congratulations! You have successfully: Deployed HDF 2.0 Enabled SSL for Nifi and explored file-based authorization for Nifi Installed Ranger and switched to Ranger-based authorization for Nifi Enabled kerberos for your HDF cluster using Active Directory Logged into Nifi using AD credentials ... View more

In the previous articles, we showed how to deploy an HDF 2.0 cluster, enable SSL for Nifi and setup the Ranger Nifi plugin. Now we will build on the same cluster and show how to enable kerberos from Ambari using MIT KDC. Summary To achieve this, the high level steps we will follow are: Setup MITC KDC Run Ambari security wizard Create principal for nifiadmin user in KDC Create Ranger policy for nifiadmin user Delete certificate from browser Login to Nifi using KDC principal credentials Steps 1. Setup MIT KDC High level steps to setup KDC: Install KDC rpms Configure KDC (krb5.conf) Create KDC database Start krb5kdc/kadmin services Create admin principal Make user an administrator by adding to kadm5.acl Restart krb5kdc/kadmin services Script to automate KDC setup (run below on Ambari node) export realm=HORTONWORKS.COM export domain=hortonworks.com export kdcpassword="BadPass#1" curl -sSL https://gist.github.com/abajwa-hw/f8b83e1c12abb1564531e00836b098fa/raw | sudo -E sh Test KDC is up by running below on Ambari node: kadmin -p admin/admin -w BadPass#1 -r HORTONWORKS.COM -q "get_principal admin/admin" 2. Run Ambari Security Wizard Launch security wizard via Ambari (under Admin > Kerberos) and enter below: The ‘Configure Kerberos’ page is the only one you will need to update. Enter the below then click Next on all remaining screens. KDC host: FQDN of KDC (Ambari) node Realm name: HORTONWORKS.COM Kadmin host: FQDN of KDC (Ambari) node Admin principal: admin/admin Password: BadPass#1 On ‘Configure Identities’ page, users will be shown the option to customize the keytabs/principals for all components: The Nifi ones are under Advanced tab: Click Next to proceed using the default keytab/principal names Click Next to proceed through all remaining steps of the wizard. What’s happening to Nifi under the covers when security wizard runs? a) NiFi principal and keytabs will be automatically be created/distributed across the cluster where needed by Ambari b) Kerberos-related nifi.properties fields will automatically be updated: NiFi.kerberos.service.principal NiFi.kerberos.keytab.location NiFi.kerberos.krb5.file NiFi.kerberos.authentication.expiration c) Login provider will also be switched to kerberos under the covers d) As part of the process, other HDF components were also kerberized including ‘Ambari Infra’ service. This mean that Ranger audits are now being written to kerberized Solr After security wizard completes, NiFi’s kerberos details will appear alongside other components (under Admin > Kerberos). At this point, Kerberos security will be enabled for all components running on the cluster: On a node running Nifi, you can run below commands to: ...verify the keytab was generated and list its principal # klist -kt /etc/security/keytabs/nifi.service.keytab Keytab name: FILE:/etc/security/keytabs/nifi.service.keytab KVNO Timestamp Principal ---- ------------------- ------------------------------------------------------ 1 09/28/2016 04:55:08 nifi/abajwa-hdf-qe-hdfsecured-1.openstacklocal@HORTONWORKS.COM 1 09/28/2016 04:55:08 nifi/abajwa-hdf-qe-hdfsecured-1.openstacklocal@HORTONWORKS.COM 1 09/28/2016 04:55:08 nifi/abajwa-hdf-qe-hdfsecured-1.openstacklocal@HORTONWORKS.COM 1 09/28/2016 04:55:08 nifi/abajwa-hdf-qe-hdfsecured-1.openstacklocal@HORTONWORKS.COM 1 09/28/2016 04:55:08 nifi/abajwa-hdf-qe-hdfsecured-1.openstacklocal@HORTONWORKS.COM ...verify the nifi configs for kerberos were automatically populated: # cat /etc/nifi/conf/nifi.properties | grep kerberos nifi.kerberos.krb5.file=/etc/krb5.conf nifi.kerberos.service.keytab.location=/etc/security/keytabs/nifi.service.keytab nifi.kerberos.service.principal=nifi/abajwa-hdf-qe-hdfsecured-1.openstacklocal@HORTONWORKS.COM nifi.kerberos.spnego.authentication.expiration=12 hours nifi.kerberos.spnego.keytab.location=/etc/security/keytabs/spnego.service.keytab nifi.kerberos.spnego.principal=HTTP/abajwa-hdf-qe-hdfsecured-1.openstacklocal@HORTONWORKS.COM nifi.security.user.login.identity.provider=kerberos-provider ...verify that the login-identity-provider or Nifi has now been switched to kerberos # tail /etc/nifi/conf/login-identity-providers.xml <provider> <identifier>kerberos-provider</identifier> <class>org.apache.nifi.kerberos.KerberosProvider</class> <property name="Default Realm">HORTONWORKS.COM</property> <property name="Authentication Expiration">12 hours</property> </provider> 3. Login to Nifi UI without certificate Now that kerberos is enabled, lets try to login without using certificate First create a principal in KDC for nifiadmin. From the node running KDC (same one as Ambari) run below and enter your desired password (e.g. BadPass#1): kadmin.local -q "addprinc nifiadmin" Create the user in Ranger by navigating to below url and entering below http://<Ranger_node>:6080/index.html#!/user/create The username should be in the format userprinc@KDC_REALM (e.g. nifiadmin@HORTONWORKS.COM) Create Ranger policy for new user In Ranger, under ‘Access Manager, click ‘HDF-nifi’ Click Edit button on the policy we previously added nifiadmin@HORTONWORKS to Add the newly created nifiadmin@HORTONWORKS.COM’ user to the policy, and click Save Delete previously imported .p12 certificates from your browser e.g. if using Chrome on OSX you can delete previously imported certificates using ‘Keychain Access’ application Restart Chrome and open Nifi UI. It should now display a login page If not, try opening “Incognito Window” Enter username as nifiadmin@HORTONWORKS.COM and the password you set The Nifi UI should open now and you will be logged in as that user You can see who you are logged in as by checking top-right corner of Nifi UI This completes the tutorial. If you made it this far in the series, congratulations! You have successfully: Deployed HDF 2.0 Enabled SSL for Nifi and explored file-based authorization for Nifi Installed Ranger and switched to Ranger-based authorization for Nifi Enabled kerberos for your HDF cluster Logged into Nifi using KDC credentials ... View more