UPDATE: This article has been vetted against HDF 2.0 - HDF 3.2. Minor updates have been made for additional clarity on use of NiFi CA for establishing trust with Ranger.
Prerequisites
- NiFi has been installed running with SSL enabled (with certificates manually installed or using the NiFi Certificate Authority). You will need the keystore/truststore names, locations, aliases, identity (DN) and passwords used when enabling SSL for NiFi. Ensure that all nodes have the same keystore/truststore passwords applied and like named locations in order to apply ranger nifi plugin configurations consistently via Ambari.
- Ranger has been installed and configured with security enabled. For instructions on setting up SSL for Ranger please review Configure Ambari Ranger SSL . Note the name, location, aliases, identity (DN) and passwords used when creating keystores and truststores for Ranger
- If Kerberos will be used recommend that this is enabled for the HDF cluster before proceeding.
Part 1 - Establishing trust between NiFi nodes and Ranger
In order for NiFi nodes to communicate over SSL with Ranger, and Ranger to communicate with secured NiFi, certificates should be imported from the Ranger host to NiFi nodes and vice versa. In these instructions we will use the same keystore/truststore used to secure Ranger in order to communicate with NiFi; however it is possible to also generate additional keystore/truststores that are dedicated solely to NiFi communication.
1. Create certificate files from Ranger’s keystore – Use the following command to generate a certificate file:
{java.home}/bin/keytool -export -keystore {ranger.keystore.file} -alias {keystore-alias} -file {cert.filename}
Example:
/usr/jdk64/jdk1.8.0_77/bin/keytool -export -keystore /etc/security/certs/ranger/ranger-admin-keystore.jks -alias rangeradmin -file /etc/security/certs/ranger/ranger-admin-trust.cer
2. Import the generated Ranger certificate file into the trust stores for all nifi nodes in the cluster:
{java.home}/bin/keytool -import -file {ranger.cert.filename} -alias {ranger.keystore.alias} -keystore {nifi.node.ssl.truststore} -storepass {nifi.node.ssl.truststore.password}
Example:
/usr/jdk64/jdk1.8.0_77/bin/keytool -import -file /etc/security/certs/ranger/ranger-admin-trust.cer -alias rangeradmin -keystore /usr/hdf/current/nifi/conf/truststore.jks –storepass {nifi.truststore.password}
3. Create certificate files for import into Ranger’s trust store. There are two ways to approach this:
a) If NiFi Certificate Authority is in use, a certificate from the CA can be generated and imported into Ranger's trust store using the following steps:
i) Create a certificate file from NiFi-CA using command below:
{java.home}/bin/keytool -export -keystore {nifi-ca.keystore.file} -alias {nifi-ca.keystore-alias} -file {nifi-ca-cert.filename}
ii) Import the NiFi CA certificate into Ranger's truststore* using the below command:
{java.home}/bin/keytool -import -file {nifi-ca.cert.filename} -alias {nifi-ca.keystore.alias} -keystore {ranger.ssl.truststore} -storepass {ranger.ssl.truststore.password}
b) If an external CA or self signed certificates are used and manual keystores and truststores were provided for each NiFi node then perform the following:
i) Create certificate files from each nifi node's keystore using the command below:
{java.home}/bin/keytool -export -keystore {nifi.keystore.file} -alias {nifi.keystore-alias} -file {cert.filename}
ii) Import the nifi certificate files into Ranger's truststore*(Repeat for each cert generated. Remember any duplicate aliases may need to be changed using changealias command before importing new ones.)
{java.home}/bin/keytool -import -file {nifi.cert.filename} -alias {nifi.keystore.alias} -keystore {ranger.ssl.truststore} -storepass {ranger.ssl.truststore.password}
*NOTE truststore used by Ranger may be default truststore cacerts file located under {java_home}/jre/lib/security/cacerts*
Part 2 – Enabling Ranger NiFi Plugin via Ambari
Enabling the Ranger-NiFi Plugin should lead to Ambari creating a Service Repository entry in Ranger which will store information for Ranger to communicate with NiFi and store the authorized identity of the NiFi node[s] that will communicate with Ranger.
From the Ambari UI perform the following steps:
1. Under the Ranger configuration section go to the “Ranger Plugin” tab and switch the NiFi Ranger Plugin toggle to “ON”. When prompted save the configuration.
2. If Ranger Auditing will be used, under the Ranger configuration section go to the “Ranger Audit” tab and, if not already enabled, switch the “Audit to Solr” toggle to “On”. This will produce options to enter connection properties for a Solr instance. To use with Ambari Infra (Internal SolrCloud) enable the “SolrCloud” toggles to “ON” as well. Ambari will pre-populate the zookeeper connection string values and credentials. If an External Solr is used the connection values will need to be provided. When prompted save the configuration.
3. Under the NiFi configuration screen go to the ranger-nifi-plugin-properties section. This section stores all the information needed to support Ranger communication with NiFi (to retrieve NiFi REST endpoint data) .
Complete the following in the ranger-nifi-plugin-properties section:
a) Confirm that “Ranger repository config password” and “Ranger repository config user” are pre-populated. These values are set by default by Ambari and refer to Ranger’s admin user and password
b) Authentication - Enter “SSL” if not already detected and pre-populated by Ambari. This will indicate to Ranger that NiFi is running with SSL
c) Keystore for Ranger Service Accessing NiFi - Provide the keystore filename with location path that Ranger will use for SSL communications with NiFi. This should correspond to the keystore used to generate a certificate in Part 1, Step 1.
d) Keystore password - Enter the password for the above keystore
e) Keystore Type – Enter the keystore type for the provided keystore (e.g. JKS)
f) Truststore for Ranger Service Accessing NiFi – Enter the filename with location path of the truststore for the Ranger service
g) Truststore password – Enter the password for the above truststore
h) Truststore type – Enter the truststore type for the provided truststore (e.g. JKS)
i) Owner for Certificate – Enter the identity (Distinguished Name or DN) of the certificate used by ranger
j) Policy user for NiFi – This should be set by default to the value “nifi”
k) Enable Ranger for NiFi – This should be checked (enabled to true)
4. Next go to the ranger-nifi-policymgr-ssl section. This section stores the information NiFi will use to communicate with the secured Ranger service.
Complete the following in the ranger-nifi-policymgr-ssl section:
a) owner.for.certificate – Enter the identity (Distinguished Name or DN) of the nifi node(s) that will communicate with Ranger. Referring to multiple nodes identities this value use regex by adding a regex prefix along with the expression (E.g.: CN=regex:ydavis-kb-ranger-nifi-demo-[1-9]\.openstacklocal, OU=NIFI to match multiple DN using 1 through 9). This value is not required if Kerberos is enabled on HDF. Update: This regular expression feature will be available in HDF 2.0.1.
b) xasecure.policymgr.clientssl.keystore – Enter the keystore location and filename that NiFi will use to communicate with Ranger. This keystore reference should be the same file used to create and import a certificate into Ranger. (Ensure that for multi-node cluster this keystore location is consistent across all nifi node hosts)
c) xasecure.policymgr.clientssl.keystore.credential.file – This value is populated by default and is used by the plugin to generate a file to store credential information. No change to this value is required.
d) xasecure.policymgr.clientssl.truststore – Enter the truststore location and filename that NiFi will use to communicate with Ranger.
e) xasecure.policymgr.clientssl.truststore.credential.file - This value is populated by default and is used by the plugin to generate a file to store credential information. No change to this value is required.
f) xasecure.policymgr.clientssl.truststore.password – Enter the password for the provided truststore file.
5. The other two sections for Ranger NiFi plugin (ranger-nifi-security and ranger-nifi-audit) do not require additional configuration however can be reviewed for the following:
Confirm the following in ranger-nifi-audit section:
a) Audit to SOLR is enabled (if Ranger Audit was enabled in Part 2, Step 2)
b) xasecure.audit.destination.solr.urls is completed (if an external Solr instance was referenced in Step 2)
c) xasecure.audit.destination.solr.zookeepers is completed and matches the connection string (if Ambari Infra or external SolrCloud was enabled in Step 2)
d) xasecure.audit.is.enabled is set to true
Confirm the following in the ranger-nifi-security section:
a) ranger.plugin.nifi.policy.rest.ss.config.file is set to ranger-policymgr-ssl.xml
b) ranger.plugin.nifi.policy.rest.url refers to the ambari variable for Ranger service {{policy_mgr_url}} (any replacement here means that a Ranger service external to the HDF installation is the target)
6. Save all NiFi configuration changes
7. Restart all required services and ensure that Ambari indicates services have restarted successfully
Part 3 – Confirm Ranger Configuration and Setting up Policies
1. Go to the Ranger Admin UI and using the Access Manager menu select “Resource Based Policies”. Confirm that an entry for NiFi exists in the NiFi Service Manager. The entry name is dynamically created based on the Ambari cluster name (see example below).
2. Select the edit button next to the service repository entry and confirm that the properties from the ranger-nifi-plugin-properties are accurately populated. Also confirm the NiFi URL provided (usually 1 node is used)
3. Confirm that the commonNameForCertificate value is the CN value from the Owner for Certificate property from ranger-nifi-plugin-properties.
4. Using the Ranger Menu go to the “Audit” screen and select the plugin tab. You should see one or more entries from each node in the cluster showing NiFi syncing with ranger policies.
5. If not using Usersync in Ranger, manually create new users in Ranger which correspond to the authentication method used to secure NiFi. For example when using Kerberos Authentication in NiFi ensure that the users created match with the Kerberos principal.
To create a user perform the following tasks:
a) In the Ranger Admin go to Settings menu and select “User/Groups”
b) Click the “Add New User” button
c) Complete the User Detail screen providing the User Name as the identity for the appropriate NiFi authentication method (e.g. Client DN, LDAP DN or Kerberos principal). Password and First Name is required by Ranger but is not used by NiFi. The Role selected should be User (groups are not used by the plugin at this time)
d) Save the new user and repeat for any other users who need access to NiFi
6. Users entries must also be created for each node in the NiFi cluster. Repeat the “Add User” step; however for User Name provide the Distinguished Name for each node as shown below:
7. In order for NiFi nodes to be authorized to communicate within the cluster a Proxy policy should be created. In the Ranger Access Manager menu select “Resource Based Policies” and select the NiFi service repository entry link. On the policy screen select the “Add New Policy” button
8. Provide the following for Policy Details:
a) Policy Name – provide a name for the policy (e.g. proxy)
b) NiFi Resource Identifier – Enter “/proxy” for NiFi’s proxy endpoint
c) Audit Logging – should be set to yes if logging was previously enabled
d) Allow conditions section – In the “Select User” field choose each NiFi node user that was previously created. For the Permissions field select “Write”.
e) Add the new policy
9. Once these authorizations have been created, it is now possible to confirm that Ranger can communicate with NiFi (attempting to do so before nodes were authorized would result in a communication error). Go to the Access Manager menu, select “Resource Based Policies” and select the edit button next to the NiFi service repository entry link.
10. Click on the Test Connection button located just below the Config Properties entries. Ranger should be able to Connect to NiFi successfully.
11. Now configure other user policies for accessing NiFi. To configure a NiFi admin/super user (or admin rights), a user can be added to the all – nifi-resource policy that was created by default. In the Ranger Access Manager menu select “Resource Based Policies” and select the NiFi service repository entry link. Then select the edit button next to the “all – nifi-resource” entry.
12. In the Allow Conditions section select the user(s) which will be applied to this policy. Also add both Read and Write permissions.
13. Save the policy with the new settings and confirm that the configured user can access NiFi with given rights by logging into NiFi on a node. Repeat login on each node in cluster to confirm policy is applied throughout.
14. Confirm that login access as well as proxy communication of nodes were audited in Ranger using the Audit screen and navigating to the “Access” tab
At this point Ranger can be used to administer policy for NiFi.
Troubleshooting
If there are problems with NiFi communication with Ranger, review the xa_secure.log (on Ranger's installation) as well as NiFi's nifi-app.log to determine the source of the issue. Many times this is due to certificates not being imported into Ranger's truststore or, if kerberos was not enabled, the commonNameForCertificate (in the NiFi service repository entry in Ranger) value is inaccurate.
If there are problems with Ranger communication with NiFi this also could be due to certificates not being imported into NiFi nodes or the Ranger certificate not appropriately being identified. The in addition to the previously mentioned logs the nifi-user.log will be useful to review as well.
