In the previous article, we showed how to enable SSL and set up identity mappings for Apache Nifi on the previously installed HDF 2.x or 3.0 cluster. Here, we will build on the same cluster and show how to install Apache Ranger and setup the Ranger Nifi plugin. For simplicity, we will assume this is a demo environment where there is no requirement to enable SSL for Ranger. If instead you would like to use secured Ranger with NiFi, follow steps here
Summary
At a high level, Apache Ranger provides a centralized platform to define, administer and manage security policies consistently across Hadoop components. In the case of HDF, it enables the administrator to create/manage authorization policies for Kafka, Storm and Nifi from the same web interface (or REST APIs).
To achieve this, the high level steps we will follow are:
- Ranger install prerequisites
- Ranger install
- Update Nifi Ranger repo
- Test Ranger plugin
- Create Ranger uses and policies
- Test Nifi access as nifiadmin user
The official documentation for this can be found here
Tested with HDF 2.x and 3.0
Step Details
1. Ranger install prerequisites:
a) Make sure Logsearch or external Solr is installed/running before installing Ranger (used to store audits)
In our case, we had deployed the cluster with Logsearch so will use that option
b) Configure RDBMS for Ranger (used to store policies)
in our case we will use the same PostGres used by Ambari. So from the Ambari node, run below:
ranger_user=rangeradmin #set this to DB user you wish to own Ranger schema in RDBMS
ranger_pass=BadPass#1 #set this to password you wish to use
yum install -y postgresql-jdbc*
chmod 644 /usr/share/java/postgresql-jdbc.jar
echo "CREATE DATABASE ranger;" | sudo -u postgres psql -U postgres
echo "CREATE USER ${ranger_user} WITH PASSWORD '${ranger_pass}';" | sudo -u postgres psql -U postgres
echo "ALTER DATABASE ranger OWNER TO ${ranger_user};" | sudo -u postgres psql -U postgres
echo "GRANT ALL PRIVILEGES ON DATABASE ranger TO ${ranger_user};" | sudo -u postgres psql -U postgres
sed -i.bak s/ambari,mapred/${ranger_user},ambari,mapred/g /var/lib/pgsql/data/pg_hba.conf
cat /var/lib/pgsql/data/postgresql.conf | grep listen_addresses
#make sure listen_addresses='*'
ambari-server setup --jdbc-db=postgres --jdbc-driver=/usr/share/java/postgresql-jdbc.jar
service ambari-server stop
service postgresql restart
service ambari-server start2. Ranger install: Start Ambari ‘Add service’ wizard and select Ranger and choose any host to install install Ranger components on.
a) On the configuration screen there are few things to set:On ‘Ranger Admin’ tab set below and run ‘Test connection’:
- Db flavor: POSTGRES
- Host: FQDN of Ambari node
- Database Administrator (DBA) username: rangeradmin
- Passwords: BadPass#1
b) ‘Ranger User Info’ tab is where you would optionally configure Ranger to pull users from Active Directory or LDAP (see here for sample steps on how we setup our AD)
- ‘Common configs’ sub-tab
- ‘User configs’ sub-tab
c) On ‘Ranger Plugin’ tab enable plugins for Nifi, Storm, Kafka. (Note the plugins for Storm/Kafka will not be enabled until kerberos is enabled on the cluster)
d) On ‘Ranger Audit’ tab provide Solr details. In our case, since Logsearch/Ambari_infra components were installed, just turn on SolrCloud - Ambari will autodetect the Zookeeper string
e) On ‘Ranger Tagsync’ tab, no changes needed
f) On ‘Advanced’, no changes needed. If you wanted to setup ability to use AD/LDAP credentials to log into Ranger you can configure this (and other advanced features) here
g) Click Next > Proceed Anyway > Deploy to start the Ranger install and wait for it to complete
h) Once installed, Ambari will show that Storm, Kafka, Nifi need to be restarted.
Use the “Restart All Required” button (new in Ambari 2.4) to do this:
3. Update Nifi Ranger repo: This is needed to enable auto-completion when creating policies in Ranger for Nifi. Note that if this step is skipped, Ranger plugin will still work as usual - it just impacts lookups when creating Nifi policies from the Ranger web interface. If SSL for Ranger will not be setup, you should consider just skipping this step.
To access the Nifi repo in Ranger:
a) Open Ranger using Quicklink in Ambari
b) In Ranger > Access Manager > Nifi > click Edit icon
c) Notice most of the configs are empty. If you try a test connect, it will fail with below:
d) On Ranger host, run below to find the keystore/truststore details (like path, type and password)
cat /usr/hdf/current/nifi/conf/nifi.properties | grep nifi.security
e) Ensure ranger user can access key/truststores by running below on Ranger host:
chmod o+r /usr/hdf/current/nifi/conf/keystore.jks /usr/hdf/current/nifi/conf/truststore.jks
Note: in secure environments Ranger should not access Nifi keystore/truststore: there should be a separate keystore/truststore for Ranger to use as part of enabling SSL for it.
Also note that these files could be re-generated by Nifi CA, resetting the permissions
f) Update as shown :
- Keystore path, type, password
- Truststore path, type, password
g) Now the test connection should return 403 error:
This is an authorization error from Ranger (since we have not yet created any policies).
Click Save to commit the changes you made
4. Test Ranger plugin
Attempting to open Nifi UI results in "Access denied" due to insufficient permissions:
Navigate to the ‘Audit’ tab in Ranger UI and notice that the requesting user showed up on Ranger audit. This shows the Ranger Nifi plugin is working
Notice how Ranger is showing details such as below for multiple HDF components:
- what time access attempt occurred
- user/IP who attempted the access
- resource that was attempted to be accessed
- whether access for allowed or denied
Also notice that Nifi now shows up as one of the registered plugins (under ‘Plugins’ tab)
5. Create Ranger uses and policies
To be able to access the Nifi U we will need to create a number of objects. The details below assume you have already setup identity mappings for Nifi (as described in previous article), but you should be able follow similar steps even if you have not.
i) Ranger users for admin and node identities (in a real customer env, you would not be manually creating these: they would be synced over from Active Directory/LDAP)
- nifiadmin@CLOUD.HORTONWORKS.COM
- node1.fqdn@CLOUD.HORTONWORKS.COM
- node2.fqdn@CLOUD.HORTONWORKS.COM
- node3.fqdn@CLOUD.HORTONWORKS.COM
ii) Read policy on /flow for node1-3 identities
iii) Read/write policy on /proxy for node1-3 identities
iv) Read/write policy on /data/* for node1-3 identities (needed to list/delete queue)
v) Read/write policy on /* for nifiadmin identity (needed to make nifiadmin an admin)
More details on what Ranger policies to create can be found here
Option 1: Run script below from Ranger node to create above using Ranger’s REST APIs
export hosts="node1.fqdn node2.fqdn node3.fqdn" #set hostnames of nodes running Nifi export admin="nifiadmin" #set your desired Nifi admin user export realm="CLOUD.HORTONWORKS.COM" #set domain of certificate export cluster="HDF" #set cluster name #download/run script curl -sSL https://gist.github.com/abajwa-hw/2b59db1a850406616d4583f44bad0a78/raw | sudo -E sh
End result:
Option 2: Manually create users and policies
Create local users in Ranger for all requesting users from Ranger UI under Settings > Users/Groups
Assuming you setup identity mapping earlier, create the users appropriately e.g. node-1.fqdn@CLOUD.HORTONWORKS.COM,node-2.fqdn@CLOUD.HORTONWORKS.COM,node-3.fqdn@CLOUD.HORTONWORKS.COM
Alternatively if you do not wish to use node identities, you would enter the long form of the identity as the username (e.g. CN=nifiadmin, OU=CLOUD.HORTONWORKS.COM; CN=node-1.fqdn, OU=CLOUD.HORTONWORKS.COM; CN=node-1.fqdn, OU=CLOUD.HORTONWORKS.COM; CN=node-1.fqdn, OU=CLOUD.HORTONWORKS.COM)
Now create Ranger policies for node identities for each host:
- /flow - read
- /proxy - read/write
- /data/* - read/write
To do this, access Nifi policies in Ranger by navigating to Ranger > Access Manager > Nifi > HDF_nifi
Then click ‘Add New Policy’ to display below form:
Create a new READ policy for node identities on /flow:
Similarly, create a new READ/WRITE policy for node identities on /proxy:
Similarly create a new READ/WRITE policy for node identities on /data/*:
We still need to manually add nifiadmin user to the global policy To do this, click the ‘HDF_nifi’ link highlighted:
Then click Edit icon on “all-nifi-resource” policy:
Under ‘Select User’ add nifiadmin@CLOUD.HORTONWORKS.COM and provide Read/Write access, then Save.
6. Test Nifi acess as nifiadmin user
Whether you created the users/policies via the script or manually, at this point the Nifi policy page should appear as below:
Note that it may take up to 30s after creating the policies in Ranger UI for them to take affect.
How to confirm that the new policies were downloaded by the Nifi Ranger plugin after they we created them?
You can do this by checking the first ‘Export Date’ for nifi service under Audit > Plugins tab in Ranger: when this timestamp shows a time after the changes were made, it means the new policies have been downloaded and should be in effect.
Open Nifi UI via Quicklink and confirm it now opens.
Confirm via Ranger audits that Ranger now allows access
With this you have successfully installed Ranger and configured Nifi to use the Ranger authorizer
Other things to try: Try disabling the policies created one by one, waiting 30s and refreshing Nifi UI to see what breaks.
Tips:
1. To authorize separate users/group access to different parts of a flow, implement multiple process groups and then:
- Grant user/group access to modify the NiFi flow with a policy for /process-groups/<root-group-id> with RW
- Create a separate a policy for /provenance/process-groups/<root-group-id> (with each of the cluster node DNs) for read access
2. Troubleshooting tip: When the Ranger plugin is enabled and you are encountering permission errors trying to login to Nifi or performing a certain action within Nifi, check Ranger audits to check for any 'Denied' requests. In the event that you encounter these, Ranger will tell you exactly what user was trying to access what resource which will help you create the right policy to avoid the issue
What next?
If you haven't already, review what Ranger policies you can create for Nifi here: https://community.hortonworks.com/articles/60842/hdf-20-defining-nifi-policies-in-ranger.html
Next, we will enable kerberos on the cluster and show how users can then login to Nifi without certificates (using AD/KDC credentials):
- Steps for enabling security on HDF using Active directory:
- Steps for enabling security on HDF using KDC:
