Kerberized HDF 2.1.1 integration with kerberized HDP 2.5.3 cluster

[#1]

This article helps with configuration of HDF processors to integrate with HDP components

Integration for puthivestreaming processor:

Pre-requisites: In HDP ambari, enable the below properties for hive.

hive.txn.manager = org.apache.hadoop.hive.ql.lockmgr.DbTxnManager 
hive.compactor.initiator.on = true
hive.compactor.worker.threads > 0

BUG: HiveStreaming processor not picking the updated hive-meta store principal from hive-site. xml

Resolution:

1. Copy hive-site.xml, core-site.xml, hdfs-site.xml to the conf directory of NiFi
2. Clear the Hive Configuration Resources property
3. Create an ExecuteStream processor on the canvas scheduled to run as often as the ticket needs to be refreshed (every hour should do for most setups) with the following groovy script, replacing [email protected] with your principal and /etc/nifi.headless.keytab with your keytab

Steps: Create /etc/hdp/tgt.groovy and copy the code below into the file on all nifi nodes.

Note: Change the kerberos principal and keytab location as necessary

import org.apache.nifi.nar.NarClassLoader
import org.apache.nifi.nar.NarClassLoaders NarClassLoaders.instance.extensionClassLoaders.each { c ->
if (c instanceof NarClassLoader && c.workingDirectory. absolutePath.contains('nifi-hive')) {
   def originalClassloader = Thread.currentThread().
getContextClassLoader();
Thread.currentThread().setContextClassLoader(c); try {
def configClass = c.loadClass('org.apache.hadoop.conf. Configuration', true)
def hiveConfigurator = c.loadClass('org.apache.nifi.util. hive.HiveConfigurator', true).newInstance();
     def config = hiveConfigurator.getConfigurationFromFiles('')
     hiveConfigurator.preload(config)
     c.loadClass('org.apache.hadoop.security.UserGroupInformation'
, true).getMethod('setConfiguration', configClass).invoke(null, config)
c.loadClass('org.apache.hadoop.security.UserGroupInformation' , true).getMethod('loginUserFromKeytab', String.class, String.clas s).invoke(null, '[email protected]', '/etc/security/keytabs/nifi. headless.keytab')
     log.info('Successfully logged in')
session.transfer(session.create(), REL_SUCCESS) } catch (Exception e) {
     log.error('Unable to login with keytab', e)
session.transfer(session.create(), REL_FAILURE) } finally {
     Thread.currentThread().setContextClassLoader
(originalClassloader);
} }
}

Do the following on all the nifi nodes,

chown nifi:nifi /etc/hdp/tgt.groovy; chmod +x /etc/hdp/tgt.groovy;
cp /etc/hdp/hive-site.xml /etc/nifi/conf/; cp /etc/hdp/core-site.xml /etc/nifi/conf/; cp /etc/hdp /hdfs-site.xml /etc/nifi/conf/;
chown nifi:nifi /etc/nifi/conf/hive-site.xml /etc/nifi/conf/core-site.xml /etc/nifi/conf/hdfs-site. xml; 

Integration for hbase processor:

Add hostname and IP of all HDP nodes to /etc/hosts file in all nifi nodes

<Hostname1>	<IpAddress1>
<Hostname2>	<IpAddress2>

Integration for kafka processor:

Create a file under the location "/etc/hdp/zookeeper-jaas.conf" and copy the code below

Note: Change the kerberos principal and keytab location as necessary

Client { com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true keyTab="/etc/security/keytabs/nifi.headless.keytab" storeKey=true
useTicketCache=false
principal="[email protected]";
};
KafkaClient { com.sun.security.auth.module.Krb5LoginModule required useTicketCache=true
renewTicket=true
serviceName="kafka"
useKeyTab=true keyTab="/etc/security/keytabs/nifi.headless.keytab" principal="[email protected]";
};

Add below configuration to advanced nifi-bootstrap.conf in HDP ambari and restart the nifi service.

java.arg.20=-Djava.security.auth.login.config=/etc/hdp/zookeeper-jaas.conf