Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

NiFi processors cannot connect to Zookeeper

avatar
New Contributor

I am integrating Apache NiFi 1.9.2 (secure cluster) with HDP 3.1.4. HDP contains Zookeeper 3.4.6 with SASL auth (Kerberos). NiFi nodes successfully connect to this Zookeeper, sync flow and log heartbeats.
Meanwhile, NiFi processors using Zookeeper are not able to connect. GenerateTableFetch throws:

 

org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (Could not open client transport for any of the Server URI's in ZooKeeper: Unable to read HiveServer2 uri from ZooKeeper)

 

 ListHDFS fails attempting to write its state to ZooKeeper:

 

o.apache.nifi.processors.hadoop.ListHDFS ListHDFS  Failed to retrieve timestamp of last listing from the State Manager. Will not perform listing until this is accomplished.

 

But GetHDFS (which doesn't communicate with zk) works OK.
Enable HiveConnector works without errors.

Thank you in before for any ideas and support.

1 ACCEPTED SOLUTION

avatar
New Contributor

The solution is to use Hive3ConnectionPool from nifi-hive3-nar library
Hive 1.* processors, e.g., GenerateTableFetch and SelectHiveQL, work with Hive3 connector.

View solution in original post

2 REPLIES 2

avatar
New Contributor

I was missing zk connect string at conf/state-management.xml. Now it looks like this:

 

...
<cluster-provider>
    <id>zk-provider</i>
    <class>
        org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider
    </class>
     <property name="Connect String">my zk connect string</property>
     <property name="Root Node">/nifi</property>
     <property name="Session Timeout">30 seconds</property>
     <property name="Access Control">Open</property>
</cluster-provider> 

 

 

ListHDFS works OK now. But GenerateTableFetch still throws:

2021-03-26 18:06:04,432 INFO [Timer-Driven Process Thread-2] org.apache.hive.jdbc.HiveConnection Will try to open client transport with JDBC Uri: jdbc:hive2://hive.server2.instance.uri=hiveserver1_hostname:10000;hive.server2.authentication=KERBEROS;hive.server2.transport.mode=binary;hive.server2.thrift.sasl.qop=auth;hive.server2.thrift.bind.host=hiveserver1_hostname;hive.server2.thrift.port=10000;hive.server2.use.SSL=false;hive.server2.authentication.kerberos.principal=hive/_HOST@DOMAIN/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2
2021-03-26 18:06:04,432 INFO [Timer-Driven Process Thread-2] org.apache.hive.jdbc.HiveConnection Could not open client transport with JDBC Uri: jdbc:hive2://hive.server2.instance.uri=hiveserver1_hostname:10000;hive.server2.authentication=KERBEROS;hive.server2.transport.mode=binary;hive.server2.thrift.sasl.qop=auth;hive.server2.thrift.bind.host=hiveserver1_hostname;hive.server2.thrift.port=10000;hive.server2.use.SSL=false;hive.server2.authentication.kerberos.principal=hive/_HOST@DOMAIN/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2
2021-03-26 18:06:04,432 INFO [Timer-Driven Process Thread-2] o.a.c.f.imps.CuratorFrameworkImpl Starting
2021-03-26 18:06:04,441 INFO [Timer-Driven Process Thread-2-EventThread] o.a.c.f.state.ConnectionStateManager State change: CONNECTED
2021-03-26 18:06:04,446 ERROR [Timer-Driven Process Thread-2] o.a.nifi.dbcp.hive.HiveConnectionPool HiveConnectionPool[id=4cfbf700-6449-3d37-9ab1-263a7528f352] Error getting Hive connection: org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (Could not open client transport for any of the Server URI's in ZooKeeper: Unable to read HiveServer2 uri from ZooKeeper)
org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (Could not open client transport for any of the Server URI's in ZooKeeper: Unable to read HiveServer2 uri from ZooKeeper)

Actually, it read Hive server2 uri from zk, but couldn't connect.

avatar
New Contributor

The solution is to use Hive3ConnectionPool from nifi-hive3-nar library
Hive 1.* processors, e.g., GenerateTableFetch and SelectHiveQL, work with Hive3 connector.