Our Community is getting an upgrade! To get everything ready for the relaunch, we’ll be placing the site in read-only mode starting September 21st.
We really appreciate your understanding while we get things set up behind the scenes. Catch up on all the exciting details about the move here.
Need help or have questions? Drop us a line at [email protected]

Support Questions

Find answers, ask questions, and share your expertise
Announcements
Share your experience with Cloudera on G2 and get a $25 Amazon Gift card.
Hi, I'm CLEO! Something exciting is coming to the Community. Stay Tuned!

NiFi processors cannot connect to Zookeeper

avatar
New Contributor

I am integrating Apache NiFi 1.9.2 (secure cluster) with HDP 3.1.4. HDP contains Zookeeper 3.4.6 with SASL auth (Kerberos). NiFi nodes successfully connect to this Zookeeper, sync flow and log heartbeats.
Meanwhile, NiFi processors using Zookeeper are not able to connect. GenerateTableFetch throws:

 

org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (Could not open client transport for any of the Server URI's in ZooKeeper: Unable to read HiveServer2 uri from ZooKeeper)

 

 ListHDFS fails attempting to write its state to ZooKeeper:

 

o.apache.nifi.processors.hadoop.ListHDFS ListHDFS  Failed to retrieve timestamp of last listing from the State Manager. Will not perform listing until this is accomplished.

 

But GetHDFS (which doesn't communicate with zk) works OK.
Enable HiveConnector works without errors.

Thank you in before for any ideas and support.

1 ACCEPTED SOLUTION

avatar
New Contributor

The solution is to use Hive3ConnectionPool from nifi-hive3-nar library
Hive 1.* processors, e.g., GenerateTableFetch and SelectHiveQL, work with Hive3 connector.

View solution in original post

2 REPLIES 2

avatar
New Contributor

I was missing zk connect string at conf/state-management.xml. Now it looks like this:

 

...
<cluster-provider>
    <id>zk-provider</i>
    <class>
        org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider
    </class>
     <property name="Connect String">my zk connect string</property>
     <property name="Root Node">/nifi</property>
     <property name="Session Timeout">30 seconds</property>
     <property name="Access Control">Open</property>
</cluster-provider> 

 

 

ListHDFS works OK now. But GenerateTableFetch still throws:

2021-03-26 18:06:04,432 INFO [Timer-Driven Process Thread-2] org.apache.hive.jdbc.HiveConnection Will try to open client transport with JDBC Uri: jdbc:hive2://hive.server2.instance.uri=hiveserver1_hostname:10000;hive.server2.authentication=KERBEROS;hive.server2.transport.mode=binary;hive.server2.thrift.sasl.qop=auth;hive.server2.thrift.bind.host=hiveserver1_hostname;hive.server2.thrift.port=10000;hive.server2.use.SSL=false;hive.server2.authentication.kerberos.principal=hive/_HOST@DOMAIN/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2
2021-03-26 18:06:04,432 INFO [Timer-Driven Process Thread-2] org.apache.hive.jdbc.HiveConnection Could not open client transport with JDBC Uri: jdbc:hive2://hive.server2.instance.uri=hiveserver1_hostname:10000;hive.server2.authentication=KERBEROS;hive.server2.transport.mode=binary;hive.server2.thrift.sasl.qop=auth;hive.server2.thrift.bind.host=hiveserver1_hostname;hive.server2.thrift.port=10000;hive.server2.use.SSL=false;hive.server2.authentication.kerberos.principal=hive/_HOST@DOMAIN/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2
2021-03-26 18:06:04,432 INFO [Timer-Driven Process Thread-2] o.a.c.f.imps.CuratorFrameworkImpl Starting
2021-03-26 18:06:04,441 INFO [Timer-Driven Process Thread-2-EventThread] o.a.c.f.state.ConnectionStateManager State change: CONNECTED
2021-03-26 18:06:04,446 ERROR [Timer-Driven Process Thread-2] o.a.nifi.dbcp.hive.HiveConnectionPool HiveConnectionPool[id=4cfbf700-6449-3d37-9ab1-263a7528f352] Error getting Hive connection: org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (Could not open client transport for any of the Server URI's in ZooKeeper: Unable to read HiveServer2 uri from ZooKeeper)
org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (Could not open client transport for any of the Server URI's in ZooKeeper: Unable to read HiveServer2 uri from ZooKeeper)

Actually, it read Hive server2 uri from zk, but couldn't connect.

avatar
New Contributor

The solution is to use Hive3ConnectionPool from nifi-hive3-nar library
Hive 1.* processors, e.g., GenerateTableFetch and SelectHiveQL, work with Hive3 connector.