Unable to connect to Hive using PutHiveQL

[#1]

Hi,

I am trying to use PutHiveQL in my Nifi (1.5.0) in HDP 2.6.5.0 and getting a NullPointerException.

Here is what i have configured.

2018-08-16 11:53:08,370 INFO [Timer-Driven Process Thread-8] o.a.c.f.imps.CuratorFrameworkImpl Starting
2018-08-16 11:53:08,372 INFO [Timer-Driven Process Thread-8-EventThread] o.a.c.f.state.ConnectionStateManager State change: CONNECTED
2018-08-16 11:53:08,372 INFO [Timer-Driven Process Thread-8] o.a.hive.jdbc.ZooKeeperHiveClientHelper Selected HiveServer2 instance with uri: hive.server2.authentication=NONE;hive.server2.transport.mode=binary;hive.server2.thrift.sasl.qop=auth;hive.server2.thrift.bind.host=ip-10-0-11-XXX.us-west-1.compute.internal;hive.server2.thrift.port=10000;hive.server2.use.SSL=false
2018-08-16 11:53:08,372 INFO [Curator-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl backgroundOperationsLoop exiting
2018-08-16 11:53:08,373 INFO [Timer-Driven Process Thread-8] org.apache.hive.jdbc.Utils Resolved authority: hive.server2.authentication=NONE;hive.server2.transport.mode=binary;hive.server2.thrift.sasl.qop=auth;hive.server2.thrift.bind.host=ip-10-0-11-XXX.us-west-1.compute.internal;hive.server2.thrift.port=10000;hive.server2.use.SSL=false
2018-08-16 11:53:08,374 INFO [Timer-Driven Process Thread-8] org.apache.hive.jdbc.HiveConnection Will try to open client transport with JDBC Uri: jdbc:hive2://hive.server2.authentication=NONE;hive.server2.transport.mode=binary;hive.server2.thrift.sasl.qop=auth;hive.server2.thrift.bind.host=ip-10-0-11-XXX.us-west-1.compute.internal;hive.server2.thrift.port=10000;hive.server2.use.SSL=false/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2
2018-08-16 11:53:08,374 ERROR [Timer-Driven Process Thread-8] o.apache.nifi.processors.hive.PutHiveQL PutHiveQL[id=4360aed5-0165-1000-0194-15362e7113fe] Failed to execute due to java.lang.NullPointerException: java.lang.NullPointerException
java.lang.NullPointerException: null
	at org.apache.thrift.transport.TSocket.open(TSocket.java:170)
	at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:266)
	at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
	at org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:204)
	at org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:176)
	at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105)
	at org.apache.commons.dbcp.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:38)
	at org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:582)
	at org.apache.commons.dbcp.BasicDataSource.validateConnectionFactory(BasicDataSource.java:1556)
	at org.apache.commons.dbcp.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:1545)
	at org.apache.commons.dbcp.BasicDataSource.createDataSource(BasicDataSource.java:1388)
	at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044)
	at org.apache.nifi.dbcp.hive.HiveConnectionPool.getConnection(HiveConnectionPool.java:312)
	at sun.reflect.GeneratedMethodAccessor425.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:89)
	at com.sun.proxy.$Proxy151.getConnection(Unknown Source)
	at org.apache.nifi.processors.hive.PutHiveQL.lambda$new$1(PutHiveQL.java:191)
	at org.apache.nifi.processor.util.pattern.Put.onTrigger(Put.java:96)
	at org.apache.nifi.processors.hive.PutHiveQL.lambda$onTrigger$6(PutHiveQL.java:274)
	at org.apache.nifi.processor.util.pattern.PartialFunctions.onTrigger(PartialFunctions.java:114)
	at org.apache.nifi.processor.util.pattern.RollbackOnFailure.onTrigger(RollbackOnFailure.java:184)
	at org.apache.nifi.processors.hive.PutHiveQL.onTrigger(PutHiveQL.java:274)
	at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1122)
	at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
	at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
	at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:128)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

Here is the test Nifi Workflow i am using to test hive connection

My Generate flow file has a simple "Select count(*) from Table_1" Statement.

Here is the Configuration i have used for HiveConnectionPool

Database Connection URL = jdbc:hive2://10.0.1.XXX:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2

Hive Configuration Resources: /etc/hadoop/conf/core-site.xml,/etc/hadoop/conf/hdfs-site.xml,/etc/hive2/conf/hive-site.xml

Database User: No Value set

Password : No Value Set

Thanks

Sateesh

[#2]

You need SelectHiveQL for select statements. PutHiveQl is different.

PutHiveQL Description: Executes a HiveQL DDL/DML command (UPDATE, INSERT, e.g.). The content of an incoming FlowFile is expected to be the HiveQL command to execute. The HiveQL command may use the ? to escape parameters. In this case, the parameters to use must exist as FlowFile attributes with the naming convention hiveql.args.N.type and hiveql.args.N.value, where N is a positive integer. The hiveql.args.N.type is expected to be a number indicating the JDBC Type. The content of the FlowFile is expected to be in UTF-8 format.

If this answer is helpful please choose Accept.

[#3]

Hi,

Thanks for quick response. Yes i changed the processor to SelectHiveQL and it is still throwing the NullPointerException.

Yes i selected the wrong processor. But how ever the error was due to not able to connect to Hive Server.

2018-08-16 13:53:48,941 ERROR [Timer-Driven Process Thread-9] o.a.nifi.processors.hive.SelectHiveQL SelectHiveQL[id=43dbea09-0165-1000-b3c3-ee2edf7cd302] org.apache.nifi.processors.hive.SelectHiveQL$$Lambda$534/1002654252@5d81e50c failed to process due to java.lang.NullPointerException; rolling back session: {}
java.lang.NullPointerException: null
	at org.apache.thrift.transport.TSocket.open(TSocket.java:170)
	at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:266)

[#4]

@sateesh kommineni glad you were able to get it sorted out!