Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Unable to connect to Hive using PutHiveQL

Unable to connect to Hive using PutHiveQL

New Contributor

Hi,

I am trying to use PutHiveQL in my Nifi (1.5.0) in HDP 2.6.5.0 and getting a NullPointerException.

Here is what i have configured.

2018-08-16 11:53:08,370 INFO [Timer-Driven Process Thread-8] o.a.c.f.imps.CuratorFrameworkImpl Starting
2018-08-16 11:53:08,372 INFO [Timer-Driven Process Thread-8-EventThread] o.a.c.f.state.ConnectionStateManager State change: CONNECTED
2018-08-16 11:53:08,372 INFO [Timer-Driven Process Thread-8] o.a.hive.jdbc.ZooKeeperHiveClientHelper Selected HiveServer2 instance with uri: hive.server2.authentication=NONE;hive.server2.transport.mode=binary;hive.server2.thrift.sasl.qop=auth;hive.server2.thrift.bind.host=ip-10-0-11-XXX.us-west-1.compute.internal;hive.server2.thrift.port=10000;hive.server2.use.SSL=false
2018-08-16 11:53:08,372 INFO [Curator-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl backgroundOperationsLoop exiting
2018-08-16 11:53:08,373 INFO [Timer-Driven Process Thread-8] org.apache.hive.jdbc.Utils Resolved authority: hive.server2.authentication=NONE;hive.server2.transport.mode=binary;hive.server2.thrift.sasl.qop=auth;hive.server2.thrift.bind.host=ip-10-0-11-XXX.us-west-1.compute.internal;hive.server2.thrift.port=10000;hive.server2.use.SSL=false
2018-08-16 11:53:08,374 INFO [Timer-Driven Process Thread-8] org.apache.hive.jdbc.HiveConnection Will try to open client transport with JDBC Uri: jdbc:hive2://hive.server2.authentication=NONE;hive.server2.transport.mode=binary;hive.server2.thrift.sasl.qop=auth;hive.server2.thrift.bind.host=ip-10-0-11-XXX.us-west-1.compute.internal;hive.server2.thrift.port=10000;hive.server2.use.SSL=false/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2
2018-08-16 11:53:08,374 ERROR [Timer-Driven Process Thread-8] o.apache.nifi.processors.hive.PutHiveQL PutHiveQL[id=4360aed5-0165-1000-0194-15362e7113fe] Failed to execute due to java.lang.NullPointerException: java.lang.NullPointerException
java.lang.NullPointerException: null
	at org.apache.thrift.transport.TSocket.open(TSocket.java:170)
	at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:266)
	at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
	at org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:204)
	at org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:176)
	at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105)
	at org.apache.commons.dbcp.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:38)
	at org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:582)
	at org.apache.commons.dbcp.BasicDataSource.validateConnectionFactory(BasicDataSource.java:1556)
	at org.apache.commons.dbcp.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:1545)
	at org.apache.commons.dbcp.BasicDataSource.createDataSource(BasicDataSource.java:1388)
	at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044)
	at org.apache.nifi.dbcp.hive.HiveConnectionPool.getConnection(HiveConnectionPool.java:312)
	at sun.reflect.GeneratedMethodAccessor425.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:89)
	at com.sun.proxy.$Proxy151.getConnection(Unknown Source)
	at org.apache.nifi.processors.hive.PutHiveQL.lambda$new$1(PutHiveQL.java:191)
	at org.apache.nifi.processor.util.pattern.Put.onTrigger(Put.java:96)
	at org.apache.nifi.processors.hive.PutHiveQL.lambda$onTrigger$6(PutHiveQL.java:274)
	at org.apache.nifi.processor.util.pattern.PartialFunctions.onTrigger(PartialFunctions.java:114)
	at org.apache.nifi.processor.util.pattern.RollbackOnFailure.onTrigger(RollbackOnFailure.java:184)
	at org.apache.nifi.processors.hive.PutHiveQL.onTrigger(PutHiveQL.java:274)
	at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1122)
	at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
	at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
	at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:128)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)





Here is the test Nifi Workflow i am using to test hive connection

My Generate flow file has a simple "Select count(*) from Table_1" Statement.

Here is the Configuration i have used for HiveConnectionPool

Database Connection URL = jdbc:hive2://10.0.1.XXX:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2

Hive Configuration Resources: /etc/hadoop/conf/core-site.xml,/etc/hadoop/conf/hdfs-site.xml,/etc/hive2/conf/hive-site.xml

Database User: No Value set

Password : No Value Set

Thanks

Sateesh


screenshot-from-2018-08-16-12-04-12.png
3 REPLIES 3

Re: Unable to connect to Hive using PutHiveQL

Expert Contributor

You need SelectHiveQL for select statements. PutHiveQl is different.

PutHiveQL Description: Executes a HiveQL DDL/DML command (UPDATE, INSERT, e.g.). The content of an incoming FlowFile is expected to be the HiveQL command to execute. The HiveQL command may use the ? to escape parameters. In this case, the parameters to use must exist as FlowFile attributes with the naming convention hiveql.args.N.type and hiveql.args.N.value, where N is a positive integer. The hiveql.args.N.type is expected to be a number indicating the JDBC Type. The content of the FlowFile is expected to be in UTF-8 format.

If this answer is helpful please choose Accept.

Re: Unable to connect to Hive using PutHiveQL

New Contributor

Hi,

Thanks for quick response. Yes i changed the processor to SelectHiveQL and it is still throwing the NullPointerException.

Yes i selected the wrong processor. But how ever the error was due to not able to connect to Hive Server.

2018-08-16 13:53:48,941 ERROR [Timer-Driven Process Thread-9] o.a.nifi.processors.hive.SelectHiveQL SelectHiveQL[id=43dbea09-0165-1000-b3c3-ee2edf7cd302] org.apache.nifi.processors.hive.SelectHiveQL$$Lambda$534/1002654252@5d81e50c failed to process due to java.lang.NullPointerException; rolling back session: {}
java.lang.NullPointerException: null
	at org.apache.thrift.transport.TSocket.open(TSocket.java:170)
	at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:266)


Re: Unable to connect to Hive using PutHiveQL

Expert Contributor
@sateesh kommineni glad you were able to get it sorted out!