Problems with PutHiveStreaming processor in NiFi-1.1.0

[#1]

Hello,

I am currently trying to test a flow using the PutHiveStreaming processor in NiFi-1.1.0. In my flow I am getting a csv file, inferring the Avro schema, converting the file from cvs to avro and from there sending it into the PutHiveStreaming processor. However I keep getting a "HiveWriter$ConnectFailure: Failed connecting to EndPoint" error in NiFi. I am using HDP version 2.5.3 for Hive. I have seen that there were some issues with NiFi and HDP 2.5 previously as mentioned here:https://issues.apache.org/jira/browse/NIFI-2828, but to my understanding they were fixed on version NiFi-1.1.0 which is the one I am using. Does anyone have any ideas of what could be causing this error? Any insight on this would be greatly appreciated. The full NiFi error log is the following:

ERROR [Timer-Driven Process Thread-18] o.a.n.processors.hive.PutHiveStreaming 
org.apache.nifi.util.hive.HiveWriter$ConnectFailure: Failed connecting to EndPoint {metaStoreUri='thrift://<hive-metastore-node>:9083', database='analytics', table='agent', partitionVals=[] }
	at org.apache.nifi.util.hive.HiveWriter.<init>(HiveWriter.java:80) ~[nifi-hive-processors-1.1.0.jar:1.1.0]
	at org.apache.nifi.util.hive.HiveUtils.makeHiveWriter(HiveUtils.java:45) ~[nifi-hive-processors-1.1.0.jar:1.1.0]
	at org.apache.nifi.processors.hive.PutHiveStreaming.makeHiveWriter(PutHiveStreaming.java:829) ~[nifi-hive-processors-1.1.0.jar:1.1.0]
	at org.apache.nifi.processors.hive.PutHiveStreaming.getOrCreateWriter(PutHiveStreaming.java:740) ~[nifi-hive-processors-1.1.0.jar:1.1.0]
	at org.apache.nifi.processors.hive.PutHiveStreaming.lambda$onTrigger$7(PutHiveStreaming.java:464) ~[nifi-hive-processors-1.1.0.jar:1.1.0]
	at org.apache.nifi.processors.hive.PutHiveStreaming$$Lambda$396/393641412.process(Unknown Source) ~[na:na]
	at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2082) ~[na:na]
	at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2053) ~[na:na]
	at org.apache.nifi.processors.hive.PutHiveStreaming.onTrigger(PutHiveStreaming.java:391) ~[nifi-hive-processors-1.1.0.jar:1.1.0]
	at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) [nifi-api-1.1.0.jar:1.1.0]
	at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1099) [nifi-framework-core-1.1.0.jar:1.1.0]
	at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136) [nifi-framework-core-1.1.0.jar:1.1.0]
	at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) [nifi-framework-core-1.1.0.jar:1.1.0]
	at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132) [nifi-framework-core-1.1.0.jar:1.1.0]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_45]
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_45]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_45]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_45]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_45]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_45]
	at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
Caused by: org.apache.nifi.util.hive.HiveWriter$TxnBatchFailure: Failed acquiring Transaction Batch from EndPoint: {metaStoreUri='thrift://<hive-metastore-node>:9083', database='analytics', table='agent', partitionVals=[] }
	at org.apache.nifi.util.hive.HiveWriter.nextTxnBatch(HiveWriter.java:255) ~[nifi-hive-processors-1.1.0.jar:1.1.0]
	at org.apache.nifi.util.hive.HiveWriter.<init>(HiveWriter.java:74) ~[nifi-hive-processors-1.1.0.jar:1.1.0]
	... 20 common frames omitted
Caused by: org.apache.hive.hcatalog.streaming.TransactionError: Unable to acquire lock on {metaStoreUri='thrift://<hive-metastore-node>:9083', database='analytics', table='agent', partitionVals=[] }
	at org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.beginNextTransactionImpl(HiveEndPoint.java:578) ~[hive-hcatalog-streaming-1.2.1.jar:1.2.1]
	at org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.beginNextTransaction(HiveEndPoint.java:547) ~[hive-hcatalog-streaming-1.2.1.jar:1.2.1]
	at org.apache.nifi.util.hive.HiveWriter.nextTxnBatch(HiveWriter.java:252) ~[nifi-hive-processors-1.1.0.jar:1.1.0]
	... 21 common frames omitted
Caused by: org.apache.thrift.transport.TTransportException: null
	at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132) ~[libthrift-0.9.2.jar:0.9.2]
	at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) ~[libthrift-0.9.2.jar:0.9.2]
	at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429) ~[libthrift-0.9.2.jar:0.9.2]
	at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318) ~[libthrift-0.9.2.jar:0.9.2]
	at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219) ~[libthrift-0.9.2.jar:0.9.2]
	at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69) ~[libthrift-0.9.2.jar:0.9.2]
	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_lock(ThriftHiveMetastore.java:3906) ~[hive-metastore-1.2.1.jar:1.2.1]
	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.lock(ThriftHiveMetastore.java:3893) ~[hive-metastore-1.2.1.jar:1.2.1]
	at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.lock(HiveMetaStoreClient.java:1863) ~[hive-metastore-1.2.1.jar:1.2.1]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_45]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_45]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_45]
	at java.lang.reflect.Method.invoke(Method.java:497) ~[na:1.8.0_45]
	at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:152) ~[hive-metastore-1.2.1.jar:1.2.1]
	at com.sun.proxy.$Proxy246.lock(Unknown Source) ~[na:na]
	at org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.beginNextTransactionImpl(HiveEndPoint.java:573) ~[hive-hcatalog-streaming-1.2.1.jar:1.2.1]
	... 23 common frames omitted

[#2]

@Adda Fuentes Looks like similar issue , can you please check what nar file you have in <nifi_home>/lib folder? e.g. nifi-hive-nar-1.1.0.2.1.2.0-10.nar

[#3]

@milind pandit the hive nar file my version of nifi has is nifi-hive-nar-1.1.0.nar

[#4 · Accepted solution]

Besides the issue in NIFI-2828, there is also a Hive version incompatibility between Apache NiFi and HDP 2.5. Apache NiFi is built with Apache Hive 1.2 and its Apache Hadoop dependencies. However HDP 2.5 has a "newer" version of Hive, based on 1.2 but also with many additions, including a change to the Thrift messages sent between client and server.

In order to fix this, I recommend using a Hortonworks Data Flow (HDF) version (2.1.0.0 or above) that is based on NiFi 1.1.0; HDF is built using HDP dependencies/versions for Hive and its Hadoop dependencies. Alternatively you can build your own NiFi from source, using the "hortonworks" profile and overriding the hive.version and hive.hadoop.version properties:

mvn clean install -Phortonworks -Dhive.version=1.2.1000.2.5.0.0-1245 -Dhive.hadoop.version=2.7.3.2.5.0.0-1245 -DskipTests

[#5]

@Matt Burgess thanks for the answer and the help. I wanted to ask if it is possible to run the NiFi 1.1.0 version of HDF instead of the Apache one independently without using the full HDF?

[#6]

You can get the NiFi-only version of HDF at https://hortonworks.com/downloads/#dataflow

[#7]

thanks a lot for the help! I will try this.