Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Problems with PutHiveStreaming processor in NiFi-1.1.0

avatar
Expert Contributor

Hello,

I am currently trying to test a flow using the PutHiveStreaming processor in NiFi-1.1.0. In my flow I am getting a csv file, inferring the Avro schema, converting the file from cvs to avro and from there sending it into the PutHiveStreaming processor. However I keep getting a "HiveWriter$ConnectFailure: Failed connecting to EndPoint" error in NiFi. I am using HDP version 2.5.3 for Hive. I have seen that there were some issues with NiFi and HDP 2.5 previously as mentioned here:https://issues.apache.org/jira/browse/NIFI-2828, but to my understanding they were fixed on version NiFi-1.1.0 which is the one I am using. Does anyone have any ideas of what could be causing this error? Any insight on this would be greatly appreciated. The full NiFi error log is the following:

ERROR [Timer-Driven Process Thread-18] o.a.n.processors.hive.PutHiveStreaming 
org.apache.nifi.util.hive.HiveWriter$ConnectFailure: Failed connecting to EndPoint {metaStoreUri='thrift://<hive-metastore-node>:9083', database='analytics', table='agent', partitionVals=[] }
	at org.apache.nifi.util.hive.HiveWriter.<init>(HiveWriter.java:80) ~[nifi-hive-processors-1.1.0.jar:1.1.0]
	at org.apache.nifi.util.hive.HiveUtils.makeHiveWriter(HiveUtils.java:45) ~[nifi-hive-processors-1.1.0.jar:1.1.0]
	at org.apache.nifi.processors.hive.PutHiveStreaming.makeHiveWriter(PutHiveStreaming.java:829) ~[nifi-hive-processors-1.1.0.jar:1.1.0]
	at org.apache.nifi.processors.hive.PutHiveStreaming.getOrCreateWriter(PutHiveStreaming.java:740) ~[nifi-hive-processors-1.1.0.jar:1.1.0]
	at org.apache.nifi.processors.hive.PutHiveStreaming.lambda$onTrigger$7(PutHiveStreaming.java:464) ~[nifi-hive-processors-1.1.0.jar:1.1.0]
	at org.apache.nifi.processors.hive.PutHiveStreaming$$Lambda$396/393641412.process(Unknown Source) ~[na:na]
	at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2082) ~[na:na]
	at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2053) ~[na:na]
	at org.apache.nifi.processors.hive.PutHiveStreaming.onTrigger(PutHiveStreaming.java:391) ~[nifi-hive-processors-1.1.0.jar:1.1.0]
	at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) [nifi-api-1.1.0.jar:1.1.0]
	at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1099) [nifi-framework-core-1.1.0.jar:1.1.0]
	at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136) [nifi-framework-core-1.1.0.jar:1.1.0]
	at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) [nifi-framework-core-1.1.0.jar:1.1.0]
	at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132) [nifi-framework-core-1.1.0.jar:1.1.0]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_45]
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_45]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_45]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_45]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_45]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_45]
	at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
Caused by: org.apache.nifi.util.hive.HiveWriter$TxnBatchFailure: Failed acquiring Transaction Batch from EndPoint: {metaStoreUri='thrift://<hive-metastore-node>:9083', database='analytics', table='agent', partitionVals=[] }
	at org.apache.nifi.util.hive.HiveWriter.nextTxnBatch(HiveWriter.java:255) ~[nifi-hive-processors-1.1.0.jar:1.1.0]
	at org.apache.nifi.util.hive.HiveWriter.<init>(HiveWriter.java:74) ~[nifi-hive-processors-1.1.0.jar:1.1.0]
	... 20 common frames omitted
Caused by: org.apache.hive.hcatalog.streaming.TransactionError: Unable to acquire lock on {metaStoreUri='thrift://<hive-metastore-node>:9083', database='analytics', table='agent', partitionVals=[] }
	at org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.beginNextTransactionImpl(HiveEndPoint.java:578) ~[hive-hcatalog-streaming-1.2.1.jar:1.2.1]
	at org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.beginNextTransaction(HiveEndPoint.java:547) ~[hive-hcatalog-streaming-1.2.1.jar:1.2.1]
	at org.apache.nifi.util.hive.HiveWriter.nextTxnBatch(HiveWriter.java:252) ~[nifi-hive-processors-1.1.0.jar:1.1.0]
	... 21 common frames omitted
Caused by: org.apache.thrift.transport.TTransportException: null
	at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132) ~[libthrift-0.9.2.jar:0.9.2]
	at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) ~[libthrift-0.9.2.jar:0.9.2]
	at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429) ~[libthrift-0.9.2.jar:0.9.2]
	at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318) ~[libthrift-0.9.2.jar:0.9.2]
	at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219) ~[libthrift-0.9.2.jar:0.9.2]
	at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69) ~[libthrift-0.9.2.jar:0.9.2]
	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_lock(ThriftHiveMetastore.java:3906) ~[hive-metastore-1.2.1.jar:1.2.1]
	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.lock(ThriftHiveMetastore.java:3893) ~[hive-metastore-1.2.1.jar:1.2.1]
	at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.lock(HiveMetaStoreClient.java:1863) ~[hive-metastore-1.2.1.jar:1.2.1]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_45]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_45]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_45]
	at java.lang.reflect.Method.invoke(Method.java:497) ~[na:1.8.0_45]
	at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:152) ~[hive-metastore-1.2.1.jar:1.2.1]
	at com.sun.proxy.$Proxy246.lock(Unknown Source) ~[na:na]
	at org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.beginNextTransactionImpl(HiveEndPoint.java:573) ~[hive-hcatalog-streaming-1.2.1.jar:1.2.1]
	... 23 common frames omitted


1 ACCEPTED SOLUTION

avatar
Master Guru

Besides the issue in NIFI-2828, there is also a Hive version incompatibility between Apache NiFi and HDP 2.5. Apache NiFi is built with Apache Hive 1.2 and its Apache Hadoop dependencies. However HDP 2.5 has a "newer" version of Hive, based on 1.2 but also with many additions, including a change to the Thrift messages sent between client and server.

In order to fix this, I recommend using a Hortonworks Data Flow (HDF) version (2.1.0.0 or above) that is based on NiFi 1.1.0; HDF is built using HDP dependencies/versions for Hive and its Hadoop dependencies. Alternatively you can build your own NiFi from source, using the "hortonworks" profile and overriding the hive.version and hive.hadoop.version properties:

mvn clean install -Phortonworks -Dhive.version=1.2.1000.2.5.0.0-1245 -Dhive.hadoop.version=2.7.3.2.5.0.0-1245 -DskipTests

View solution in original post

6 REPLIES 6

avatar

@Adda Fuentes Looks like similar issue , can you please check what nar file you have in <nifi_home>/lib folder? e.g. nifi-hive-nar-1.1.0.2.1.2.0-10.nar

avatar
Expert Contributor

@milind pandit the hive nar file my version of nifi has is nifi-hive-nar-1.1.0.nar

avatar
Master Guru

Besides the issue in NIFI-2828, there is also a Hive version incompatibility between Apache NiFi and HDP 2.5. Apache NiFi is built with Apache Hive 1.2 and its Apache Hadoop dependencies. However HDP 2.5 has a "newer" version of Hive, based on 1.2 but also with many additions, including a change to the Thrift messages sent between client and server.

In order to fix this, I recommend using a Hortonworks Data Flow (HDF) version (2.1.0.0 or above) that is based on NiFi 1.1.0; HDF is built using HDP dependencies/versions for Hive and its Hadoop dependencies. Alternatively you can build your own NiFi from source, using the "hortonworks" profile and overriding the hive.version and hive.hadoop.version properties:

mvn clean install -Phortonworks -Dhive.version=1.2.1000.2.5.0.0-1245 -Dhive.hadoop.version=2.7.3.2.5.0.0-1245 -DskipTests

avatar
Expert Contributor

@Matt Burgess thanks for the answer and the help. I wanted to ask if it is possible to run the NiFi 1.1.0 version of HDF instead of the Apache one independently without using the full HDF?

avatar
Master Guru

You can get the NiFi-only version of HDF at https://hortonworks.com/downloads/#dataflow

avatar
Expert Contributor

thanks a lot for the help! I will try this.