question Re: issue with "Puthivestreaming" Processor version 1.4 - Using a stand alone Version of Nifi. in Archives of Support Questions (Read Only)

issue with "Puthivestreaming" Processor version 1.4 - Using a stand alone Version of Nifi.

abhinav_joshi — Mon, 30 Apr 2018 13:27:59 GMT

Hi All ,

Need some help here with the "PutHiveStreaming" Processor. I have followed all the prerequisites to set up the data flow onto "PutHiveStreaming" processor however I continue to get the following error :-

Hive Streaming connect/write error, flow file will be penalized and routed to retry. org.apache.nifi.util.hive.HiveWriter$ConnectFailure: Failed connecting to EndPoint

I have seen this is a fairly common error across the Nifi Forums and I guess @Matt Burgess had posted a NAR file to fix this issue however the NAR file was of version 1.0 and in my case wont work as I am using version 1.4 which is a stand alone open source version of Apache Nifi ( not HDP ). Our requirement is to extract data from database and stream it into a partitioned table on Hive. We have used the streaming option to write onto HDFS using QueryDatabaseTable -> ConvertAvrotoOrc --> putHDFS however this would just land files onto HDFS , we need to directly insert data onto the partitioned table and make data available to the end users in real time.

@Matt Burgess

@Shu

Do you have any idea on how to get this processor working by any chance

Re: issue with "Puthivestreaming" Processor version 1.4 - Using a stand alone Version of Nifi.

abhinav_joshi — Tue, 01 May 2018 09:37:17 GMT

Ok , we just got this fixed , It seemed to be some issue with the thrift server on Hive. all good 🙂

Re: issue with "Puthivestreaming" Processor version 1.4 - Using a stand alone Version of Nifi.

srinatha_ananth — Sat, 18 Aug 2018 03:26:11 GMT

Abhinav,

I am seeing the same error, How did you fix this?

Re: issue with "Puthivestreaming" Processor version 1.4 - Using a stand alone Version of Nifi.

abhinav_joshi — Tue, 18 Sep 2018 09:42:37 GMT

Hi @Srinatha Anantharaman - we actually got in touch with our Hadoop admin and opened all firewalls for the thrift metastore server after which it worked, Hope this helps