question Re: NiFi processor putHiveStreaming exception. in Archives of Support Questions (Read Only)

NiFi processor putHiveStreaming exception.

mpandit — Thu, 24 Nov 2016 06:01:35 GMT

I am getting following exception while invoking the putHiveStreaming processor.

016-11-23 04:16:43,024 ERROR [Timer-Driven Process Thread-4] o.a.n.processors.hive.PutHiveStreaming PutHiveStreaming[id=8f50a88f-0158-1000-4941-4b83f9b57917] Hive Streaming connect/write error, flow file will be penalized and routed to retry 2016-11-23 04:16:43,025 ERROR [Timer-Driven Process Thread-4] o.a.n.processors.hive.PutHiveStreaming org.apache.nifi.util.hive.HiveWriter$ConnectFailure: Failed connecting to EndPoint {metaStoreUri='thrift://sandbox.hortonworks.com:9083', database='default', table='store_order', partitionVals=[] } at org.apache.nifi.util.hive.HiveWriter.<init>(HiveWriter.java:80) ~[nifi-hive-processors-1.0.0.2.0.0.0-579.jar:1.0.0.2.0.0.0-579] at org.apache.nifi.util.hive.HiveUtils.makeHiveWriter(HiveUtils.java:45) ~[nifi-hive-processors-1.0.0.2.0.0.0-579.jar:1.0.0.2.0.0.0-579] at org.apache.nifi.processors.hive.PutHiveStreaming.makeHiveWriter(PutHiveStreaming.java:827) ~[nifi-hive-processors-1.0.0.2.0.0.0-579.jar:1.0.0.2.0.0.0-579] at org.apache.nifi.processors.hive.PutHiveStreaming.getOrCreateWriter(PutHiveStreaming.java:738) ~[nifi-hive-processors-1.0.0.2.0.0.0-579.jar:1.0.0.2.0.0.0-579]

Re: NiFi processor putHiveStreaming exception.

gkeys — Thu, 24 Nov 2016 06:10:09 GMT

(Assuming you are running nifi locally and putting to the sandbox)

I had the same issue when putting anything to the sandbox (PutHiveStreaming, PutHDFS -- for PutHDFS NiFi began writing the file but would immediate suffer broken pipe, leaving 0 byte files in HDFS).

I solved this by following Simon Ball's article http://www.simonellistonball.com/technology/nifi-sandbox-hdfs-hdp/ which simplifies communicating with the vm by using remote processor groups on each side. Very straightfoward and worked the first time.

Re: NiFi processor putHiveStreaming exception.

mpandit — Thu, 24 Nov 2016 11:59:22 GMT

@Greg Keys NiFi is also running on sandbox. It was issue with my hive table. I see the PutHiveStreaming is successful and the files are getting written successfully. When I select from hive table it doesn't show any data, not sure why?

Re: NiFi processor putHiveStreaming exception.

gkeys — Thu, 24 Nov 2016 20:20:51 GMT

Not sure if this helps -- just tossing this out there -- but a few things which you probably already know:

From https://community.hortonworks.com/articles/52856/stream-data-into-hive-like-a-king-using-nifi.html

In order for Hive Streaming to work the following has to be in place:

Table is stored as ORC
Transactional Property is set to “True”
The Table is Bucketed

When I ran that demo I noticed:

InferAvroSchema processor:

City,Edition,Sport,sub_sport,Athlete,country,Gender,Event,Event_gender,Medal - produced nulls in the Hive table for columns in caps
I made them all lower case and got the values in the Hive table

Finally ... a recent post:

https://community.hortonworks.com/questions/68068/hive-streamaing.html

Re: NiFi processor putHiveStreaming exception.

mpandit — Sat, 17 Dec 2016 06:06:43 GMT

I was running into the issue NIFI-2828. I ended up using Hive NAR provided by @Matt Burgess as mentioned here https://community.hortonworks.com/questions/59681/puthivestreaming-nifi-processor-various-errors.html. It must have been already fixed with latest NiFi version.

Re: NiFi processor putHiveStreaming exception.

mustafa_qizilba — Tue, 21 Aug 2018 23:33:47 GMT

In HDP 3.0, please use PutHive3Streaming, PutHive3QL and SelectHiveQL.

Cheers.

Re: NiFi processor putHiveStreaming exception.

alpeshghori — Wed, 05 Sep 2018 23:00:08 GMT

@Greg Keys, The solution you provided for the table definition and file format worked for me? But any insights why we need to have it as part of the data load?