NiFi processor putHiveStreaming exception.

[#1]

I am getting following exception while invoking the putHiveStreaming processor.

016-11-23 04:16:43,024 ERROR [Timer-Driven Process Thread-4] o.a.n.processors.hive.PutHiveStreaming PutHiveStreaming[id=8f50a88f-0158-1000-4941-4b83f9b57917] Hive Streaming connect/write error, flow file will be penalized and routed to retry 2016-11-23 04:16:43,025 ERROR [Timer-Driven Process Thread-4] o.a.n.processors.hive.PutHiveStreaming org.apache.nifi.util.hive.HiveWriter$ConnectFailure: Failed connecting to EndPoint {metaStoreUri='thrift://sandbox.hortonworks.com:9083', database='default', table='store_order', partitionVals=[] } at org.apache.nifi.util.hive.HiveWriter.<init>(HiveWriter.java:80) ~[nifi-hive-processors-1.0.0.2.0.0.0-579.jar:1.0.0.2.0.0.0-579] at org.apache.nifi.util.hive.HiveUtils.makeHiveWriter(HiveUtils.java:45) ~[nifi-hive-processors-1.0.0.2.0.0.0-579.jar:1.0.0.2.0.0.0-579] at org.apache.nifi.processors.hive.PutHiveStreaming.makeHiveWriter(PutHiveStreaming.java:827) ~[nifi-hive-processors-1.0.0.2.0.0.0-579.jar:1.0.0.2.0.0.0-579] at org.apache.nifi.processors.hive.PutHiveStreaming.getOrCreateWriter(PutHiveStreaming.java:738) ~[nifi-hive-processors-1.0.0.2.0.0.0-579.jar:1.0.0.2.0.0.0-579]

[#2]

(Assuming you are running nifi locally and putting to the sandbox)

I had the same issue when putting anything to the sandbox (PutHiveStreaming, PutHDFS -- for PutHDFS NiFi began writing the file but would immediate suffer broken pipe, leaving 0 byte files in HDFS).

I solved this by following Simon Ball's article http://www.simonellistonball.com/technology/nifi-sandbox-hdfs-hdp/ which simplifies communicating with the vm by using remote processor groups on each side. Very straightfoward and worked the first time.

[#3]

@Greg Keys NiFi is also running on sandbox. It was issue with my hive table. I see the PutHiveStreaming is successful and the files are getting written successfully. When I select from hive table it doesn't show any data, not sure why?

[#4]

Not sure if this helps -- just tossing this out there -- but a few things which you probably already know:

From https://community.hortonworks.com/articles/52856/stream-data-into-hive-like-a-king-using-nifi.html

In order for Hive Streaming to work the following has to be in place:

• Table is stored as ORC
• Transactional Property is set to “True”
• The Table is Bucketed

When I ran that demo I noticed:

InferAvroSchema processor:

• City,Edition,Sport,sub_sport,Athlete,country,Gender,Event,Event_gender,Medal - produced nulls in the Hive table for columns in caps
• I made them all lower case and got the values in the Hive table

Finally ... a recent post:

https://community.hortonworks.com/questions/68068/hive-streamaing.html

[#5 · Accepted solution]

I was running into the issue NIFI-2828. I ended up using Hive NAR provided by @Matt Burgess as mentioned here https://community.hortonworks.com/questions/59681/puthivestreaming-nifi-processor-various-errors.htm.... It must have been already fixed with latest NiFi version.

[#6]

In HDP 3.0, please use PutHive3Streaming, PutHive3QL and SelectHiveQL.

Cheers.

[#7]

@Greg Keys, The solution you provided for the table definition and file format worked for me? But any insights why we need to have it as part of the data load?