Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

NiFi processor putHiveStreaming exception.

Solved Go to solution
Highlighted

NiFi processor putHiveStreaming exception.

I am getting following exception while invoking the putHiveStreaming processor.

016-11-23 04:16:43,024 ERROR [Timer-Driven Process Thread-4] o.a.n.processors.hive.PutHiveStreaming PutHiveStreaming[id=8f50a88f-0158-1000-4941-4b83f9b57917] Hive Streaming connect/write error, flow file will be penalized and routed to retry 2016-11-23 04:16:43,025 ERROR [Timer-Driven Process Thread-4] o.a.n.processors.hive.PutHiveStreaming org.apache.nifi.util.hive.HiveWriter$ConnectFailure: Failed connecting to EndPoint {metaStoreUri='thrift://sandbox.hortonworks.com:9083', database='default', table='store_order', partitionVals=[] } at org.apache.nifi.util.hive.HiveWriter.<init>(HiveWriter.java:80) ~[nifi-hive-processors-1.0.0.2.0.0.0-579.jar:1.0.0.2.0.0.0-579] at org.apache.nifi.util.hive.HiveUtils.makeHiveWriter(HiveUtils.java:45) ~[nifi-hive-processors-1.0.0.2.0.0.0-579.jar:1.0.0.2.0.0.0-579] at org.apache.nifi.processors.hive.PutHiveStreaming.makeHiveWriter(PutHiveStreaming.java:827) ~[nifi-hive-processors-1.0.0.2.0.0.0-579.jar:1.0.0.2.0.0.0-579] at org.apache.nifi.processors.hive.PutHiveStreaming.getOrCreateWriter(PutHiveStreaming.java:738) ~[nifi-hive-processors-1.0.0.2.0.0.0-579.jar:1.0.0.2.0.0.0-579]

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: NiFi processor putHiveStreaming exception.

I was running into the issue NIFI-2828. I ended up using Hive NAR provided by @Matt Burgess as mentioned here https://community.hortonworks.com/questions/59681/puthivestreaming-nifi-processor-various-errors.htm.... It must have been already fixed with latest NiFi version.

View solution in original post

6 REPLIES 6
Highlighted

Re: NiFi processor putHiveStreaming exception.

Guru

(Assuming you are running nifi locally and putting to the sandbox)

I had the same issue when putting anything to the sandbox (PutHiveStreaming, PutHDFS -- for PutHDFS NiFi began writing the file but would immediate suffer broken pipe, leaving 0 byte files in HDFS).

I solved this by following Simon Ball's article http://www.simonellistonball.com/technology/nifi-sandbox-hdfs-hdp/ which simplifies communicating with the vm by using remote processor groups on each side. Very straightfoward and worked the first time.

Highlighted

Re: NiFi processor putHiveStreaming exception.

@Greg Keys NiFi is also running on sandbox. It was issue with my hive table. I see the PutHiveStreaming is successful and the files are getting written successfully. When I select from hive table it doesn't show any data, not sure why?

Highlighted

Re: NiFi processor putHiveStreaming exception.

Guru

Not sure if this helps -- just tossing this out there -- but a few things which you probably already know:

From https://community.hortonworks.com/articles/52856/stream-data-into-hive-like-a-king-using-nifi.html

In order for Hive Streaming to work the following has to be in place:

  • Table is stored as ORC
  • Transactional Property is set to “True”
  • The Table is Bucketed

When I ran that demo I noticed:

InferAvroSchema processor:

  • City,Edition,Sport,sub_sport,Athlete,country,Gender,Event,Event_gender,Medal - produced nulls in the Hive table for columns in caps
  • I made them all lower case and got the values in the Hive table

Finally ... a recent post:

https://community.hortonworks.com/questions/68068/hive-streamaing.html

Highlighted

Re: NiFi processor putHiveStreaming exception.

I was running into the issue NIFI-2828. I ended up using Hive NAR provided by @Matt Burgess as mentioned here https://community.hortonworks.com/questions/59681/puthivestreaming-nifi-processor-various-errors.htm.... It must have been already fixed with latest NiFi version.

View solution in original post

Re: NiFi processor putHiveStreaming exception.

In HDP 3.0, please use PutHive3Streaming, PutHive3QL and SelectHiveQL.

Cheers.

Highlighted

Re: NiFi processor putHiveStreaming exception.

New Contributor

@Greg Keys, The solution you provided for the table definition and file format worked for me? But any insights why we need to have it as part of the data load?

Don't have an account?
Coming from Hortonworks? Activate your account here