question Re: Unable to load CSV data in to Hive table using Puthivestreaming using Nifi ? in Support Questions

Unable to load CSV data in to Hive table using Puthivestreaming using Nifi ?

bhanu_balaji — Wed, 30 Nov 2016 15:18:29 GMT

Below are the errors i'm getting while loading CSV file to HIVE table using NIFI.

1.Hive streaming connect/write error, flowfile will be penalized and routed to retry.

2.Error connecting to Hive endpoint:table routes at thrift.

3.Failed to create hive writer for endpoint.

Any solutions for the above much appreciated& Thanks a lot in advance.

Re: Unable to load CSV data in to Hive table using Puthivestreaming using Nifi ?

pgreiff — Wed, 30 Nov 2016 15:36:51 GMT

Bhanu, can you please share the entry of the nifi-app.log for the error to see what exactly happened?

Re: Unable to load CSV data in to Hive table using Puthivestreaming using Nifi ?

avijeetd — Wed, 30 Nov 2016 16:22:08 GMT

please check this, this helped me

https://community.hortonworks.com/questions/59681/puthivestreaming-nifi-processor-various-errors.html#answer-59737

Re: Unable to load CSV data in to Hive table using Puthivestreaming using Nifi ?

pgreiff — Wed, 30 Nov 2016 17:08:24 GMT

Ah, Hive has reached the maximum number of open transactions. There is the parameter hive.max.open.txns that limits the number (compare to https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions ). Default Value is 100000 - this should be high enough, and I guess you did not change it, but maybe you want to check it.

Not sure what is going on in your environment but maybe you want to have a look Chapter 6 of http://hortonworks.com/hadoop-tutorial/using-hive-acid-transactions-insert-update-delete-data/ to see how to manage your open transactions.

Hope this helps.

Re: Unable to load CSV data in to Hive table using Puthivestreaming using Nifi ?

bhanu_balaji — Wed, 30 Nov 2016 17:08:25 GMT

Bhanu @ Please find my nifi_app.log as an attachment.

Re: Unable to load CSV data in to Hive table using Puthivestreaming using Nifi ?

pgreiff — Wed, 30 Nov 2016 17:08:26 GMT

Thanks Bhanu, the error comes from Hive. Can you please also have a look at the hive metastore log file and share any related error message?

Re: Unable to load CSV data in to Hive table using Puthivestreaming using Nifi ?

bhanu_balaji — Wed, 30 Nov 2016 17:08:26 GMT

Bhanu@please find hivemetastore as an attachment.

Re: Unable to load CSV data in to Hive table using Puthivestreaming using Nifi ?

bhanu_balaji — Wed, 30 Nov 2016 17:42:51 GMT

I have changed Open txns limit & enabled Run compactor to ' TRUE' & changed the numbers of threads used by compactor ='5'.Now i'm able to load data in to my Hive table. Thanks a lot for help Peter.It worked like an charm.

Re: Unable to load CSV data in to Hive table using Puthivestreaming using Nifi ?

pgreiff — Wed, 30 Nov 2016 17:53:04 GMT

Very good, glad to help. I transformed the comment that helped finally to an answer and would be happy if you accept it 😉 Thanks.

Re: Unable to load CSV data in to Hive table using Puthivestreaming using Nifi ?

bhanu_balaji — Wed, 30 Nov 2016 18:03:11 GMT

Sure & Thank you very much indeed.