Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Unable to load CSV data in to Hive table using Puthivestreaming using Nifi ?

Solved Go to solution

Unable to load CSV data in to Hive table using Puthivestreaming using Nifi ?

New Contributor

Below are the errors i'm getting while loading CSV file to HIVE table using NIFI.

1.Hive streaming connect/write error, flowfile will be penalized and routed to retry.

2.Error connecting to Hive endpoint:table routes at thrift.

3.Failed to create hive writer for endpoint.

Any solutions for the above much appreciated& Thanks a lot in advance.

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: Unable to load CSV data in to Hive table using Puthivestreaming using Nifi ?

Contributor

Ah, Hive has reached the maximum number of open transactions. There is the parameter hive.max.open.txns that limits the number (compare to https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions ). Default Value is 100000 - this should be high enough, and I guess you did not change it, but maybe you want to check it.

Not sure what is going on in your environment but maybe you want to have a look Chapter 6 of http://hortonworks.com/hadoop-tutorial/using-hive-acid-transactions-insert-update-delete-data/ to see how to manage your open transactions.

Hope this helps.

9 REPLIES 9

Re: Unable to load CSV data in to Hive table using Puthivestreaming using Nifi ?

Contributor

Bhanu, can you please share the entry of the nifi-app.log for the error to see what exactly happened?

Re: Unable to load CSV data in to Hive table using Puthivestreaming using Nifi ?

Expert Contributor
Highlighted

Re: Unable to load CSV data in to Hive table using Puthivestreaming using Nifi ?

Contributor

Ah, Hive has reached the maximum number of open transactions. There is the parameter hive.max.open.txns that limits the number (compare to https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions ). Default Value is 100000 - this should be high enough, and I guess you did not change it, but maybe you want to check it.

Not sure what is going on in your environment but maybe you want to have a look Chapter 6 of http://hortonworks.com/hadoop-tutorial/using-hive-acid-transactions-insert-update-delete-data/ to see how to manage your open transactions.

Hope this helps.

Re: Unable to load CSV data in to Hive table using Puthivestreaming using Nifi ?

New Contributor

Bhanu @ Please find my nifi_app.log as an attachment.

Re: Unable to load CSV data in to Hive table using Puthivestreaming using Nifi ?

Contributor

Thanks Bhanu, the error comes from Hive. Can you please also have a look at the hive metastore log file and share any related error message?

Re: Unable to load CSV data in to Hive table using Puthivestreaming using Nifi ?

New Contributor

Bhanu@please find hivemetastore as an attachment.

Re: Unable to load CSV data in to Hive table using Puthivestreaming using Nifi ?

New Contributor

I have changed Open txns limit & enabled Run compactor to ' TRUE' & changed the numbers of threads used by compactor ='5'.Now i'm able to load data in to my Hive table. Thanks a lot for help Peter.It worked like an charm.

Re: Unable to load CSV data in to Hive table using Puthivestreaming using Nifi ?

Contributor

Very good, glad to help. I transformed the comment that helped finally to an answer and would be happy if you accept it ;) Thanks.

Re: Unable to load CSV data in to Hive table using Puthivestreaming using Nifi ?

New Contributor

Sure & Thank you very much indeed.

Don't have an account?
Coming from Hortonworks? Activate your account here