Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Unable to load CSV data in to Hive table using Puthivestreaming using Nifi ?

avatar

Below are the errors i'm getting while loading CSV file to HIVE table using NIFI.

1.Hive streaming connect/write error, flowfile will be penalized and routed to retry.

2.Error connecting to Hive endpoint:table routes at thrift.

3.Failed to create hive writer for endpoint.

Any solutions for the above much appreciated& Thanks a lot in advance.

1 ACCEPTED SOLUTION

avatar
Rising Star

Ah, Hive has reached the maximum number of open transactions. There is the parameter hive.max.open.txns that limits the number (compare to https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions ). Default Value is 100000 - this should be high enough, and I guess you did not change it, but maybe you want to check it.

Not sure what is going on in your environment but maybe you want to have a look Chapter 6 of http://hortonworks.com/hadoop-tutorial/using-hive-acid-transactions-insert-update-delete-data/ to see how to manage your open transactions.

Hope this helps.

View solution in original post

9 REPLIES 9

avatar
Rising Star

Bhanu, can you please share the entry of the nifi-app.log for the error to see what exactly happened?

avatar
Super Collaborator

avatar
Rising Star

Ah, Hive has reached the maximum number of open transactions. There is the parameter hive.max.open.txns that limits the number (compare to https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions ). Default Value is 100000 - this should be high enough, and I guess you did not change it, but maybe you want to check it.

Not sure what is going on in your environment but maybe you want to have a look Chapter 6 of http://hortonworks.com/hadoop-tutorial/using-hive-acid-transactions-insert-update-delete-data/ to see how to manage your open transactions.

Hope this helps.

avatar

Bhanu @ Please find my nifi_app.log as an attachment.

avatar
Rising Star

Thanks Bhanu, the error comes from Hive. Can you please also have a look at the hive metastore log file and share any related error message?

avatar

Bhanu@please find hivemetastore as an attachment.

avatar

I have changed Open txns limit & enabled Run compactor to ' TRUE' & changed the numbers of threads used by compactor ='5'.Now i'm able to load data in to my Hive table. Thanks a lot for help Peter.It worked like an charm.

avatar
Rising Star

Very good, glad to help. I transformed the comment that helped finally to an answer and would be happy if you accept it 😉 Thanks.

avatar

Sure & Thank you very much indeed.