Support Questions
Find answers, ask questions, and share your expertise

Using NiFi with Hive in HDP

Using NiFi with Hive in HDP

Hi everyone!

I have installed HDP (2.6.5.0-292) cluster from Ambari, where are many services, including Apache Hive. Additionally, I have installed outside the Apache NiFi cluster, which listens to proxy logs and then has to load them into the Hive database.

My question is: Is it possible that in such a configuration in NiFi, the PutHiveStreaming component will be able to connect with Hive?

I have not been able to connect with Hive for a long time, and there is such a message in NiFi UI:

PutHiveStreaming[id=7f71da89-0165-1000-5dd9-bd6d6784a1e9] Hive Streaming connect/write error, flow file will be penalized and routed to retry. org.apache.nifi.util.hive.HiveWriter$ConnectFailure: Failed connecting to EndPoint {metaStoreUri='thrift://server:9083', database='hive', table='squid_data_orc', partitionVals=[] }: org.apache.nifi.processors.hive.PutHiveStreaming$ShouldRetryException: Hive Streaming connect/write error, flow file will be penalized and routed to retry. org.apache.nifi.util.hive.HiveWriter$ConnectFailure: Failed connecting to EndPoint {metaStoreUri='thrift://server:9083', database='hive', table='squid_data_orc', partitionVals=[] }

This is the current configuration for the PutHiveStreaming component:

Hive Metastore URI: thrift://server:9083
Hive Configuration Resources: /etc/hive/2.6.5.0-292/0/hive-site.xml
Database Name: hive
Table Name: squid_data_orc
Others settings are default.

88401-puthivestreaming.png

1 REPLY 1
Highlighted

Re: Using NiFi with Hive in HDP

Hello @Wojtek !

Did you look at this post?
https://community.hortonworks.com/articles/88309/using-puthivestreaming-processor-in-nifi.html
Quick question, on every nifi node, you have this path
/etc/hive/2.6.5.0-292/0/hive-site.xml ? If so, could you double-check if is there any thrift pointing to your server:9083?

And also, could you check if in the nifi-app.log or in the hiveserver2.logs there are more details related to your issue.
Hope this helps!