Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Using NiFi with Hive in HDP

Highlighted

Using NiFi with Hive in HDP

New Contributor

Hi everyone!

I have installed HDP (2.6.5.0-292) cluster from Ambari, where are many services, including Apache Hive. Additionally, I have installed outside the Apache NiFi cluster, which listens to proxy logs and then has to load them into the Hive database.

My question is: Is it possible that in such a configuration in NiFi, the PutHiveStreaming component will be able to connect with Hive?

I have not been able to connect with Hive for a long time, and there is such a message in NiFi UI:

PutHiveStreaming[id=7f71da89-0165-1000-5dd9-bd6d6784a1e9] Hive Streaming connect/write error, flow file will be penalized and routed to retry. org.apache.nifi.util.hive.HiveWriter$ConnectFailure: Failed connecting to EndPoint {metaStoreUri='thrift://server:9083', database='hive', table='squid_data_orc', partitionVals=[] }: org.apache.nifi.processors.hive.PutHiveStreaming$ShouldRetryException: Hive Streaming connect/write error, flow file will be penalized and routed to retry. org.apache.nifi.util.hive.HiveWriter$ConnectFailure: Failed connecting to EndPoint {metaStoreUri='thrift://server:9083', database='hive', table='squid_data_orc', partitionVals=[] }

This is the current configuration for the PutHiveStreaming component:

Hive Metastore URI: thrift://server:9083
Hive Configuration Resources: /etc/hive/2.6.5.0-292/0/hive-site.xml
Database Name: hive
Table Name: squid_data_orc
Others settings are default.

88401-puthivestreaming.png

1 REPLY 1

Re: Using NiFi with Hive in HDP

Hello @Wojtek !

Did you look at this post?
https://community.hortonworks.com/articles/88309/using-puthivestreaming-processor-in-nifi.html
Quick question, on every nifi node, you have this path
/etc/hive/2.6.5.0-292/0/hive-site.xml ? If so, could you double-check if is there any thrift pointing to your server:9083?

And also, could you check if in the nifi-app.log or in the hiveserver2.logs there are more details related to your issue.
Hope this helps!