Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

NiFi: How to load data into Hive Partitioned Tables

Highlighted

NiFi: How to load data into Hive Partitioned Tables

I am able to bring CSV files from local to hdfs and then created Hive External Table on top of the files/ folder.

Now want to load data in Hive Partitioned tables.

Processor: PutHiveStreaming is giving error show in the attached snapshot.

Looking forwards for a quick help.

Thanks.

86579-puthivestreaming-error.jpg

4 REPLIES 4

Re: NiFi: How to load data into Hive Partitioned Tables

Super Collaborator

To what output format have you set the 'SelectHiveQL' processor? It supportst csv and Avro, and you must set it to Avro, as the 'PutHiveStreaming' processor expects the input to be in Avro format.

Highlighted

Re: NiFi: How to load data into Hive Partitioned Tables

Hi Harald,

Thanks for participating.

Let me share the screen shot of both processors as well.

Processor: SelectHiveQL

86583-selecthiveql.jpeg

Processor: PutHiveStreaming

86584-puthivestreaming.jpeg

Also changed Auto-Create Partitions to true as well for testing but still the same error.

Highlighted

Re: NiFi: How to load data into Hive Partitioned Tables

I have;

1) One VM for HDP as single node cluster

2) One VM for Separate instance for NiFi

Everything is working fine, just not the thrift, unable to write in Hive partition table. I read somewhere that Hive jar files from HDP need to be copied into NiFi instance. Can anyone guide in this direction as well?

https://community.hortonworks.com/questions/830/how-to-do-i-get-the-hive-jdbc-driver-for-my-clinet.h...

Looking forwards.

Highlighted

Re: NiFi: How to load data into Hive Partitioned Tables

Issue Resolved for me.

In HDP 3.0, please use PutHive3Streaming, PutHive3QL and SelectHiveQL.

Cheers.

Don't have an account?
Coming from Hortonworks? Activate your account here