Support Questions
Find answers, ask questions, and share your expertise

NiFi: How to load data into Hive Partitioned Tables

I am able to bring CSV files from local to hdfs and then created Hive External Table on top of the files/ folder.

Now want to load data in Hive Partitioned tables.

Processor: PutHiveStreaming is giving error show in the attached snapshot.

Looking forwards for a quick help.




Super Collaborator

To what output format have you set the 'SelectHiveQL' processor? It supportst csv and Avro, and you must set it to Avro, as the 'PutHiveStreaming' processor expects the input to be in Avro format.

Hi Harald,

Thanks for participating.

Let me share the screen shot of both processors as well.

Processor: SelectHiveQL


Processor: PutHiveStreaming


Also changed Auto-Create Partitions to true as well for testing but still the same error.

I have;

1) One VM for HDP as single node cluster

2) One VM for Separate instance for NiFi

Everything is working fine, just not the thrift, unable to write in Hive partition table. I read somewhere that Hive jar files from HDP need to be copied into NiFi instance. Can anyone guide in this direction as well?

Looking forwards.

Issue Resolved for me.

In HDP 3.0, please use PutHive3Streaming, PutHive3QL and SelectHiveQL.