Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

How to execute Hive DDL Command from Apache NIFI ?

Solved Go to solution

How to execute Hive DDL Command from Apache NIFI ?

Explorer

I am trying to create some tables in Hive from Apache NIFI but I didn't find any exact Processor for that. However I found a processor name PutHiveQl which can be used for DDL/DML operation but I didn't find any property in which I can write the query. If it is the right processor for this purpose then how it can be used in my case.

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: How to execute Hive DDL Command from Apache NIFI ?

Super Guru

@Vivek Singh

PutHiveQL processor is used to execute HiveDDL/DML commands and the processor expects incoming flowfile content would be HiveQL command.

You can keep your create table statement by using GenerateFlowfile processor (or) replacetext processor ..etc and feed the success relation to PutHiveQL processor then the processor executes the content of flowfile and creates the table.

Flow:

77532-flow.png

GenerateFlowfile configs:

77533-gff.png

PuthiveQL configs:

Configure/enable HiveConnection pool and if you are having more than one HiveDDL/DML command in the flowfile content then use ; as delimiter then the processor will execute those commands with the specified delimiter.

In NiFi convertAvroTo ORC processor adds hive.ddl attribute based on the flowfile content we can make use of that attribute and then use ReplaceText processor to create new flowfile content and execute the hive ddl statement using PutHiveQL processor.

Please refer to this link for more details regarding generating/executing hive.ddl statements using NiFi.

View solution in original post

1 REPLY 1
Highlighted

Re: How to execute Hive DDL Command from Apache NIFI ?

Super Guru

@Vivek Singh

PutHiveQL processor is used to execute HiveDDL/DML commands and the processor expects incoming flowfile content would be HiveQL command.

You can keep your create table statement by using GenerateFlowfile processor (or) replacetext processor ..etc and feed the success relation to PutHiveQL processor then the processor executes the content of flowfile and creates the table.

Flow:

77532-flow.png

GenerateFlowfile configs:

77533-gff.png

PuthiveQL configs:

Configure/enable HiveConnection pool and if you are having more than one HiveDDL/DML command in the flowfile content then use ; as delimiter then the processor will execute those commands with the specified delimiter.

In NiFi convertAvroTo ORC processor adds hive.ddl attribute based on the flowfile content we can make use of that attribute and then use ReplaceText processor to create new flowfile content and execute the hive ddl statement using PutHiveQL processor.

Please refer to this link for more details regarding generating/executing hive.ddl statements using NiFi.

View solution in original post

Don't have an account?
Coming from Hortonworks? Activate your account here