Nifi PutHiveQL from SelectHiveQL

[#1]

Hello

I have the Hive Rawdata External Table and need populate swap data from select * from rawdata.

The process is execute by crontab daily, and i want migrate the process to Nifi.

Whats best pratice for this question?

Thanks.

[#2 · Accepted solution]

@Danilo Sousa

If you want to move data between two hive tables then you don't need to use SelectHiveQL processor at all.

You create hive statement like below

insert into <db_name>.<final_table> select * from <db_name>.<rawdata>

Then execute the above statement using PutHiveQL processor.

To incrementally run this process then you need to store the state i.e. until what time you have already processed the data from rawdata table. Then only select the new data after the state value.

Please refer to this and this link for more details how to incrementally copy data in hive.

[#3]

@Shu

This is working.

Thanks for your time.

Att.

[#4]

@Shu

I can cascade the executions?

After finish table A, start table B?

Thanks

[#5]

@Danilo Sousa

if you want to execute sequentially then you can use Success of PutHiveQL processor to trigger another job(i.e. start table B).

Flow:

1.GenerateFlowfile //start with tableA
2.PutHiveQL
3.ReplaceText //to prepare tableB statement
4.PutHiveQL