question Re: Nifi PutHiveQL from SelectHiveQL in Archives of Support Questions (Read Only)

Nifi PutHiveQL from SelectHiveQL

danilosousadba — Wed, 01 Aug 2018 03:14:15 GMT

Hello

I have the Hive Rawdata External Table and need populate swap data from select * from rawdata.

The process is execute by crontab daily, and i want migrate the process to Nifi.

Whats best pratice for this question?

Thanks.

Re: Nifi PutHiveQL from SelectHiveQL

Shu_ashu — Wed, 01 Aug 2018 06:41:07 GMT

@Danilo Sousa

If you want to move data between two hive tables then you don't need to use SelectHiveQL processor at all.

You create hive statement like below

insert into <db_name>.<final_table> select * from <db_name>.<rawdata>

Then execute the above statement using PutHiveQL processor.

To incrementally run this process then you need to store the state i.e. until what time you have already processed the data from rawdata table. Then only select the new data after the state value.

Please refer to this and this link for more details how to incrementally copy data in hive.

Re: Nifi PutHiveQL from SelectHiveQL

danilosousadba — Wed, 01 Aug 2018 21:24:40 GMT

@Shu

This is working.

Thanks for your time.

Att.

Re: Nifi PutHiveQL from SelectHiveQL

danilosousadba — Fri, 03 Aug 2018 00:49:41 GMT

@Shu

I can cascade the executions?

After finish table A, start table B?

Thanks

Re: Nifi PutHiveQL from SelectHiveQL

Shu_ashu — Fri, 03 Aug 2018 05:14:00 GMT

@Danilo Sousa

if you want to execute sequentially then you can use Success of PutHiveQL processor to trigger another job(i.e. start table B).

Flow:

1.GenerateFlowfile //start with tableA
2.PutHiveQL
3.ReplaceText //to prepare tableB statement
4.PutHiveQL