question Re: NiFi PutHive3Streaming and Hive's minor compaction in Archives of Support Questions (Read Only)

NiFi PutHive3Streaming and Hive's minor compaction

kei_miyauchi_ — Tue, 20 Nov 2018 17:50:03 GMT

Datas which are inserted into Hive by INSERT INTO command are gotten minor compaction.
But the data inserted into Hive using NiFi's PutHive3Streaming aren't gotten minor compaction even if there is enough deltas.
Is it possible to make minor compaction(NiFi) work?

Re: NiFi PutHive3Streaming and Hive's minor compaction

Shu_ashu — Tue, 20 Nov 2018 22:22:17 GMT

@Kei Miyauchi

If you want to trigger minor/major compactions from NiFi then feed the Success relationship PutHiveStreaming processor to Replace Text process and configure ReplaceText processor with below configs:

replacement strategy

always replace

and replacement value as

alter table <db_name>.<table_name> compact 'minor';

Then using PutHiveQL processor execute the minor compaction.

Flow:

--other processors
--> PutHivestreaming 
--> ReplaceText processor
--> PutHiveQL

By following this way we are initializing minor compaction from NiFi.

Take a look into this SupportKB related to Minor Compactions are not working in Hive, set the recommended global configs to make minor compactions work.

Re: NiFi PutHive3Streaming and Hive's minor compaction

kei_miyauchi_ — Wed, 21 Nov 2018 12:37:42 GMT

@Shu

Thank you, I'll try that.
The link isn't working. You meant this page?
https://community.hortonworks.com/content/supportkb/193756/automatic-minor-compaction-on-hive-is-not-working.html

And, do you have any idea to trigger ReplaceText and PutHiveQL only after some flowfile passed PutHiveStreaming?
I think invoking minor compaction for each flowfile is too much when a lot of flowfile comes.

Re: NiFi PutHive3Streaming and Hive's minor compaction

Shu_ashu — Wed, 21 Nov 2018 22:03:37 GMT

@Kei Miyauchi

Yes, i meant https://community.hortonworks.com/content/supportkb/193756/automatic-minor-compaction-on-hive-is-not-working.html this page.

Use merge content processor after PutHiveStreaming and configure the processor to wait for minimum 10 flowfiles (or some other number) and merge them into one then feed the merged relation to ReplaceText processor, by using merge content processor we are going to wait for atleast 10 flowfiles and then triggering minor compaction.

Flow:

--other processors
-->PutHivestreaming
--> MergeContent
-->ReplaceText processor
-->PutHiveQL

Re: NiFi PutHive3Streaming and Hive's minor compaction

kei_miyauchi_ — Mon, 26 Nov 2018 08:02:24 GMT

@Shu

That works!

Thank you for all you replies.