Tuning PutHive3Streaming NiFi processor
We using PutHive3Streaming processor to send data to Hive from NiFi, I have an issue where we are getting LOTS of small delta files on our busier feeds which is causing issues with compaction etc.
I have used a series of merges in NiFi to ensure each flowfile contains many thousand records but it still creates many delta files.
I wondered if anyone had any advice on tuning 'Records Per Transaction' and 'Transactions per Batch' options on the PutHive3Streaming processor, I believe this could help with my issue but have had mixed/confusing results from testing. There isn't a great deal of information on best practice that I have found.
Has anyone else had similar issues/found adjustments helpful?

