question Re: Accelerate working processor PutHiveStreaming in Archives of Support Questions (Read Only)

Accelerate working processor PutHiveStreaming

Kyivstar — Wed, 22 Feb 2017 18:12:03 GMT

Whether there is possibility to accelerate work processor PutHiveStreaming ? running in parallel ?

Scheduling --> Concurrent Tasks : posible only single tasks.

Re: Accelerate working processor PutHiveStreaming

aervits — Fri, 24 Feb 2017 00:46:22 GMT

I'm going to attempt to answer this as there are far better experts here. You can improve performance in a few ways, you can parallelize by running PutHiveStreaming processor on a couple of nodes or tweak some of the parameters in the processor.

https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.hive.PutHiveStreaming/

Probably one property I'd tweak first is Transactions per batch. See if yours is set too low?

Re: Accelerate working processor PutHiveStreaming

ekoifman — Fri, 24 Feb 2017 03:33:47 GMT

PutHiveStreaming relies on Streaming API which has 2 relevant concepts: number of events per transaction and number of transactions per batch. Generally, the more events you write per transaction the faster the ingest. I don't see the 1st of these properties in the NiFi doc referenced above - perhaps there is some NiFi specific property that controls this.

Re: Accelerate working processor PutHiveStreaming

mburgess — Fri, 24 Feb 2017 03:41:46 GMT

As of NIFI-3418, NiFi will allow the user to set both of the aforementioned properties.

Re: Accelerate working processor PutHiveStreaming

mananstar2001 — Thu, 07 Sep 2017 19:56:26 GMT

Even though adding both the suggested properties, the output rate of Hive Streaming processor still seems to be slow, we are getting a mere 2 tps output rate for processor. the input queue contains more than 10k messages, and processor properties are : transactions per batch = 1k , records per transactions = 100k.

Re: Accelerate working processor PutHiveStreaming

abhinav_joshi — Tue, 15 May 2018 11:54:43 GMT

I have set my Transactions per batch to 10. Is that too low ? I am really looking at ways to process my queues faster . it always seems to be piling slowly over time