Support Questions

Find answers, ask questions, and share your expertise

Control rate processor- Apache nifi

Explorer

Dear,

Could you please explain me how the control rate processor works.

Can we use control rate processor while writing data(large file:80GB)  into hdfs(using PUTHDFS) to reduce the network bandwidth. The data rate should be 300mbps as we are using all the bandwidth of the network now while writing data into hdfs.

 

Thank you 

2 REPLIES 2

Expert Contributor

ControlRate processor is explained here: 

https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.12.1/org.apach...

 

The ControlRate processor allows you to actually control the throughput of a dataflow. It does not slow the processing. The ControlRate processor will allow data to queue in its input side and based on its configured setting only allow x number of FlowFiles through over y amount of time. lets say it is configured to let 5 KB of data through every 1 minute. If you feed it a 37 MB file, it does not transfer just pieces of that FlowFile. It will feed through the entire 37 MB FlowFile and then not allow another FlowFile through until the average data per 1 minute is 5 KB.

 

Above explanation extracted from another post: https://community.cloudera.com/t5/Support-Questions/ControlRate-and-BackPressure-seems-not-work-for-...

Explorer

Thank you. I found the same but I wanted to know if I can use control rate processor in my scenario.

I want to reduce the bandwidth coming from upstream to PUTHDFS. Can we limit the bandwidth in nifi? 

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.