Support Questions

Find answers, ask questions, and share your expertise

increase performance of putHDFS

Hello there,

I am using NiFi 1.8.0

I wanna transfer data (about 50,000,000 millions records or upper than it , 6G) from my local to HDFS.

I am using tailFile, mergContent and HDFS processors. With the default config of NiFi and processors (thread = 4) I transfer data about 2:20 ( 390 event per seconds) but data generator 's time is 900,000.

How can I increase performance for this situation?



@Matt Clarke

Please help me for this problem.

thank you


You can try to increase concurrent tasks in Processor along with heap size (in conf/bootstrap.conf). Also split your process on two. For example Tail and Merge will work on one server and PutHDFS on another. To make a link between them use port and "Remote input port" group. I hope my advice helps.

@Zholaman Kubaliyev

Thanks for your answering.

I installed NiFi on CentOS7 and I am using single NiFi instance.

heap size = 2G


for these situation, How can I solve it?

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.