increase performance of putHDFS

[#1]

Hello there,

I am using NiFi 1.8.0

I wanna transfer data (about 50,000,000 millions records or upper than it , 6G) from my local to HDFS.

I am using tailFile, mergContent and HDFS processors. With the default config of NiFi and processors (thread = 4) I transfer data about 2:20 ( 390 event per seconds) but data generator 's time is 900,000.

How can I increase performance for this situation?

[#2]

@Matt Clarke

Please help me for this problem.

thank you

[#3]

You can try to increase concurrent tasks in Processor along with heap size (in conf/bootstrap.conf). Also split your process on two. For example Tail and Merge will work on one server and PutHDFS on another. To make a link between them use port and "Remote input port" group. I hope my advice helps.

[#4]

@Zholaman Kubaliyev

Thanks for your answering.

I installed NiFi on CentOS7 and I am using single NiFi instance.

heap size = 2G

thead=4

for these situation, How can I solve it?