I am using NiFi 1.8.0
I wanna transfer data (about 50,000,000 millions records or upper than it , 6G) from my local to HDFS.
I am using tailFile, mergContent and HDFS processors. With the default config of NiFi and processors (thread = 4) I transfer data about 2:20 ( 390 event per seconds) but data generator 's time is 900,000.
How can I increase performance for this situation?
You can try to increase concurrent tasks in Processor along with heap size (in conf/bootstrap.conf). Also split your process on two. For example Tail and Merge will work on one server and PutHDFS on another. To make a link between them use port and "Remote input port" group. I hope my advice helps.