Collect edr-logs and ingest it in HDFS using NIFI

[#1]

1. I'm trying to collect the logs from EDR and store it in hdfs using NIFI using the following nifi dataflow: Its working well but the size of the stored file was so small though I combined records together

 

2. I intended to merge records before do the compression but the merge process not receiving data from the listenUDP processor

3. I tried listenUDPrecord insted of listenUDP but it didn't listen to any data

[#2]

@Mercy,

your MergeRecord Processor does not perform any action because you set a minimum number of records equal to a very big value. You need to know that the Property Minimum Number of Records is a hard limit, whereas the Maximum Number of Records is a soft limit. This basically means that your processor will wait until the queue coming in the processor has at least X records present, where X = the value you set in the processor as Minimum.

If you want to merge those files and not wait until you reach that mentioned value, you have two options:
1) You either decrease the value set in Minimum Number of Records.
2) You set a Max Bin Age. This property works as a counter. If you set here 120 minutes or 75 seconds, the processor will wait the specific amount of time and merge the presents record, ignoring the value set for Minimum Number of Record.

[#3]

Thank you for the reply, but my main concern is to have large file before
input to HDFS.