question Re: Output to HDFS separate events one per line in Support Questions

Output to HDFS separate events one per line

elloyd — Tue, 07 Mar 2017 03:04:53 GMT

I have been hunting online for awhile trying to find how one would tell Nifi to format the HDFS output with one event per line instead of one event immediately after the next. I am trying to set up a virtual index through HUNK and its registering my entire output file as one event.

Thanks in advance.

Re: Output to HDFS separate events one per line

MattWho — Tue, 07 Mar 2017 03:10:47 GMT

@Eric Lloyd

Did you try using a "Demarcator" between your merged content? The MergeContent processor allows you to designate a Demarcator. Perhaps use new line or something else unique that can then be used later to differentiate between bits of content in the same HDFS file.

Thanks,

Matt

Re: Output to HDFS separate events one per line

elloyd — Tue, 07 Mar 2017 03:15:14 GMT

Once again, thanks Matt! I think I'm finally starting to wrap my brain around how to adjust parameters to alter file flow.

Re: Output to HDFS separate events one per line

elloyd — Wed, 19 Apr 2017 22:05:34 GMT

I know this was from awhile ago but I just noticed with this solution, the appended data starts right at the end of the previous event causing data loss cause two events get smooshed together. Do you have this issue when you do this and how do you fix it?

Example:

Apr 11 05:00:39 fw01 /kernel: KERN_ARP_ADDR_CHANGE: arp info overwritten for 10.11.x.x from 00:xx:xx:xx:xx:cd to 00:xx:xx:xx:xx:a3Apr 11 05:00:39 fw01 /kernel: KERN_ARP_ADDR_CHANGE: arp info overwritten for 10.11.x.x from 00:xx:xx:xx:xx:a3 to 00:xx:xx:xx:xx:cd