Cloudera Community
m_soury
user rank
New Contributor

Re: Nifi TailFile Processor does not detect vast i...

by New Contributor in Support Questions
‎07-25-2018 01:00 PM
‎07-25-2018 01:00 PM
Dear @Matt Clarke Thank you for your great help. Actually, based on your help I used a SplitText after a TailFile processor, which ingest Squid logs (based on Apache Metron), with the following properties: Line Split Count: 1 Maximum Fragment Size: (No value set) Header Line Count: 0 Header Line Marker Characters: (No value set) Remove Trailing Newlines: false I checked all types of the relationship (including: failure / original / splits) in the "Connection Details" between SplitText and PutKafka processors. The problem is the SplitText duplicates the input data. Please find the attachment which is completely obvious. Actually, there is an extra newline character at the end of each received FlowFile (if I increase the insertion rate, the extra newline only would be added after a bunch of lines in a FlowFile). Any advice? ... View more

Nifi TailFile Processor does not detect vast inser...

by New Contributor in Support Questions
‎07-11-2018 01:45 PM
‎07-11-2018 01:45 PM
We have setup Nifi site-to-site to transfer Snort logs (alert.csv) to a remote Nifi server, using Apache Metron (Hortonworks Cybersecurity). Once Snort is running, it produces a vast amount of logs in CSV format, more than 500 records only in 5 seconds for instance. The problem is that Nifi Tailfile cannot detect and separate all 500 records, it may detect around 20 changes (totally include 500 records). Current configurations are attached to this message.How can we configure the Tailfile processor to detect every single line as a new packet (event)? Any help would be appreciated. ... View more
Labels:
Contact Me
Online
Offline
Last Visited
‎11-03-2018 04:05 PM
Community Statistics
Member Since ‎04-10-2018 01:52 PM
Last Visited ‎11-03-2018 04:05 PM
Posts 3