Member since
01-05-2017
153
Posts
10
Kudos Received
2
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 4482 | 02-20-2018 07:40 PM | |
| 3306 | 05-04-2017 06:46 PM |
07-11-2017
03:08 PM
Hey Matt Your solution works very well. Its not as pretty as GrokReader true, but it is just Splitting the content rather than putting it into attributes or switching the format of the log file to JSON. I have one unusual result. It is putting a newline in between each event. I don't think this is going to be a big issue for us but is there a reason you can think of that it would do that? Is it doing that on yours?
... View more
07-11-2017
02:51 PM
I am hesitant about this because ExtractGrok its options are to either put the log file portions into attributes or to rewrite it to the log file as JSON, neither of which I want to do. I just wanted to split based on the timestamp rather than on line.
... View more
07-11-2017
02:49 PM
Interesting. This might be what we need because GrokReader is putting everything into attributes which isn't what we are looking for.
... View more
07-10-2017
03:42 PM
Hello, we are attempting to log a stack trace event using Nifi of the following type: Note that continues until it hits a line such as "... 23 more" We currently have a setup where we read each log file, line by line and if we find a timestamp, we count that line as an event. We regex to extract the timestamp. You can see where this would cause an issue as it creates a new line after "unwinding now" and then that next line and the subsequent lines after do not have timestamps. Currently we are just filtering the non-timestamp lines into a folder but we'd like to be able to put them with the other non-timestamp lines. Is there a way in Nifi to say "Take everything between two timestamps as an event despite if it has newlines in it" but still use the SplitText processor to manage the grouping of the lines (or an alternative?) Has anyone else had to deal with Stack Trace events not processing correctly? Below is a screenshot of the relevant portion of the flow:
... View more
Labels:
- Labels:
-
Apache NiFi
06-22-2017
05:26 PM
Awesome thanks again !
... View more
06-22-2017
05:04 PM
That was it! I don't understand the difference between setting it to Primary Node from All Nodes. I thought it would duplicate all the output data.
... View more
06-22-2017
05:01 PM
But if I say "run on all nodes" doesn't that duplicate the data output ?? Or am I misunderstanding...
... View more
06-22-2017
03:20 PM
We currently have a setup of 3 Nifis in a cluster. I have MiniFi on a sending server sending to the Input port in the screenshot called LoggerMiniFi2. The data always gets stuck right after the Input Port. By stuck I mean it queues up and then just sits there, never empties. We cannot just drop the data, its important data so setting an expiration time doesn't make sense. You can see in the screenshot below under Status History that the data has sat in there with no bytes out for almost 15 minutes. Can anyone provide any troubleshooting tips on this, perhaps a cause? Note, I have also tried removing the SplitText processor and just connecting the Input Port to the ExtractText and it still just stalled out right after the Input Port. I'm at a loss of what could be causing this. Note, I have put the SplitText configs down below just for reference as well.
... View more
Labels:
- Labels:
-
Apache NiFi
06-22-2017
03:04 PM
I have a question regarding your solution of using backpressure and expiration with MiniFi... this seems like a terrible solution because then you lose the data. We cannot just throw away our data here if it gets stuck in a queue so how do you suggest handling this with MiniFi?
... View more