About robnew666

robnew666 · ‎03-14-2023

I am trying to use Nifi to send data via tcp to relevant filebeat modules, but I am finding that Nifi is adding unwanted data to the logs. Has anyone come across this problem if using Nifi to send to Elasticsearch before and if so, what might be a good solution to preventing Nifi from doing this? (separate pipeline in Elastic to drop/rename/add etc or to use Logstash from Nifi or add a new module to not pick up Nifi data...?)

robnew666 · ‎03-13-2023

I am trying to use Nifi to send data via tcp to relevant filebeat modules, but I am finding that Nifi is adding unwanted data to the logs. Has anyone come across this problem if using Nifi to send to Elasticsearch before and if so, what might be a good solution to preventing Nifi from doing this? (separate pipeline in Elastic to drop/rename/add etc or to use Logstash from Nifi or add a new module to not pick up Nifi data...?)

robnew666 · ‎11-23-2020

I have Zeek logs being ingested and being sent to Splunk via a Splunk Forwarder. I want to be able to catch this also in NiFi to be able to do some extra stuff to it, but I cannot see it using the usual processors as I think it is because of it monitoring the zeek logs constantly, and pushing them across, so it might seem to NiFi that there is no end of the file. There are delimiters within the Zeek logs - { }, but I am wondering if anyone else has tried this before with any success, as it seems I am the only one wanting to be able to do this. Whether it is because of the logs being sent across via the Splunk Forwarder, or because of the way the Zeek(bro) logs being monitored.

robnew666 · ‎11-04-2020

Came up with a fix for this as it was a text file - to splittext each line (1) at the beginning before converting to a JSON file. (think it was the only way to do this as couldn't split after).

robnew666 · ‎11-02-2020

I am trying a ParseCEF & a ParseSysLog processor with a text file and it falls down at the point of Parsing wrt only seeing the first line and then stopping. Are there any examples of parsing a text file to JSON and also recognising the multiple lines after the Parse processor so that there are multiple lines but in the same file. (I realise you can do a SplitText processor before the Parsing, but don't want to go this route.

robnew666 · ‎10-29-2020

Thanks, I am trying some stuff now to parse data using the JoltSpec/JoltTransformJSON processor that could help me with this issue, but thanks for this help, hopefully can get things running more smoothly soon. 🙂

robnew666 · ‎10-29-2020

Thanks, I cant put any data up here, but I think we are getting close. I can pull all data through which then marries it up with the schema, so it is formatted as data associated with the schema. I am then wondering how I can then add another QueryRecord Processor below which I can then add individual sql lines which can pull out coloumns from this data, ie SELECT *, FROM FLOWFILE WHERE table_name = 'rt'

robnew666 · ‎10-27-2020

I am trying to reference an Avro Schema that already works from a CEF file and take out a part of that file (time as rt) by referencing the Schema, but I am a bit rusty with my SQL so it's not going to plan rn - does anyone have any examples of doing this in NiFi?

Online	Offline
Last Visited	‎03-14-2023 05:38 AM

Member Since	‎10-15-2020 02:18 AM
Last Visited	‎03-14-2023 05:38 AM
Posts	11

Cloudera Community

Re: Parse processor not recognising multiple lines

Nifi to Elasticsearch via Filebeat module using Pu...

Re: Using NiFi instead of Logstash

What processor for Zeek logs via Splunk Forwarder

Re: Parse processor not recognising multiple lines

Parse processor not recognising multiple lines

Re: How to use SQL within a QueryRecord Processor ...

Re: How to use SQL within a QueryRecord Processor ...

How to use SQL within a QueryRecord Processor to q...