Can NiFi route raw packets - like UDP

[#1]

Hi there,

I am planning on using NiFi for various types of data ingestion, but was wondering if I could use NiFi for raw UDP forwarding.

For example if my NiFi server is accepting syslog/UDP can I process the syslog and send to say elasticsearch and also forward that raw unprocessed syslog/UDP flow?

Also if I wants to forward/route raw collectd/UDP binary protocol flows could I do that with NiFi?

Thanks!

[#2]

@Stephen Greszczyszyn

-

NiFi is designed to be data agnostic meaning it has no dependency on any specific type(s) of data. This is accomplished by wrapping ingested content in a NiFi FlowFile. A NiFi FlowFile consists of two parts:
-

1. FlowFile content (This is the bytes of data which are simply written to claims in the content repository)

2. FlowFile attributes/metatada (This is information about the FlowFile and its content)

-

While NiFi does not have and dependency on data types, various processors that are available in NiFi likely will. So you will need to take a closer look at the documentation for any processor you use that will need to interact with the FlowFile content. NiFi has some syslog based processors already.

-

When it comes to writing the raw data NiFi simply transmits the bytes. If the target will accept the raw data, then all is good.

-

Thank you,

Matt

-

If you found this answer addressed your question, please take a moment to login in and click the "ACCEPT" link.

[#3]

@Stephen Greszczyszyn

The word "process" can mean many things. What kind of processing are you trying to do?
-
The content of your syslog data is just standard Ascii, correct? If so, then it can be read by many processors. So thread question is what are you trying to do with it?

-

I am assuming your syslog ingest may consist of many log lines per FlowFile. If that is the case you may want to "process" these FlowFiles as records. Maybe start by looking at the various "Record" based processors. The GrokReader is probably what you want to configure the record based processors to use in order to parse your syslog content.

-

Thanks,

Matt

[#4]

I'm starting to trial NiFi as a data ingestion engine. I would like to input the following datatypes:

1) collectd (UDP): I don't think that NiFi has a collectd parser, so I will need to direct these raw UDP streams to a locally running Telegraf and Logstash instance for parsing.

2) Syslog (UDP): I would like to experiment with routing raw syslog UDP packets (unprocessed) to destinations as well as filtering/parsing the syslog data using the NiFi syslog modules.

3) Netflow (UDP): I would like to take a heavy raw Netflow stream and test performance to forward only a subset of Netflow data based on a list of protocol types that I'm interested in (mapped against one of the Netflow data key values).

What I haven't been able to understand in the documentation is how I can redirect raw UDP packet flows, (listening and then forwarding to two destinations), without having to process the particular data in the UDP packet.

[#5]

Thanks @Matt Clarke. I guess I have a bit of reading to do to find the right processor that just routes raw UDP packets.