Replacing Splunk Universal forwarder with NiFi?

[#1]

NiFi has a putSplunk processor that should do what I want (send data to an indexer) BUT it doesn't have any place for me to specify sourcetype, or index, and it only has one "Host" field, whereas I usually use autolb with 2 indexers. Can I do this? If so, how? Thanks!

[#2]

PutSplunk sends data to Splunk over a TCP or UDP input, its essentially a combination of PutTcp and PutUdp wrapped into a single processor.

Wouldn't the type and index be specified when you create the input in Splunk?

As far as load balancing, you could possibly use DistributeLoad processor to route to two different PutSplunk processors pointing at different hosts, or stick a load balancer like haproxy or nginx in front of the indexers and point PutSplunk at the load balancer.

[#3]

Hi Bryan,

The source type the data needs to be tagged with, and the index it needs to be sent to are both specified by the Splunk Universal Forwarder (SUF) in its inputs.conf file before the data is sent to Splunk. I don't understand what you mean by "the type and index are specified when you create the input in Splunk".

I am trying to replace the splunk universal forwarder completely with NiFi.

The load balancing tips are helpful.

[#4]

Wouldn't you define a new network input in Splunk enterprise, as described here: http://docs.splunk.com/Documentation/Splunk/latest/Data/Monitornetworkports

Part of defining the input you would be choosing UDP or TCP, and specifying the type and index, which is essentially configuring the inputs.conf for Splunk enterprise.

NiFi is just writing bytes to a socket via UDP or TCP, its not modifying the data in anyway (unless you do so earlier in the flow).

You mentioned inputs.conf on the universal forwarder, isn't that for data coming into the universal forwarder? not for data sent from the forwarder to the indexer.

[#5]

@Alex K IMHO the best way, to send data to Splunk is through the HTTP Event Collector.

I'm using the AttributeToJSON processor to create a JSON file with an Attribute List of host, sourcetype, source, event

The JSON file is then sent to HEC using the InvokeHTTP processor.

Works like a charm 🙂

[#6]

@Mika Borner Hey Mika I'm trying to do the same procedure host, sourcetype, source. Is there any way you could help me with configuring AttributeToJSON and the InvokeHTTP processor, please