Support Questions
Find answers, ask questions, and share your expertise

Send data using nifi to Metron

Highlighted

Send data using nifi to Metron

Explorer

I configured nifi process to ingest data from a file to kafka broker to metron. Nifi send data to kafka without any error. But it will not push to metron parsers. Please help me on this. But if I run below two commands it send data parsers without any issues.

cat /var/log/squid/access.log |${HDP_HOME}/kafka-broker/bin/kafka-console-producer.sh --broker-list $BROKERLIST --topic squid

${HDP_HOME}/kafka-broker/bin/kafka-console-consumer.sh --zookeeper $ZOOKEEPER --topic squid --from-beginning

7 REPLIES 7
Highlighted

Re: Send data using nifi to Metron

Guru

If you're sending a whole file it may not parse properly.

Metron expects one line per message on Kafka, so the best bet is to use the NiFi SplitText processor before PublishKafka. Note that you can also use the demarcator option in PublishKafka instead of SplitText if you are not using the Kafka key to send things like tenant meta data to Metron.

It would also be worth checking that the data is coming through to kafka properly from NiFi with a quick console consumer check without the from-beginning while sending.

Highlighted

Re: Send data using nifi to Metron

Explorer

Hi Simon,

Thanks a lot. I added "\n" to message demarcator. Now it is working with out any issue. thanks a lot for your quick response. Really appreciate that.

Cheers,

Suresh

Highlighted

Re: Send data using nifi to Metron

Explorer

@Simon Elliston Ball

I was trying to send data in CEF from using "\n" demarcator. But it doesnot parsed correctly in metron CEF paser. Could you please send me a proper demarcator which is working for CEF format.

Sample log format :

CEF:0|Microsoft|Microsoft Windows||Microsoft-Windows-Security-Auditing:4624

CEF:0|Microsoft|Microsoft Windows||Microsoft-Windows-Security-Auditing:4634

Cheers,

Suresh

Highlighted

Re: Send data using nifi to Metron

Guru
@Suresh Rupasinghe

are you sure the windows logs use \n as a delimiter, more likely to be using windows line endings (\r\n)

Highlighted

Re: Send data using nifi to Metron

Explorer

@Simon Elliston Ball

it is end like below.

CEF:0|Microsoft|Microsoft Windows||Microsoft-Windows-............. aid=393wdw18BABCAB3JFk8LDzQ\=\=
Highlighted

Re: Send data using nifi to Metron

Guru

@Suresh Rupasinghe that does not show a line ending, just the end of some base64 encoded data by the looks of it. Please try using the windows line ending (\r\n) vs the unix line endings (\n) as the delimiter for lines.

Re: Send data using nifi to Metron

Explorer

@Simon Elliston Ball

Ok thanks Simon. I will check it. As you mentioned above i used SplitText and now it is working without any issue.

Cheers,

Suresh