Ingest Yaf to Metron

[#1]

I configured Yaf on node instance. And I run command to ingest Yaf event to Kafka. When I check storm log on Metron node but I don't find yaf.

nohup /usr/local/bin/yaf --in eth0 --become-user yaf --live pcap | /usr/local/bin/yafscii --tabular | /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list node1:6667 --topic yaf &

Can you help me?

[#2]

@Lee Adrian, can you check if you are getting any output from the kafka-console-consumer?

/usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh -z node1:2181 --topic yaf

This would be the first step to ensure that yaf is generating data indeed.

[#3]

Hi @asubramanian, I try to follow your suggest but it's not ingest yaf.

[#4]

Hi @Lee Adrian, in that case looks like your yaf sensor is not generating any input.

Can you check if there is any output coming from the command:

nohup /usr/local/bin/yaf --in eth0 --become-user yaf --live pcap | /usr/local/bin/yafscii --tabular

If there is no output, then you need to look at configuring YAF to get it working.

[#5]

Hi @asubramanian,

When I run this command it have output.

nohup /usr/local/bin/yaf --in eth0 --become-user yaf --live pcap | /usr/local/bin/yafscii --tabular

But when I run this command it don't have output

nohup /usr/local/bin/yaf --in eth0 --become-user yaf --live pcap | /usr/local/bin/yafscii --tabular | /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list node1:6667 --topic yaf &

[#6]

Hi @asubramanian.,

If I start yaf by command:

/etc/init.d/yaf start

And then I running command:

nohub /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list node1:6667 --topic yaf &

It's show log

[2017-06-29 10:19:32,326] ERROR Error when sending message to topic yaf with key: null, value: 52 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.TimeoutException: Batch containing 190 record(s) expired due to timeout while requesting metadata from brokers for yaf-0

[#7]

@asubramanian I installed Metron using Ambari. How do I configure snort,bro and yaf ?? Could you please tell me

[#8]

Hi @Rahul P you can use bro-plugin-kafka for bro and flume for snort to forward events to metron.

[#9]

@Lee Adrian

Well i tried using bro plugin for kafka, but I am getting errors. Do you have any documentation with steps to follow ?

[#10]

@Rahul P

You try to the guide bro plugin. But you change ["metadata.broker.list"] = "localhost:9092" to ["metadata.broker.list"] = "node1:6667"

[#11]

@asubramanian

I installed Metron using Ambari. I want to install Yaf as a standalone application on a centos/ubuntu client. How can I transfer generated logs (probably IPFIX logs) to the metron server? Actually, am I configuring Yaf correctly (on a separate machine)? Is it a right architecture?
Currently, I have installed bro and snort on a separate machine and I am able to send logs to the metron (using BroKafka Plugin and Nifi Site-to-Site respectively), but regarding the Yaf, I am still unclear about the way of transferring logs to the server.