Support Questions
Find answers, ask questions, and share your expertise

Ingest Yaf to Metron

Ingest Yaf to Metron

Explorer

I configured Yaf on node instance. And I run command to ingest Yaf event to Kafka. When I check storm log on Metron node but I don't find yaf.

nohup /usr/local/bin/yaf --in eth0 --become-user yaf --live pcap | /usr/local/bin/yafscii --tabular | /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list node1:6667 --topic yaf &

Can you help me?

10 REPLIES 10

Re: Ingest Yaf to Metron

Super Collaborator

@Lee Adrian, can you check if you are getting any output from the kafka-console-consumer?

/usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh -z node1:2181 --topic yaf

This would be the first step to ensure that yaf is generating data indeed.

Re: Ingest Yaf to Metron

Explorer

Hi @asubramanian, I try to follow your suggest but it's not ingest yaf.

Re: Ingest Yaf to Metron

Super Collaborator

Hi @Lee Adrian, in that case looks like your yaf sensor is not generating any input.

Can you check if there is any output coming from the command:

nohup /usr/local/bin/yaf --in eth0 --become-user yaf --live pcap | /usr/local/bin/yafscii --tabular

If there is no output, then you need to look at configuring YAF to get it working.

Re: Ingest Yaf to Metron

Explorer

Hi @asubramanian,

When I run this command it have output.

nohup /usr/local/bin/yaf --in eth0 --become-user yaf --live pcap | /usr/local/bin/yafscii --tabular

But when I run this command it don't have output

nohup /usr/local/bin/yaf --in eth0 --become-user yaf --live pcap | /usr/local/bin/yafscii --tabular | /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list node1:6667 --topic yaf &

Re: Ingest Yaf to Metron

Explorer

Hi @asubramanian.,

If I start yaf by command:

/etc/init.d/yaf start

And then I running command:

nohub /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list node1:6667 --topic yaf &

It's show log

[2017-06-29 10:19:32,326] ERROR Error when sending message to topic yaf with key: null, value: 52 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.TimeoutException: Batch containing 190 record(s) expired due to timeout while requesting metadata from brokers for yaf-0

Re: Ingest Yaf to Metron

@asubramanian I installed Metron using Ambari. How do I configure snort,bro and yaf ?? Could you please tell me

Re: Ingest Yaf to Metron

Explorer

Hi @Rahul P you can use bro-plugin-kafka for bro and flume for snort to forward events to metron.

Re: Ingest Yaf to Metron

@Lee Adrian

Well i tried using bro plugin for kafka, but I am getting errors. Do you have any documentation with steps to follow ?

Re: Ingest Yaf to Metron

Explorer

@Rahul P

You try to the guide bro plugin. But you change ["metadata.broker.list"] = "localhost:9092" to ["metadata.broker.list"] = "node1:6667"