Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

metron-bro-plugin-kafka is not producing anything even though bro is generating logs

Highlighted

metron-bro-plugin-kafka is not producing anything even though bro is generating logs

New Contributor

New Metron user. I set up a beefy box as a single node instance of Metron for learning purposes mostly following the instructions at https://cwiki.apache.org/confluence/display/METRON/Metron+0.4.1+with+HDP+2.5+bare-metal+install+on+C... with the primary exception that I installed the latest Bro release (2.5.3) from rpms before realizing that I needed the source for building the Kafka plugin so I built Bro from source, symlinked /opt/bro to /usr/local/bro, then built and installed the Kafka plugin. The end of my local.bro looks like:

redef Kafka::logs_to_send = set(Conn::LOG);

redef Kafka::topic_name = "bro";

redef Kafka::tag_json = T;

redef Kafka::kafka_conf = table( ["metadata.broker.list"] = "127.0.0.1:6667" );

I know its picking this up because I was getting errors about misconfigurations here, such as when I forgot to change the IP address from the placeholder, and now bro starts without error. I send traffic to bro via tcp-replay, and I see that is working because I am getting the expected logs in /usr/local/bro/logs/current, such as conn.log . Before doing that I set up a Kafka cli consumer:

[root@ip-172-31-38-75 ~]# /usr/hdp/2.5.3.0-37/kafka/bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic bro

{metadata.broker.list=ip-172-31-38-75.us-east-2.compute.internal:6667, request.timeout.ms=30000, client.id=console-consumer-35841, security.protocol=PLAINTEXT}

I would expect to get spammed by conn logs on that consumer, but... nothing.

Any thoughts on where I went wrong or how I can go about debugging this?

Thanks!

Terry

4 REPLIES 4

Re: metron-bro-plugin-kafka is not producing anything even though bro is generating logs

New Contributor

how exactly are you starting bro? I was doing bro -i eth0 & . and nothing got posted to kafka. I changed it to

bro -i eth0 /usr/local/bro/share/bro/site/local.bro & . and the messages started appearing in kafka

Re: metron-bro-plugin-kafka is not producing anything even though bro is generating logs

New Contributor

Hi Terry, do you figure out this, I have the same problem.

Re: metron-bro-plugin-kafka is not producing anything even though bro is generating logs

New Contributor

Hi terry, i am facing same issue, i can see logs generated by bro, but it is not pushing anything to kafka topic, can you please share how you solved this.

Thanks.

Re: metron-bro-plugin-kafka is not producing anything even though bro is generating logs

New Contributor

If i guess it right the issue is with rdkafka, it bonds with kafka on port 9092 but in case of hortonworks, kafka listens on port 6667 all you have to do, redefine your kafka port in your bro script and redeploy bro with broctl.

#Write this below the @load statements in your bro script. This will override your rdkafka config and you can see your logs emitting to kafka topic.

 redef Kafka::topic_name = "bro-kafka-topic";
redef Kafka::tag_json = T;
redef Kafka::kafka_conf = table(
    ["metadata.broker.list"] = "localhost:6667",
    ["client.id"] = "bro"
);
Don't have an account?
Coming from Hortonworks? Activate your account here