Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Metron 0.7.1 bro_parser topology not processing

Highlighted

Metron 0.7.1 bro_parser topology not processing

Contributor

Hi,

I've been struggling with this issue and hoping someone can shed some light here.

I'm running Metron 0.7.1 with Bro 2.5.5; Sensors yaf and snort are working and I can see the data in dashboard.

Bro is running; sending logs to Kafka topic "bro"; I can see lots of logs when I run the kafka-console-consumer.sh; however, when I check bro parser topology I got nothing.


I checked the storm worker log for bro and there were no error and it seems to be idle;

2019-06-03 14:27:56.199 o.a.k.c.p.ProducerConfig Thread-5-parserBolt-executor[8 8] [WARN] The configuration request.required.acks = 1 was supplied but isn't a known config.

2019-06-03 14:27:56.199 o.a.k.c.u.AppInfoParser Thread-5-parserBolt-executor[8 8] [INFO] Kafka version : 0.10.0.2.5.0.0-1245

2019-06-03 14:27:56.199 o.a.k.c.u.AppInfoParser Thread-5-parserBolt-executor[8 8] [INFO] Kafka commitId : dae559f56f07e2cd

2019-06-03 14:27:56.204 o.a.s.d.executor Thread-5-parserBolt-executor[8 8] [INFO] Prepared bolt parserBolt:(8)


I used the metron/bin/load_tool.sh to check the throughput of bro and I see numbers; why my bro parser in the storm is not picking up any?

Monitoring bro every 10000 ms

Summarizing over the last 5 monitoring periods (50000ms)

2019/06/03 15:31:33 - 10 eps throughput measured for bro (Mean: 10, Std Dev: 0)

2019/06/03 15:31:43 - 11 eps throughput measured for bro (Mean: 10, Std Dev: 0)

2019/06/03 15:31:53 - 6 eps throughput measured for bro (Mean: 9, Std Dev: 2)

2019/06/03 15:32:03 - 18 eps throughput measured for bro (Mean: 11, Std Dev: 4)

2019/06/03 15:32:13 - 3 eps throughput measured for bro (Mean: 9, Std Dev: 5)

2019/06/03 15:32:23 - 3 eps throughput measured for bro (Mean: 8, Std Dev: 6)

2019/06/03 15:32:33 - 4 eps throughput measured for bro (Mean: 6, Std Dev: 6)




109094-screen-shot-2019-06-03-at-32901-pm.png


1 REPLY 1

Re: Metron 0.7.1 bro_parser topology not processing

Contributor

After more troubleshooting and by adding debug level to the storm topology, I'm seeing a continual of this message

2019-06-11 11:55:01.562 o.a.m.p.b.WriterHandler Thread-12-parserBolt-executor[5 5] [DEBUG] Flushing message queues older than their batchTimeouts

by default the topology.message.timeout.sec is 30

It looks like all my messages have been flushed. Does this mean my message is older than 30s? I thought it's reasonable to have messages that are older than 30s.

Please shine a light and thank you.