Error while pushing Bro logs to Kafka & Storm in Apache Metron

lookup128 · ‎07-23-2018

I am using Apache Metron to push Bro logs to the Apache Kafka using the following link (metron-bro-plugin-kafka):

https://github.com/apache/metron-bro-plugin-kafka

I installed Bro on a CentOS VM and able to receive data in Kafka on the server side. The data then send to the Apache Storm. There is a problem inside the Storm which it seems that it cannot parse the received log and cast it to a JSONObject. Short description of the error is as follow:

java.lang.IllegalStateException:
Unable to parse Message:
{"ts":1532184126.429716,"uid":"CIYoFN1y9Q3D2iJVnc","id.orig_h":"192.168.181.130","id.orig_p":49483,"id.resp_h":"192.168.181.2","id.resp_p":53,

The relevant part of the log is as follow:

2018-07-21 14:45:29.002 o.a.s.d.executor [INFO] TRANSFERING
tuple [dest: 3 tuple: source: parserBolt:5, stream: error, id: {},
[{"exception":"java.lang.IllegalStateException: Unable to parse
Message: {\"ts\":1532184126.434496,\"uid\":\"CIYoFN1y9Q3D2iJVnc\",\"id.orig_h\":\"192.168.181.130\",\"id.orig_p\":49483,\"id.resp_h\":\"192.168.181.2\",\"id.resp_p\":53,\"proto\":\"udp\",\"trans_id\":24981,\"query\":\"safebrowsing-cache.google.com\",\"rcode\":0,\"rcode_name\":\"NOERROR\",\"AA\":false,\"TC\":false,\"RD\":false,\"RA\":true,\"Z\":0,\"answers\":[\"safebrowsing.cache.l.google.com\",\"2a00:1450:4014:80c::200e\"],\"TTLs\":[5.0,5.0],\"rejected\":false}","failed_sensor_type":"bro","stack":"java.lang.IllegalStateException:
Unable to parse Message: {\"ts\":1532184126.434496,\"uid\":\"CIYoFN1y9Q3D2iJVnc\",\"id.orig_h\":\"192.168.181.130\",\"id.orig_p\":49483,\"id.resp_h\":\"192.168.181.2\",\"id.resp_p\":53,\"proto\":\"udp\",\"trans_id\":24981,\"query\":\"safebrowsing-cache.google.com\",\"rcode\":0,\"rcode_name\":\"NOERROR\",\"AA\":false,\"TC\":false,\"RD\":false,\"RA\":true,\"Z\":0,\"answers\":[\"safebrowsing.cache.l.google.com\",\"2a00:1450:4014:80c::200e\"],\"TTLs\":[5.0,5.0],\"rejected\":false}\n\tat
org.apache.metron.parsers.bro.BasicBroParser.parse(BasicBroParser.java:145)\n\tat
org.apache.metron.parsers.interfaces.MessageParser.parseOptional(MessageParser.java:45)\n\tat
org.apache.metron.parsers.bolt.ParserBolt.execute(ParserBolt.java:177)\n\tat
org.apache.storm.daemon.executor$fn__6571$tuple_action_fn__6573.invoke(executor.clj:734)\n\tat
org.apache.storm.daemon.executor$mk_task_receiver$fn__6492.invoke(executor.clj:466)\n\tat
org.apache.storm.disruptor$clojure_handler$reify__6005.onEvent(disruptor.clj:40)\n\tat
org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451)\n\tat
org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430)\n\tat
org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)\n\tat
org.apache.storm.daemon.executor$fn__6571$fn__6584$fn__6637.invoke(executor.clj:853)\n\tat
org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484)\n\tat
clojure.lang.AFn.run(AFn.java:22)\n\tat
java.lang.Thread.run(Thread.java:745)\nCaused by: java.lang.ClassCastException:
java.lang.Boolean cannot be cast to org.json.simple.JSONObject\n\tat
org.apache.metron.parsers.bro.BasicBroParser.parse(BasicBroParser.java:88)\n\t...
12
more\n","hostname":"localhost","raw_message":"{\"ts\":1532184126.434496,\"uid\":\"CIYoFN1y9Q3D2iJVnc\",\"id.orig_h\":\"192.168.181.130\",\"id.orig_p\":49483,\"id.resp_h\":\"192.168.181.2\",\"id.resp_p\":53,\"proto\":\"udp\",\"trans_id\":24981,\"query\":\"safebrowsing-cache.google.com\",\"rcode\":0,\"rcode_name\":\"NOERROR\",\"AA\":false,\"TC\":false,\"RD\":false,\"RA\":true,\"Z\":0,\"answers\":[\"safebrowsing.cache.l.google.com\",\"2a00:1450:4014:80c::200e\"],\"TTLs\":[5.0,5.0],\"rejected\":false}","error_hash":"5b13edd4ca52633f169c276ac5cf728b46c19de2d02b31cf330f6d640c6bfc86","error_type":"parser_error","message":"Unable
to parse Message:
{\"ts\":1532184126.434496,\"uid\":\"CIYoFN1y9Q3D2iJVnc\",\"id.orig_h\":\"192.168.181.130\",\"id.orig_p\":49483,\"id.resp_h\":\"192.168.181.2\",\"id.resp_p\":53,\"proto\":\"udp\",\"trans_id\":24981,\"query\":\"safebrowsing-cache.google.com\",\"rcode\":0,\"rcode_name\":\"NOERROR\",\"AA\":false,\"TC\":false,\"RD\":false,\"RA\":true,\"Z\":0,\"answers\":[\"safebrowsing.cache.l.google.com\",\"2a00:1450:4014:80c::200e\"],\"TTLs\":[5.0,5.0],\"rejected\":false}","source.type":"error","timestamp":1532168129001}]]2018-07-21 14:45:29.002 o.a.s.d.executor [ERROR] java.lang.IllegalStateException: Unable to parse Message:
{"ts":1532184126.434496,"uid":"CIYoFN1y9Q3D2iJVnc","id.orig_h":"192.168.181.130","id.orig_p":49483,"id.resp_h":"192.168.181.2","id.resp_p":53,"proto":"udp","trans_id":24981,"query":"safebrowsing-cache.google.com","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":false,"RA":true,"Z":0,"answers":["safebrowsing.cache.l.google.com","2a00:1450:4014:80c::200e"],"TTLs":[5.0,5.0],"rejected":false}  at
org.apache.metron.parsers.bro.BasicBroParser.parse(BasicBroParser.java:145)
~[stormjar.jar:?]  at
org.apache.metron.parsers.interfaces.MessageParser.parseOptional(MessageParser.java:45)
~[stormjar.jar:?]  at
org.apache.metron.parsers.bolt.ParserBolt.execute(ParserBolt.java:177)
[stormjar.jar:?]  at
org.apache.storm.daemon.executor$fn__6571$tuple_action_fn__6573.invoke(executor.clj:734)
[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]  at
org.apache.storm.daemon.executor$mk_task_receiver$fn__6492.invoke(executor.clj:466)
[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]  at
org.apache.storm.disruptor$clojure_handler$reify__6005.onEvent(disruptor.clj:40)
[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]  at
org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451)
[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]  at
org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430)
[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]  at
org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]  at
org.apache.storm.daemon.executor$fn__6571$fn__6584$fn__6637.invoke(executor.clj:853)
[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]  at
org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484)
[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]  at
clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]  at
java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]Caused by: java.lang.ClassCastException: java.lang.Boolean
cannot be cast to org.json.simple.JSONObject  at
org.apache.metron.parsers.bro.BasicBroParser.parse(BasicBroParser.java:88)
~[stormjar.jar:?]  ... 12 more

Do I need to make any change in the parser config? Any help would be appreciated.

drodriguez · ‎10-22-2018

I have this exact same problem.

mustafakmal · ‎11-06-2018

Were you able to solve this? Having the same exact problem

drodriguez · ‎11-06-2018

@Mustafa Akmal

I was able to solve this by adding a simple configuration on local.bro:

redef Kafka::tag_json = T;

mustafakmal · ‎11-06-2018

Hey. Thanks for the help. I added this to the local.bro file but now getting this error.

drodriguez · ‎11-14-2018

Could you share your local.bro?