Created 07-23-2018 11:39 AM
I am using Apache Metron to push Bro logs to the Apache Kafka using the following link (metron-bro-plugin-kafka):
https://github.com/apache/metron-bro-plugin-kafka
I installed Bro on a CentOS VM and able to receive data in Kafka on the server side. The data then send to the Apache Storm. There is a problem inside the Storm which it seems that it cannot parse the received log and cast it to a JSONObject. Short description of the error is as follow:
java.lang.IllegalStateException: Unable to parse Message: {"ts":1532184126.429716,"uid":"CIYoFN1y9Q3D2iJVnc","id.orig_h":"192.168.181.130","id.orig_p":49483,"id.resp_h":"192.168.181.2","id.resp_p":53,
The relevant part of the log is as follow:
2018-07-21 14:45:29.002 o.a.s.d.executor [INFO] TRANSFERING tuple [dest: 3 tuple: source: parserBolt:5, stream: error, id: {}, [{"exception":"java.lang.IllegalStateException: Unable to parse Message: {\"ts\":1532184126.434496,\"uid\":\"CIYoFN1y9Q3D2iJVnc\",\"id.orig_h\":\"192.168.181.130\",\"id.orig_p\":49483,\"id.resp_h\":\"192.168.181.2\",\"id.resp_p\":53,\"proto\":\"udp\",\"trans_id\":24981,\"query\":\"safebrowsing-cache.google.com\",\"rcode\":0,\"rcode_name\":\"NOERROR\",\"AA\":false,\"TC\":false,\"RD\":false,\"RA\":true,\"Z\":0,\"answers\":[\"safebrowsing.cache.l.google.com\",\"2a00:1450:4014:80c::200e\"],\"TTLs\":[5.0,5.0],\"rejected\":false}","failed_sensor_type":"bro","stack":"java.lang.IllegalStateException: Unable to parse Message: {\"ts\":1532184126.434496,\"uid\":\"CIYoFN1y9Q3D2iJVnc\",\"id.orig_h\":\"192.168.181.130\",\"id.orig_p\":49483,\"id.resp_h\":\"192.168.181.2\",\"id.resp_p\":53,\"proto\":\"udp\",\"trans_id\":24981,\"query\":\"safebrowsing-cache.google.com\",\"rcode\":0,\"rcode_name\":\"NOERROR\",\"AA\":false,\"TC\":false,\"RD\":false,\"RA\":true,\"Z\":0,\"answers\":[\"safebrowsing.cache.l.google.com\",\"2a00:1450:4014:80c::200e\"],\"TTLs\":[5.0,5.0],\"rejected\":false}\n\tat org.apache.metron.parsers.bro.BasicBroParser.parse(BasicBroParser.java:145)\n\tat org.apache.metron.parsers.interfaces.MessageParser.parseOptional(MessageParser.java:45)\n\tat org.apache.metron.parsers.bolt.ParserBolt.execute(ParserBolt.java:177)\n\tat org.apache.storm.daemon.executor$fn__6571$tuple_action_fn__6573.invoke(executor.clj:734)\n\tat org.apache.storm.daemon.executor$mk_task_receiver$fn__6492.invoke(executor.clj:466)\n\tat org.apache.storm.disruptor$clojure_handler$reify__6005.onEvent(disruptor.clj:40)\n\tat org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451)\n\tat org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430)\n\tat org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)\n\tat org.apache.storm.daemon.executor$fn__6571$fn__6584$fn__6637.invoke(executor.clj:853)\n\tat org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484)\n\tat clojure.lang.AFn.run(AFn.java:22)\n\tat java.lang.Thread.run(Thread.java:745)\nCaused by: java.lang.ClassCastException: java.lang.Boolean cannot be cast to org.json.simple.JSONObject\n\tat org.apache.metron.parsers.bro.BasicBroParser.parse(BasicBroParser.java:88)\n\t... 12 more\n","hostname":"localhost","raw_message":"{\"ts\":1532184126.434496,\"uid\":\"CIYoFN1y9Q3D2iJVnc\",\"id.orig_h\":\"192.168.181.130\",\"id.orig_p\":49483,\"id.resp_h\":\"192.168.181.2\",\"id.resp_p\":53,\"proto\":\"udp\",\"trans_id\":24981,\"query\":\"safebrowsing-cache.google.com\",\"rcode\":0,\"rcode_name\":\"NOERROR\",\"AA\":false,\"TC\":false,\"RD\":false,\"RA\":true,\"Z\":0,\"answers\":[\"safebrowsing.cache.l.google.com\",\"2a00:1450:4014:80c::200e\"],\"TTLs\":[5.0,5.0],\"rejected\":false}","error_hash":"5b13edd4ca52633f169c276ac5cf728b46c19de2d02b31cf330f6d640c6bfc86","error_type":"parser_error","message":"Unable to parse Message: {\"ts\":1532184126.434496,\"uid\":\"CIYoFN1y9Q3D2iJVnc\",\"id.orig_h\":\"192.168.181.130\",\"id.orig_p\":49483,\"id.resp_h\":\"192.168.181.2\",\"id.resp_p\":53,\"proto\":\"udp\",\"trans_id\":24981,\"query\":\"safebrowsing-cache.google.com\",\"rcode\":0,\"rcode_name\":\"NOERROR\",\"AA\":false,\"TC\":false,\"RD\":false,\"RA\":true,\"Z\":0,\"answers\":[\"safebrowsing.cache.l.google.com\",\"2a00:1450:4014:80c::200e\"],\"TTLs\":[5.0,5.0],\"rejected\":false}","source.type":"error","timestamp":1532168129001}]]2018-07-21 14:45:29.002 o.a.s.d.executor [ERROR] java.lang.IllegalStateException: Unable to parse Message: {"ts":1532184126.434496,"uid":"CIYoFN1y9Q3D2iJVnc","id.orig_h":"192.168.181.130","id.orig_p":49483,"id.resp_h":"192.168.181.2","id.resp_p":53,"proto":"udp","trans_id":24981,"query":"safebrowsing-cache.google.com","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":false,"RA":true,"Z":0,"answers":["safebrowsing.cache.l.google.com","2a00:1450:4014:80c::200e"],"TTLs":[5.0,5.0],"rejected":false} at org.apache.metron.parsers.bro.BasicBroParser.parse(BasicBroParser.java:145) ~[stormjar.jar:?] at org.apache.metron.parsers.interfaces.MessageParser.parseOptional(MessageParser.java:45) ~[stormjar.jar:?] at org.apache.metron.parsers.bolt.ParserBolt.execute(ParserBolt.java:177) [stormjar.jar:?] at org.apache.storm.daemon.executor$fn__6571$tuple_action_fn__6573.invoke(executor.clj:734) [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245] at org.apache.storm.daemon.executor$mk_task_receiver$fn__6492.invoke(executor.clj:466) [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245] at org.apache.storm.disruptor$clojure_handler$reify__6005.onEvent(disruptor.clj:40) [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245] at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451) [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245] at org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430) [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245] at org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73) [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245] at org.apache.storm.daemon.executor$fn__6571$fn__6584$fn__6637.invoke(executor.clj:853) [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245] at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484) [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245] at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?] at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]Caused by: java.lang.ClassCastException: java.lang.Boolean cannot be cast to org.json.simple.JSONObject at org.apache.metron.parsers.bro.BasicBroParser.parse(BasicBroParser.java:88) ~[stormjar.jar:?] ... 12 more
Do I need to make any change in the parser config? Any help would be appreciated.
Created 10-22-2018 11:08 AM
I have this exact same problem.
Created 11-06-2018 09:23 AM
Were you able to solve this? Having the same exact problem
Created 11-06-2018 12:23 PM
I was able to solve this by adding a simple configuration on local.bro:
redef Kafka::tag_json = T;
Created on 11-06-2018 01:21 PM - edited 08-17-2019 11:24 PM
Hey. Thanks for the help. I added this to the local.bro file but now getting this error.
Created 11-14-2018 12:08 PM
Could you share your local.bro?