Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Metron new Parser Topology

Metron new Parser Topology

New Contributor

Hello community,

I have deployed Apache Metron in a single node Vagrant VM. Everything works as expected so far (I'm able to read Bro, Snort, Yaf sensors, enrich the data and index it). However, now I have Suricata data streaming into Metron. Suricata data is being queue at Metron's Kafka for parsing. Furthermore, I have a Logstash that filters Suricata data and writes JSON messages to Kafka.

See an example of a Kafka message (as inputing to the parser topology)

{"in_packets":17,"data_type":"suricata_event","in_bytes":13055,"dvc_type":"suricata","conn_state":"closed","out_packets":17,"@version":"1","id":"9dcca116-86e5-4044-b033-8bdb7053bd7e","ip_dst_addr":"10.5.0.14","nproto":"TCP","raw":"{\"tcp\":{\"fin\":true,\"tcp_flags_tc\":\"1b\",\"ack\":true,\"syn\":true,\"tcp_flags\":\"1b\",\"psh\":true,\"state\":\"closed\",\"tcp_flags_ts\":\"1b\"},\"proto\":\"TCP\",\"source\":\"/var/log/suricata/eve.json\",\"data_type\":\"suricata_event\",\"tags\":[\"beats_input_codec_json_applied\"],\"dest_port\":6690,\"dvc_type\":\"suricata\",\"beat\":{\"name\":\"redbox1\",\"version\":\"6.2.3\",\"hostname\":\"redbox1\"},\"@version\":\"1\",\"dvc_host\":\"redbox1\",\"src_ip\":\"10.5.0.148\",\"src_port\":46560,\"timestamp\":\"2018-04-11T12:44:22.003733+0000\",\"@timestamp\":\"2018-04-11T12:55:05.147Z\",\"prospector\":{\"type\":\"log\"},\"app_proto\":\"failed\",\"offset\":20805212,\"dest_ip\":\"10.5.0.14\",\"host\":\"redbox1\",\"flow_id\":2078884479188050,\"event_type\":\"flow\",\"flow\":{\"bytes_toserver\":4001,\"age\":0,\"end\":\"2018-04-11T12:43:21.821489+0000\",\"start\":\"2018-04-11T12:43:21.670802+0000\",\"reason\":\"timeout\",\"alerted\":false,\"bytes_toclient\":13055,\"pkts_toclient\":17,\"state\":\"closed\",\"pkts_toserver\":17}}","additional_atts":"{\"reason\":\"timeout\"}","duration":0,"ip_src_port":46560,"ip_dst_port":6690,"@timestamp":"2018-04-12T12:21:10.192Z","ip_src_addr":"10.5.0.148","eventid":2078884479188050,"out_bytes":4001,"event_type":"flow","dvc_time":1523450662003,"eventtime":1523450601670,"aproto":"failed"}

Then, I have defined my "suricata" Sensor in Metron's Management UI using a "JSONMap" Parser and a sample message is correctly parsed.

When I try to start the suricata sensor, it does not start correctly. I tried both using the Metron Management UI and the command line:

[vagrant@node1 /]$ sudo sh usr/metron/0.4.3/bin/start_parser_topology.sh -k $KAFKA_HOST:6667 -z $ZOOKEEPER_HOST:2181 -s suricata

After this, I get this error:

Running: /usr/jdk64/jdk1.8.0_112/bin/java -server -Ddaemon.name= -Dstorm.options= -Dstorm.home=/usr/hdp/2.6.4.0-91/storm -Dstorm.log.dir=/var/log/storm -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -Dstorm.conf.file= -cp /usr/hdp/2.6.4.0-91/storm/lib/log4j-core-2.8.2.jar:/usr/hdp/2.6.4.0-91/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.6.4.0-91/storm/lib/reflectasm-1.10.1.jar:/usr/hdp/2.6.4.0-91/storm/lib/objenesis-2.1.jar:/usr/hdp/2.6.4.0-91/storm/lib/asm-5.0.3.jar:/usr/hdp/2.6.4.0-91/storm/lib/disruptor-3.3.2.jar:/usr/hdp/2.6.4.0-91/storm/lib/storm-core-1.1.0.2.6.4.0-91.jar:/usr/hdp/2.6.4.0-91/storm/lib/slf4j-api-1.7.21.jar:/usr/hdp/2.6.4.0-91/storm/lib/log4j-slf4j-impl-2.8.2.jar:/usr/hdp/2.6.4.0-91/storm/lib/log4j-api-2.8.2.jar:/usr/hdp/2.6.4.0-91/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.6.4.0-91/storm/lib/ring-cors-0.1.5.jar:/usr/hdp/2.6.4.0-91/storm/lib/clojure-1.7.0.jar:/usr/hdp/2.6.4.0-91/storm/lib/storm-rename-hack-1.1.0.2.6.4.0-91.jar:/usr/hdp/2.6.4.0-91/storm/lib/kryo-3.0.3.jar:/usr/hdp/2.6.4.0-91/storm/lib/zookeeper.jar:/usr/hdp/2.6.4.0-91/storm/lib/minlog-1.3.0.jar org.apache.storm.daemon.ClientJarTransformerRunner org.apache.storm.hack.StormShadeTransformer /usr/metron/0.4.3/lib/metron-parsers-0.4.3-uber.jar /tmp/077916523e5311e8b7ea525400e0ddcb.jar
Running: /usr/jdk64/jdk1.8.0_112/bin/java -Ddaemon.name= -Dstorm.options= -Dstorm.home=/usr/hdp/2.6.4.0-91/storm -Dstorm.log.dir=/var/log/storm -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib:/usr/hdp/current/storm-client/lib -Dstorm.conf.file= -cp /usr/hdp/2.6.4.0-91/storm/lib/log4j-core-2.8.2.jar:/usr/hdp/2.6.4.0-91/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.6.4.0-91/storm/lib/reflectasm-1.10.1.jar:/usr/hdp/2.6.4.0-91/storm/lib/objenesis-2.1.jar:/usr/hdp/2.6.4.0-91/storm/lib/asm-5.0.3.jar:/usr/hdp/2.6.4.0-91/storm/lib/disruptor-3.3.2.jar:/usr/hdp/2.6.4.0-91/storm/lib/storm-core-1.1.0.2.6.4.0-91.jar:/usr/hdp/2.6.4.0-91/storm/lib/slf4j-api-1.7.21.jar:/usr/hdp/2.6.4.0-91/storm/lib/log4j-slf4j-impl-2.8.2.jar:/usr/hdp/2.6.4.0-91/storm/lib/log4j-api-2.8.2.jar:/usr/hdp/2.6.4.0-91/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.6.4.0-91/storm/lib/ring-cors-0.1.5.jar:/usr/hdp/2.6.4.0-91/storm/lib/clojure-1.7.0.jar:/usr/hdp/2.6.4.0-91/storm/lib/storm-rename-hack-1.1.0.2.6.4.0-91.jar:/usr/hdp/2.6.4.0-91/storm/lib/kryo-3.0.3.jar:/usr/hdp/2.6.4.0-91/storm/lib/zookeeper.jar:/usr/hdp/2.6.4.0-91/storm/lib/minlog-1.3.0.jar:/tmp/077916523e5311e8b7ea525400e0ddcb.jar:/usr/hdp/current/storm-supervisor/conf:/usr/hdp/2.6.4.0-91/storm/bin -Dstorm.jar=/tmp/077916523e5311e8b7ea525400e0ddcb.jar -Dstorm.dependency.jars= -Dstorm.dependency.artifacts={} org.apache.metron.parsers.topology.ParserTopologyCLI -k :6667 -z :2181 -s suricata
993  [main] INFO  o.a.c.f.i.CuratorFrameworkImpl - Starting
1105 [main-EventThread] INFO  o.a.c.f.s.ConnectionStateManager - State change: CONNECTED
1803 [main] INFO  o.a.c.f.i.CuratorFrameworkImpl - Starting
1806 [main-EventThread] INFO  o.a.c.f.s.ConnectionStateManager - State change: CONNECTED
java.lang.IllegalStateException: Bolt 'parserBolt' contains a non-serializable field of type org.apache.metron.parsers.json.JSONMapParser$1, which was instantiated prior to topology creation. org.apache.metron.parsers.json.JSONMapParser$1 should be instantiated within the prepare method of 'parserBolt at the earliest.
	at org.apache.storm.topology.TopologyBuilder.createTopology(TopologyBuilder.java:128)
	at org.apache.metron.parsers.topology.ParserTopologyCLI.main(ParserTopologyCLI.java:459)
Caused by: java.lang.RuntimeException: java.io.NotSerializableException: org.apache.metron.parsers.json.JSONMapParser$1
	at org.apache.storm.utils.Utils.javaSerialize(Utils.java:238)
	at org.apache.storm.topology.TopologyBuilder.createTopology(TopologyBuilder.java:123)
	... 1 more
Caused by: java.io.NotSerializableException: org.apache.metron.parsers.json.JSONMapParser$1
	at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1184)
	at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
	at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
	at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
	at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
	at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
	at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
	at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
	at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
	at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
	at org.apache.storm.utils.Utils.javaSerialize(Utils.java:234)
	... 2 more
2 REPLIES 2

Re: Metron new Parser Topology

Contributor

By default the vagrant vm doesn't have enough storm slots open to add new parsers to it.

You are going to want to verify that, and either stop other parsers or add slots in the storm configuration in ambari.

Highlighted

Re: Metron new Parser Topology

Contributor

The bug you see in your log was just fixed in master: https://github.com/apache/metron/pull/991