Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Error in enrichment threatinteljoinbolt

Highlighted

Error in enrichment threatinteljoinbolt

Explorer

After metron running ,always show up this error in storm UI likes screenshot below:

13435-1.pngThen i get information below from enrichment log:

2017-03-09 10:22:12.451 o.a.m.e.b.JoinBolt [ERROR] [Metron] Unable to join messages: {"adapter.threatinteladapter.end.ts":"1489026132420","adapter.threatinteladapter.begin.ts":"1489026132420","source.type":"snort"} org.apache.metron.common.dsl.ParseException: Unable to pop an empty stack at org.apache.metron.common.stellar.StellarCompiler.popStack(StellarCompiler.java:421) ~[stormjar.jar:?] at org.apache.metron.common.stellar.StellarCompiler.exitNotFunc(StellarCompiler.java:171) ~[stormjar.jar:?] at org.apache.metron.common.stellar.generated.StellarParser$NotFuncContext.exitRule(StellarParser.java:458) ~[stormjar.jar:?] at org.antlr.v4.runtime.Parser.triggerExitRuleEvent(Parser.java:422) ~[stormjar.jar:?] at org.antlr.v4.runtime.Parser.exitRule(Parser.java:632) ~[stormjar.jar:?] at org.apache.metron.common.stellar.generated.StellarParser.transformation(StellarParser.java:158) ~[stormjar.jar:?] at org.apache.metron.common.stellar.BaseStellarProcessor.parse(BaseStellarProcessor.java:92) ~[stormjar.jar:?] at org.apache.metron.common.stellar.StellarPredicateProcessor.parse(StellarPredicateProcessor.java:54) ~[stormjar.jar:?] at org.apache.metron.threatintel.triage.ThreatTriageProcessor.apply(ThreatTriageProcessor.java:58) ~[stormjar.jar:?] at org.apache.metron.enrichment.bolt.ThreatIntelJoinBolt.joinMessages(ThreatIntelJoinBolt.java:133) ~[stormjar.jar:?] at org.apache.metron.enrichment.bolt.ThreatIntelJoinBolt.joinMessages(ThreatIntelJoinBolt.java:38) ~[stormjar.jar:?] at org.apache.metron.enrichment.bolt.JoinBolt.execute(JoinBolt.java:113) [stormjar.jar:?] at backtype.storm.daemon.executor$fn__6259$tuple_action_fn__6261.invoke(executor.clj:684) [storm-core-0.10.0.2.4.3.0-227.jar:0.10.0.2.4.3.0-227] at backtype.storm.daemon.executor$mk_task_receiver$fn__6182.invoke(executor.clj:431) [storm-core-0.10.0.2.4.3.0-227.jar:0.10.0.2.4.3.0-227] at backtype.storm.disruptor$clojure_handler$reify__4313.onEvent(disruptor.clj:58) [storm-core-0.10.0.2.4.3.0-227.jar:0.10.0.2.4.3.0-227] at backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:125) [storm-core-0.10.0.2.4.3.0-227.jar:0.10.0.2.4.3.0-227] at backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:99) [storm-core-0.10.0.2.4.3.0-227.jar:0.10.0.2.4.3.0-227] at backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80) [storm-core-0.10.0.2.4.3.0-227.jar:0.10.0.2.4.3.0-227] at backtype.storm.daemon.executor$fn__6259$fn__6272$fn__6323.invoke(executor.clj:813) [storm-core-0.10.0.2.4.3.0-227.jar:0.10.0.2.4.3.0-227] at backtype.storm.util$async_loop$fn__545.invoke(util.clj:479) [storm-core-0.10.0.2.4.3.0-227.jar:0.10.0.2.4.3.0-227] at clojure.lang.AFn.run(AFn.java:22) [clojure-1.6.0.jar:?] at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77] 2017-03-09 10:22:12.452 b.s.d.executor [ERROR] org.apache.metron.common.dsl.ParseException: Unable to pop an empty stack at org.apache.metron.common.stellar.StellarCompiler.popStack(StellarCompiler.java:421) ~[stormjar.jar:?] at org.apache.metron.common.stellar.StellarCompiler.exitNotFunc(StellarCompiler.java:171) ~[stormjar.jar:?] at org.apache.metron.common.stellar.generated.StellarParser$NotFuncContext.exitRule(StellarParser.java:458) ~[stormjar.jar:?] at org.antlr.v4.runtime.Parser.triggerExitRuleEvent(Parser.java:422) ~[stormjar.jar:?] at org.antlr.v4.runtime.Parser.exitRule(Parser.java:632) ~[stormjar.jar:?] at org.apache.metron.common.stellar.generated.StellarParser.transformation(StellarParser.java:158) ~[stormjar.jar:?] at org.apache.metron.common.stellar.BaseStellarProcessor.parse(BaseStellarProcessor.java:92) ~[stormjar.jar:?] at org.apache.metron.common.stellar.StellarPredicateProcessor.parse(StellarPredicateProcessor.java:54) ~[stormjar.jar:?] at org.apache.metron.threatintel.triage.ThreatTriageProcessor.apply(ThreatTriageProcessor.java:58) ~[stormjar.jar:?] at org.apache.metron.enrichment.bolt.ThreatIntelJoinBolt.joinMessages(ThreatIntelJoinBolt.java:133) ~[stormjar.jar:?] at org.apache.metron.enrichment.bolt.ThreatIntelJoinBolt.joinMessages(ThreatIntelJoinBolt.java:38) ~[stormjar.jar:?] at org.apache.metron.enrichment.bolt.JoinBolt.execute(JoinBolt.java:113) [stormjar.jar:?] at backtype.storm.daemon.executor$fn__6259$tuple_action_fn__6261.invoke(executor.clj:684) [storm-core-0.10.0.2.4.3.0-227.jar:0.10.0.2.4.3.0-227] at backtype.storm.daemon.executor$mk_task_receiver$fn__6182.invoke(executor.clj:431) [storm-core-0.10.0.2.4.3.0-227.jar:0.10.0.2.4.3.0-227] at backtype.storm.disruptor$clojure_handler$reify__4313.onEvent(disruptor.clj:58) [storm-core-0.10.0.2.4.3.0-227.jar:0.10.0.2.4.3.0-227] at backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:125) [storm-core-0.10.0.2.4.3.0-227.jar:0.10.0.2.4.3.0-227] at backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:99) [storm-core-0.10.0.2.4.3.0-227.jar:0.10.0.2.4.3.0-227] at backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80) [storm-core-0.10.0.2.4.3.0-227.jar:0.10.0.2.4.3.0-227] at backtype.storm.daemon.executor$fn__6259$fn__6272$fn__6323.invoke(executor.clj:813) [storm-core-0.10.0.2.4.3.0-227.jar:0.10.0.2.4.3.0-227] at backtype.storm.util$async_loop$fn__545.invoke(util.clj:479) [storm-core-0.10.0.2.4.3.0-227.jar:0.10.0.2.4.3.0-227] at clojure.lang.AFn.run(AFn.java:22) [clojure-1.6.0.jar:?] at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77] 2017-03-09 10:22:54.899 b.s.m.n.Server [INFO] Getting metrics for server on port 6704 2017-03-09 10:23:03.473 s.k.ZkCoordinator [INFO] Task [1/1] Refreshing partition manager connections 2017-03-09 10:23:03.475 s.k.DynamicBrokersReader [INFO] Read partition info from zookeeper: GlobalPartitionInformation{partitionMap={0=node4:6667}} 2017-03-09 10:23:03.475 s.k.KafkaUtils [INFO] Task [1/1] assigned [Partition{host=node4:6667, partition=0}] 2017-03-09 10:23:03.476 s.k.ZkCoordinator [INFO] Task [1/1] Deleted partition managers: [] 2017-03-09 10:23:03.476 s.k.ZkCoordinator [INFO] Task [1/1] New partition managers: [] 2017-03-09 10:23:03.476 s.k.ZkCoordinator [INFO] Task [1/1] Finished refreshing 2017-03-09 10:23:54.899 b.s.m.n.Server [INFO] Getting metrics for server on port 6704 2017-03-09 10:24:03.478 s.k.ZkCoordinator [INFO] Task [1/1] Refreshing partition manager connections 2017-03-09 10:24:03.481 s.k.DynamicBrokersReader [INFO] Read partition info from zookeeper: GlobalPartitionInformation{partitionMap={0=node4:6667}} 2017-03-09 10:24:03.481 s.k.KafkaUtils [INFO] Task [1/1] assigned [Partition{host=node4:6667, partition=0}] 2017-03-09 10:24:03.481 s.k.ZkCoordinator [INFO] Task [1/1] Deleted partition managers: [] 2017-03-09 10:24:03.481 s.k.ZkCoordinator [INFO] Task [1/1] New partition managers: [] 2017-03-09 10:24:03.481 s.k.ZkCoordinator [INFO] Task [1/1] Finished refreshing

how could i do to avoid this error? any help would be greatly appreciated!

3 REPLIES 3
Highlighted

Re: Error in enrichment threatinteljoinbolt

Explorer

@cstella @Jon Zeolla @David Lyle @Ambud Sharma can you take a look for me ?

Re: Error in enrichment threatinteljoinbolt

Explorer

Can you please post your snort enrichment configs in zookeeper? (either the output of a zk_load_configs.sh -m DUMP or what is at $METRON_HOME/config/zookeeper/enrichment/snort.json)

Highlighted

Re: Error in enrichment threatinteljoinbolt

Explorer

Thank you !This is the content in $METRON_HOME/config/zookeeper/enrichment/snort.json

{ "index": "snort", "batchSize": 1, "enrichment" : { "fieldMap": { "geo": ["ip_dst_addr", "ip_src_addr"], "host": ["host"] } }, "threatIntel" : { "fieldMap": { "hbaseThreatIntel": ["ip_src_addr", "ip_dst_addr"] }, "fieldToTypeMap": { "ip_src_addr" : ["malicious_ip"], "ip_dst_addr" : ["malicious_ip"] }, "triageConfig" : { "riskLevelRules" : { "not(IN_SUBNET(ip_dst_addr, '192.168.0.0/24'))" : 10 }, "aggregator" : "MAX" } } }

Don't have an account?
Coming from Hortonworks? Activate your account here