Member since
10-11-2016
29
Posts
1
Kudos Received
4
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 1131 | 11-21-2016 02:29 PM | |
| 1802 | 11-04-2016 01:12 PM | |
| 1781 | 10-18-2016 09:21 AM | |
| 731 | 10-14-2016 01:05 PM |
10-17-2016
02:08 PM
I am running the full-dev-platform of Metron version 0.2.0BETA and have added squid log data as per wiki guide and some help from @cduby with some configuration issues. Now trying to extend this to add threat intelligence alerting based upon wiki guide but am having issues with no enrichment data being added from the HBase table containing the CSV data, the geo enrichments are being added to the data however. Also the url in the example elasticsearch index is shown as "atmape.ru" and in my index it shows as "http://www.atmape.ru". Enrichment config for squid is in zookeeper as below; ENRICHMENT Config: squid
{
"index" : "squid",
"batchSize" : 5,
"enrichment" : {
"fieldMap" : {
"geo" : [ "ip_dst_addr", "ip_src_addr" ],
"host" : [ "host" ]
},
"fieldToTypeMap" : { },
"config" : { }
},
"threatIntel" : {
"fieldMap" : {
"hbaseThreatIntel" : [ "ip_src_addr", "ip_dst_addr", "url" ]
},
"fieldToTypeMap" : {
"ip_src_addr" : [ "malicious_ip" ],
"ip_dst_addr" : [ "malicious_ip" ],
"url" : [ "zeusList" ]
},
"config" : { },
"triageConfig" : {
"riskLevelRules" : { },
"aggregator" : "MAX",
"aggregationConfig" : { }
}
},
"configuration" : { }
}
... View more
Labels:
- Labels:
-
Apache Metron
10-17-2016
10:36 AM
@cduby Thanks for all your help along the way I think I am finally up and running now. Found the issue with the enrichments, it was that the squid logs I had generated were missing the destination IP address, once I regenerated these, cleared the kafka queues and restarted the topologies the data started flowing through into elastic index. Then to get around the timestamp issue I had to curl in a template to elastic to create a template for the squid data with the timestamp field specified as a date as below; curl -XPUT http://node1:9200/_template/squid -d '{"template":"squid*","mappings": {"squid*": {"properties": {"timestamp": { "type": "date" }}}}}'
... View more
10-14-2016
01:05 PM
In case anyone has the same issue I resolved by manually starting flume via shell access to the node; /usr/hdp/current/flume-server/bin/flume-ng agent -n snort -c /usr/hdp/current/flume-server/conf -f /usr/hdp/current/flume-server/conf/flume-snort.conf
... View more
10-14-2016
09:24 AM
@cduby Thanks for that have removed the historical index from elastic so I now have no squid indexes, however I am back to the previous problem with the enrichmentJoinBolt, I have checked and MySQL is running and no errors are showing in the geoEnrichmentBolt, I am also getting data from bro and yaf showing up in elastic. Have worked through the troubleshooting article and cannot see any problems, the only thing I can think of is that the enrichment config I have provided has something incorrect in it although I cannot see what. Pasted below is the error in the Storm geo enrichment bolt and a few of the logs either side, there are no other errors in Storm UI. 2016-10-14 09:11:47 b.s.d.executor [INFO] Prepared bolt simpleHBaseEnrichmentBolt:(8)
2016-10-14 09:12:07 o.a.m.e.b.JoinBolt [ERROR] [Metron] Unable to join messages: {"enrichments.geo.ip_dst_addr":"","adapter.geoadapter.end.ts":"1476436327576","enrichments.geo.ip_src_addr":"","adapter.geoadapter.begin.ts":"1476436327576","source.type":"squid"}
java.lang.NullPointerException: null
at org.apache.metron.enrichment.bolt.EnrichmentJoinBolt.joinMessages(EnrichmentJoinBolt.java:76) ~[stormjar.jar:na]
at org.apache.metron.enrichment.bolt.EnrichmentJoinBolt.joinMessages(EnrichmentJoinBolt.java:33) ~[stormjar.jar:na]
at org.apache.metron.enrichment.bolt.JoinBolt.execute(JoinBolt.java:111) ~[stormjar.jar:na]
at backtype.storm.daemon.executor$fn__7014$tuple_action_fn__7016.invoke(executor.clj:670) [storm-core-0.10.0.2.3.0.0-2557.jar:0.10.0.2.3.0.0-2557]
at backtype.storm.daemon.executor$mk_task_receiver$fn__6937.invoke(executor.clj:426) [storm-core-0.10.0.2.3.0.0-2557.jar:0.10.0.2.3.0.0-2557]
at backtype.storm.disruptor$clojure_handler$reify__6513.onEvent(disruptor.clj:58) [storm-core-0.10.0.2.3.0.0-2557.jar:0.10.0.2.3.0.0-2557]
at backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:125) [storm-core-0.10.0.2.3.0.0-2557.jar:0.10.0.2.3.0.0-2557]
at backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:99) [storm-core-0.10.0.2.3.0.0-2557.jar:0.10.0.2.3.0.0-2557]
at backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80) [storm-core-0.10.0.2.3.0.0-2557.jar:0.10.0.2.3.0.0-2557]
at backtype.storm.daemon.executor$fn__7014$fn__7027$fn__7078.invoke(executor.clj:808) [storm-core-0.10.0.2.3.0.0-2557.jar:0.10.0.2.3.0.0-2557]
at backtype.storm.util$async_loop$fn__545.invoke(util.clj:475) [storm-core-0.10.0.2.3.0.0-2557.jar:0.10.0.2.3.0.0-2557]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.6.0.jar:na]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_40]
2016-10-14 09:12:07 b.s.d.executor [ERROR]
java.lang.NullPointerException: null
at org.apache.metron.enrichment.bolt.EnrichmentJoinBolt.joinMessages(EnrichmentJoinBolt.java:76) ~[stormjar.jar:na]
at org.apache.metron.enrichment.bolt.EnrichmentJoinBolt.joinMessages(EnrichmentJoinBolt.java:33) ~[stormjar.jar:na]
at org.apache.metron.enrichment.bolt.JoinBolt.execute(JoinBolt.java:111) ~[stormjar.jar:na]
at backtype.storm.daemon.executor$fn__7014$tuple_action_fn__7016.invoke(executor.clj:670) [storm-core-0.10.0.2.3.0.0-2557.jar:0.10.0.2.3.0.0-2557]
at backtype.storm.daemon.executor$mk_task_receiver$fn__6937.invoke(executor.clj:426) [storm-core-0.10.0.2.3.0.0-2557.jar:0.10.0.2.3.0.0-2557]
at backtype.storm.disruptor$clojure_handler$reify__6513.onEvent(disruptor.clj:58) [storm-core-0.10.0.2.3.0.0-2557.jar:0.10.0.2.3.0.0-2557]
at backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:125) [storm-core-0.10.0.2.3.0.0-2557.jar:0.10.0.2.3.0.0-2557]
at backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:99) [storm-core-0.10.0.2.3.0.0-2557.jar:0.10.0.2.3.0.0-2557]
at backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80) [storm-core-0.10.0.2.3.0.0-2557.jar:0.10.0.2.3.0.0-2557]
at backtype.storm.daemon.executor$fn__7014$fn__7027$fn__7078.invoke(executor.clj:808) [storm-core-0.10.0.2.3.0.0-2557.jar:0.10.0.2.3.0.0-2557]
at backtype.storm.util$async_loop$fn__545.invoke(util.clj:475) [storm-core-0.10.0.2.3.0.0-2557.jar:0.10.0.2.3.0.0-2557]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.6.0.jar:na]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_40]
2016-10-14 09:12:42 b.s.m.n.Server [INFO] Getting metrics for server on port 6703
2016-10-14 09:12:45 s.k.ZkCoordinator [INFO] Task [1/1] Refreshing partition manager connections
2016-10-14 09:12:45 s.k.DynamicBrokersReader [INFO] Read partition info from zookeeper: GlobalPartitionInformation{partitionMap={0=node1:6667}}
... View more
10-13-2016
02:27 PM
Have setup an instance of Metron in a single instance VM, bro and yaf data is flowing through into elastic search indexes, however there seems to be an error with flume starting up to ingest the snort logs. I am getting the below error in the flume logs, however I cannot see a reference to a /snort folder in the flume-snort.conf file. 13 Oct 2016 14:06:33,238 ERROR [main] (org.apache.flume.node.Application.main:307) - A fatal error occurred while running. Exception follows.
org.apache.commons.cli.ParseException: The specified configuration file does not exist: /snort
Any help would be greatly appreciated.
... View more
Labels:
- Labels:
-
Apache Flume
-
Apache Metron
10-13-2016
08:40 AM
Fixed the storm issue it was to do with a backup I took of the storm local data directory when I was having problems starting the storm supervisor. I restored the nimbus/stormdist data from my backup and it is started up correctly now. So next step is to look at the indexing in elasticsearch.
... View more
10-13-2016
08:36 AM
Fixed now thanks, the issue was that I had backed up my storm local data previously because of an issue with the supervisor starting. Once I copied the nimbus/stormdist folder back into the storm local data folder and started nimbus again it all came up correctly.
... View more
10-13-2016
08:12 AM
Actually I have found a section further up in the log that might be the actual error; 2016-10-13T08:08:54.289+0000 b.s.zookeeper [INFO] node1 gained leadership, checking if it has all the topology code locally.
2016-10-13T08:08:54.297+0000 b.s.zookeeper [INFO] active-topology-ids [yaf-1-1476261289,bro-11-1476195353,squid-15-1476196296,enrichment-17-1476215724,snort-13-1476195443] local-topology-ids [enrichment-10-1476302158,bro-8-1476302148,yaf-7-1476302143,snort-9-1476302153] diff-topology [yaf-1-1476261289,bro-11-1476195353,squid-15-1476196296,enrichment-17-1476215724,snort-13-1476195443]
2016-10-13T08:08:54.299+0000 b.s.zookeeper [INFO] code for all active topologies not available locally, giving up leadership.
... View more
10-13-2016
08:04 AM
@Santhosh B Gowda Thanks for your response. Nimbus is showing as running without errors in Ambari and the service looks to be up correctly. I am getting an error in the nimbus.log as below; 2016-10-13T08:00:58.228+0000 o.a.t.s.AbstractNonblockingServer$FrameBuffer [ERROR] Unexpected throwable while invoking!
java.lang.RuntimeException: No nimbus leader participant host found, have you started your nimbus hosts?
at backtype.storm.zookeeper$to_NimbusInfo.invoke(zookeeper.clj:233) ~[storm-core-0.10.0.2.3.0.0-2557.jar:0.10.0.2.3.0.0-2557]
at backtype.storm.zookeeper$zk_leader_elector$reify__1009.getLeader(zookeeper.clj:305) ~[storm-core-0.10.0.2.3.0.0-2557.jar:0.10.0.2.3.0.0-2557]
at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source) ~[na:na]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_40]
at java.lang.reflect.Method.invoke(Method.java:497) ~[na:1.8.0_40]
at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93) ~[clojure-1.6.0.jar:na]
at clojure.lang.Reflector.invokeNoArgInstanceMember(Reflector.java:313) ~[clojure-1.6.0.jar:na]
at backtype.storm.daemon.nimbus$fn__6231$exec_fn__1296__auto__$reify__6250.getClusterInfo(nimbus.clj:1349) ~[storm-core-0.10.0.2.3.0.0-2557.jar:0.10.0.2.3.0.0-2557]
at backtype.storm.generated.Nimbus$Processor$getClusterInfo.getResult(Nimbus.java:1812) ~[storm-core-0.10.0.2.3.0.0-2557.jar:0.10.0.2.3.0.0-2557]
at backtype.storm.generated.Nimbus$Processor$getClusterInfo.getResult(Nimbus.java:1796) ~[storm-core-0.10.0.2.3.0.0-2557.jar:0.10.0.2.3.0.0-2557]
at org.apache.thrift7.ProcessFunction.process(ProcessFunction.java:39) ~[storm-core-0.10.0.2.3.0.0-2557.jar:0.10.0.2.3.0.0-2557]
at org.apache.thrift7.TBaseProcessor.process(TBaseProcessor.java:39) ~[storm-core-0.10.0.2.3.0.0-2557.jar:0.10.0.2.3.0.0-2557]
at backtype.storm.security.auth.SimpleTransportPlugin$SimpleWrapProcessor.process(SimpleTransportPlugin.java:159) ~[storm-core-0.10.0.2.3.0.0-2557.jar:0.10.0.2.3.0.0-2557]
at org.apache.thrift7.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:518) ~[storm-core-0.10.0.2.3.0.0-2557.jar:0.10.0.2.3.0.0-2557]
at org.apache.thrift7.server.Invocation.run(Invocation.java:18) [storm-core-0.10.0.2.3.0.0-2557.jar:0.10.0.2.3.0.0-2557]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_40]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_40]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_40]
Then the below is occurring in the ui.log 2016-10-13T08:02:08.239+0000 b.s.u.NimbusClient [WARN] Ignoring exception while trying to get leader nimbus info from node1. will retry with a different seed host.
org.apache.thrift7.transport.TTransportException: null
at org.apache.thrift7.transport.TIOStreamTransport.read(TIOStreamTransport.java:132) ~[storm-core-0.10.0.2.3.0.0-2557.jar:0.10.0.2.3.0.0-2557]
at org.apache.thrift7.transport.TTransport.readAll(TTransport.java:86) ~[storm-core-0.10.0.2.3.0.0-2557.jar:0.10.0.2.3.0.0-2557]
at org.apache.thrift7.transport.TFramedTransport.readFrame(TFramedTransport.java:129) ~[storm-core-0.10.0.2.3.0.0-2557.jar:0.10.0.2.3.0.0-2557]
at org.apache.thrift7.transport.TFramedTransport.read(TFramedTransport.java:101) ~[storm-core-0.10.0.2.3.0.0-2557.jar:0.10.0.2.3.0.0-2557]
at org.apache.thrift7.transport.TTransport.readAll(TTransport.java:86) ~[storm-core-0.10.0.2.3.0.0-2557.jar:0.10.0.2.3.0.0-2557]
at org.apache.thrift7.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429) ~[storm-core-0.10.0.2.3.0.0-2557.jar:0.10.0.2.3.0.0-2557]
at org.apache.thrift7.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318) ~[storm-core-0.10.0.2.3.0.0-2557.jar:0.10.0.2.3.0.0-2557]
at org.apache.thrift7.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219) ~[storm-core-0.10.0.2.3.0.0-2557.jar:0.10.0.2.3.0.0-2557]
at org.apache.thrift7.TServiceClient.receiveBase(TServiceClient.java:69) ~[storm-core-0.10.0.2.3.0.0-2557.jar:0.10.0.2.3.0.0-2557]
at backtype.storm.generated.Nimbus$Client.recv_getClusterInfo(Nimbus.java:559) ~[storm-core-0.10.0.2.3.0.0-2557.jar:0.10.0.2.3.0.0-2557]
at backtype.storm.generated.Nimbus$Client.getClusterInfo(Nimbus.java:547) ~[storm-core-0.10.0.2.3.0.0-2557.jar:0.10.0.2.3.0.0-2557]
at backtype.storm.utils.NimbusClient.getConfiguredClientAs(NimbusClient.java:68) ~[storm-core-0.10.0.2.3.0.0-2557.jar:0.10.0.2.3.0.0-2557]
at backtype.storm.ui.core$nimbus_summary.invoke(core.clj:580) [storm-core-0.10.0.2.3.0.0-2557.jar:0.10.0.2.3.0.0-2557]
at backtype.storm.ui.core$fn__10249.invoke(core.clj:982) [storm-core-0.10.0.2.3.0.0-2557.jar:0.10.0.2.3.0.0-2557]
at compojure.core$make_route$fn__1889.invoke(core.clj:93) [storm-core-0.10.0.2.3.0.0-2557.jar:0.10.0.2.3.0.0-2557]
at compojure.core$if_route$fn__1877.invoke(core.clj:39) [storm-core-0.10.0.2.3.0.0-2557.jar:0.10.0.2.3.0.0-2557]
at compojure.core$if_method$fn__1870.invoke(core.clj:24) [storm-core-0.10.0.2.3.0.0-2557.jar:0.10.0.2.3.0.0-2557]
at compojure.core$routing$fn__1895.invoke(core.clj:106) [storm-core-0.10.0.2.3.0.0-2557.jar:0.10.0.2.3.0.0-2557]
... View more
10-13-2016
07:57 AM
@cduby Yes that is the tutorial I am following, however I have another problem now with the Storm UI throwing an error, even though Ambari is showing that all the Storm components are running fine. java.lang.RuntimeException: Could not find leader nimbus from seed hosts ["node1"]. Did you specify a valid list of nimbus hosts for config nimbus.seeds I have posted it in a separate topic as I think the enrichment issue is now resolved as I can see index data in elastic. How did you delete the index? Was that through elastic, and was it just a case of pushing the logs to the kafka topic that created the new index and resolved the issue. Thanks again for all your help, really appreciate it.
... View more
- « Previous
-
- 1
- 2
- Next »