Member since
09-26-2016
16
Posts
1
Kudos Received
1
Solution
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 1011 | 05-26-2017 09:17 PM |
07-20-2017
05:33 PM
@Sebastien Lehuede: No, I didn't find the root cause. But in my case, it was fairly easy to use the CSV parser instead of the Grok parser, with slight modifications to my telemetry source. You can find more details about the CSV parser here: https://metron.apache.org/current-book/metron-platform/metron-parsers/index.html Hope this helps.
... View more
05-26-2017
11:48 PM
So, I used 'global' instead of 'my_entity_id' with the same settings. After sometime, I can see parser_score values (though they don't seem to be computed correctly). So the issue with not getting scores seems to be related to how 'my_entity_id' values are parsed/interpreted or my potentially incorrect use of it. I still get the same exception in Stellar shell though. As a side note, all my entities (as visible by elastic search head plugin) have the same value for 'my_entity_id' in my simple test. So there's no semantic difference between using 'global' or 'my_entity_id' as the entity id in the profile. Apart from the problem with the shell, I'm curious about what I'm not doing right when using 'my_entity_id'. In other words, is there anything wrong with defining the profile below? {
"profiles": [
{
"profile": "my_profile_name",
"foreach": "my_entity_id",
"onlyif": "true",
"init" : {
"s": "OUTLIER_MAD_STATE_MERGE(PROFILE_GET('my_profile_name','my_entity_id', PROFILE_FIXED(2, 'MINUTES')))"
},
"update": {
"s": "OUTLIER_MAD_ADD(s, my_double_value)"
},
"result": "s"
}
]
} and/or the following enrichment config: {
"enrichment": {
"fieldMap": {
"stellar" : {
"config" : {
"parser_score" : "OUTLIER_MAD_SCORE(OUTLIER_MAD_STATE_MERGE(PROFILE_GET( 'my_profile_name', 'my_entity_id',
PROFILE_FIXED(5, 'MINUTES')) ), my_double_value)"
,"is_alert" : "if parser_score > 3.5 then true else is_alert"
}
}
}
,"fieldToTypeMap": { }
},
"threatIntel": {
"fieldMap": { },
"fieldToTypeMap": { },
"triageConfig" : {
"riskLevelRules" : [
{
"rule" : "parser_score > 3.5",
"score" : 10
}
],
"aggregator" : "MAX"
}
}
} Thanks in advance for your help!
... View more
05-26-2017
10:51 PM
The enrichment configuration discussed in the this link seems to have an error: {
"index": "mad",
"batchSize": 1,
"enrichment": {
"fieldMap": {
"stellar" : {
"config" : {
"parser_score" : "OUTLIER_MAD_SCORE(OUTLIER_MAD_STATE_MERGE(
PROFILE_GET( 'sketchy_mad', 'global', PROFILE_FIXED(10, 'MINUTES')) ), value)"
,"is_alert" : "if parser_score > 3.5 then true else is_alert"
}
}
}
,"fieldToTypeMap": { }
},
"threatIntel": {
"fieldMap": { },
"fieldToTypeMap": { },
"triageConfig" : {
"riskLevelRules" : [
{
"rule" : "parser_score > 3.5",
"score" : 10
}
],
"aggregator" : "MAX"
}
}
} I think the first two lines should appear in the corresponding indexing configuration and not the enrichment configuration. Using the enrichment config as is results in a parse error when pushing the configuration to zookeeper. I just wanted to confirm this is the case. If not, what do those first two lines exactly mean?
... View more
Labels:
05-26-2017
09:17 PM
I eventually got around this problem by using the CSV parser instead of grok. I was using metron 3.0 at the time and am not sure if the issue with grok got resolved or not, but I don't have any problem with the CSV parser in the current version , 3.1.
... View more
05-26-2017
09:11 PM
Using Metron 3.1 release, I am following the instructions here to create a profiler for my ingested data stream and perform statistical outlier analysis of it. I can confirm that the data is parsed, enriched, and indexed in elastic search correctly. The parsing, enrichment, indexing, and profiler topologies are all running without any errors and I can see that the profiler is writing into HBase. However, my "parser_score" as stored in elasticsearch is null, even when an outlier is pushed into the stream, and I'm trying to debug why. I'm using the Stellar shell for that purpose. Once I get the message that all functions are loaded successfully, and after waiting sufficiently (20 minutes) for the data to be populated in HBase, I run the following command to get a profile: PROFILE_GET('my_profile_name','my_entity_id',PROFILE_FIXED(2,'MINUTES')) which results in the following exception in Stellar shell: [!] Unable to execute: Found interface org.objectweb.asm.MethodVisitor, but class was expected
org.apache.metron.common.dsl.ParseException: Unable to execute: Found interface org.objectweb.asm.MethodVisitor, but class was expected
at org.apache.metron.common.stellar.StellarCompiler.getResult(StellarCompiler.java:409)
at org.apache.metron.common.stellar.BaseStellarProcessor.parse(BaseStellarProcessor.java:127)
at org.apache.metron.common.stellar.shell.StellarExecutor.execute(StellarExecutor.java:275)
at org.apache.metron.common.stellar.shell.StellarShell.executeStellar(StellarShell.java:373)
at org.apache.metron.common.stellar.shell.StellarShell.handleStellar(StellarShell.java:276)
at org.apache.metron.common.stellar.shell.StellarShell.execute(StellarShell.java:412)
at org.jboss.aesh.console.AeshProcess.run(AeshProcess.java:53)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IncompatibleClassChangeError: Found interface org.objectweb.asm.MethodVisitor, but class was expected
at com.esotericsoftware.reflectasm.ConstructorAccess.insertConstructor(ConstructorAccess.java:124)
at com.esotericsoftware.reflectasm.ConstructorAccess.get(ConstructorAccess.java:95)
at org.apache.metron.common.utils.SerDeUtils$DefaultInstantiatorStrategy.newInstantiatorOf(SerDeUtils.java:129)
at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1078)
at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1087)
at com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:570)
at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:546)
at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:790)
at org.apache.metron.common.utils.SerDeUtils.fromBytes(SerDeUtils.java:249)
at org.apache.metron.profiler.client.HBaseProfilerClient.lambda$get$4(HBaseProfilerClient.java:160)
at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
at org.apache.metron.profiler.client.HBaseProfilerClient.get(HBaseProfilerClient.java:160)
at org.apache.metron.profiler.client.HBaseProfilerClient.fetch(HBaseProfilerClient.java:139)
at org.apache.metron.profiler.client.stellar.GetProfile.apply(GetProfile.java:170)
at org.apache.metron.common.stellar.StellarCompiler.exitTransformationFunc(StellarCompiler.java:267)
at org.apache.metron.common.stellar.generated.StellarParser$TransformationFuncContext.exitRule(StellarParser.java:1689)
at org.antlr.v4.runtime.Parser.triggerExitRuleEvent(Parser.java:422)
at org.antlr.v4.runtime.Parser.exitRule(Parser.java:632)
at org.apache.metron.common.stellar.generated.StellarParser.functions(StellarParser.java:1712)
at org.apache.metron.common.stellar.generated.StellarParser.arithmetic_operands(StellarParser.java:1846)
at org.apache.metron.common.stellar.generated.StellarParser.arithmetic_expr_mul(StellarParser.java:1609)
at org.apache.metron.common.stellar.generated.StellarParser.arithmetic_expr(StellarParser.java:1469)
at org.apache.metron.common.stellar.generated.StellarParser.transformation_expr(StellarParser.java:308)
at org.apache.metron.common.stellar.generated.StellarParser.transformation(StellarParser.java:149)
at org.apache.metron.common.stellar.BaseStellarProcessor.parse(BaseStellarProcessor.java:126)
... 8 more I have tried double-quotes instead of single-quotes in the command but the result is the same. Any thought about what might be causing the issue? My profiler config is very similar to what is described here , except that instead of capturing a 'global' statistical state for my 'value', I'm capturing that state per 'my_entity_id'. Also, I'm setting the following in my profiler.properties without changing the other configuration parameters: profiler.period.duration=1
profiler.period.duration.units=MINUTES I'm not sure if the issue I'm facing in Stellar shell is related with why I'm not getting parser scores, but looks like something is not working correctly when calling "PROFILE_GET". I appreciate your help.
... View more
Labels:
03-29-2017
12:23 AM
@asubramanian I can confirm that escaping the braces does not wok. Here's a simplified version of my problem: Telemetry Input (I'm using NiFi with JoltTransform processor to run this test): {"hostname":"my.machine","timestamp":"1490746284"} Grok Pattern 1: {"hostname":"%{HOSTNAME:hostname}","timestamp":"%{NUMBER:timestamp}"} Grok Pattern 2: \{"hostname":"%{HOSTNAME:hostname}","timestamp":"%{NUMBER:timestamp}"\} They both result in the following error in Metron's parser bolt: java.lang.IllegalStateException: Grok parser Error: Grok statement produced a null message. Original message was: {"hostname":"my.machine","timestamp":"1490746284"} and the parsed message was: {} Note that Pattern 1 is a match according to the grok constructor website: http://grokconstructor.appspot.com/ I'm not sure what's going on. Is this a bug in Metron grok parser or am I doing something wrong? I greatly appreciate your help.
... View more
03-28-2017
10:43 PM
@asubramanian, Strange... After restarting everything I'm getting the initial parse error again.
... View more
03-28-2017
06:46 PM
@asubramanian, I forgot to add my indexing configuration to the previous post. Here it goes: {
"elasticsearch": {
"index": "my_telemetry_source",
"batchSize": 5,
"enabled" : true
},
"hdfs": {
"index": "my_telemetry_source",
"batchSize": 5,
"enabled" : true
}
}
... View more
03-28-2017
06:41 PM
Hi @asubramanian, Thanks for your response. I made your suggested change and I'm not getting the parsing error any more. I can't see the data getting indexed though in elastic search. This is likely unrelated to the parsing error, but I'm not sure. I'm not getting any data corresponding to my telemetry source, but am getting the following error for the snort sensor from hdfsIndexingBolt: org.apache.hadoop.security.AccessControlException: Permission denied: user=storm, access=WRITE, inode="/apps/metron/indexing/indexed/snort/enrichment-null-0-0-1490726017771.json":metron:metron:drwxrwxr-x at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:319) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:292) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:213) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:190) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1827) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1811) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkAncestorAccess(FSDirectory.java:1794) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:2598) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2533) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2417) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:729) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:405) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:640) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2313) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2309) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2307) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73) at org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1628) at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1703) at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1638) at org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:448) at org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:444) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:459) at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:387) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:909) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:890) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:787) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:776) at org.apache.metron.writer.hdfs.SourceHandler.createOutputFile(SourceHandler.java:142) at org.apache.metron.writer.hdfs.SourceHandler.initialize(SourceHandler.java:98) at org.apache.metron.writer.hdfs.SourceHandler.<init>(SourceHandler.java:65) at org.apache.metron.writer.hdfs.HdfsWriter.getSourceHandler(HdfsWriter.java:102) at org.apache.metron.writer.hdfs.HdfsWriter.write(HdfsWriter.java:77) at org.apache.metron.writer.BulkWriterComponent.write(BulkWriterComponent.java:138) at org.apache.metron.writer.bolt.BulkMessageWriterBolt.execute(BulkMessageWriterBolt.java:115) at org.apache.storm.daemon.executor$fn__6573$tuple_action_fn__6575.invoke(executor.clj:734) at org.apache.storm.daemon.executor$mk_task_receiver$fn__6494.invoke(executor.clj:466) at org.apache.storm.disruptor$clojure_handler$reify__6007.onEvent(disruptor.clj:40) at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451) at org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430) at org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73) at org.apache.storm.daemon.executor$fn__6573$fn__6586$fn__6639.invoke(executor.clj:853) at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484) at clojure.lang.AFn.run(AFn.java:22) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): Permission denied: user=storm, access=WRITE, inode="/apps/metron/indexing/indexed/snort/enrichment-null-0-0-1490726017771.json":metron:metron:drwxrwxr-x at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:319) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:292) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:213) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:190) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1827) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1811) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkAncestorAccess(FSDirectory.java:1794) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:2598) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2533) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2417) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:729) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:405) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:640) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2313) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2309) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2307) at org.apache.hadoop.ipc.Client.call(Client.java:1476) at org.apache.hadoop.ipc.Client.call(Client.java:1407) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229) at com.sun.proxy.$Proxy46.create(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create(ClientNamenodeProtocolTranslatorPB.java:296) at sun.reflect.GeneratedMethodAccessor46.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy48.create(Unknown Source) at org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1623) ... 28 more Do I need to setup permissions for storm user on hdfs? How do you suggest I test that the data from my telemetry source successfully passes the parsing phase? Thanks again for your help.
... View more
03-24-2017
10:24 PM
Hi, I'm getting the following error from Storm's ParserBolt in a single node deployment of Metron 0.3.1: java.lang.IllegalStateException: Grok parser Error: Grok statement produced a null message. Original message was: {"hostname":"myhost","timestamp":"1010101010","cpu_user":"8.522706","cpu_sys":"29.210946","cpu_nice":"0.000000","cpu_idle":"62.266348","cpu_wait":"0.000000","cpu_irq":"0.000000","cpu_soft_irq":"0.000000","cpu_stolen":"0.000000","load_one":"4.945312","load_five":"4.391113","load_fifteen":"4.385254"} and the parsed message was: {} . Check the pattern at: /apps/metron/patterns/my_telemetry_source on {"hostname":"myhost","timestamp":"1010101010","cpu_user":"8.522706","cpu_sys":"29.210946","cpu_nice":"0.000000","cpu_idle":"62.266348","cpu_wait":"0.000000","cpu_irq":"0.000000","cpu_soft_irq":"0.000000","cpu_stolen":"0.000000","load_one":"4.945312","load_five":"4.391113","load_fifteen":"4.385254"} at org.apache.metron.parsers.GrokParser.parse(GrokParser.java:174) at org.apache.metron.parsers.interfaces.MessageParser.parseOptional(MessageParser.java:45) at org.apache.metron.parsers.bolt.ParserBolt.execute(ParserBolt.java:132) at org.apache.storm.daemon.executor$fn__6573$tuple_action_fn__6575.invoke(executor.clj:734) at org.apache.storm.daemon.executor$mk_task_receiver$fn__6494.invoke(executor.clj:466) at org.apache.storm.disruptor$clojure_handler$reify__6007.onEvent(disruptor.clj:40) at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451) at org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430) at org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73) at org.apache.storm.daemon.executor$fn__6573$fn__6586$fn__6639.invoke(executor.clj:853) at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484) at clojure.lang.AFn.run(AFn.java:22) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.RuntimeException: Grok statement produced a null message. Original message was: {"hostname":"myhost","timestamp":"1010101010","cpu_user":"8.522706","cpu_sys":"29.210946","cpu_nice":"0.000000","cpu_idle":"62.266348","cpu_wait":"0.000000","cpu_irq":"0.000000","cpu_soft_irq":"0.000000","cpu_stolen":"0.000000","load_one":"4.945312","load_five":"4.391113","load_fifteen":"4.385254"} and the parsed message was: {} . Check the pattern at: /apps/metron/patterns/my_telemetry_source at org.apache.metron.parsers.GrokParser.parse(GrokParser.java:152) ... 12 more My input lines are of the following form: {"hostname":"myhost","timestamp":"1010101010","cpu_user":"8.522706","cpu_sys":"29.210946","cpu_nice":"0.000000","cpu_idle":"62.266348","cpu_wait":"0.000000","cpu_irq":"0.000000","cpu_soft_irq":"0.000000","cpu_stolen":"0.000000","load_one":"4.945312","load_five":"4.391113","load_fifteen":"4.385254"}
{"hostname":"myhost","timestamp":"1012101010","cpu_user":"8.522706","cpu_sys":"29.210946","cpu_nice":"0.000000","cpu_idle":"62.266348","cpu_wait":"0.000000","cpu_irq":"0.000000","cpu_soft_irq":"0.000000","cpu_stolen":"0.000000","load_one":"4.945312","load_five":"4.391113","load_fifteen":"4.385254"} My Grok Pattern is: MY_TELEMETRY_SOURCE_DELIMITED %{GREEDYDATA:UNWANTED}{"hostname":"%{HOSTNAME:hostname}","timestamp":"%{NUMBER:timestamp}","cpu_user":"%{NUMBER:cpu_user}","cpu_sys":"%{NUMBER:cpu_sys}","cpu_nice":"%{NUMBER:cpu_nice}","cpu_idle":"%{NUMBER:cpu_idle}","cpu_wait":"%{NUMBER:cpu_wait}","cpu_irq":"%{NUMBER:cpu_irq}","cpu_soft_irq":"%{NUMBER:cpu_soft_irq}","cpu_stolen":"%{NUMBER:cpu_stolen}","load_one":"%{NUMBER:load_one}","load_five":"%{NUMBER:load_five}","load_fifteen":"%{NUMBER:load_fifteen}"}%{GREEDYDATA:UNWANTED} Which according to the GrokConstructor (http://grokconstructor.appspot.com) matches correctly. All Metron services are up and the pattern is staged at /apps/metron/patterns/my_telemetry_source as evident by the error log. My parser configuration for the parsing topology is located at /usr/metron/0.3.1/config/zookeeper/parsers/my_telemetry_source.json and consists of the following: {
"parserClassName": "org.apache.metron.parsers.GrokParser",
"sensorTopic": "my_telemetry_source",
"parserConfig": {
"grokPath": "/apps/metron/patterns/my_telemetry_source",
"patternLabel": "MY_TELEMETRY_SOURCE_DELIMITED",
"timestampField": "timestamp"
}
} I have tried this setup both with and without applying the patternLabel, MY_TELEMETRY_SOURCE_DELIMITED, getting the same error message in both cases. I appreciate your help.
... View more
Labels:
03-08-2017
11:45 PM
Hi,
I was wondering about the best way to ingest logs in XML format into Metron. Parsing with Grok doesn't seem to be the way to go in this case. Remaining options seem to be to 1) either utilize NiFi to turn the XML into a format that Metron expects, 2) or develop a java parser for parsing XML, (that is Metron-288 issue which is not of high priority: https://issues.apache.org/jira/browse/METRON-288 ). Are these two options the only possibilities?
... View more
Labels:
10-22-2016
01:49 AM
Hi, I'm running Metron on a single node, full-dev-platform. I'm not getting any data on Kibana and am trying to debug what happened with Monit, based on the suggestions here: https://community.hortonworks.com/content/kbentry/59801/troubleshooting-missing-events-in-metron-quick-dev.html Unfortunately, I'm not able to bring up Monit UI on http://node1:2812 . Is this the right port as suggested in the above article? Calling "sudo monit summary" on the console gives me the following: Process 'yaf' Running
Process 'snort' Running
Process 'snort-logs' Execution failed
Process 'pcap-service' Running
Process 'pcap-replay' Running
Program 'pcap-parser' Execution failed
Program 'yaf-parser' Execution failed
Program 'bro-parser' Execution failed
Program 'snort-parser' Execution failed
Process 'mysql' Running
Process 'kibana' Running
Program 'indexing' Execution failed
Program 'enrichment' Execution failed
Process 'elasticsearch' Running
Process 'bro' Running
System 'node1' Running 1) How reliable is Monit's output for debugging purposes? Do you recommend using it or is there a better way? 2) It seems that the reason I'm not getting any data on Kibana is the fact that snort-parser, yaf-parser, pcap-parser (and possibly the others whose execution failed) are not running. I tried using Monit to restart them, but that didn't work as the processes fail to execute. I should add that I do not see any alerts on Ambari, indicating an error. Mysql is running, Storm has "enrichment", "pcap", "snort", and "yaf" topologies, and the following topics have been created on Kafka: bro
enrichments
indexing
indexing_error
parser_error
parser_invalid
pcap
snort
yaf Furthermore, It seems that some data is coming through the topics, as the following command in "/usr/hdp/current/kafka-broker": $ ./bin/kafka-console-consumer.sh --topic bro --zookeeper localhost:2181 results in something like {metadata.broker.list=node1:6667, request.timeout.ms=30000, client.id=console-consumer-80191, security.protocol=PLAINTEXT}
{"http": {"ts":1477100588.011956,"uid":"CK38Oa4To9c4C0Hi0k","id.orig_h":"192.168.138.158","id.orig_p":49205,"id.resp_h":"95.163.121.204","id.resp_p":80,"trans_depth":1,"method":"GET","host":"7oqnsnzwwnm6zb7y.gigapaysun.com","uri":"/11iQmfg","user_agent":"Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0)","request_body_len":0,"response_body_len":3289,"status_code":200,"status_msg":"OK","tags":[],"resp_fuids":["F9oD3k2wgqQ6WzQot1"],"resp_mime_types":["text/html"]}}
{"http": {"ts":1477100590.698489,"uid":"Cpax4X1rrfzVYDVAT4","id.orig_h":"192.168.138.158","id.orig_p":49206,"id.resp_h":"95.163.121.204","id.resp_p":80,"trans_depth":1,"method":"GET","host":"7oqnsnzwwnm6zb7y.gigapaysun.com","uri":"/img/style.css","referrer":"http://7oqnsnzwwnm6zb7y.gigapaysun.com/11iQmfg","user_agent":"Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0)","request_body_len":0,"response_body_len":4492,"status_code":200,"status_msg":"OK","tags":[],"resp_fuids":["FVYVnk4BazYbfR3sOg"],"resp_mime_types":["text/plain"]}}
{"http": {"ts":1477100592.541302,"uid":"Cpax4X1rrfzVYDVAT4","id.orig_h":"192.168.138.158","id.orig_p":49206,"id.resp_h":"95.163.121.204","id.resp_p":80,"trans_depth":2,"method":"GET","host":"7oqnsnzwwnm6zb7y.gigapaysun.com","uri":"/img/flags/it.png","referrer":"http://7oqnsnzwwnm6zb7y.gigapaysun.com/11iQmfg","user_agent":"Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0)","request_body_len":0,"response_body_len":552,"status_code":200,"status_msg":"OK","tags":[],"resp_fuids":["Ff0bri1CjR5gXsAuF4"],"resp_mime_types":["image/png"]}}
But $ ./bin/kafka-console-consumer.sh --topic snort --zookeeper localhost:2181 only results in {metadata.broker.list=node1:6667, request.timeout.ms=30000, client.id=console-consumer-49141, security.protocol=PLAINTEXT} which I guess is do to snort-logs process not getting executed (is that correct?). I appreciate any help you could provide to debug this deployment.
... View more
Labels:
10-22-2016
01:12 AM
1 Kudo
I have an installation of Metron running on a single node. I receive an error when I try to provision the VM with
either "sensors" or more specifically "yaf" tag. The error happens during this task: TASK [yaf : Compile and Install fixbuf] **************************************** and is as follows: failed: [node1] => (item=./configure) => {"changed": true, "cmd": "./configure", "delta": "0:00:00.879759", "end": "2016-10-22 00:56:24.976554", "failed": true, "invocation": {"module_args": {"_raw_params": "./configure", "_uses_shell": true, "chdir": "/tmp/libfixbuf-1.7.1", "creates": null, "executable": null, "removes": null, "warn": true}, "module_name": "command"}, "item": "./configure", "rc": 1, "start": "2016-10-22 00:56:24.096795", "stderr": "configure: error: cannot find install-sh, install.sh, or shtool in autoconf \".\"/autoconf", "stdout": "", "stdout_lines": [], "warnings": []}
failed: [node1] => (item=make) => {"changed": true, "cmd": "make", "delta": "0:00:00.014689", "end": "2016-10-22 00:56:25.601005", "failed": true, "invocation": {"module_args": {"_raw_params": "make", "_uses_shell": true, "chdir": "/tmp/libfixbuf-1.7.1", "creates": null, "executable": null, "removes": null, "warn": true}, "module_name": "command"}, "item": "make", "rc": 2, "start": "2016-10-22 00:56:25.586316", "stderr": "make: *** No rule to make target `Makefile.am', needed by `Makefile.in'. Stop.", "stdout": "", "stdout_lines": [], "warnings": []}
failed: [node1] => (item=make install) => {"changed": true, "cmd": "make install", "delta": "0:00:00.027586", "end": "2016-10-22 00:56:26.259226", "failed": true, "invocation": {"module_args": {"_raw_params": "make install", "_uses_shell": true, "chdir": "/tmp/libfixbuf-1.7.1", "creates": null, "executable": null, "removes": null, "warn": true}, "module_name": "command"}, "item": "make install", "rc": 2, "start": "2016-10-22 00:56:26.231640", "stderr": "make: *** No rule to make target `Makefile.am', needed by `Makefile.in'. Stop.", "stdout": "", "stdout_lines": [], "warnings": []} It seems to be the same error as METRON-104: https://issues.apache.org/jira/browse/METRON-104 , which is marked as resolved. Any suggestions on how to resolve the issue?
... View more
- Tags:
- CyberSecurity
- Metron
Labels:
09-27-2016
06:44 PM
Thanks for your response. The issue was resolved and Metron is successfully deployed.
... View more
09-27-2016
12:19 AM
I should add that I receive the mentioned error during the following task: TASK [ambari_config : Start the ambari cluster - wait] *************************
... View more
09-26-2016
11:26 PM
Hi, I'm trying to install Metron on a single node, following the instructions here: https://github.com/apache/incubator-metron/tree/master/metron-deployment/vagrant/full-dev-platform I receive the following error, indicating an issue with Ambari: fatal: [node1]: FAILED! => {"changed": false, "failed": true, "invocation": {"module_args": {"blueprint_name": null, "blueprint_var": null, "cluster_name": "metron_cluster", "cluster_state": "started", "configurations": null, "host": "node1", "password": "admin", "port": 8080, "username": "admin", "wait_for_complete": true}, "module_name": "ambari_cluster_state"}, "msg": "Ambari client exception occurred: No JSON object could be decoded"} Note that Ambari is up and running. I can log into it using a browser. Here's the output of platform-info.sh: Metron 0.2.0BETA
--
* master
--
commit c85c74269327487a5fe607ea85ebc56d3b5650ef
Author: cstella <cestella@gmail.com>
Date: Mon Sep 26 15:14:53 2016 -0400
METRON-452: Add rudimentary configuration management functions to Stellar closes apache/incubator-metron#269
--
metron-deployment/vagrant/full-dev-platform/Vagrantfile | 6 ++++--
1 file changed, 4 insertions(+), 2 deletions(-)
--
ansible 2.0.0.2
config file = <MY_FOLDER>/incubator-metron/metron-deployment/vagrant/full-dev-platform/ansible.cfg
configured module search path = ../../extra_modules
--
Vagrant 1.8.5
--
Python 2.7.10
--
Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-10T08:41:47-08:00)
Maven home: /usr/local/Cellar/maven/3.3.9/libexec
Java version: 1.8.0_73, vendor: Oracle Corporation
Java home: /Library/Java/JavaVirtualMachines/jdk1.8.0_73.jdk/Contents/Home/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "mac os x", version: "10.10.5", arch: "x86_64", family: "mac"
--
Darwin <MY_MAC>.local 14.5.0 Darwin Kernel Version 14.5.0: Mon Aug 29 21:14:16 PDT 2016; root:xnu-2782.50.6~1/RELEASE_X86_64 x86_64 It is perhaps unrelated but worth mentioning that I have made the following changes in my Vagrantfile: 1) set the number of cpus to 2 instead of the default 4. 2) set config.ssh.insert_key = false I appreciate your help.
... View more
Labels:
