Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Issues with Metron Profiler & Stellar

Issues with Metron Profiler & Stellar

Explorer

Using Metron 3.1 release, I am following the instructions here to create a profiler for my ingested data stream and perform statistical outlier analysis of it. I can confirm that the data is parsed, enriched, and indexed in elastic search correctly. The parsing, enrichment, indexing, and profiler topologies are all running without any errors and I can see that the profiler is writing into HBase. However, my "parser_score" as stored in elasticsearch is null, even when an outlier is pushed into the stream, and I'm trying to debug why.

I'm using the Stellar shell for that purpose. Once I get the message that all functions are loaded successfully, and after waiting sufficiently (20 minutes) for the data to be populated in HBase, I run the following command to get a profile:

PROFILE_GET('my_profile_name','my_entity_id',PROFILE_FIXED(2,'MINUTES'))

which results in the following exception in Stellar shell:

[!] Unable to execute: Found interface org.objectweb.asm.MethodVisitor, but class was expected
org.apache.metron.common.dsl.ParseException: Unable to execute: Found interface org.objectweb.asm.MethodVisitor, but class was expected
at org.apache.metron.common.stellar.StellarCompiler.getResult(StellarCompiler.java:409)
at org.apache.metron.common.stellar.BaseStellarProcessor.parse(BaseStellarProcessor.java:127)
at org.apache.metron.common.stellar.shell.StellarExecutor.execute(StellarExecutor.java:275)
at org.apache.metron.common.stellar.shell.StellarShell.executeStellar(StellarShell.java:373)
at org.apache.metron.common.stellar.shell.StellarShell.handleStellar(StellarShell.java:276)
at org.apache.metron.common.stellar.shell.StellarShell.execute(StellarShell.java:412)
at org.jboss.aesh.console.AeshProcess.run(AeshProcess.java:53)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IncompatibleClassChangeError: Found interface org.objectweb.asm.MethodVisitor, but class was expected
at com.esotericsoftware.reflectasm.ConstructorAccess.insertConstructor(ConstructorAccess.java:124)
at com.esotericsoftware.reflectasm.ConstructorAccess.get(ConstructorAccess.java:95)
at org.apache.metron.common.utils.SerDeUtils$DefaultInstantiatorStrategy.newInstantiatorOf(SerDeUtils.java:129)
at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1078)
at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1087)
at com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:570)
at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:546)
at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:790)
at org.apache.metron.common.utils.SerDeUtils.fromBytes(SerDeUtils.java:249)
at org.apache.metron.profiler.client.HBaseProfilerClient.lambda$get$4(HBaseProfilerClient.java:160)
at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
at org.apache.metron.profiler.client.HBaseProfilerClient.get(HBaseProfilerClient.java:160)
at org.apache.metron.profiler.client.HBaseProfilerClient.fetch(HBaseProfilerClient.java:139)
at org.apache.metron.profiler.client.stellar.GetProfile.apply(GetProfile.java:170)
at org.apache.metron.common.stellar.StellarCompiler.exitTransformationFunc(StellarCompiler.java:267)
at org.apache.metron.common.stellar.generated.StellarParser$TransformationFuncContext.exitRule(StellarParser.java:1689)
at org.antlr.v4.runtime.Parser.triggerExitRuleEvent(Parser.java:422)
at org.antlr.v4.runtime.Parser.exitRule(Parser.java:632)
at org.apache.metron.common.stellar.generated.StellarParser.functions(StellarParser.java:1712)
at org.apache.metron.common.stellar.generated.StellarParser.arithmetic_operands(StellarParser.java:1846)
at org.apache.metron.common.stellar.generated.StellarParser.arithmetic_expr_mul(StellarParser.java:1609)
at org.apache.metron.common.stellar.generated.StellarParser.arithmetic_expr(StellarParser.java:1469)
at org.apache.metron.common.stellar.generated.StellarParser.transformation_expr(StellarParser.java:308)
at org.apache.metron.common.stellar.generated.StellarParser.transformation(StellarParser.java:149)
at org.apache.metron.common.stellar.BaseStellarProcessor.parse(BaseStellarProcessor.java:126)
... 8 more

I have tried double-quotes instead of single-quotes in the command but the result is the same.

Any thought about what might be causing the issue?

My profiler config is very similar to what is described here , except that instead of capturing a 'global' statistical state for my 'value', I'm capturing that state per 'my_entity_id'.

Also, I'm setting the following in my profiler.properties without changing the other configuration parameters:

profiler.period.duration=1 
profiler.period.duration.units=MINUTES

I'm not sure if the issue I'm facing in Stellar shell is related with why I'm not getting parser scores, but looks like something is not working correctly when calling "PROFILE_GET".

I appreciate your help.

2 REPLIES 2
Highlighted

Re: Issues with Metron Profiler & Stellar

Explorer

So, I used 'global' instead of 'my_entity_id' with the same settings. After sometime, I can see parser_score values (though they don't seem to be computed correctly). So the issue with not getting scores seems to be related to how 'my_entity_id' values are parsed/interpreted or my potentially incorrect use of it. I still get the same exception in Stellar shell though.

As a side note, all my entities (as visible by elastic search head plugin) have the same value for 'my_entity_id' in my simple test. So there's no semantic difference between using 'global' or 'my_entity_id' as the entity id in the profile. Apart from the problem with the shell, I'm curious about what I'm not doing right when using 'my_entity_id'.

In other words, is there anything wrong with defining the profile below?

{
  "profiles": [
    {
      "profile": "my_profile_name",
      "foreach": "my_entity_id",
      "onlyif": "true",
      "init" : {
        "s": "OUTLIER_MAD_STATE_MERGE(PROFILE_GET('my_profile_name','my_entity_id', PROFILE_FIXED(2, 'MINUTES')))"
      },
      "update": {
        "s": "OUTLIER_MAD_ADD(s, my_double_value)"
      },
      "result": "s"
    }
  ]
}

and/or the following enrichment config:

{
  "enrichment": {
    "fieldMap": {
      "stellar" : {
        "config" : {
          "parser_score" : "OUTLIER_MAD_SCORE(OUTLIER_MAD_STATE_MERGE(PROFILE_GET( 'my_profile_name', 'my_entity_id', 
            PROFILE_FIXED(5, 'MINUTES')) ), my_double_value)"
         ,"is_alert" : "if parser_score > 3.5 then true else is_alert"
        }
      }
    }
  ,"fieldToTypeMap": { }
  },
  "threatIntel": {
    "fieldMap": { },
    "fieldToTypeMap": { },
    "triageConfig" : {
      "riskLevelRules" : [
        {
          "rule" : "parser_score > 3.5",
          "score" : 10
        }
      ],
      "aggregator" : "MAX"
    }
  }
}

Thanks in advance for your help!

Highlighted

Re: Issues with Metron Profiler & Stellar

New Contributor

You need to tell the profiler client that you are using 1 minute periods , otherwise it thinks you are using 15 minute default periods and so it cant generate the right row keys that match. you can either set that property for the profiler client

'profiler.client.period.duration':'1','profiler.client.period.duration.units':'MINUTES'

in your global properties at $METRON_HOME/config/zookeeper/global.json or pass it in directly to PROFILE_FIXED

Like

PROFILE_GET('profile1','entity1', PROFILE_FIXED(30,'DAYS',{'profiler.client.period.duration':'1','profiler.client.period.duration.units':'MINUTES'}))
Don't have an account?
Coming from Hortonworks? Activate your account here