Support Questions
Find answers, ask questions, and share your expertise
Announcements
Check out our newest addition to the community, the Cloudera Innovation Accelerator group hub.

Metron Statistics Documentation Error

Explorer

The enrichment configuration discussed in the this link seems to have an error:

{
  "index": "mad",
  "batchSize": 1,
  "enrichment": {
    "fieldMap": {
      "stellar" : {
        "config" : {
          "parser_score" : "OUTLIER_MAD_SCORE(OUTLIER_MAD_STATE_MERGE(
PROFILE_GET( 'sketchy_mad', 'global', PROFILE_FIXED(10, 'MINUTES')) ), value)"
         ,"is_alert" : "if parser_score > 3.5 then true else is_alert"
        }
      }
    }
  ,"fieldToTypeMap": { }
  },
  "threatIntel": {
    "fieldMap": { },
    "fieldToTypeMap": { },
    "triageConfig" : {
      "riskLevelRules" : [
        {
          "rule" : "parser_score > 3.5",
          "score" : 10
        }
      ],
      "aggregator" : "MAX"
    }
  }
}

I think the first two lines should appear in the corresponding indexing configuration and not the enrichment configuration. Using the enrichment config as is results in a parse error when pushing the configuration to zookeeper.

I just wanted to confirm this is the case. If not, what do those first two lines exactly mean?

1 ACCEPTED SOLUTION

Explorer

Hi guys. Yes, it would appear that doc example is outdated. "index" and "batchSize" belong in the indexing config. Here is a sample for bro from the current source:

cat metron-platform/metron-enrichment/src/main/config/zookeeper/enrichments/bro.json
{
  "enrichment" : {
    "fieldMap": {
      "geo": ["ip_dst_addr", "ip_src_addr"],
      "host": ["host"]
    }
  },
  "threatIntel": {
    "fieldMap": {
      "hbaseThreatIntel": ["ip_src_addr", "ip_dst_addr"]
    },
    "fieldToTypeMap": {
      "ip_src_addr" : ["malicious_ip"],
      "ip_dst_addr" : ["malicious_ip"]
    }
  }
}
cat metron-platform/metron-indexing/src/main/config/zookeeper/indexing/bro.json
{
  "hdfs" : {
    "index": "bro",
    "batchSize": 5,
    "enabled" : true
  },
  "elasticsearch" : {
    "index": "bro",
    "batchSize": 5,
    "enabled" : true
  },
  "solr" : {
    "index": "bro",
    "batchSize": 5,
    "enabled" : false
  }
}

View solution in original post

3 REPLIES 3

Cloudera Employee

Hi @HS, Thank you for your comment on our Metron documentation. Sorry the delay in responding to you. I've raised your question to the Dev team and I hope to have an answer for you shortly.

Explorer

Hi guys. Yes, it would appear that doc example is outdated. "index" and "batchSize" belong in the indexing config. Here is a sample for bro from the current source:

cat metron-platform/metron-enrichment/src/main/config/zookeeper/enrichments/bro.json
{
  "enrichment" : {
    "fieldMap": {
      "geo": ["ip_dst_addr", "ip_src_addr"],
      "host": ["host"]
    }
  },
  "threatIntel": {
    "fieldMap": {
      "hbaseThreatIntel": ["ip_src_addr", "ip_dst_addr"]
    },
    "fieldToTypeMap": {
      "ip_src_addr" : ["malicious_ip"],
      "ip_dst_addr" : ["malicious_ip"]
    }
  }
}
cat metron-platform/metron-indexing/src/main/config/zookeeper/indexing/bro.json
{
  "hdfs" : {
    "index": "bro",
    "batchSize": 5,
    "enabled" : true
  },
  "elasticsearch" : {
    "index": "bro",
    "batchSize": 5,
    "enabled" : true
  },
  "solr" : {
    "index": "bro",
    "batchSize": 5,
    "enabled" : false
  }
}

Explorer

I submitted a fix to this README. You can view progress here

@HS, the Metron community always looking for contributors and future committers and is extremely helpful (imho) in getting users involved. I see you've been active on the Metron boards here on HCC and we would be happy to get you open source community credit should you also choose to submit Jiras/PRs in the future. Best, Mike.