Created 05-26-2017 10:51 PM
The enrichment configuration discussed in the this link seems to have an error:
{
"index": "mad",
"batchSize": 1,
"enrichment": {
"fieldMap": {
"stellar" : {
"config" : {
"parser_score" : "OUTLIER_MAD_SCORE(OUTLIER_MAD_STATE_MERGE(
PROFILE_GET( 'sketchy_mad', 'global', PROFILE_FIXED(10, 'MINUTES')) ), value)"
,"is_alert" : "if parser_score > 3.5 then true else is_alert"
}
}
}
,"fieldToTypeMap": { }
},
"threatIntel": {
"fieldMap": { },
"fieldToTypeMap": { },
"triageConfig" : {
"riskLevelRules" : [
{
"rule" : "parser_score > 3.5",
"score" : 10
}
],
"aggregator" : "MAX"
}
}
}
I think the first two lines should appear in the corresponding indexing configuration and not the enrichment configuration. Using the enrichment config as is results in a parse error when pushing the configuration to zookeeper.
I just wanted to confirm this is the case. If not, what do those first two lines exactly mean?
Created 05-31-2017 02:18 PM
Hi guys. Yes, it would appear that doc example is outdated. "index" and "batchSize" belong in the indexing config. Here is a sample for bro from the current source:
cat metron-platform/metron-enrichment/src/main/config/zookeeper/enrichments/bro.json { "enrichment" : { "fieldMap": { "geo": ["ip_dst_addr", "ip_src_addr"], "host": ["host"] } }, "threatIntel": { "fieldMap": { "hbaseThreatIntel": ["ip_src_addr", "ip_dst_addr"] }, "fieldToTypeMap": { "ip_src_addr" : ["malicious_ip"], "ip_dst_addr" : ["malicious_ip"] } } }
cat metron-platform/metron-indexing/src/main/config/zookeeper/indexing/bro.json { "hdfs" : { "index": "bro", "batchSize": 5, "enabled" : true }, "elasticsearch" : { "index": "bro", "batchSize": 5, "enabled" : true }, "solr" : { "index": "bro", "batchSize": 5, "enabled" : false } }
Created 05-30-2017 11:21 PM
Hi @HS, Thank you for your comment on our Metron documentation. Sorry the delay in responding to you. I've raised your question to the Dev team and I hope to have an answer for you shortly.
Created 05-31-2017 02:18 PM
Hi guys. Yes, it would appear that doc example is outdated. "index" and "batchSize" belong in the indexing config. Here is a sample for bro from the current source:
cat metron-platform/metron-enrichment/src/main/config/zookeeper/enrichments/bro.json { "enrichment" : { "fieldMap": { "geo": ["ip_dst_addr", "ip_src_addr"], "host": ["host"] } }, "threatIntel": { "fieldMap": { "hbaseThreatIntel": ["ip_src_addr", "ip_dst_addr"] }, "fieldToTypeMap": { "ip_src_addr" : ["malicious_ip"], "ip_dst_addr" : ["malicious_ip"] } } }
cat metron-platform/metron-indexing/src/main/config/zookeeper/indexing/bro.json { "hdfs" : { "index": "bro", "batchSize": 5, "enabled" : true }, "elasticsearch" : { "index": "bro", "batchSize": 5, "enabled" : true }, "solr" : { "index": "bro", "batchSize": 5, "enabled" : false } }
Created 05-31-2017 10:01 PM
I submitted a fix to this README. You can view progress here
@HS, the Metron community always looking for contributors and future committers and is extremely helpful (imho) in getting users involved. I see you've been active on the Metron boards here on HCC and we would be happy to get you open source community credit should you also choose to submit Jiras/PRs in the future. Best, Mike.