Metron Threat Intelligence enrichment not triggering alert

[#1]

I am running the full-dev-platform of Metron version 0.2.0BETA and have added squid log data as per wiki guide and some help from @cduby with some configuration issues.

Now trying to extend this to add threat intelligence alerting based upon wiki guide but am having issues with no enrichment data being added from the HBase table containing the CSV data, the geo enrichments are being added to the data however. Also the url in the example elasticsearch index is shown as "atmape.ru" and in my index it shows as "http://www.atmape.ru".

Enrichment config for squid is in zookeeper as below;

ENRICHMENT Config: squid
{
  "index" : "squid",
  "batchSize" : 5,
  "enrichment" : {
    "fieldMap" : {
      "geo" : [ "ip_dst_addr", "ip_src_addr" ],
      "host" : [ "host" ]
    },
    "fieldToTypeMap" : { },
    "config" : { }
  },
  "threatIntel" : {
    "fieldMap" : {
      "hbaseThreatIntel" : [ "ip_src_addr", "ip_dst_addr", "url" ]
    },
    "fieldToTypeMap" : {
      "ip_src_addr" : [ "malicious_ip" ],
      "ip_dst_addr" : [ "malicious_ip" ],
      "url" : [ "zeusList" ]
    },
    "config" : { },
    "triageConfig" : {
      "riskLevelRules" : { },
      "aggregator" : "MAX",
      "aggregationConfig" : { }
    }
  },
  "configuration" : { }
}

[#2]

@Aaron Harris

You can use the opentaxii service to load threat intelligence data into Hbase. This article describes how to:

https://community.hortonworks.com/articles/59698/pushing-stixtaxii-feeds-from-opentaxii-server-into....

You can also find more information other ways to load data in:

https://github.com/apache/incubator-metron/tree/master/metron-platform/metron-data-management

[#3]

@cduby

Apologies I wasn't very clear in my description of the problem, the threat intelligence has loaded into HBase fine, I can scan the 'threatintel' table in HBase and it returns the csv threat data that was uploaded.

However when I ingest squid logs, they still look identical to before I added the threat enrichment to the squid enrichment config.

[#4 · Accepted solution]

It seems the version of Metron I was using had some inconsistencies with the fieldTransformation being used in the enrichment config, it didn't recognise STELLAR as the transformation language.

I downloaded the latest version of the source code (0.2.1BETA instead of 0.2.0BETA), followed the original process for building the full cluster and configuring Metron to add a telemetry source, after this I could then follow the steps to add the threat intel configuration and it is now enriching the data correctly.

Thanks for all your help along the way @cduby it has been very much appreciated.

[#5]

@Aaron Harris Glad to help and glad you got it working.