Community Articles

gvetticaden1 · ‎05-05-2016

In previous article of the sereies, Enriching Telemetry Events, we walked through how to enrich a domain element of a given telemetry event with WhoIs data like home country, company associated with domain, etc. In this article, we will enrich with a special type of data called threat intel feeds. When a given telemetry event matches data in a threat Intel feed, an alert is generated.

Again, the customers requirement are the following:

The proxy events from Squid logs needs to ingested in real-time.
The proxy logs has to be parsed into a standardized JSON structure that Metron can understand.
In real-time, the squid proxy event needs to be enriched so that the domain named are enriched with the IP information
In real-time, the IP with in the proxy event must be checked against for threat intel feeds.
If there is a threat intel hit, an alert needs to be raised.
The end user must be able to see the new telemetry events and the alerts from the new data source.
All of this requirements will need to be implemented easily without writing any new java code.

In this article, we will walk you through how to do 4 and 5.

Threat Intel Framework Explained

Metron currently provides an extensible framework to plug in threat intel sources. Each threat intel source has two components: an enrichment data source and and enrichment bolt. The threat intelligence feeds are bulk loaded and streamed into a threat intelligence store similarly to how the enrichment feeds are loaded. The keys are loaded in a key-value format. The key is the indicator and the value is the JSON formatted description of what the indicator is. It is recommended to use a threat feed aggregator such as Soltra to dedup and normalize the feeds via Stix/Taxii. Metron provides an adapter that is able to read Soltra-produced Stix/Taxii feeds and stream them into Hbase, which is the data store of choice to back high speed threat intel lookups of Metron. Metron additionally provides a flat file and Stix bulk loader that can normalize, dedup, and bulk load or stream threat intel data into Hbase even without the use of a threat feed aggregator.

The below diagram illustrates the architecture:

Step 1: Threat Intel Feed Source

Metron is designed to work with Stix/Taxii threat feeds, but can also be bulk loaded with threat data from a CSV file. In this example we will explore the CSV example. The same loader framework that is used for enrichment here is used for threat intelligence. Similarly to enrichments we need to setup a data.csv file, the extractor config JSON and the enrichment config JSON.

For this example we will be using a Zeus malware tracker list located here: https://zeustracker.abuse.ch/blocklist.php?download=domainblocklist.

Copy the data form the above link into a file called domainblocklist.txt on your VM.

Run the following command to parse the above file to a csv file called domainblocklist.csv

cat domainblocklist.txt | grep -v "^#" | grep -v "^$" | grep -v "^https" | awk '{print $1",abuse.ch”}' > domainblocklist.csv

Now that we have the "Threat Intel Feed Source" , we need to now configure an extractor config file that describes the the source. Create a file called extractor_config_temp.json and put the following contents in it.

{
  "config" : {
    "columns" : {
        "domain" : 0
        ,"source" : 1
    }
    ,"indicator_column" : "domain"
    ,"type" : "zeusList"
    ,"separator" : ","
  }
  ,"extractor" : "CSV"
}

Run the following to remove the non-ascii characters we run the following:

iconv -c -f utf-8 -t ascii extractor_config_temp.json -o extractor_config.json

Step 2: Configure Element to Threat Intel Feed Mapping

We now have to configure what element of a tuple and what threat intel feed to cross-reference with.This configuration will be stored in zookeeper.

The config looks like the following:

{
  "zkQuorum" : "node1:2181"
 ,"sensorToFieldList" : {
    "bro" : {
           "type" : "THREAT_INTEL"
          ,"fieldToEnrichmentTypes" : {
             "url" : [ "zeusList" ]
                                      }
             }
           }
}

Cut and paste this file into a file called "enrichment_config_temp.json" on the virtual machine. Because copying and pasting from this blog will include some non-ascii invisible characters, to strip them out please run

iconv -c -f utf-8 -t ascii enrichment_config_temp.json -o enrichment_config.json

iconv -c -f utf-8 -t ascii enrichment_config_temp.json -o enrichment_config.json

Step 3: Run the Threat Intel Loader

Now that we have the threat intel source and threat intel config defined, we can now run the loader to move the data from the threat intel source to the Metron threat intel Store and store the enrichment config in zookeeper.

/usr/metron/0.1BETA/bin/flatfile_loader.sh -n enrichment_config.json -i abuse.csv -t threatintel -c t -e extractor_config.json

After this, the threat intel data will be loaded in Hbase and a Zookeeper mapping will be established. The data will be populated into Hbase table called threatintel. To verify that the logs were properly ingested into Hbase run the following command:

hbase shell
scan 'threatintel'

You should see the table bulk loaded with data from the CSV file. Now check if Zookeeper enrichment tag was properly populated:

/usr/metron/0.1BETA/bin/zk_load_configs.sh -z localhost:2181

Generate some data by using the squid client to execute http requests (do this about 20 times)

squidclient http://www.alamman.com
squidclient http://www.atmape.ru

View the Threat Alerts in Metron UI

When the logs are ingested we get messages that has a hit against threat intel:

Notice a couple of characteristics about this message. It has is_alert=true, which designates it as an alert message.

Now that we have alerts coming through we need to visualize them in Kibana. First, we need to setup a pinned query to look for messages where is_alert=true:

And then once we point the alerts table to this pinned query it looks like this:

Cloudera Community