Pushing STIX/Taxii feeds from Opentaxii server into HBASE

[#1]

Pre-requisite

• Working Metron cluster - deployed via ansible-playbook or via Ambari + Mpack.
• The node on which opentaxii service is being deployed should have access to HBASE.

Step 1 - Deploy Opentaxii Role (Optional - if not deployed)

a) Create a playbook to deploy the opentaxii role

[root@metron-test ~]# cat metron/metron-deployment/playbooks/install-opentaxii.yml
- hosts: metron
  become: true
  roles:
    - role: opentaxii

b) Deploy using ansible-playbook

[root@metron-test ~]# ansible-playbook -i ~/metron-deployment/inventory/metron_example  playbooks/install-opentaxii.yml -e ansible_python_interpreter=python -e ansible_user=root  -e ansible_ssh_private_key_file=/path/to/private-keypair.pem -vvv

c) Verify the service has been deployed successfully using the command:

service opentaxii status

This should show the list of subscribed services along with threat feed counts. Here is a sample output:

[root@metron-test]# service opentaxii status
guest.phishtank_com                                888
guest.Abuse_ch                                     0
guest.CyberCrime_Tracker                           0
guest.EmergingThreats_rules                        0
guest.Lehigh_edu                                   0
guest.MalwareDomainList_Hostlist                   0
guest.blutmagie_de_torExits                        648
guest.dataForLast_7daysOnly                        1124
guest.dshield_BlockList                            0

Note:

In case the following is noticed

[root@node1 ~]# service opentaxii status
Checking opentaxii...                             Running
Services not defined

Refer to METRON-484 for more details and a workaround.

Step 2 - Fetch Latest Opentaxii Feeds

Use the following command to fetch the latest hailataxii feeds into the opentaxii server

service opentaxii sync <service-name> [YYYY-MM-DD]
For e.g.
service opentaxii sync guest.phishtank_com 
service opentaxii sync guest.Abuse_ch 2016-08-01

Note: The date (YYYY-MM-DD) indicates the time from when the threat intel feeds is to be pulled. If not suffixed, then the sync command picks up feeds available for the current day.

The above process can be repeated for all the subscribed services.

Step 3 - Load Opentaxii Feeds into HBASE

Create sample extractor.json and connection_config.json files as follows:

[root@metron-test]# cat ~/extractor.json
{
  "config": {
    "columns": {
      "ip": 0
    },
    "indicator_column": "ip",
    "type" : "malicious_ip",
    "separator" : ","
  },
  "extractor" : "STIX"
}
[root@metron-test]# cat ~/connection_config.json
{
  "endpoint" : "http://localhost:9000/services/discovery"
  ,"username" : "guest"
  ,"password" : "guest"
  ,"type" : "DISCOVER"
  ,"collection" : "guest.MalwareDomainList_Hostlist"
  ,"table" : "threatintel"
  ,"columnFamily" : "t"
  ,"allowedIndicatorTypes" : [ "domainname:FQDN", "address:IPV_4_ADDR" ]
}

Now, push the hailataxii feeds from the opentaxii server into HBASE using the following script:

/usr/metron/<METRON_VERSION>/bin/threatintel_taxii_load.sh -b <START_TIME> -c /path/to/connection_config.json -e /path/to/extractor.json -p <TIME_INTERVAL_MSECS>
For e.g.
/usr/metron/0.2.0BETA/bin/threatintel_taxii_load.sh -b "2016-08-01 00:00:00" -c ~/connection_config.json -e ~/extractor.json -p 10000

Step 4 - Verify in HBASE

Query the Hbase table to check for the threat intel feeds.

echo "scan 'threatintel'" | hbase shell

[#2]

Thanks for the wonderful tutorial. I managed to get both the taxii server and threatintel_taxii_load.sh to work after patching metron-data-management.jar with 0.4.1's one.

17/11/14 12:03:24 INFO taxii.TaxiiHandler: Loading configuration: TaxiiConnectionConfig{endpoint=http://localhost:9000/services/discovery, port=443, proxy=null, username='guest', password='******', type=DISCOVER, allowedIndicatorTypes=, collection='pool', subscriptionId='null', beginTime=Sat Nov 11 00:00:00 SGT 2017, table=threat_intel:t}
17/11/14 12:03:24 INFO taxii.TaxiiHandler: Initializing client..
17/11/14 12:03:24 INFO taxii.TaxiiHandler: Discovering endpoint
17/11/14 12:03:25 INFO taxii.TaxiiHandler: Request made : org.mitre.taxii.messages.xml11.DiscoveryRequest => org.mitre.taxii.messages.xml11.DiscoveryResponse (expected org.mitre.taxii.messages.xml11.DiscoveryResponse)
17/11/14 12:03:25 INFO taxii.TaxiiHandler: Discovered endpoint as http://localhost:9000/services/poll
17/11/14 12:03:25 INFO taxii.TaxiiHandler: Configured, starting polling http://localhost:9000/services/poll for pool
17/11/14 12:03:25 INFO taxii.TaxiiHandler: Polling...11/14/17 12:03 PM
17/11/14 12:03:25 INFO taxii.TaxiiHandler: Begin Time: 2017-11-10T16:00:00Z
17/11/14 12:03:26 INFO taxii.TaxiiHandler: Request made : org.mitre.taxii.messages.xml11.PollRequest => org.mitre.taxii.messages.xml11.PollResponse (expected org.mitre.taxii.messages.xml11.PollResponse)
17/11/14 12:03:26 INFO taxii.TaxiiHandler: Got Poll Response with 1917 blocks
17/11/14 12:03:26 INFO taxii.TaxiiHandler: Processed 99  in 0 ms, avg time: 0
17/11/14 12:03:26 INFO taxii.TaxiiHandler: Processed 99  in 0 ms, avg time: 0
17/11/14 12:03:26 INFO taxii.TaxiiHandler: Processed 99  in 0 ms, avg time: 0
17/11/14 12:03:26 INFO taxii.TaxiiHandler: Processed 99  in 0 ms, avg time: 0
17/11/14 12:03:26 INFO taxii.TaxiiHandler: Processed 99  in 0 ms, avg time: 0
17/11/14 12:03:26 INFO taxii.TaxiiHandler: Processed 99  in 0 ms, avg time: 0
17/11/14 12:03:26 INFO taxii.TaxiiHandler: Processed 99  in 0 ms, avg time: 0
17/11/14 12:03:26 INFO taxii.TaxiiHandler: Processed 99  in 0 ms, avg time: 0
17/11/14 12:03:26 INFO taxii.TaxiiHandler: Processed 99  in 0 ms, avg time: 0
17/11/14 12:03:26 INFO taxii.TaxiiHandler: Processed 99  in 0 ms, avg time: 0
17/11/14 12:03:26 INFO taxii.TaxiiHandler: Processed 99  in 0 ms, avg time: 0
17/11/14 12:03:26 INFO taxii.TaxiiHandler: Processed 99  in 0 ms, avg time: 0
17/11/14 12:03:26 INFO taxii.TaxiiHandler: Processed 99  in 0 ms, avg time: 0
17/11/14 12:03:26 INFO taxii.TaxiiHandler: Processed 99  in 0 ms, avg time: 0
17/11/14 12:03:26 INFO taxii.TaxiiHandler: Processed 99  in 0 ms, avg time: 0
17/11/14 12:03:26 INFO taxii.TaxiiHandler: Processed 99  in 0 ms, avg time: 0
17/11/14 12:03:26 INFO taxii.TaxiiHandler: Processed 99  in 0 ms, avg time: 0
17/11/14 12:03:26 INFO taxii.TaxiiHandler: Processed 99  in 0 ms, avg time: 0
17/11/14 12:03:26 INFO taxii.TaxiiHandler: Processed 99  in 0 ms, avg time: 0
...

However after polling for blocks, HBase does not seem to be populated with the polled blocks.

hbase(main):010:0> count 'threat_intel'
0 row(s) in 0.0090 seconds
=> 0

My connection_config.json:

{
    "endpoint": "http://localhost:9000/services/discovery",
    "username": "guest",
    "password": "guest",
    "type": "DISCOVER",
    "collection": "pool",
    "table": "threat_intel",
    "columnFamily": "t",
    "allowedIndicatorTypes": [ ]
}

My extractor.json:

{
    "config": {
        "zk_quorum": "node2:2181",
        "stix_address_categories": "IPV_4_ADDR"
    },
    "extractor": "STIX"
}

Prior to running threatintel_taxii_load.sh, I created the new HBase table `threat_intel` with column family `t`. I have a feeling that Metron's StixExtractor is not extracting the STIX indicators properly (I'm using all guest collections from Hail a TAXII). In the meantime, will be trying to use flatfile_loader.sh to load some threat intel into HBase using CSVs.

[#3]

Thank you for the excellent tutorial. I got the set up working with my taxii server along with threatintel_taxii_load.sh with `./threatintel_taxii_load.sh -b "2017-11-11 00:00:00" -c ~/connection.json -e ~/extractor.json -p 10000`:

17/11/14 12:18:06 INFO taxii.TaxiiHandler: Loading configuration: TaxiiConnectionConfig{endpoint=http://localhost:9000/services/discovery, port=443, proxy=null, username='guest', password='******', type=DISCOVER, allowedIndicatorTypes=, collection='pool', subscriptionId='null', beginTime=Sat Nov 11 00:00:00 SGT 2017, table=threat_intel:t}
17/11/14 12:18:06 INFO taxii.TaxiiHandler: Initializing client..
17/11/14 12:18:06 INFO taxii.TaxiiHandler: Discovering endpoint
17/11/14 12:18:07 INFO taxii.TaxiiHandler: Request made : org.mitre.taxii.messages.xml11.DiscoveryRequest => org.mitre.taxii.messages.xml11.DiscoveryResponse (expected org.mitre.taxii.messages.xml11.DiscoveryResponse)
17/11/14 12:18:07 INFO taxii.TaxiiHandler: Discovered endpoint as http://localhost:9000/services/poll
17/11/14 12:18:07 INFO taxii.TaxiiHandler: Configured, starting polling http://localhost:9000/services/poll for pool
17/11/14 12:18:07 INFO taxii.TaxiiHandler: Polling...11/14/17 12:18 PM
17/11/14 12:18:07 INFO taxii.TaxiiHandler: Begin Time: 2017-11-10T16:00:00Z
17/11/14 12:18:08 INFO taxii.TaxiiHandler: Request made : org.mitre.taxii.messages.xml11.PollRequest => org.mitre.taxii.messages.xml11.PollResponse (expected org.mitre.taxii.messages.xml11.PollResponse)
17/11/14 12:18:08 INFO taxii.TaxiiHandler: Got Poll Response with 1917 blocks
17/11/14 12:18:08 INFO taxii.TaxiiHandler: Processed 99  in 0 ms, avg time: 0
17/11/14 12:18:08 INFO taxii.TaxiiHandler: Processed 99  in 0 ms, avg time: 0
... Continues to poll ever

However after the blocks have been processed, they do not seem to be stored into HBase. I also tried creating a `threat_intel` table with column family `t` prior to running `threatintel_taxii_load.sh`.

hbase(main):006:0> scan 'threat_intel'
ROW                   COLUMN+CELL                                               
0 row(s) in 0.0220 seconds

My connection.json:

{
    "endpoint": "http://localhost:9000/services/discovery",
    "username": "guest",
    "password": "guest",
    "type": "DISCOVER",
    "collection": "pool",
    "table": "threat_intel",
    "columnFamily": "t",
    "allowedIndicatorTypes": [ ]
}

My extractor.json

{
    "config": {
        "zk_quorum": "node2:2181",
        "stix_address_categories": "IPV_4_ADDR"
    },
    "extractor": "STIX"
}

I have a feeling that it may be the StixExtractor.java not being able to extract the indicators (IPs), or perhaps it could be HBase having issues. I'll be trying to load in threat intel from CSV files using the flatfile_loader.sh to check whether HBase gets populated.

[#4]

Thank you for the excellent tutorial. I got the set up working with my taxii server along with threatintel_taxii_load.sh with `./threatintel_taxii_load.sh -b "2017-11-11 00:00:00" -c ~/connection.json -e ~/extractor.json -p 10000`:

However after the blocks have been processed, they do not seem to be stored into HBase. I also tried creating a `threat_intel` table with column family `t` prior to running `threatintel_taxii_load.sh`.

hbase(main):006:0> scan 'threat_intel'

ROW COLUMN+CELL 0 row(s) in 0.0220 seconds

My connection.json:

{

"endpoint": "http://localhost:9000/services/discovery",

"username": "guest",

"password": "guest",

"type": "DISCOVER",

"collection": "pool",

"table": "threat_intel",

"columnFamily": "t",

"allowedIndicatorTypes": [ ]

}

My extractor.json:

{

"config": {

"zk_quorum": "node2:2181",

"stix_address_categories": "IPV_4_ADDR"

},

"extractor": "STIX"

}

I have a feeling that it may be the StixExtractor.java not being able to extract the indicators (IPs), or perhaps it could be HBase having issues. I'll be trying to load in threat intel from CSV files using the flatfile_loader.sh to check whether HBase gets populated.

[#5]

Thank you for the excellent tutorial. I got the set up working with my taxii server along with threatintel_taxii_load.sh with "./threatintel_taxii_load.sh -b "2017-11-11 00:00:00" -c ~/connection.json -e ~/extractor.json -p 10000"

However after the blocks have been processed, they do not seem to be stored into HBase. I also tried creating a "threat_intel" table with column family "t" prior to running "threatintel_taxii_load.sh". hbase(main):

006:0> scan 'threat_intel'

ROW COLUMN+CELL

0 row(s) in 0.0220 seconds

My connection.json:

{ "endpoint": "http://localhost:9000/services/discovery", "username": "guest", "password": "guest", "type": "DISCOVER", "collection": "pool", "table": "threat_intel", "columnFamily": "t", "allowedIndicatorTypes": [ ] }

My extractor.json: { "config": { "zk_quorum": "node2:2181", "stix_address_categories": "IPV_4_ADDR" }, "extractor": "STIX" }

I have a feeling that it may be the StixExtractor.java not being able to extract the indicators (IPs), or perhaps it could be HBase having issues. I'll be trying to load in threat intel from CSV files using the flatfile_loader.sh to check whether HBase gets populated.