Pushing STIX/Taxii feeds from Opentaxii server into HBASE
Pre-requisite
- Working Metron cluster - deployed via ansible-playbook or via Ambari + Mpack.
- The node on which opentaxii service is being deployed should have access to HBASE.
Step 1 - Deploy Opentaxii Role (Optional - if not deployed)
a) Create a playbook to deploy the opentaxii role
[root@metron-test ~]# cat metron/metron-deployment/playbooks/install-opentaxii.yml
- hosts: metron
become: true
roles:
- role: opentaxii
b) Deploy using ansible-playbook
[root@metron-test ~]# ansible-playbook -i ~/metron-deployment/inventory/metron_example playbooks/install-opentaxii.yml -e ansible_python_interpreter=python -e ansible_user=root -e ansible_ssh_private_key_file=/path/to/private-keypair.pem -vvv
c) Verify the service has been deployed successfully using the command:
service opentaxii status
This should show the list of subscribed services along with threat feed counts. Here is a sample output:
[root@metron-test]# service opentaxii status
guest.phishtank_com 888
guest.Abuse_ch 0
guest.CyberCrime_Tracker 0
guest.EmergingThreats_rules 0
guest.Lehigh_edu 0
guest.MalwareDomainList_Hostlist 0
guest.blutmagie_de_torExits 648
guest.dataForLast_7daysOnly 1124
guest.dshield_BlockList 0
Note:
In case the following is noticed
[root@node1 ~]# service opentaxii status
Checking opentaxii... Running
Services not defined
Refer to METRON-484 for more details and a workaround.
Step 2 - Fetch Latest Opentaxii Feeds
Use the following command to fetch the latest hailataxii feeds into the opentaxii server
service opentaxii sync <service-name> [YYYY-MM-DD]
For e.g.
service opentaxii sync guest.phishtank_com
service opentaxii sync guest.Abuse_ch 2016-08-01
Note: The date (YYYY-MM-DD) indicates the time from when the threat intel feeds is to be pulled. If not suffixed, then the sync command picks up feeds available for the current day.
The above process can be repeated for all the subscribed services.
Step 3 - Load Opentaxii Feeds into HBASE
Create sample extractor.json and connection_config.json files as follows:
[root@metron-test]# cat ~/extractor.json
{
"config": {
"columns": {
"ip": 0
},
"indicator_column": "ip",
"type" : "malicious_ip",
"separator" : ","
},
"extractor" : "STIX"
}
[root@metron-test]# cat ~/connection_config.json
{
"endpoint" : "http://localhost:9000/services/discovery"
,"username" : "guest"
,"password" : "guest"
,"type" : "DISCOVER"
,"collection" : "guest.MalwareDomainList_Hostlist"
,"table" : "threatintel"
,"columnFamily" : "t"
,"allowedIndicatorTypes" : [ "domainname:FQDN", "address:IPV_4_ADDR" ]
}
Now, push the hailataxii feeds from the opentaxii server into HBASE using the following script:
/usr/metron/<METRON_VERSION>/bin/threatintel_taxii_load.sh -b <START_TIME> -c /path/to/connection_config.json -e /path/to/extractor.json -p <TIME_INTERVAL_MSECS>
For e.g.
/usr/metron/0.2.0BETA/bin/threatintel_taxii_load.sh -b "2016-08-01 00:00:00" -c ~/connection_config.json -e ~/extractor.json -p 10000
Step 4 - Verify in HBASE
Query the Hbase table to check for the threat intel feeds.
echo "scan 'threatintel'" | hbase shell
