Pre-requisite Working Metron cluster - deployed via ansible-playbook or via Ambari + Mpack. The node on which opentaxii service is being deployed should have access to HBASE. Step 1 - Deploy Opentaxii Role (Optional - if not deployed) a) Create a playbook to deploy the opentaxii role [root@metron-test ~]# cat metron/metron-deployment/playbooks/install-opentaxii.yml - hosts: metron become: true roles: - role: opentaxii b) Deploy using ansible-playbook [root@metron-test ~]# ansible-playbook -i ~/metron-deployment/inventory/metron_example playbooks/install-opentaxii.yml -e ansible_python_interpreter=python -e ansible_user=root -e ansible_ssh_private_key_file=/path/to/private-keypair.pem -vvv c) Verify the service has been deployed successfully using the command: service opentaxii status This should show the list of subscribed services along with threat feed counts. Here is a sample output: [root@metron-test]# service opentaxii status guest.phishtank_com 888 guest.Abuse_ch 0 guest.CyberCrime_Tracker 0 guest.EmergingThreats_rules 0 guest.Lehigh_edu 0 guest.MalwareDomainList_Hostlist 0 guest.blutmagie_de_torExits 648 guest.dataForLast_7daysOnly 1124 guest.dshield_BlockList 0 Note: In case the following is noticed [root@node1 ~]# service opentaxii status Checking opentaxii... Running Services not defined Refer to METRON-484 for more details and a workaround. Step 2 - Fetch Latest Opentaxii Feeds Use the following command to fetch the latest hailataxii feeds into the opentaxii server service opentaxii sync <service-name> [YYYY-MM-DD] For e.g. service opentaxii sync guest.phishtank_com service opentaxii sync guest.Abuse_ch 2016-08-01 Note: The date (YYYY-MM-DD) indicates the time from when the threat intel feeds is to be pulled. If not suffixed, then the sync command picks up feeds available for the current day. The above process can be repeated for all the subscribed services. Step 3 - Load Opentaxii Feeds into HBASE Create sample extractor.json and connection_config.json files as follows: [root@metron-test]# cat ~/extractor.json { "config": { "columns": { "ip": 0 }, "indicator_column": "ip", "type" : "malicious_ip", "separator" : "," }, "extractor" : "STIX" } [root@metron-test]# cat ~/connection_config.json { "endpoint" : "http://localhost:9000/services/discovery" ,"username" : "guest" ,"password" : "guest" ,"type" : "DISCOVER" ,"collection" : "guest.MalwareDomainList_Hostlist" ,"table" : "threatintel" ,"columnFamily" : "t" ,"allowedIndicatorTypes" : [ "domainname:FQDN", "address:IPV_4_ADDR" ] } Now, push the hailataxii feeds from the opentaxii server into HBASE using the following script: /usr/metron/<METRON_VERSION>/bin/threatintel_taxii_load.sh -b <START_TIME> -c /path/to/connection_config.json -e /path/to/extractor.json -p <TIME_INTERVAL_MSECS> For e.g. /usr/metron/0.2.0BETA/bin/threatintel_taxii_load.sh -b "2016-08-01 00:00:00" -c ~/connection_config.json -e ~/extractor.json -p 10000 Step 4 - Verify in HBASE Query the Hbase table to check for the threat intel feeds. echo "scan 'threatintel'" | hbase shell ... View more