Thank you for the excellent tutorial. I got the set up working with my taxii server along with threatintel_taxii_load.sh with "./threatintel_taxii_load.sh -b "2017-11-11 00:00:00" -c ~/connection.json -e ~/extractor.json -p 10000" However after the blocks have been processed, they do not seem to be stored into HBase. I also tried creating a "threat_intel" table with column family "t" prior to running "threatintel_taxii_load.sh". hbase(main): 006:0> scan 'threat_intel' ROW COLUMN+CELL 0 row(s) in 0.0220 seconds My connection.json: { "endpoint": "http://localhost:9000/services/discovery", "username": "guest", "password": "guest", "type": "DISCOVER", "collection": "pool", "table": "threat_intel", "columnFamily": "t", "allowedIndicatorTypes": [ ] } My extractor.json: { "config": { "zk_quorum": "node2:2181", "stix_address_categories": "IPV_4_ADDR" }, "extractor": "STIX" } I have a feeling that it may be the StixExtractor.java not being able to extract the indicators (IPs), or perhaps it could be HBase having issues. I'll be trying to load in threat intel from CSV files using the flatfile_loader.sh to check whether HBase gets populated. ... View more

Thank you for the excellent tutorial. I got the set up working with my taxii server along with threatintel_taxii_load.sh with `./threatintel_taxii_load.sh -b "2017-11-11 00:00:00" -c ~/connection.json -e ~/extractor.json -p 10000`: However after the blocks have been processed, they do not seem to be stored into HBase. I also tried creating a `threat_intel` table with column family `t` prior to running `threatintel_taxii_load.sh`. hbase(main):006:0> scan 'threat_intel' ROW COLUMN+CELL 0 row(s) in 0.0220 seconds My connection.json: { "endpoint": "http://localhost:9000/services/discovery", "username": "guest", "password": "guest", "type": "DISCOVER", "collection": "pool", "table": "threat_intel", "columnFamily": "t", "allowedIndicatorTypes": [ ] } My extractor.json: { "config": { "zk_quorum": "node2:2181", "stix_address_categories": "IPV_4_ADDR" }, "extractor": "STIX" } I have a feeling that it may be the StixExtractor.java not being able to extract the indicators (IPs), or perhaps it could be HBase having issues. I'll be trying to load in threat intel from CSV files using the flatfile_loader.sh to check whether HBase gets populated. ... View more

Thank you for the excellent tutorial. I got the set up working with my taxii server along with threatintel_taxii_load.sh with `./threatintel_taxii_load.sh -b "2017-11-11 00:00:00" -c ~/connection.json -e ~/extractor.json -p 10000`: 17/11/14 12:18:06 INFO taxii.TaxiiHandler: Loading configuration: TaxiiConnectionConfig{endpoint=http://localhost:9000/services/discovery, port=443, proxy=null, username='guest', password='******', type=DISCOVER, allowedIndicatorTypes=, collection='pool', subscriptionId='null', beginTime=Sat Nov 11 00:00:00 SGT 2017, table=threat_intel:t} 17/11/14 12:18:06 INFO taxii.TaxiiHandler: Initializing client.. 17/11/14 12:18:06 INFO taxii.TaxiiHandler: Discovering endpoint 17/11/14 12:18:07 INFO taxii.TaxiiHandler: Request made : org.mitre.taxii.messages.xml11.DiscoveryRequest => org.mitre.taxii.messages.xml11.DiscoveryResponse (expected org.mitre.taxii.messages.xml11.DiscoveryResponse) 17/11/14 12:18:07 INFO taxii.TaxiiHandler: Discovered endpoint as http://localhost:9000/services/poll 17/11/14 12:18:07 INFO taxii.TaxiiHandler: Configured, starting polling http://localhost:9000/services/poll for pool 17/11/14 12:18:07 INFO taxii.TaxiiHandler: Polling...11/14/17 12:18 PM 17/11/14 12:18:07 INFO taxii.TaxiiHandler: Begin Time: 2017-11-10T16:00:00Z 17/11/14 12:18:08 INFO taxii.TaxiiHandler: Request made : org.mitre.taxii.messages.xml11.PollRequest => org.mitre.taxii.messages.xml11.PollResponse (expected org.mitre.taxii.messages.xml11.PollResponse) 17/11/14 12:18:08 INFO taxii.TaxiiHandler: Got Poll Response with 1917 blocks 17/11/14 12:18:08 INFO taxii.TaxiiHandler: Processed 99 in 0 ms, avg time: 0 17/11/14 12:18:08 INFO taxii.TaxiiHandler: Processed 99 in 0 ms, avg time: 0 ... Continues to poll ever However after the blocks have been processed, they do not seem to be stored into HBase. I also tried creating a `threat_intel` table with column family `t` prior to running `threatintel_taxii_load.sh`. hbase(main):006:0> scan 'threat_intel' ROW COLUMN+CELL 0 row(s) in 0.0220 seconds My connection.json: { "endpoint": "http://localhost:9000/services/discovery", "username": "guest", "password": "guest", "type": "DISCOVER", "collection": "pool", "table": "threat_intel", "columnFamily": "t", "allowedIndicatorTypes": [ ] } My extractor.json { "config": { "zk_quorum": "node2:2181", "stix_address_categories": "IPV_4_ADDR" }, "extractor": "STIX" } I have a feeling that it may be the StixExtractor.java not being able to extract the indicators (IPs), or perhaps it could be HBase having issues. I'll be trying to load in threat intel from CSV files using the flatfile_loader.sh to check whether HBase gets populated. ... View more

Thanks for the wonderful tutorial. I managed to get both the taxii server and threatintel_taxii_load.sh to work after patching metron-data-management.jar with 0.4.1's one. 17/11/14 12:03:24 INFO taxii.TaxiiHandler: Loading configuration: TaxiiConnectionConfig{endpoint=http://localhost:9000/services/discovery, port=443, proxy=null, username='guest', password='******', type=DISCOVER, allowedIndicatorTypes=, collection='pool', subscriptionId='null', beginTime=Sat Nov 11 00:00:00 SGT 2017, table=threat_intel:t} 17/11/14 12:03:24 INFO taxii.TaxiiHandler: Initializing client.. 17/11/14 12:03:24 INFO taxii.TaxiiHandler: Discovering endpoint 17/11/14 12:03:25 INFO taxii.TaxiiHandler: Request made : org.mitre.taxii.messages.xml11.DiscoveryRequest => org.mitre.taxii.messages.xml11.DiscoveryResponse (expected org.mitre.taxii.messages.xml11.DiscoveryResponse) 17/11/14 12:03:25 INFO taxii.TaxiiHandler: Discovered endpoint as http://localhost:9000/services/poll 17/11/14 12:03:25 INFO taxii.TaxiiHandler: Configured, starting polling http://localhost:9000/services/poll for pool 17/11/14 12:03:25 INFO taxii.TaxiiHandler: Polling...11/14/17 12:03 PM 17/11/14 12:03:25 INFO taxii.TaxiiHandler: Begin Time: 2017-11-10T16:00:00Z 17/11/14 12:03:26 INFO taxii.TaxiiHandler: Request made : org.mitre.taxii.messages.xml11.PollRequest => org.mitre.taxii.messages.xml11.PollResponse (expected org.mitre.taxii.messages.xml11.PollResponse) 17/11/14 12:03:26 INFO taxii.TaxiiHandler: Got Poll Response with 1917 blocks 17/11/14 12:03:26 INFO taxii.TaxiiHandler: Processed 99 in 0 ms, avg time: 0 17/11/14 12:03:26 INFO taxii.TaxiiHandler: Processed 99 in 0 ms, avg time: 0 17/11/14 12:03:26 INFO taxii.TaxiiHandler: Processed 99 in 0 ms, avg time: 0 17/11/14 12:03:26 INFO taxii.TaxiiHandler: Processed 99 in 0 ms, avg time: 0 17/11/14 12:03:26 INFO taxii.TaxiiHandler: Processed 99 in 0 ms, avg time: 0 17/11/14 12:03:26 INFO taxii.TaxiiHandler: Processed 99 in 0 ms, avg time: 0 17/11/14 12:03:26 INFO taxii.TaxiiHandler: Processed 99 in 0 ms, avg time: 0 17/11/14 12:03:26 INFO taxii.TaxiiHandler: Processed 99 in 0 ms, avg time: 0 17/11/14 12:03:26 INFO taxii.TaxiiHandler: Processed 99 in 0 ms, avg time: 0 17/11/14 12:03:26 INFO taxii.TaxiiHandler: Processed 99 in 0 ms, avg time: 0 17/11/14 12:03:26 INFO taxii.TaxiiHandler: Processed 99 in 0 ms, avg time: 0 17/11/14 12:03:26 INFO taxii.TaxiiHandler: Processed 99 in 0 ms, avg time: 0 17/11/14 12:03:26 INFO taxii.TaxiiHandler: Processed 99 in 0 ms, avg time: 0 17/11/14 12:03:26 INFO taxii.TaxiiHandler: Processed 99 in 0 ms, avg time: 0 17/11/14 12:03:26 INFO taxii.TaxiiHandler: Processed 99 in 0 ms, avg time: 0 17/11/14 12:03:26 INFO taxii.TaxiiHandler: Processed 99 in 0 ms, avg time: 0 17/11/14 12:03:26 INFO taxii.TaxiiHandler: Processed 99 in 0 ms, avg time: 0 17/11/14 12:03:26 INFO taxii.TaxiiHandler: Processed 99 in 0 ms, avg time: 0 17/11/14 12:03:26 INFO taxii.TaxiiHandler: Processed 99 in 0 ms, avg time: 0 ... However after polling for blocks, HBase does not seem to be populated with the polled blocks. hbase(main):010:0> count 'threat_intel' 0 row(s) in 0.0090 seconds => 0 My connection_config.json: { "endpoint": "http://localhost:9000/services/discovery", "username": "guest", "password": "guest", "type": "DISCOVER", "collection": "pool", "table": "threat_intel", "columnFamily": "t", "allowedIndicatorTypes": [ ] } My extractor.json: { "config": { "zk_quorum": "node2:2181", "stix_address_categories": "IPV_4_ADDR" }, "extractor": "STIX" } Prior to running threatintel_taxii_load.sh, I created the new HBase table `threat_intel` with column family `t`. I have a feeling that Metron's StixExtractor is not extracting the STIX indicators properly (I'm using all guest collections from Hail a TAXII). In the meantime, will be trying to use flatfile_loader.sh to load some threat intel into HBase using CSVs. ... View more