Created on 10-04-2016 09:32 AM
a) Create a playbook to deploy the opentaxii role
[root@metron-test ~]# cat metron/metron-deployment/playbooks/install-opentaxii.yml - hosts: metron become: true roles: - role: opentaxii
b) Deploy using ansible-playbook
[root@metron-test ~]# ansible-playbook -i ~/metron-deployment/inventory/metron_example playbooks/install-opentaxii.yml -e ansible_python_interpreter=python -e ansible_user=root -e ansible_ssh_private_key_file=/path/to/private-keypair.pem -vvv
c) Verify the service has been deployed successfully using the command:
service opentaxii status
This should show the list of subscribed services along with threat feed counts. Here is a sample output:
[root@metron-test]# service opentaxii status guest.phishtank_com 888 guest.Abuse_ch 0 guest.CyberCrime_Tracker 0 guest.EmergingThreats_rules 0 guest.Lehigh_edu 0 guest.MalwareDomainList_Hostlist 0 guest.blutmagie_de_torExits 648 guest.dataForLast_7daysOnly 1124 guest.dshield_BlockList 0
Note:
In case the following is noticed
[root@node1 ~]# service opentaxii status Checking opentaxii... Running Services not defined
Refer to METRON-484 for more details and a workaround.
Use the following command to fetch the latest hailataxii feeds into the opentaxii server
service opentaxii sync <service-name> [YYYY-MM-DD] For e.g. service opentaxii sync guest.phishtank_com service opentaxii sync guest.Abuse_ch 2016-08-01
Note: The date (YYYY-MM-DD) indicates the time from when the threat intel feeds is to be pulled. If not suffixed, then the sync command picks up feeds available for the current day.
The above process can be repeated for all the subscribed services.
Create sample extractor.json and connection_config.json files as follows:
[root@metron-test]# cat ~/extractor.json { "config": { "columns": { "ip": 0 }, "indicator_column": "ip", "type" : "malicious_ip", "separator" : "," }, "extractor" : "STIX" } [root@metron-test]# cat ~/connection_config.json { "endpoint" : "http://localhost:9000/services/discovery" ,"username" : "guest" ,"password" : "guest" ,"type" : "DISCOVER" ,"collection" : "guest.MalwareDomainList_Hostlist" ,"table" : "threatintel" ,"columnFamily" : "t" ,"allowedIndicatorTypes" : [ "domainname:FQDN", "address:IPV_4_ADDR" ] }
Now, push the hailataxii feeds from the opentaxii server into HBASE using the following script:
/usr/metron/<METRON_VERSION>/bin/threatintel_taxii_load.sh -b <START_TIME> -c /path/to/connection_config.json -e /path/to/extractor.json -p <TIME_INTERVAL_MSECS> For e.g. /usr/metron/0.2.0BETA/bin/threatintel_taxii_load.sh -b "2016-08-01 00:00:00" -c ~/connection_config.json -e ~/extractor.json -p 10000
Query the Hbase table to check for the threat intel feeds.
echo "scan 'threatintel'" | hbase shell
Created on 10-15-2016 12:28 PM
connector.json and extractor.json created as per this article. ansible pushed opentaxii as per this article. the service is started and i was able to pull several thousand elements from a few of the taxii servers. i can connect to localhost:9000 and there is a service waiting for input.
when i enter this:
0.2.1BETA/bin/threatintel_taxii_load.sh -b "2016-10-13 00:00:00" -c taxii_connector.json -e taxii_extractor.json -p 10000
i get this (bold for emphasis):
WARNING: Use "yarn jar" to launch YARN applications.
16/10/15 12:14:55 INFO taxii.TaxiiHandler: Loading configuration: TaxiiConnectionConfig{endpoint=http://localhost:9000/services/discovery, port=443, proxy=null, username='guest', password='******', type=DISCOVER, allowedIndicatorTypes=domainname:FQDN,address:IPV_4_ADDR, collection='guest.MalwareDomainList_Hostlist', subscriptionId='null', beginTime=Thu Oct 13 00:00:00 UTC 2016, table=threatintel:t}
16/10/15 12:14:55 INFO taxii.TaxiiHandler: Initializing client..
16/10/15 12:14:55 INFO taxii.TaxiiHandler: Discovering endpoint
16/10/15 12:14:57 INFO taxii.TaxiiHandler: Request made : org.mitre.taxii.messages.xml11.DiscoveryRequest => org.mitre.taxii.messages.xml11.DiscoveryResponse (expected org.mitre.taxii.messages.xml11.DiscoveryResponse)
16/10/15 12:14:57 INFO taxii.TaxiiHandler: Discovered endpoint as http://localhost:9000/services/poll
16/10/15 12:14:57 INFO taxii.TaxiiHandler: Configured, starting polling http://localhost:9000/services/poll for guest.MalwareDomainList_Hostlist
16/10/15 12:14:57 INFO taxii.TaxiiHandler: Polling...10/15/16 12:14 PM
16/10/15 12:14:57 INFO taxii.TaxiiHandler: Begin Time: 2016-10-13T00:00:00Z
16/10/15 12:14:57 INFO taxii.TaxiiHandler: Request made : org.mitre.taxii.messages.xml11.PollRequest => org.mitre.taxii.messages.xml11.PollResponse (expected org.mitre.taxii.messages.xml11.PollResponse)
16/10/15 12:14:57 INFO taxii.TaxiiHandler: Got Poll Response with 0 blocks
16/10/15 12:15:07 INFO taxii.TaxiiHandler: Polling...10/15/16 12:15 PM
16/10/15 12:15:07 INFO taxii.TaxiiHandler: Begin Time: 2016-10-15T12:14:57Z
16/10/15 12:15:07 ERROR taxii.TaxiiHandler: Connection pool shut down
java.lang.IllegalStateException: Connection pool shut down
at org.apache.metron.httpcore.dataload.util.Asserts.check(Asserts.java:34) at org.apache.metron.httpcore.dataload.pool.AbstractConnPool.lease(AbstractConnPool.java:169) at org.apache.metron.httpcore.dataload.impl.conn.PoolingHttpClientConnectionManager.requestConnection(PoolingHttpClientConnectionManager.java:217) at org.apache.metron.httpcore.dataload.impl.execchain.MainClientExec.execute(MainClientExec.java:158) at org.apache.metron.httpcore.dataload.impl.execchain.ProtocolExec.execute(ProtocolExec.java:195) at org.apache.metron.httpcore.dataload.impl.execchain.RetryExec.execute(RetryExec.java:85) at org.apache.metron.httpcore.dataload.impl.execchain.RedirectExec.execute(RedirectExec.java:108) at org.apache.metron.httpcore.dataload.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:186) at org.apache.metron.httpcore.dataload.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) at org.mitre.taxii.client.HttpClient.callTaxiiService(HttpClient.java:297) at org.apache.metron.dataloads.nonbulk.taxii.TaxiiHandler.call(TaxiiHandler.java:336) at org.apache.metron.dataloads.nonbulk.taxii.TaxiiHandler.call(TaxiiHandler.java:242) at org.apache.metron.dataloads.nonbulk.taxii.TaxiiHandler.run(TaxiiHandler.java:171) at java.util.TimerThread.mainLoop(Timer.java:555) at java.util.TimerThread.run(Timer.java:505)
Exception in thread "Timer-0" java.lang.RuntimeException: Unable to make request
at org.apache.metron.dataloads.nonbulk.taxii.TaxiiHandler.run(TaxiiHandler.java:214) at java.util.TimerThread.mainLoop(Timer.java:555) at java.util.TimerThread.run(Timer.java:505)
Caused by: java.lang.IllegalStateException: Connection pool shut down
at org.apache.metron.httpcore.dataload.util.Asserts.check(Asserts.java:34) at org.apache.metron.httpcore.dataload.pool.AbstractConnPool.lease(AbstractConnPool.java:169) at org.apache.metron.httpcore.dataload.impl.conn.PoolingHttpClientConnectionManager.requestConnection(PoolingHttpClientConnectionManager.java:217) at org.apache.metron.httpcore.dataload.impl.execchain.MainClientExec.execute(MainClientExec.java:158) at org.apache.metron.httpcore.dataload.impl.execchain.ProtocolExec.execute(ProtocolExec.java:195) at org.apache.metron.httpcore.dataload.impl.execchain.RetryExec.execute(RetryExec.java:85) at org.apache.metron.httpcore.dataload.impl.execchain.RedirectExec.execute(RedirectExec.java:108) at org.apache.metron.httpcore.dataload.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:186) at org.apache.metron.httpcore.dataload.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) at org.mitre.taxii.client.HttpClient.callTaxiiService(HttpClient.java:297) at org.apache.metron.dataloads.nonbulk.taxii.TaxiiHandler.call(TaxiiHandler.java:336) at org.apache.metron.dataloads.nonbulk.taxii.TaxiiHandler.call(TaxiiHandler.java:242) at org.apache.metron.dataloads.nonbulk.taxii.TaxiiHandler.run(TaxiiHandler.java:171) ... 2 more
am i missing somthing?
i am not behind a proxy, this is a cloud based server, there is no real firewall between me and the internet, there is no local firewall, there is no SSL interception, i can access other sites with no issues, and metron as a whole is functional.
thanks for any insight.
Created on 07-27-2017 04:56 PM
What version of ansible did you use to get the opentaxii feed? i tried running it on 2.0.0.2 and still getting service not defined error while checking the opentaxii status, the service is running though. I followed the work around steps but still the same error.
Created on 08-08-2017 04:22 PM
I have got taxii to work but i keep getting error message "Exception in thread "main" java.lang.IllegalStateException: Extractor must be a STIX Extractor" when i try to push the feed to Hbase
Here's what the extractor file looks like
{
"config": {
"zk_quorum": "node1:2181",
"columns": {
"stix_address_categories" : "IPV_4_ADDR"
},
"indicator_column": "ip",
"type" : "malicious_ip",
"separator" : ","
}
,"extractor" : "STIX"
}
Created on 08-22-2017 07:31 AM
Have you solve this problem. I got the same mistake. The version of Metron is 0.4.0.
Created on 08-22-2017 08:48 AM
@ankur V and @leo lee - looks like you are hitting into https://issues.apache.org/jira/browse/METRON-1026. This has been fixed with latest bits of metron. Can you give it a try?
Created on 08-22-2017 11:04 AM
Okay ,Thank you so much.
Created on 08-22-2017 01:15 PM
I got the follow mistake when load opentaxii feeds to hbase. Metron version is 0.4.0
[root@node1 ~]# /usr/metron/0.4.0/bin/threatintel_taxii_load.sh -b "2016-08-01 00:00:00" -c ~/connection_config.json -e ~/extractor.json -p 10000
17/08/22 03:35:01 WARN extractor.TransformFilterExtractorDecorator: Unable to setup zookeeper client - zk_quorum url not provided. **This will limit some Stellar functionality**
Exception in thread "main" java.lang.IllegalStateException: Extractor must be a STIX Extractor
at org.apache.metron.dataloads.nonbulk.taxii.TaxiiLoader.main(TaxiiLoader.java:202)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:233)
at org.apache.hadoop.util.RunJar.main(RunJar.java:148)
so what can i do to solve this? Any help will be appreciated.
Created on 08-22-2017 02:08 PM
@leo leeWe fixed the issue by modifying the JAR files and replacing the metron-data-management-0.4.0.jar in /usr/metron/0.4.0/lib with the modified jar follow the instructions noted in https://github.com/apache/metron/pull/643/files see if that helps if not let me know i can upload the JAR file (its 100M though)
Created on 08-22-2017 02:09 PM
Thanks for the headsup @asubramanian
Created on 08-24-2017 06:11 AM
Thanks to your help,I handled this correctly.
Created on 11-14-2017 07:26 PM
Thanks for the wonderful tutorial. I managed to get both the taxii server and threatintel_taxii_load.sh to work after patching metron-data-management.jar with 0.4.1's one.
17/11/14 12:03:24 INFO taxii.TaxiiHandler: Loading configuration: TaxiiConnectionConfig{endpoint=http://localhost:9000/services/discovery, port=443, proxy=null, username='guest', password='******', type=DISCOVER, allowedIndicatorTypes=, collection='pool', subscriptionId='null', beginTime=Sat Nov 11 00:00:00 SGT 2017, table=threat_intel:t} 17/11/14 12:03:24 INFO taxii.TaxiiHandler: Initializing client.. 17/11/14 12:03:24 INFO taxii.TaxiiHandler: Discovering endpoint 17/11/14 12:03:25 INFO taxii.TaxiiHandler: Request made : org.mitre.taxii.messages.xml11.DiscoveryRequest => org.mitre.taxii.messages.xml11.DiscoveryResponse (expected org.mitre.taxii.messages.xml11.DiscoveryResponse) 17/11/14 12:03:25 INFO taxii.TaxiiHandler: Discovered endpoint as http://localhost:9000/services/poll 17/11/14 12:03:25 INFO taxii.TaxiiHandler: Configured, starting polling http://localhost:9000/services/poll for pool 17/11/14 12:03:25 INFO taxii.TaxiiHandler: Polling...11/14/17 12:03 PM 17/11/14 12:03:25 INFO taxii.TaxiiHandler: Begin Time: 2017-11-10T16:00:00Z 17/11/14 12:03:26 INFO taxii.TaxiiHandler: Request made : org.mitre.taxii.messages.xml11.PollRequest => org.mitre.taxii.messages.xml11.PollResponse (expected org.mitre.taxii.messages.xml11.PollResponse) 17/11/14 12:03:26 INFO taxii.TaxiiHandler: Got Poll Response with 1917 blocks 17/11/14 12:03:26 INFO taxii.TaxiiHandler: Processed 99 in 0 ms, avg time: 0 17/11/14 12:03:26 INFO taxii.TaxiiHandler: Processed 99 in 0 ms, avg time: 0 17/11/14 12:03:26 INFO taxii.TaxiiHandler: Processed 99 in 0 ms, avg time: 0 17/11/14 12:03:26 INFO taxii.TaxiiHandler: Processed 99 in 0 ms, avg time: 0 17/11/14 12:03:26 INFO taxii.TaxiiHandler: Processed 99 in 0 ms, avg time: 0 17/11/14 12:03:26 INFO taxii.TaxiiHandler: Processed 99 in 0 ms, avg time: 0 17/11/14 12:03:26 INFO taxii.TaxiiHandler: Processed 99 in 0 ms, avg time: 0 17/11/14 12:03:26 INFO taxii.TaxiiHandler: Processed 99 in 0 ms, avg time: 0 17/11/14 12:03:26 INFO taxii.TaxiiHandler: Processed 99 in 0 ms, avg time: 0 17/11/14 12:03:26 INFO taxii.TaxiiHandler: Processed 99 in 0 ms, avg time: 0 17/11/14 12:03:26 INFO taxii.TaxiiHandler: Processed 99 in 0 ms, avg time: 0 17/11/14 12:03:26 INFO taxii.TaxiiHandler: Processed 99 in 0 ms, avg time: 0 17/11/14 12:03:26 INFO taxii.TaxiiHandler: Processed 99 in 0 ms, avg time: 0 17/11/14 12:03:26 INFO taxii.TaxiiHandler: Processed 99 in 0 ms, avg time: 0 17/11/14 12:03:26 INFO taxii.TaxiiHandler: Processed 99 in 0 ms, avg time: 0 17/11/14 12:03:26 INFO taxii.TaxiiHandler: Processed 99 in 0 ms, avg time: 0 17/11/14 12:03:26 INFO taxii.TaxiiHandler: Processed 99 in 0 ms, avg time: 0 17/11/14 12:03:26 INFO taxii.TaxiiHandler: Processed 99 in 0 ms, avg time: 0 17/11/14 12:03:26 INFO taxii.TaxiiHandler: Processed 99 in 0 ms, avg time: 0 ...
However after polling for blocks, HBase does not seem to be populated with the polled blocks.
hbase(main):010:0> count 'threat_intel' 0 row(s) in 0.0090 seconds => 0
My connection_config.json:
{ "endpoint": "http://localhost:9000/services/discovery", "username": "guest", "password": "guest", "type": "DISCOVER", "collection": "pool", "table": "threat_intel", "columnFamily": "t", "allowedIndicatorTypes": [ ] }
My extractor.json:
{ "config": { "zk_quorum": "node2:2181", "stix_address_categories": "IPV_4_ADDR" }, "extractor": "STIX" }
Prior to running threatintel_taxii_load.sh, I created the new HBase table `threat_intel` with column family `t`. I have a feeling that Metron's StixExtractor is not extracting the STIX indicators properly (I'm using all guest collections from Hail a TAXII). In the meantime, will be trying to use flatfile_loader.sh to load some threat intel into HBase using CSVs.
Created on 11-14-2017 07:26 PM
Thank you for the excellent tutorial. I got the set up working with my taxii server along with threatintel_taxii_load.sh with `./threatintel_taxii_load.sh -b "2017-11-11 00:00:00" -c ~/connection.json -e ~/extractor.json -p 10000`:
17/11/14 12:18:06 INFO taxii.TaxiiHandler: Loading configuration: TaxiiConnectionConfig{endpoint=http://localhost:9000/services/discovery, port=443, proxy=null, username='guest', password='******', type=DISCOVER, allowedIndicatorTypes=, collection='pool', subscriptionId='null', beginTime=Sat Nov 11 00:00:00 SGT 2017, table=threat_intel:t} 17/11/14 12:18:06 INFO taxii.TaxiiHandler: Initializing client.. 17/11/14 12:18:06 INFO taxii.TaxiiHandler: Discovering endpoint 17/11/14 12:18:07 INFO taxii.TaxiiHandler: Request made : org.mitre.taxii.messages.xml11.DiscoveryRequest => org.mitre.taxii.messages.xml11.DiscoveryResponse (expected org.mitre.taxii.messages.xml11.DiscoveryResponse) 17/11/14 12:18:07 INFO taxii.TaxiiHandler: Discovered endpoint as http://localhost:9000/services/poll 17/11/14 12:18:07 INFO taxii.TaxiiHandler: Configured, starting polling http://localhost:9000/services/poll for pool 17/11/14 12:18:07 INFO taxii.TaxiiHandler: Polling...11/14/17 12:18 PM 17/11/14 12:18:07 INFO taxii.TaxiiHandler: Begin Time: 2017-11-10T16:00:00Z 17/11/14 12:18:08 INFO taxii.TaxiiHandler: Request made : org.mitre.taxii.messages.xml11.PollRequest => org.mitre.taxii.messages.xml11.PollResponse (expected org.mitre.taxii.messages.xml11.PollResponse) 17/11/14 12:18:08 INFO taxii.TaxiiHandler: Got Poll Response with 1917 blocks 17/11/14 12:18:08 INFO taxii.TaxiiHandler: Processed 99 in 0 ms, avg time: 0 17/11/14 12:18:08 INFO taxii.TaxiiHandler: Processed 99 in 0 ms, avg time: 0 ... Continues to poll ever
However after the blocks have been processed, they do not seem to be stored into HBase. I also tried creating a `threat_intel` table with column family `t` prior to running `threatintel_taxii_load.sh`.
hbase(main):006:0> scan 'threat_intel' ROW COLUMN+CELL 0 row(s) in 0.0220 seconds
My connection.json:
{ "endpoint": "http://localhost:9000/services/discovery", "username": "guest", "password": "guest", "type": "DISCOVER", "collection": "pool", "table": "threat_intel", "columnFamily": "t", "allowedIndicatorTypes": [ ] }
My extractor.json
{ "config": { "zk_quorum": "node2:2181", "stix_address_categories": "IPV_4_ADDR" }, "extractor": "STIX" }
I have a feeling that it may be the StixExtractor.java not being able to extract the indicators (IPs), or perhaps it could be HBase having issues. I'll be trying to load in threat intel from CSV files using the flatfile_loader.sh to check whether HBase gets populated.
Created on 11-14-2017 07:26 PM
Thank you for the excellent tutorial. I got the set up working with my taxii server along with threatintel_taxii_load.sh with `./threatintel_taxii_load.sh -b "2017-11-11 00:00:00" -c ~/connection.json -e ~/extractor.json -p 10000`:
However after the blocks have been processed, they do not seem to be stored into HBase. I also tried creating a `threat_intel` table with column family `t` prior to running `threatintel_taxii_load.sh`.
hbase(main):006:0> scan 'threat_intel'
ROW COLUMN+CELL 0 row(s) in 0.0220 seconds
My connection.json:
{
"endpoint": "http://localhost:9000/services/discovery",
"username": "guest",
"password": "guest",
"type": "DISCOVER",
"collection": "pool",
"table": "threat_intel",
"columnFamily": "t",
"allowedIndicatorTypes": [ ]
}
My extractor.json:
{
"config": {
"zk_quorum": "node2:2181",
"stix_address_categories": "IPV_4_ADDR"
},
"extractor": "STIX"
}
I have a feeling that it may be the StixExtractor.java not being able to extract the indicators (IPs), or perhaps it could be HBase having issues. I'll be trying to load in threat intel from CSV files using the flatfile_loader.sh to check whether HBase gets populated.
Created on 11-14-2017 07:26 PM
Thank you for the excellent tutorial. I got the set up working with my taxii server along with threatintel_taxii_load.sh with "./threatintel_taxii_load.sh -b "2017-11-11 00:00:00" -c ~/connection.json -e ~/extractor.json -p 10000"
However after the blocks have been processed, they do not seem to be stored into HBase. I also tried creating a "threat_intel" table with column family "t" prior to running "threatintel_taxii_load.sh". hbase(main):
006:0> scan 'threat_intel'
ROW COLUMN+CELL
0 row(s) in 0.0220 seconds
My connection.json:
{ "endpoint": "http://localhost:9000/services/discovery", "username": "guest", "password": "guest", "type": "DISCOVER", "collection": "pool", "table": "threat_intel", "columnFamily": "t", "allowedIndicatorTypes": [ ] }
My extractor.json: { "config": { "zk_quorum": "node2:2181", "stix_address_categories": "IPV_4_ADDR" }, "extractor": "STIX" }
I have a feeling that it may be the StixExtractor.java not being able to extract the indicators (IPs), or perhaps it could be HBase having issues. I'll be trying to load in threat intel from CSV files using the flatfile_loader.sh to check whether HBase gets populated.
User | Count |
---|---|
758 | |
379 | |
316 | |
309 | |
268 |