Member since
08-02-2019
131
Posts
92
Kudos Received
13
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
2665 | 12-03-2018 09:33 PM | |
3390 | 04-11-2018 02:26 PM | |
1994 | 05-09-2017 09:35 PM | |
835 | 03-31-2017 12:59 PM | |
1612 | 11-21-2016 08:58 PM |
10-20-2016
04:43 PM
@Attila Kanto This looks like the information I was looking for. Can you post this as an answer and I will accept it?
... View more
10-18-2016
02:44 PM
What are the options for automating the provisioning of HDP cluster users on openstack? Can cloud break do this?
... View more
Labels:
- Labels:
-
Hortonworks Cloudbreak
10-18-2016
02:41 PM
@Attila Kanto For iSilon, we need to add a node to the cluster using Ambari manual registration after the cluster is provisioned. So first we would need to create a blueprint with no data or name nodes, deploy it on open stack, and then after the cluster is created add the data and name node using manual registration. It would probably be ok to have the manual registration as a manual step but would be even better if it could be automated.
... View more
10-18-2016
12:15 PM
@Aaron Harris Glad to help and glad you got it working.
... View more
10-17-2016
09:42 PM
@Aaron Harris You can use the opentaxii service to load threat intelligence data into Hbase. This article describes how to: https://community.hortonworks.com/articles/59698/pushing-stixtaxii-feeds-from-opentaxii-server-into.html You can also find more information other ways to load data in: https://github.com/apache/incubator-metron/tree/master/metron-platform/metron-data-management
... View more
10-17-2016
11:51 AM
@Aaron Harris Glad you are up and running!
... View more
10-14-2016
02:51 PM
@Aaron Harris First check HBase in Ambari to make sure it is green. The threat intelligence enrichments are using hbase. Another thing to check is the squid log that is sent to kafka. One of the things I found with squid is that if you aren't constantly sending http requests to squid the logs roll over and there are no messages in the latest log. In a production system where squid is routing user http request the log won't be empty. I think you may be running into this problem: Check the messages going to the squid topic. It looks like they might be missing some information such as the source and dest ips. An easy way to fix this is to do the squid requests again and populate the most recent log. The squid messages should look something like this: [vagrant@node1 ~]$ /usr/hdp/2.4.2.0-258/kafka/bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic squid --from-beginning {metadata.broker.list=node1:6667, request.timeout.ms=30000, client.id=console-consumer-31722, security.protocol=PLAINTEXT} 1476285641.838 1439 127.0.0.1 TCP_MISS/200 457194 GET http://www.aliexpress.com/af/shoes.html? - DIRECT/104.81.164.40 text/html 1476285642.545 704 127.0.0.1 TCP_MISS/200 40385 GET http://www.help.1and1.co.uk/domains-c40986/transfer-domains-c79878 - DIRECT/212.227.34.3 text/html 1476285644.617 2068 127.0.0.1 TCP_MISS/200 177264 GET http://www.pravda.ru/science/ - DIRECT/185.103.135.90 text/html Then check the squid messages going to the enrichments topic. They should look something like this: [vagrant@node1 ~]$ /usr/hdp/2.4.2.0-258/kafka/bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic enrichments --from-beginning | grep squid {"full_hostname":"www.aliexpress.com","code":200,"method":"GET","url":"http:\/\/www.aliexpress.com\/af\/shoes.html?","source.type":"squid","elapsed":1439,"ip_dst_addr":"104.81.164.40","original_string":"1476285641.838 1439 127.0.0.1 TCP_MISS\/200 457194 GET http:\/\/www.aliexpress.com\/af\/shoes.html? - DIRECT\/104.81.164.40 text\/html","bytes":457194,"domain_without_subdomains":"aliexpress.com","action":"TCP_MISS","ip_src_addr":"127.0.0.1","timestamp":1476285641838} {"full_hostname":"www.help.1and1.co.uk","code":200,"method":"GET","url":"http:\/\/www.help.1and1.co.uk\/domains-c40986\/transfer-domains-c79878","source.type":"squid","elapsed":704,"ip_dst_addr":"212.227.34.3","original_string":"1476285642.545 704 127.0.0.1 TCP_MISS\/200 40385 GET http:\/\/www.help.1and1.co.uk\/domains-c40986\/transfer-domains-c79878 - DIRECT\/212.227.34.3 text\/html","bytes":40385,"domain_without_subdomains":"1and1.co.uk","action":"TCP_MISS","ip_src_addr":"127.0.0.1","timestamp":1476285642545}
... View more
10-13-2016
01:16 PM
@Aaron Harris >How did you delete the index? Was that through elastic, and was it just a case of pushing the logs to the kafka topic that created the new index and resolved the issue See Step 3 of the tutorial. You an use the rest service to delete the index: curl -XDELETE node1:9200/squid*
... View more
10-12-2016
08:09 PM
@Aaron Harris Are you following the tutorial below: https://cwiki.apache.org/confluence/display/METRON/2016/06/22/Metron+Tutorial+-+Fundamentals+Part+7%3A+Dashboarding+with+Kibana After deleting the index, I got a similar error. I generated some more log entries to the squid topic. Then I was able configure the index and the timestamp appeared.
... View more
10-11-2016
08:43 PM
@Aaron Harris Is there any more context to the error log. For example a log entry that says Joining problem? Also is your geo enrichment topology running cleanly? Check out the Storm UI. Go to the enrichment topology and see if you have any errors with the geo bolt. A common issue is that that geoenrichment bolt requires mysql. If it isn't running the geo bolt will fail. Check out Michael Young's article : https://community.hortonworks.com/content/kbentry/59801/troubleshooting-missing-events-in-metron-quick-dev.html It is geared toward quick dev but much of it applies to other deployments as well.
... View more