Resources

Original Post DC/OS - a new kind of operating system that spans all of the servers in a physical or cloud-based datacenter, and runs on top of any Linux distribution. Source Projects More details https://docs.mesosphere.com/overview/components/ Let's cover Mesos in this post Frameworks (Application running on mesos) http://mesos.apache.org/documentation/latest/frameworks/ I used http://mesos.apache.org/gettingstarted/ to install Mesos in my local machine. I am launching c++, java and python framework in this demo. Slide Share http://www.slideshare.net/tomasbart/introduction-to-apache-mesos

IntroductionMirrorMaker can be used to replicate messages of a defined list of topics across clusters.ConfigurationWe need to configure both the producer and the consumer through config files. The config file for the producer will define the destination cluster Kafka brokers, whereas the consumer one will point to the Zookeeper servers from the source cluster. The following configuration also assumes that security is enabled (Kerberos) so the PLAINTEXTSASL protocol will be used:mm_producer-1.properties:bootstrap.servers=dest-kafkabrk1:6667,dest-kafkabrk2:6667,dest-kafkabrk3:6667producer.type=asyncqueue.time=1000000queue.enqueueTimeout.ms=-1security.protocol=PLAINTEXTSASL#sasl.kerberos.service.name=kafkamm_consumer-1.properties:zookeeper.connect=src-zkhost1:2181,src-zkhost2:2181,src-zkhost3:2181zk.connectiontimeout.ms=1000000bootstrap.servers=src-kafkabrk1:6667,src-kafkabrk2:6667,src-kafkabrk3:6667consumer.timeout.ms=-1security.protocol=PLAINTEXTSASLgroup.id=kafka-mirror#sasl.kerberos.s

1.Login to Ambari UI using admin credentials [admin/admin]2.Check the alert definition using below command http://<ambari_fqdn>:8080/api/v1/clusters/<cluster-name>/alert_definitions/3.Get the respective alert definition you wan to modify from the above output. Here for example lets say we will modify "Hive Metastore".Here we will change 'check.command.timeout' value from default 60 to 120 4.Copy the output of point2 to a file say "test.json"5.First - Modify the test.json and remove the 2nd line of the file which starts with "href" eg. is below -"href" : "http://<ambari-fqdn>:8080/api/v1/clusters/sandbox/alert_definitions/51",6.Second - edit the value of 'check.command.timeout' in test.json from 60.0 to 120.0{ "name" : "check.command.timeout", "display_name" : "Check command timeout", "units" : "seconds", "value" : 120.0, "description" : "The maximum time before check command will be killed by timeout", "type" :

Introduction: Here is a small demo how NiFi can let you retrieve Stock quotes on voice commands and respond back with the help of Mac Dictation.- Here you can view the screen recording session that Demonstrates how it works! NiFi + Mac Dictation DemoPrerequisite: 1) Assuming you already have latest version of NiFi [0.5.x or later] downloaded on your Mac. 2) Make sure you have Mac dictation feature, and its working! Steps: 1) Use GetHttp processor to Retrieve any Stock price from “Google Finance”, you can use the below URL to fetch YAHOO stock price: http://finance.google.com/finance/info?client=ig&q=YHOO - Processor Properties would look like: 2) Use ReplaceText and EvaluateJsonPath processors to trim json result and extract stock value and variation in stock price today. Use UpdateAttribute processor to save values to attributes. 3) Now you can use a RouteOnAttribute to decide when to trigger mail or/and give voice feedback. 4) Finally Use PutEmail process

Using Sqoop (Sql from/to Hadoop), you can use a password file instead of a plaintext password, which is more secure :<arg>--password-file</arg> <arg>hdfs://NAMENODE/teradata.password</arg> You may end with error like 3737 [main] ERROR org.apache.sqoop.teradata.TeradataSqoopExportHelper - Exception running Teradata export jobcom.teradata.connector.common.exception.ConnectorException: java.sql.SQLException: [Teradata Database] [TeraJDBC 15.00.00.20] [Error 8017] [SQLState 28000] The UserId, Password or Account is invalid. If you set your “Password” pass with vi you’ll end with a line feed control character, making the password invalid. To find if there’s a LF ending the password file, use od (display file in octal format): [root@localhost ~]# od -c teradata.password 0000000 P a s s w o r d \n 0000011You’ll have to delete your newline control character using tr : [root@localhost ~]# tr -d '\n' < teradata.password > teradata.password.new [r

You can now visualize any Zeppelin notebook using Zeppelinhub viewer. https://www.zeppelinhub.com/viewerpersonal likes:1. No need to sign up or register just paste a link2. I've been posting my zeppelin notebooks to github but everyone that wants to visualize them or interact with them needs to download, move to environment, import into their instance of zeppelin. Not anymore just paste the link.3. Less of a need to take screenshots and create a powerpoint just send the hyperlinkexamples: Stock Variance Notebook github - https://github.com/kirkhas/zeppelin-notebooks/blob/master/stock-variance/note.jsonvsStock Variance Notebook zephub - https://www.zeppelinhub.com/viewer/notebooks/aHR0cHM6Ly9yYXcuZ2l0aHVidXNlcmNvbnRlbnQuY29tL2tpcmtoYXMvemVwcGVsaW4tbm90ZWJvb2tzL21hc3Rlci9zdG9jay12YXJpYW5jZS9ub3RlLmpzb24Credit Card Fraud Transactions git - https://github.com/vakshorton/CreditCardTransactionMonitor/blob/master/Zeppelin/notebook/2BGDWYZV9/note.jsonvshttps://www.zeppelinhub.com/viewer/

A question encountered by most organizations when attempting to take on new Big Data Initiatives or adapt current operations to keep up with the pace of innovation is What approach to take in architecting their streaming workflows and applications. Because Twitter data is free and easily accessible I chose to examine twitter events in this article; however, twitter events are simply json messages and can be conceptualized as machine generated json packets for a more generalized streaming architecture. Lets move forward with an evaluation of NiFi vs Python ingesting streaming events from twitter. When approaching a problem there are often several tools and methods that could ultimately yield similar results. In this article I will look at the problem of ingesting streaming twitter events and show how HDF and python can be used to create an application which achieves the goal. I will first lay out a solution using both platforms, then I will point out different aspects and co

Hello I am so sorry but I could not find out change passsword section. How can I change my password ?Actually I also want to change my security question Thanks

Modern data driven applications require a "Connected Platform" capable of bringing data in and out to/from the Internet of Things, mobile users, and social media in real time. In order to monetize all of that real time data the platform must have the ability to process Petabytes of data to create adaptive learning algorithms and apply those algorithms in real time as the data streams in and out of the platform. However, the modern data application cannot be effectively utilized or operated without an application tier that allows the business to visualize, interact, and act on the massive volumes of data and insight coming in and out in real time and accumulating in storage. The Hortonworks Connected Platform "HDP+HDF" has the capability to act as a PaaS that can host the application tier of the modern data application along side of all of the data processing. It is possible to use Slider to run a dockerized application managed by Yarn inside of the Hadoop cluster similar to an applicat

I ran into some issues using the latest iODBC 3.52.10 version from www.iodbc.org (mxkozzzz.dmg)Instead of~/.odbc.ini ~/.odbcinst.ini ~/.hortonworks.hiveodbc.iniit uses (without leading dots):~/Library/ODBC/odbc.ini ~/Library/ODBC/odbcinst.ini ~/Library/ODBC/hortonworks.hiveodbc.iniNote: iODBCwill link ~/.odbc.ini and ~/.odbcinst.ini to the ~/Library/ODBC/ versionsMy setup:~/Library/ODBC/odbcinst.ini - edit at least host, port and user ~/Library/ODBC/odbcinst.ini - keep as is ~/Library/ODBC/hortonworks.hiveodbc.ini - Change ODBCInstLib to the fully qualified path of iODBC's libiodbcinst.dylib under /usr/local/iODBC/lib/[Driver]## - Note that this default DriverManagerEncoding of UTF-32 is for iODBC.DriverManagerEncoding=UTF-32ErrorMessagesPath=/usr/lib/hive/lib/native/hiveodbc/ErrorMessages/LogLevel=0LogPath=SwapFilePath=/tmp# iODBCODBCInstLib=/usr/local/iODBC/lib/libiodbcinst.dylib{ignore the "5." in the output above - no idea how to get rid of it}

When Storm developers want to share data across multiple bolts or cache a large amount of state information in a single bolt, one of the common choices is to use HBase. In a unsecured HDP cluster, the related code in the HBase Bolt is very intuitive:public class myHBaseBolt implements IRichBolt { ... private OutputCollector collector; private Connection connection; private Table myTable; ... public void prepare(Map stormConf, TopologyContext context, OutputCollector collector) { this.collector = collector; try { this.connection = ConnectionFactory.createConnection(HBaseConfiguration.create()); this.myTable = connection.getTable(TableName.valueOf(MY_TABLE_NAME)); ... } catch (Exception e) { ... } }}However, in a Kerberized HDP cluster, we need to configure lots of information in our code to enable secured connection. First, we need to configure storm keytab and principal for the hbase client in storm bolt. For example, in the myTopology.java code:Map<String, Object>

IntroductionApache Storm is a free and open source distributed realtime computation system. Storm makes it easy to reliably process unbounded streams of data. In this article we explore the support for windowing operations and stateful processing that were recently added to Apache Storm.Windowing support Windowing computations is one of the common use cases in stream processing where the unbounded stream of data is split into finite sets based on some criteria (e.g. time) and computation is applied on each group of events. An example would be to compute the top trending twitter topic in the last hour. Windowing is primarily used for aggregations, joins, pattern matching and more. Windows can be seen as an in-memory table where events are added and evicted based on some policies. So far Apache Storm relied on developers to built their own windowing logic. There were no recommended or high level abstractions that developers could use to define a Window in a standard way in a Topology

In previous article of the sereies, Enriching Telemetry Events, we walked through how to enrich a domain element of a given telemetry event with WhoIs data like home country, company associated with domain, etc. In this article, we will enrich with a special type of data called threat intel feeds. When a given telemetry event matches data in a threat Intel feed, an alert is generated. Again, the customers requirement are the following: The proxy events from Squid logs needs to ingested in real-time.The proxy logs has to be parsed into a standardized JSON structure that Metron can understand.In real-time, the squid proxy event needs to be enriched so that the domain named are enriched with the IP informationIn real-time, the IP with in the proxy event must be checked against for threat intel feeds.If there is a threat intel hit, an alert needs to be raised.The end user must be able to see the new telemetry events and the alerts from the new data source.All of this requirements will nee

I'm a long-time user of Apache Bigtop. My experience with Hadoop and Bigtop predates Ambari. I started using Bigtop with version 0.3. I remember pulling bigtop.repo file and install Hadoop, Pig and Hive for some quick development. Bigtop makes it convenient and easy. Bigtop has matured since then and there are now multiple ways of deployment. There's still a way to pull repo and install manually but there's better ways now with Vagrant and Docker. I won't rehash how to deploy Bigtop using Docker as it was beautifly described here. Admittedly, I'm running it on Mac and was not able to provision a cluster using Docker. I did not try with non-OSX. This post is about Vagrant. Let's get started:Install VirtualBox and VagrantDownload 1.1.0 releasewget http://www.apache.org/dist/bigtop/bigtop-1.1.0/bigtop-1.1.0-project.tar.gzuncompress the tarballtar -xvzf bigtop-1.1.0-project.tar.gzchange directory to bigtop-1.1.0/bigtop-deploy/vm/vagrant-puppet-vmcd bigtop-1.1.0/bigtop-deploy/vm/vagrant-pup

aka YAND; Yet Another Nifi Demo The recording explains it all really, but see below for detailed instructions on how to duplicate it. Recording of SAMS in action ## Background I recently attended YASK, the Hortonworker sales kickoff, and as a part of the training we were to spend some time learning and presenting the corporate sales pitch. This is 35 slides covering the major story points of the business, and takes about 20mins to blast through if you don't stop for questions. It isn't for field use, but it's a good educational tool to get new people up to speed. Instead of heading back to my hotel room the night before, I got distracted towards the pub by a regional SE director. After a couple of beers we got around to discussing the schedule for the next day, and I joked that it would be pretty funny to use Nifi to do the pitch a la @Jeremy Dyer. Grant (the director) uttered the immortal phrase "I'd be really disappointed if you didn't", and now we're here. ## Credits P

One of the key design principles of Apache Metron is that it should be easily extensible. We envision many users using Metron as a platform and building custom capabilities on top of it; one of which will be to add new telemetry data sources. In this multi-part article series, we will walk you through how to add a new data telemetry data source: Squid proxy logs.This multi-part article series consists of the following: This Article: Sets up the use case for this multi-part article seriesUse Case 1: Collecting and Parsing Telemetry Events - This tutorial walks you through how to collect/ingest events into Metron and then parse them.Use Case 2: Enriching Telemetry Data - Describes how to enrich elements of telemetry events with Apache Metron.Use Case 3: Adding/Enriching/Validating with Threat Intel Feeds - Describes how to add new threat intel feeds to the system and how those feeds can be used to cross-reference every telemetry event that comes in. When a hit occurs, an alert will be ge

User demouser needs access to /landingzone. Super user HDFS created a directory called /landingzone and user demouser does not have access to it. We will use Ranger to control the access.

Data Federation discussions are becoming more and more common place as organizations embark on their Big Data Journey. New data platforms like the Hortonworks Connected platform (HDP+HDF), NoSQL, and NewSQL data stores are reducing the cost and difficulty of storing and working with vast volumes of data. This is empowering organizations to leverage and monetize their data like never before. However, legacy data infrastructures still play an important role in the overall technology architecture. In order to fully realize the power of the new and traditional data platforms, it is often necessary to integrate the data. One obvious approach is to simply move the data from where it sits in the existing data platform over to the target data platform. However, in many cases it is desirable to leave the data in place and enable a "Federation" tier to act as a single point of access to data from multiple sources. For details on the concepts and implementation of Data Federation see https://comm

Metron TP1 Features The following are key capabilities available in Metron TP1 broken up across its four key functional themes.How do I get Started?You can spin up the Metron TP1 in two ways: Ansible based Vagrant Single Node VM Install This the best place to play with Metron First.Detailed instructions how to do the install can be found in the following HCC Article: Apache Metron TP 1 Install Instructions- Single Node Vagrant Deployment Fully Automated 10 Node Ansible Based Install on AWS using Ambari Blueprints and AWS APIs If you want a more realistic setup of the Metron app, use this approach. Keep in mind that this install will spin up 10 m4.xlarge EC2 instance by defaultDetailed instructions how to do the install can be found in the following HCC Article: Apache Metron - First Steps in the CloudWhere do I get Help?Hortonworks has created new Track called CyberSecurity in the Hortonworks Community Connection (HCC). The link to the this new track in HCC is the following: HCC Cyb

Falcon by default coes with authorization turned off. To turn on set the following through ambari falcon config:*.falcon.security.authorization.enabled = true*.falcon.security.authorization.superusergroup = <linux group>In my example I am using linux group usersIn my example oozie,ambari-qa,tez,falcon,hue,guest are in the group "users". The purpose of this group is to only allow users within this group to view, edit, and delete each others material. Any user outside this group should not have access.Now logging in a falcon who is part of users group:Falcon user has created a cluster "authTest" and feed "feed1" Lets view it:Great so falcon is see the feed and cluster. Now lets go in with user hdfs who is NOT part of group usersLogged in as user hdfs who is NOT part of group users. This user will do a simple search for everything cluster/feed entity which exist in falcon.So hdfs user search does not return anything since the use is not allowed. Now lets log in with user tez wh

I have seen phantom or false alerts being generated by Ambari Alerts that clear on the next testing cycle without any evidence of an actual outage of the component. This issue may be related to a race condition within the python scripts where the check for whether the kerberos ticket is still valid occurs right at the end of the 5 minute lifetime and becomes invalid before the actual alert check can be run against the component. This causes the script to error-out which is taken as a failure of the alert check. In order to try to avoid this race condition, it has been suggested and there has been some early evidence to support the fact that increasing the kerberos lifespan setting to be a different, longer time than the alert check interval (which by default is 5 minutes).The parameter "-l 5m" (that's a lower-case L) is used within these two python scripts to set the lifetime to 5 minutes:/usr/lib/ambari-server/lib/resource_management/libraries/functions/curl_krb_request.py /var/lib

When performing a Rolling or Express Upgrade, failures can naturally happen because large clusters are bound to have problematic hosts. Here are 10 easy tips to prevent, diagnose and fix errors.Before upgrading the stack ... 1. Always upgrade Ambari to the most recent version, even if it's a dot release. Often, there are fixes and optimizations that make the stack upgrade smoother. 2. Ensure all services are up, service checks are passing, there are no critical alerts, etc. This helps ensure that the cluster is fully operational and helps to isolate any failures. 3. Pre-Install the bits and make sure all hosts have enough disk space. You can check that the version is found on all hosts. E.g., hdp-select versions | grep 2.5.0.0 | sort | tail -1 4. Do not ignore warnings. Starting in Ambari 2.2.2, there's a flag in ambari.properties file that allows users to bypass PreCheck errors, make sure it is either not present or set to false, stack.upgrade.bypass.prechecks=false 5. T

IntroductionApache Tez has become a very important framework and API to support batch and interactive over terabytes and petabytes of data for many engines within HDP such as Pig, Hive, Java. Cascading and others, with performance advantages at scale over Map Reduce and even Spark at certain volumes of data.For more on Apache Tez see http://hortonworks.com/hadoop/tez/This article is meant to outline the best practice in configuring and tuning Tez, and why you would set certain values in certain properties to get performance at scale, with step by step instructions.With this in place you would hopefully prevent out of memory errors when you execute your Hive Queries or Pig Scripts as seen in https://community.hortonworks.com/questions/5780/hive-on-tez-query-map-output-outofmemoryerror-java.html https://community.hortonworks.com/questions/12067/what-is-the-workaround-when-getting-hive-outofmemo.htmlTez Memory DemystifiedI find a diagram usually helps to understand why you would set cert

So you've just got all excited about the Metron Tech Preview, have downloaded the code and maybe have even followed the Apache Metron - First Steps in the Cloud article, but then EC2 instances pop up where you don't expect to see them!Unfortunately at this stage the Metron Tech Preview deployment ignores some of the systemwide settings that you might have in your .aws directory including the preferred region and availability zone.Thankfully all the settings are within the setup scripts instead so this is easy enough to correct.There are two settings I'd suggest updating:./deployment/amazon-ec2/conf/defaults.yamlContains the key settings you will need to change towards the top of the file:# ec2env: test-metronregion: us-west-2instance_type: m4.xlargeimage: ami-05cf2265For example, to change this to run in the EU Ireland region your config settings would need to look like this:# ec2env: mynamehere-metronregion: eu-west-1instance_type: m4.xlargeimage: ami-edb9069eNote that I've also ch

In a multiple application environment (MapReduce, Tez, others), it can appear that some application get stuck (blocked, deadlocked), it is likely your YARN / MAPREDUCE settings need to be reviewed. First of all, setup YARN queue, check the value of mapreduce.job.reduce.slowstart.completedmaps (interesting to change it to 0.9) and enable preemption.