Community Articles

Follow the instructions here on how to download and import the vm into virtual boxOnce you have imported the vm select the vm and click on setting Then click on networkTo assign a IP in the attach to down drop list select "Bridge Adapter"Then under option Promiscuous Mode select "Allow All"Now start your vmOnce the machine is up verify you have a IP addressNow you have IP for your vm. have fun.

The release of version 1.0 marks another major milestone for Storm. Since becoming an Apache project in Sept 2013, much work has gone into maturing the feature set and also improving performance by reworking or tweaking various components. Some of the notable changes that contribute to improved performance are: Switch from ZeroMQ to Netty for inter-worker messaging. Employing batching in Disruptor queues (used for intra-worker messaging) Optimizations in the clojure code such as employing type hints and reducing expensive clojure lookups in perf sensitive areas. In this blog we shall take a look at performance improvements in Storm since its incubation into Apache. To quantify this, we shall compare the performance numbers of Storm v.0.9.0.1, which was the last pre-apache release, with the most recent Storm v1.0.1. Storm v.0.9.0.1 has also been used as a reference point for performance comparisons against Heron. Given the existence of recent efforts to benchmark Storm “at

Adding HDF (with Apache NiFi) to your HDP 2.5 Sandbox is very quick, painless and easy.Get the most recent Hortonworks DataFlow (download😞wget http://d3d0kdwqv675cq.cloudfront.net/HDF/centos6/1.x/updates/1.2.0.1/HDF-1.2.0.1-1.tar.gztar -xvf HDF-1.2.0.1-1.tar.gzcd HDF-1.2.0.1-1/nifi/Then change the port used by NiFi in the conf/nifi.properties file to:nifi.web.http.port=8090Install NiFi as a Linux Servicebin/nifi.sh installsudo service nifi startNiFi home: /opt/HDF-1.2.0.1-1/nifiBootstrap Config File: /opt/HDF-1.2.0.1-1/nifi/conf/bootstrap.conf2016-07-04 02:18:00,005 INFO [main] org.apache.nifi.bootstrap.Command Starting Apache NiFi...2016-07-04 02:18:00,006 INFO [main] org.apache.nifi.bootstrap.Command Working Directory: /opt/HDF-1.2.0.1-1/nifiYou can check the status of single NiFi server via status command:[root@sandbox nifi]# sudo service nifi statusnifi.sh: JAVA_HOME not set; results may varyJava home:NiFi home: /opt/HDF-1.2.0.1-1/nifiBootstrap Config File: /opt/HDF-1.2.0.1-1/nifi

Using Yahoo Kafka ManagerGit clone the project (you need Java 8 to build). Then use SBT to do a clean distribution.This will take a while as it downloads a lot of jars.<code>kafka-manager.zkhosts="sandbox.hortonworks.com:2181" The build will produce a Zip file, unzip it, update configuration file (conf/application.conf) and then you can run it.../kafka-manager/target/universal/kafka-manager-1.3.0.8.zipunzip ../kafka-manager/target/universal/kafka-manager-1.3.0.8.zipkafka-manager-1.3.0.8 git:(master) ✗ vi conf/application.confkafka-manager-1.3.0.8 git:(master) ✗ bin/kafka-manager -Dconfig.file=conf/application.confAccess the Kafka Manager from Chromehttp://localhost:9000/ Running Kafka Manager Resourceshttps://github.com/yahoo/kafka-managerhttp://edbaker.weebly.com/blog/install-and-evaluation-of-yahoos-kafka-managerhttp://chennaihug.org/knowledgebase/yahoo-kafka-manager/https://cwiki.apache.org/confluence/display/KAFKA/EcosystemToolsFor Testing Kafka with a command-line client

From the Sandbox as Root/usr/hdp/current/kafka-broker/bin/kafka-topics.sh--create --zookeeper sandbox.hortonworks.com:2181 --replication-factor 1--partitions 1 --topic peopleTest The Topic[root@sandbox kafka]# /usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh --topic people --zookeeper sandbox.hortonworks.com:2181{metadata.broker.list=sandbox.hortonworks.com:6667, request.timeout.ms=30000, client.id=console-consumer-10628, security.protocol=PLAINTEXT}Resources:https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.2/bk_secure-kafka-ambari/content/ch_secure-kafka-create-topics.htmlhttps://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.2/bk_secure-kafka-ambari/content/ch_secure-kafka-produce-events.html

In order to submit jobs to Spark, so called "fat jars" (containing all dependencies) are quite useful. If you develop your code in Scala, "sbt" (http://www.scala-sbt.org) is a great choice to build your project. The following relies on the newest version, sbt 0.13 For fat jars you first need "sbt-assembly" (https://github.com/sbt/sbt-assembly). Assuming you have the standard sbt folder structure, the easiest way is to add a file "assembly.sbt" into the "project" folder containing one line addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.14.3") The project structure now looks like (most probably without the "target" folder which will be created upon building the project) MyProject+-- build.sbt+-- project| +-- assembly.sbt+-- src| +-- main| +-- scala| +-- MyProject.scala+-- target For building Spark Kafka Streaming jobs on HDP 2.4.2, this is the build file "build.sbt" name := "MyProject"version := "0.1"scalaVersion := "2.10.6"resolvers += "Hortonworks

In HDP 2.5 Sandbox, I did a quick walk through.First thing I liked was the Visualization:The Data Explorer provided a nice query tool to view tables and graphs.The Data Visualization tab provides some nice graphing capabilities.After you run your queries you can look at the Tez results to see how it ran, it's a nice way to see what you may need to optimize.The Hive Ambari View is getting to be a very solid tool for working with Hive. From DDL (creating tables is easy) to viewing data, to updates and inserts.

We have Hive over Hbase table and lets say there are few columns with INT datatype, data loaded from Hive. Now if we would like to delete data based on values present in that particular column(INT), is not possible. It is because values are converted to Binary, even HBase API filter(SingleColumnValueFilter) would return wrong result if we query that particular column values from HBase. Problem to solve: How purge Hive INT datatype column data from HBase?This is the first textual series containing the resolution of above problem. Next series i'll create a small video on running code and cover other datatypes too.In such scenario we cant use standard API and unable to apply filters on binary column values, Solution :- Below JRuby program code.So you have already heard many advantages of storing data in HBase(specially binary block format) and create Hive table on top of that to query your data. I am not going to explain use case for this, why we required HBase over Hive but simple reason

Introduction This article continues where part 1 left off. It describes a few more configuration settings that can be enabled in CDP, HDP, CDH, or Apache Hadoop clusters to help the NameNode scale better. Audience This article is for Hadoop administrators who are familiar with HDFS and its components. If you are using Ambari or Cloudera Manager you should know how to manage services and configurations. It is assumed that you have read part 1 of the article. RPC Handler Count The Hadoop RPC server consists of a single RPC queue per port and multiple handler (worker) threads that dequeue and process requests. If the number of handlers is insufficient, then the RPC queue starts building up and eventually overflows. You may start seeing task failures and eventually job failures and unhappy users. It is recommended that the RPC handler count be set to 20 * log2(Cluster Size) with an upper limit of 200. e.g. for a 250 node cluster you should initialize this to 20 * log2(250) = 160. The RPC h

Query JSON using Spark Imagine you are ingesting JSON msgs but each one has different tag names or even a different structure. This is very common because JSON is a flexible nested structure. However we commonly interact with data in a flat table like structure using SQL. The decision becomes to either parse the dynamic data into a physical schema (on write) or apply a schema at runtime (on read). Ultimately the decision will likely be made based on the number of writes vs reads. However there is one major advantage to using Spark to apply schema on read to JSON events, it alleviates the parsing step. Typically you have to hand code all the tags in the JSON msgs and map each one to a schema column. This may require meeting with upstream teams or third parties to get the DDL/xsd or schema definition. It also doesn't protect you from msgs you haven't seen or new tags being added to existing JSON structures. Sparks schema on read handles all of this as well as flattens the structur

This tutorial will show how to export data out of hbase table into csv format. We will use airport data from american statical association available here. Assume you have a sandbox up and running lets start. First ssh into your sandbox and switch user to hdfssudo su - hdfsThen grab the airport data by issues a wgetwget http://stat-computing.org/dataexpo/2009/airports.csvFor my example the file is located/home/hdfs/airports.csvNow lets create a hbase table called "airports" with column family "info". Do this in hbase shellNow that the table is created lets load it. Get out of hbase shell. as user hdfs run the following to load the tablehbase org.apache.hadoop.hbase.mapreduce.ImportTsv -Dimporttsv.separator=, -Dimporttsv.columns="HBASE_ROW_KEY,info:iata,info:airport,info:city,info:country,info:lat,info:long" airports hdfs://sandbox.hortonworks.com:/tmp/airports.csvThat will kick off map reduce job to load airport table in hbase. once that is done you can do a quick verify in hbase

ObjectiveDeploy a 4-node HDP 2.4.2 cluster with Apache Ambari 2.2.2, Vagrant and VirtualBox on OS X host. This is helpful for development and proof of concepts.ScopeThis approach has been tested on OS X host, but it should work on all supported Vagrant and VirtualBox environments.Pre-requisitesMinimum 9 GB of RAM for the HDP 2.4.2 clusterDownload and install Vagrant for your host OS: https://www.vagrantup.com/downloads.htmlDownload and install VirtualBox for your host OS: https://www.virtualbox.org/wiki/DownloadsDownload and install git client for your hostOpen a command shell and change to the folder where you plan to clone the github repositoryClone the following git repositorygit clone https://github.com/cstanca1/hdp2_4_2-vagrant.gitCreate and Start VMsChange directory to /hdp_2.4.2-vagrant, the folder that includes Vagrantfile and create a /data folder: mkdir dataThis /data folder will be needed for guest VMs to share with the host. Vagrant (via Vagrantfile) is configured to use Ce

We have seem many incidents of overloaded HDFS namenode due to 1) misconfigurations or 2) “bad” MR jobs or Hive queries that create large amount of RPC requests in a short period of time. There are quite a few features that have been introduced in HDP 2.3/2.4 to protect HDFS namenode. This article summarize the deployment steps of these features with an incomplete list of known issues and possible solutions for them. Enable Async Audit LoggingDedicated Service RPC PortDedicated Lifeline RPC Port for HAEnable FairCallQueue on Client RPC PortEnable RPC Client Backoff on Client RPC portEnable RPC Caller Context to track the “bad” jobsEnable Response time based backoff with DecayedRpcSchedulerCheck JMX for namenode client RPC call queue length and average queue timeCheck JMX for namenode DecayRpcScheduler when FCQ is enabledNNtop (HDFS-6982)1. Enable Async Audit LoggingEnable async audit logging by setting "dfs.namenode.audit.log.async" to true in hdfs-site.xml. This can minimize the impa

Hortonworks Data Platform ArtifactsDeveloping solutions with Hadoop commonly requires the use of multiple different HDP component libraries. Whether you’re building solutions with Pig, Spark, Cascading, or HBase, at some point extensions will need to be created, and those artifacts for each component will need to be used. This guide serves as an overview of where to find those artifacts and how to get them quickly integrated with your preferred build tool, and IDE.ReposotoriesAt Hortonworks, we store all of our artifacts in a public Sonatype Nexus repository. That repository can be easily accessed and searched for commonly used library, source code, and javadoc archives simply by navigating to http://repo.hortonworks.com.ArtifactsJar files containing compiled classes, source, and javadocs are all available in our public repository, and finding the right artifact with right version is as easy as searching the repository for classes you need to resolve. For example, If creating a sol

Storm provides rich set of built-in metrics which are provided to Storm UI, and also metrics consumer. However these built-in metrics are measured for each task, while we are occasionally configuring a task to have multiple calls for external storages, and/or heavy computation sequentially. Let’s say we have a Bolt which crawls the page from the web, and store page to external storage.public class StoreCrawledPageBolt extends BaseRichBolt { @Override public void execute(Tuple input) { String url = tuple.getString(0); // the execute latency for crawlPage and storePage can be vary String html = crawlPage(url); storePage(url, html); collector.ack(input); } // skip method implementations...}The latency for crawling page from web server is vary, which takes several milliseconds to tens of seconds which might be better to treat as time-out. The latency for storing page data to external storage would be relatively faster, but it might be slower for specific reason like m

Driven first-hand experience: While researching for a Hive Query Metrics UI found this tool called driven. Step1: Register for a free trial at http://www.driven.io/hive/ Step2: Upon clicking the verification link sent over email came to this page: Click on “Install the Driven Agent” Step3: On clicking that two things will happen: You will be taken to the configuration and install instructions You will get an email with username and password Step4: Click on https://trial.driven.io/index.html#/apps from screenshot 2 You will be in the login screen. Then enter the id and password you got in your email. Step5: After logging in you will prompted to change the password. Change that and click submit. Step6: You will be logged into the user interface. But since the agent is not installed you cannot view anything. You will be also provided with your API Key. Step7: Hive is installed in the cluster in a node named hiveserver. Am going to install

One common problem in SQL is that you join two tables and get duplicate column names from the two tables. When you now for example want to create a CTAS you will get "duplicate column name" errors. You also often want to exclude the join key from the result set since it is by definition duplicate.Database schemas often prefix column names with a letter from the table to fix at least the first issue. Like TPCH: lineitem.l_key and orders.o_key. A common approach to fix this is to explicitely specify all column names in your SELECT list. However it looks like Hive has some cool/dirty tricks up its sleeve to make this easier. Regular Expressions to specify column names.Testsetup:describe tb1;id intname stringdescribe tb2;id intage intYou then have to disable the use of quotes in identifiers because that interferes with the regex.set hive.support.quoted.identifiers=none;And now you can use Java Regex to select all columns from the right

Disaster recovery plan or a business process contingency plan is a set of well-defined process or procedures that needs to be executed so that the effects of a disaster will be minimized and the organization will be able to either maintain or quickly resume mission-critical operations.Disaster usually comes in several forms and need to planned for recovery accordingly:Catastrophic failure at the data center level, requiring failover to a backup locationNeeding to restore a previous copy of your data due to user error or accidental deletionThe ability to restore a point-in-time copy of your data for auditing purposesDisclaimer: 1. This article is solely my personal take on disaster recovery in a Hadoop cluster 2. Disaster Recovery is specialized subject in itself. Do not Implement something based on this article in production until you have a good understanding on what you are implementing.Key objectives: Minimal or no downtime for production cluster Ensure High Availability

There're already several articles describing how to setup NiFi to connect Kafka or HDFS. However, following scattered piece of documentations to complete Kerberizing HDP, and setting ACL of Zookeeper, Kafka, HDFS, and Kerberos was not an easy task for me. So, the motivation of this article is to share rough but throughout operations needed to set it up right. You can elaborate each step by digging related documentation further. Example configuration and NiFi template are available, too. Some of these steps may not be required, and some were forgotten to be here, but these were the steps to Kerberize my HDP sandbox VM: Install by downloading the latest HDF Start Kafka, stop maintenance mode. Restart all affected services Install a new MIT KDC [1] KDC Enable Kerberos from Ambari Proceed with Ambari Kerberos wizard Check Pig failed to pass the test, but continued with Complete button anyway Start services manually that didn't start automatically Configured Kafka for Kerberos

Chronos is a replacement for cron. A fault tolerant job scheduler for Mesos which handles dependencies and ISO8601 based schedules Marathon is a framework for Mesos that is designed to launch long-running applications, and, in Mesosphere, serves as a replacement for a traditional system In Mesosphere, Chronos compliments Marathon as it provides another way to run applications, according to a schedule or other conditions, such as the completion of another job. It is also capable of scheduling jobs on multiple Mesos slave nodes, and provides statistics about job failures and successes. Source Install https://mesos.github.io/chronos/docs/ and gist Demo Part 1 - https://www.linkedin.com/pulse/data-center-operating-system-dcos-part-1-neeraj-sabharwalPart 2 - https://www.linkedin.com/pulse/apache-marathon-part-2-neeraj-sabharwal

You need Mesos to run this - Post 1 What is Apache Marathon? Marathon is a production-grade container orchestration platform for Mesosphere'sDatacenter Operating System (DCOS) and Apache Mesos. I am launching multiple applications using Marathon and Mesos is providing the framework to launch those applications. Demo More readinghttps://mesosphere.github.io/marathon/Gist & Application example

DC/OS - a new kind of operating system that spans all of the servers in a physical or cloud-based datacenter, and runs on top of any Linux distribution. Source Projects More details https://docs.mesosphere.com/overview/components/ Let's cover Mesos in this post Frameworks (Application running on mesos) http://mesos.apache.org/documentation/latest/frameworks/ I used http://mesos.apache.org/gettingstarted/ to install Mesos in my local machine. I am launching c++, java and python framework in this demo. Mesos demo More reading

This tutorial is great:https://github.com/hortonworks-gallery/ambari-vnc-serviceEclipse Pluginhttps://github.com/winghc/hadoop2x-eclipse-pluginJDK 7 is best for most use case and Scala 2.10.Maven and SBT are necessary as well.Setup your Environmenthttps://dzone.com/articles/spark-and-scala-resourceshttps://dzone.com/articles/whats-on-your-laptopLots of options:This is an eclipse project for Hbase Coprocessorhttps://github.com/tspannhw/hbasecoprocessorArtem has a great project for testinghttps://github.com/dbist/HBaseUnitTestOnce all the ports are open and not firewalled it’s usually straight forward.Eclipse to Sparkhttps://community.hortonworks.com/questions/36354/eclipse-to-sandbox-1.htmlhttps://community.hortonworks.com/questions/32567/scala-with-hive-in-ecplipse-scala.htmlHadoop Eclipse Pluginhttps://community.hortonworks.com/questions/10404/hadoop-eclipse-plugin.htmlIntelliJ Project for Sparkhttps://github.com/agilemobiledev/sparkworkshophttps://community.hortonworks.com/questions/

Hive: Apache Hive is a data warehouse infrastructure built on top of Hadoop for providing data summarization, query, and analysis. HBase: Apache HBase™ is the Hadoop database, a distributed, scalable, big data store Hawq: http://hawq.incubator.apache.org/ PXF: PXF is an extensible framework that allows HAWQ to query external system data Let's learn Query federation This topic describes how to access Hive data using PXF. Link Previously, in order to query Hive tables using HAWQ and PXF, you needed to create an external table in PXF that described the target table's Hive metadata. Since HAWQ is now integrated with HCatalog, HAWQ can use metadata stored in HCatalog instead of external tables created for PXF. HCatalog is built on top of the Hive metastore and incorporates Hive's DDL. This provides several advantages: You do not need to know the table schema of your Hive tables You do not need to manually enter information about Hive table location or format If Hive table metadat

I found this article "For mobile analytics, Yahoo is in the process of replacing HBase with Druid" History : 24th Oct, 2012 To test out the setup, I have deployed druid in 2 clusters. first deployment is in my multi node cluster and 2nd deployment is using this repo. Details are on this blog Demo - PS: It's 10 minutes demo We are loading pageviews and then executing queries. See links at the bottom to download the git and code. Gif Download I use this to control gif movement Links: Page view queries and data Spin up the environment in your mac or windows "not sure about windows" Git link . This will spin up Druid, ZK, Hadoop, Postgres Gist Happy Hadooping!!!!