Resources

su hdfshadoop fs -mkdir /udfhadoop fs -put urldetector-1.0-jar-with-dependencies.jar /udf/hadoop fs -put libs/url-detector-0.1.15.jar /udf/hadoop fs -chown -R hdfs /udfhadoop fs -chgrp -R hdfs /udfhadoop fs -chmod -R 775 /udfCreate Hadoop Directories and upload the two necessary libraries. CREATE FUNCTION urldetector as 'com.dataflowdeveloper.detection.URLDetector' USING JAR 'hdfs:///udf/urldetector-1.0-jar-with-dependencies.jar', JAR 'hdfs:///udf/url-detector-0.1.15.jar';Create Hive Function with those HDFS referenced JARsselect http_user_agent,urldetector(remote_host)asurls,remote_host from AccessLogs limit 100;Test the UDF via Hive QL@Description(name="urldetector", value="_FUNC_(string) - detectsurls")public final class URLDetector extends UDF{}Java Header for the UDFset hive.cli.print.header=true;add jar urldetector-1.0-jar-with-dependencies.jar;CREATE TEMPORARY FUNCTION urldetector as 'com.dataflowdeveloper.detection.URLDetector';select urldetector(description) from sample_07 lim

In general, you have following options when running R on Hortonworks Data Platform (HDP) -o RHadoop (rmr) - R program written in MapReduce paradigm. MapReduce is not a vendor specific API and any program written with MapReduce is portable across Hadoop distributions.https://github.com/RevolutionAnalytics/RHadoop/wiki/rmro Hadoop Streaming - R program written to make use of Hadoop Streaming but the program structure still aligns with MapReduce. Above benefit still applies.o RJDBC - This example does not require the R programs to be written using MapReduce and still remains 100% native R APIs without any third party packages.Here is a tutorial with a video, sample data and R script:http://hortonworks.com/hadoop-tutorial/using-revolution-r-enterprise-tutorial-hortonworks-sandbox/Using RJDBC, the R program can have Hadoop parallelize pre-processing and filtering. R submits a query to Hive or SparkSQL making use of distributed and parallel processing. Then uses existing R models, as is &amp

Objective The objective of this article is to provide an introduction to User-Defined Functions for spatial analysis in Hive.Credit This article is based on multiple ESRI resources referenced throughout the article. Thanks also to my colleague Artem Ervits for his contribution to the proof-of-concept that lead to this article.Spatial Types Spatial types are data types that store geometry data and they have associated functions or methods that are used to access and manipulate the data using Structured Query Language (SQL). Unlike Oracle, DB2, SQL Server, or PostGreSQL, Hive does not support yet a geometry native data type. The best approach is to store spatial data as text. That will not impact the use of UDFs available for Hive.ESRI Libraries The Esri Geometry API for Java includes geometry objects (e.g. points, lines, and polygons), spatial operations (e.g. intersects, buffer), and spatial indexing. By deploying the library (as a jar) within Hadoop, you are able to build custo

On Kerberos-secured clusters managed by Ambari versions prior to 2.0.0, the hdfs.headless.keytab file is deployed on all the nodes in the cluster. From Ambari version 2.0.0 onwards, this keytab file is not deployed by default on every node. If the Ranger HBase plugin is enabled and set up to write to HDFS, Ambari's RegionServer start up script tries to access the hdfs.headless.keytab file. If the host that the RegionServer is running on does not also have the HDFS client role assigned to it, the hdfs.headless.keytab file will be absent and the RegionServer will fail to start with an error such as this in Ambari: resource_management.core.exceptions.Fail: Execution of '/usr/bin/kinit -kt /etc/security/keytabs/hdfs.headless.keytab hdfs-examplecluster@KRBEXAMPLE.COM' returned 1. kinit: Key table file '/etc/security/keytabs/hdfs.headless.keytab' not found while getting initial credentials This problem is expected to be resolved with the release of Ambari 2.4.0, but until then, the work-

I have done data analysis for one of my project using below approach and hopefully it may help you understand underlying subject. Soon i'll post my project on data analysis and detail description on technology used Python(web scraping- data collection), Hadoop, Spark and R.Data analysis is a highly iterative and non-linear process, better reflected by a series of cyclic process, in which information is learned at each step, which then informs whether (and how) to refine, and redo, the step that was just performed, or whether (and how) to proceed to the next step.Setting the SceneData analysis is a study of subjective question and study even includes developing and executing a plan for collecting data, a data analysis presumes the data have already been collected. More specifically, a study includes the development of a hypothesis or question, the designing of the data collection process (or study protocol), the collection of the data, and the analysis and interpretation of the data.Act

I have worked with almost 20 to 25 applications. Whenever i start working first i have to understand each applications naming convention and i keep thinking why we all not follow single naming convention. As Hadoop is evolving rapidly therefore would like to share my naming convention so that may be if you come to my project will feel comfortable and so as I if you follow too.Database Names:If application serve to technology then database name would be like<APPID>_<TECHNOLOGY>_TBLS<APPID>_<TECHNOLOGY>_VIEWIf application serve to vendor then database name would be like<APPID>_<VENDORNAME>_TBLS<APPID>_<VENDORNAME>_VIEWIf database application further required to divide by module then database name would be like<APPLID>_<MODULE>_TBLS<APPLID>_<MODULE>_VIEWFact Table Names:TFXXX_<FREQUENCY>_<AGRT> Note: AGGRT is will not be there for the table stores lowest granularity table. It will be added only to aggregat

Short Description:Teragen and Terasort Performance testing on AWSArticleThis article should be used with extreme care. Do not use as benchmark. I performed this test to simply run a quick 1 Terabype teragen test on AWS to determine what type of performance I can get from mapreduce on AWS with VERY LITTLE configuration tweaking/tuningOn my github page here you will find the following: teragen scripthadoop,yarn,mapred,capacity scheduler configurations used during testingHardware: (Master & Datanode)1 Master, 3 Data nodesd2.4xlarge, 16vCPU, 122GB ram, (max) 12x2000 StorageTeraGen Results:1hrs, 6mins, 38secJob Counters:Terasort Results:1hrs, 34mins, 20secTeravalidate Results:25mins, 27sec

Two simple steps do direct access custom banana dashboards:1- Copy your_custom_dashboard.json file to folder below in banana server (you can export .json from your banana UI): /opt/lucidworks-hdpsearch/solr/server/solr-webapp/webapp/banana/app/dashboards/2- Point your browser to URL like below:http://sandbox.hortonworks.com:8983/solr/banana/index.html#/dashboard/file/your_custom_dashboard.json

Follow the instructions here on how to download and import the vm into virtual boxOnce you have imported the vm select the vm and click on setting Then click on networkTo assign a IP in the attach to down drop list select "Bridge Adapter"Then under option Promiscuous Mode select "Allow All"Now start your vmOnce the machine is up verify you have a IP addressNow you have IP for your vm. have fun.

The release of version 1.0 marks another major milestone for Storm. Since becoming an Apache project in Sept 2013, much work has gone into maturing the feature set and also improving performance by reworking or tweaking various components. Some of the notable changes that contribute to improved performance are: Switch from ZeroMQ to Netty for inter-worker messaging. Employing batching in Disruptor queues (used for intra-worker messaging) Optimizations in the clojure code such as employing type hints and reducing expensive clojure lookups in perf sensitive areas. In this blog we shall take a look at performance improvements in Storm since its incubation into Apache. To quantify this, we shall compare the performance numbers of Storm v.0.9.0.1, which was the last pre-apache release, with the most recent Storm v1.0.1. Storm v.0.9.0.1 has also been used as a reference point for performance comparisons against Heron. Given the existence of recent efforts to benchmark Storm “at

Adding HDF (with Apache NiFi) to your HDP 2.5 Sandbox is very quick, painless and easy.Get the most recent Hortonworks DataFlow (download😞wget http://d3d0kdwqv675cq.cloudfront.net/HDF/centos6/1.x/updates/1.2.0.1/HDF-1.2.0.1-1.tar.gztar -xvf HDF-1.2.0.1-1.tar.gzcd HDF-1.2.0.1-1/nifi/Then change the port used by NiFi in the conf/nifi.properties file to:nifi.web.http.port=8090Install NiFi as a Linux Servicebin/nifi.sh installsudo service nifi startNiFi home: /opt/HDF-1.2.0.1-1/nifiBootstrap Config File: /opt/HDF-1.2.0.1-1/nifi/conf/bootstrap.conf2016-07-04 02:18:00,005 INFO [main] org.apache.nifi.bootstrap.Command Starting Apache NiFi...2016-07-04 02:18:00,006 INFO [main] org.apache.nifi.bootstrap.Command Working Directory: /opt/HDF-1.2.0.1-1/nifiYou can check the status of single NiFi server via status command:[root@sandbox nifi]# sudo service nifi statusnifi.sh: JAVA_HOME not set; results may varyJava home:NiFi home: /opt/HDF-1.2.0.1-1/nifiBootstrap Config File: /opt/HDF-1.2.0.1-1/nifi

Using Yahoo Kafka ManagerGit clone the project (you need Java 8 to build). Then use SBT to do a clean distribution.This will take a while as it downloads a lot of jars.<code>kafka-manager.zkhosts="sandbox.hortonworks.com:2181" The build will produce a Zip file, unzip it, update configuration file (conf/application.conf) and then you can run it.../kafka-manager/target/universal/kafka-manager-1.3.0.8.zipunzip ../kafka-manager/target/universal/kafka-manager-1.3.0.8.zipkafka-manager-1.3.0.8 git:(master) ✗ vi conf/application.confkafka-manager-1.3.0.8 git:(master) ✗ bin/kafka-manager -Dconfig.file=conf/application.confAccess the Kafka Manager from Chromehttp://localhost:9000/ Running Kafka Manager Resourceshttps://github.com/yahoo/kafka-managerhttp://edbaker.weebly.com/blog/install-and-evaluation-of-yahoos-kafka-managerhttp://chennaihug.org/knowledgebase/yahoo-kafka-manager/https://cwiki.apache.org/confluence/display/KAFKA/EcosystemToolsFor Testing Kafka with a command-line client

From the Sandbox as Root/usr/hdp/current/kafka-broker/bin/kafka-topics.sh--create --zookeeper sandbox.hortonworks.com:2181 --replication-factor 1--partitions 1 --topic peopleTest The Topic[root@sandbox kafka]# /usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh --topic people --zookeeper sandbox.hortonworks.com:2181{metadata.broker.list=sandbox.hortonworks.com:6667, request.timeout.ms=30000, client.id=console-consumer-10628, security.protocol=PLAINTEXT}Resources:https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.2/bk_secure-kafka-ambari/content/ch_secure-kafka-create-topics.htmlhttps://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.2/bk_secure-kafka-ambari/content/ch_secure-kafka-produce-events.html

In order to submit jobs to Spark, so called "fat jars" (containing all dependencies) are quite useful. If you develop your code in Scala, "sbt" (http://www.scala-sbt.org) is a great choice to build your project. The following relies on the newest version, sbt 0.13 For fat jars you first need "sbt-assembly" (https://github.com/sbt/sbt-assembly). Assuming you have the standard sbt folder structure, the easiest way is to add a file "assembly.sbt" into the "project" folder containing one line addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.14.3") The project structure now looks like (most probably without the "target" folder which will be created upon building the project) MyProject+-- build.sbt+-- project| +-- assembly.sbt+-- src| +-- main| +-- scala| +-- MyProject.scala+-- target For building Spark Kafka Streaming jobs on HDP 2.4.2, this is the build file "build.sbt" name := "MyProject"version := "0.1"scalaVersion := "2.10.6"resolvers += "Hortonworks

In HDP 2.5 Sandbox, I did a quick walk through.First thing I liked was the Visualization:The Data Explorer provided a nice query tool to view tables and graphs.The Data Visualization tab provides some nice graphing capabilities.After you run your queries you can look at the Tez results to see how it ran, it's a nice way to see what you may need to optimize.The Hive Ambari View is getting to be a very solid tool for working with Hive. From DDL (creating tables is easy) to viewing data, to updates and inserts.

We have Hive over Hbase table and lets say there are few columns with INT datatype, data loaded from Hive. Now if we would like to delete data based on values present in that particular column(INT), is not possible. It is because values are converted to Binary, even HBase API filter(SingleColumnValueFilter) would return wrong result if we query that particular column values from HBase. Problem to solve: How purge Hive INT datatype column data from HBase?This is the first textual series containing the resolution of above problem. Next series i'll create a small video on running code and cover other datatypes too.In such scenario we cant use standard API and unable to apply filters on binary column values, Solution :- Below JRuby program code.So you have already heard many advantages of storing data in HBase(specially binary block format) and create Hive table on top of that to query your data. I am not going to explain use case for this, why we required HBase over Hive but simple reason

Introduction This article continues where part 1 left off. It describes a few more configuration settings that can be enabled in CDP, HDP, CDH, or Apache Hadoop clusters to help the NameNode scale better. Audience This article is for Hadoop administrators who are familiar with HDFS and its components. If you are using Ambari or Cloudera Manager you should know how to manage services and configurations. It is assumed that you have read part 1 of the article. RPC Handler Count The Hadoop RPC server consists of a single RPC queue per port and multiple handler (worker) threads that dequeue and process requests. If the number of handlers is insufficient, then the RPC queue starts building up and eventually overflows. You may start seeing task failures and eventually job failures and unhappy users. It is recommended that the RPC handler count be set to 20 * log2(Cluster Size) with an upper limit of 200. e.g. for a 250 node cluster you should initialize this to 20 * log2(250) = 160. The RPC h

Query JSON using Spark Imagine you are ingesting JSON msgs but each one has different tag names or even a different structure. This is very common because JSON is a flexible nested structure. However we commonly interact with data in a flat table like structure using SQL. The decision becomes to either parse the dynamic data into a physical schema (on write) or apply a schema at runtime (on read). Ultimately the decision will likely be made based on the number of writes vs reads. However there is one major advantage to using Spark to apply schema on read to JSON events, it alleviates the parsing step. Typically you have to hand code all the tags in the JSON msgs and map each one to a schema column. This may require meeting with upstream teams or third parties to get the DDL/xsd or schema definition. It also doesn't protect you from msgs you haven't seen or new tags being added to existing JSON structures. Sparks schema on read handles all of this as well as flattens the structur

This tutorial will show how to export data out of hbase table into csv format. We will use airport data from american statical association available here. Assume you have a sandbox up and running lets start. First ssh into your sandbox and switch user to hdfssudo su - hdfsThen grab the airport data by issues a wgetwget http://stat-computing.org/dataexpo/2009/airports.csvFor my example the file is located/home/hdfs/airports.csvNow lets create a hbase table called "airports" with column family "info". Do this in hbase shellNow that the table is created lets load it. Get out of hbase shell. as user hdfs run the following to load the tablehbase org.apache.hadoop.hbase.mapreduce.ImportTsv -Dimporttsv.separator=, -Dimporttsv.columns="HBASE_ROW_KEY,info:iata,info:airport,info:city,info:country,info:lat,info:long" airports hdfs://sandbox.hortonworks.com:/tmp/airports.csvThat will kick off map reduce job to load airport table in hbase. once that is done you can do a quick verify in hbase

ObjectiveDeploy a 4-node HDP 2.4.2 cluster with Apache Ambari 2.2.2, Vagrant and VirtualBox on OS X host. This is helpful for development and proof of concepts.ScopeThis approach has been tested on OS X host, but it should work on all supported Vagrant and VirtualBox environments.Pre-requisitesMinimum 9 GB of RAM for the HDP 2.4.2 clusterDownload and install Vagrant for your host OS: https://www.vagrantup.com/downloads.htmlDownload and install VirtualBox for your host OS: https://www.virtualbox.org/wiki/DownloadsDownload and install git client for your hostOpen a command shell and change to the folder where you plan to clone the github repositoryClone the following git repositorygit clone https://github.com/cstanca1/hdp2_4_2-vagrant.gitCreate and Start VMsChange directory to /hdp_2.4.2-vagrant, the folder that includes Vagrantfile and create a /data folder: mkdir dataThis /data folder will be needed for guest VMs to share with the host. Vagrant (via Vagrantfile) is configured to use Ce

We have seem many incidents of overloaded HDFS namenode due to 1) misconfigurations or 2) “bad” MR jobs or Hive queries that create large amount of RPC requests in a short period of time. There are quite a few features that have been introduced in HDP 2.3/2.4 to protect HDFS namenode. This article summarize the deployment steps of these features with an incomplete list of known issues and possible solutions for them. Enable Async Audit LoggingDedicated Service RPC PortDedicated Lifeline RPC Port for HAEnable FairCallQueue on Client RPC PortEnable RPC Client Backoff on Client RPC portEnable RPC Caller Context to track the “bad” jobsEnable Response time based backoff with DecayedRpcSchedulerCheck JMX for namenode client RPC call queue length and average queue timeCheck JMX for namenode DecayRpcScheduler when FCQ is enabledNNtop (HDFS-6982)1. Enable Async Audit LoggingEnable async audit logging by setting "dfs.namenode.audit.log.async" to true in hdfs-site.xml. This can minimize the impa

Hortonworks Data Platform ArtifactsDeveloping solutions with Hadoop commonly requires the use of multiple different HDP component libraries. Whether you’re building solutions with Pig, Spark, Cascading, or HBase, at some point extensions will need to be created, and those artifacts for each component will need to be used. This guide serves as an overview of where to find those artifacts and how to get them quickly integrated with your preferred build tool, and IDE.ReposotoriesAt Hortonworks, we store all of our artifacts in a public Sonatype Nexus repository. That repository can be easily accessed and searched for commonly used library, source code, and javadoc archives simply by navigating to http://repo.hortonworks.com.ArtifactsJar files containing compiled classes, source, and javadocs are all available in our public repository, and finding the right artifact with right version is as easy as searching the repository for classes you need to resolve. For example, If creating a sol

Storm provides rich set of built-in metrics which are provided to Storm UI, and also metrics consumer. However these built-in metrics are measured for each task, while we are occasionally configuring a task to have multiple calls for external storages, and/or heavy computation sequentially. Let’s say we have a Bolt which crawls the page from the web, and store page to external storage.public class StoreCrawledPageBolt extends BaseRichBolt { @Override public void execute(Tuple input) { String url = tuple.getString(0); // the execute latency for crawlPage and storePage can be vary String html = crawlPage(url); storePage(url, html); collector.ack(input); } // skip method implementations...}The latency for crawling page from web server is vary, which takes several milliseconds to tens of seconds which might be better to treat as time-out. The latency for storing page data to external storage would be relatively faster, but it might be slower for specific reason like m

Driven first-hand experience: While researching for a Hive Query Metrics UI found this tool called driven. Step1: Register for a free trial at http://www.driven.io/hive/ Step2: Upon clicking the verification link sent over email came to this page: Click on “Install the Driven Agent” Step3: On clicking that two things will happen: You will be taken to the configuration and install instructions You will get an email with username and password Step4: Click on https://trial.driven.io/index.html#/apps from screenshot 2 You will be in the login screen. Then enter the id and password you got in your email. Step5: After logging in you will prompted to change the password. Change that and click submit. Step6: You will be logged into the user interface. But since the agent is not installed you cannot view anything. You will be also provided with your API Key. Step7: Hive is installed in the cluster in a node named hiveserver. Am going to install

One common problem in SQL is that you join two tables and get duplicate column names from the two tables. When you now for example want to create a CTAS you will get "duplicate column name" errors. You also often want to exclude the join key from the result set since it is by definition duplicate.Database schemas often prefix column names with a letter from the table to fix at least the first issue. Like TPCH: lineitem.l_key and orders.o_key. A common approach to fix this is to explicitely specify all column names in your SELECT list. However it looks like Hive has some cool/dirty tricks up its sleeve to make this easier. Regular Expressions to specify column names.Testsetup:describe tb1;id intname stringdescribe tb2;id intage intYou then have to disable the use of quotes in identifiers because that interferes with the regex.set hive.support.quoted.identifiers=none;And now you can use Java Regex to select all columns from the right