Community Articles

There are a lot of excellent talks from the summit.Deep LearningApache Spark with Machine Learning like TensorFlowDistributed Deep Learning on Hadoop Clusters (Yahoo)Apache SparkBig Data Heterogeneous Mixture Learning on Spark Integrating Apache Spark and NiFi for Data Lakes (ThinkBig)OperationsZero Downtime App Deployment Using Hadoop (Hortonworks)Debugging YARN Cluster in Production (Hortonworks)Yahoo's Experience Running Pig on Tez at ScaleThe DAP Where Yarn HBase Kafka and Spark go to Production (Cask)Extend Governance in Hadoop with Atlas Ecosystem (Hortonworks)Cost and Resource Tracking for Hadoop (Yahoo)Managing Hadoop HBase and Storm Clusters at Yahoo ScaleOperating and Supporting Apache HBase Best Practices and Improvements (Hortonworks)Scheduling Policies in Yarn (Slides)Future of DataArun Murthy, Hortonworks - Hadoop Summit 2016 San Jose - #HS16SJ - #theCUBEThe Future of Hadoop : An Enterprise View (Slides)StreamingThe Future of Storm (Hortonworks)Streaming ETL for All : Emb

We have noticed production job failures where customer upgraded their hive from .14 (HDP 2.1) to the latest version(>1.2.x) and resulted in critical jobs failing (not to mention the severity 1 case). This is due to the changes in the reserved words between the source and target hive versions.For example, Word 'date' is not a reserved word in Hive.14 but inHive 1.2.1 it is. Same is the case with REGEXP and RLIKE.Here are the reserved keywords which hive.support.sql11.reserved.keywords support. https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-ReservedKeywordsThere are two ways if the user still would like to use those reserved keywords as identifiers:1).use quoted identifiers,2).set hive.support.sql11.reserved.keywords=false.This would be the best option though code change is required. To illustrate option 1,For example,With word "user" being a keyword, We can use it as an identifier like the below SQL.SELECT createddate, `user`.screenname FROM twi

1. Fetch the latest FS Image from the Active NameNode:Look at the (NameNode directories) property in Ambari and copy the latest image to a node with free disk space and memory. (Ex: fsimage_0000000001138083674)2. Load the FS Image:On the node where you copied the FS Image. Run the below commands:export HADOOP_OPTS="-Xms16000m -Xmx16000m $HADOOP_OPTS" nohup hdfs oiv -i fsimage_0000000001138083674 -o fsimage_0000000001138083674.txt &Above command will make the FS Image available on a web server (temporary). 3. Create "ls -R" report from the FS Image:nohup hdfs dfs -ls -R webhdfs://127.0.0.1:5978/ > /data/home/hdfs/lsrreport.txt &This could take some time.Copy the data from /data/home/hdfs/lsrreport.txt to hdfs /user/hdfs/lsr/lsrreport.txt4. Analyze the ls-R output:Create required table, load data, create view and analyze:hive> add jar /usr/hdp/2.3.2.0-2950/hive/lib/hive-contrib.jar; hive> CREATE EXTERNAL TABLE lsr (permissions STRING, replication STRING, owner STRING, ow

As the Apache Hadoop project completed a decade, I will start with a fun fact. Hadoop is not an acronym- it was named after a toy elephant belonging to one of the creator’s son and now, is the fastest growing segment of the data management and analytics market! The data management and analytics market is projected to grow from $69.6B in 2015 to $132B in 2020 and the Hadoop segment is forecasted to grow 2.7 times faster than the data management and analytics market (according to a recent analyst report).Though Apache Hadoop started as a single system for batch workloads, Apache Hadoop is becoming the multi-use data platform for batch, interactive, online and streaming workloads. At the core of the Hadoop eco-system, we have Apache Hadoop File System (HDFS) to reliably store data and Apache Hadoop YARN (Yet Another Resource Negotiator) to reconcile the way applications use Hadoop system resources. While this post is focused on Apache HDFS, I will recommend Vinod Kumar Vavilapalli’s blog

You may run into slow Hadoop service start on your OS X development laptop. You can check this by opening up your service logs and looking for large (5-10 second) gaps between successive log entries at startup.DiagnosisIt often manifests as test failures for MiniDFSCluster-based tests that use short timeouts (<10 seconds).Here is an example from a NameNode log file with a 5 second stall at startup.2016-07-25 14:57:37,982 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties2016-07-25 14:57:43,060 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).Another 5 second stall during NameNode startup.2016-07-25 14:57:48,790 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true2016-07-25 14:57:53,914 INFO org.apache.hadoop.util.GSet: Computing capacity for map INodeMapResolutionIf you see this behavior you are likely running into an OS X bug. The fix is to put all your entr

This tutorial has been successfully tried on HDP-2.4.2.0 and Ambari 2.2.2.0.I have my HDP Cluster Kerberized with Namenode HA..Please follow below steps for Configuring File View on Kerberized HDP Cluster..Step 1 - Please configure your Ambari Server for Kerberos with the steps mentioned in below article. Please follow steps 1 to 5.https://community.hortonworks.com/articles/40635/configure-tez-view-for-kerberized-hdp-cluster.html.Step 2 - Please add below properties to core-site.xml via Ambari UI and restart required services..Note - If you are running Ambari Server as root user then add below propertieshadoop.proxyuser.root.groups=*hadoop.proxyuser.root.hosts=*.If you are running Ambari server as non-root user then please add below properties in core-site.xmlhadoop.proxyuser.<ambari-server-user>.groups=*hadoop.proxyuser.<ambari-server-user>.hosts=*Please replace <ambari-server-user> with user running Ambari Server in above example..I'm assuming that your ambari serve

Using the GetHTTP Processor we grab random images from the DigitalOcean's Unsplash.it free image site. I give it a random file name so we can save it uniquely in HDFS.The Entire Data Flow from GetHTTP to Final HDFS storage of image and it's metadata as JSON.ExtractMediaMetaData ProcessorThe final results:hdfs dfs -cat /mediametadata/random1469112881039.json{"Number of Components":"3","Resolution Units":"none","Image Height":"200pixels","File Name":"apache-tika-3181704319795384377.tmp","Data Precision":"8 bits","File Modified Date":"Thu Jul 21 14:54:43 UTC 2016","tiff:BitsPerSample":"8","Compression Type":"Progressive,Huffman","X-Parsed-By":"org.apache.tika.parser.DefaultParser,org.apache.tika.parser.jpeg.JpegParser","Component 1":"Y component: Quantization table 0, Sampling factors 2 horiz/2vert","Component 2":"Cb component: Quantization table 1,Sampling factors 1 horiz/1 vert","tiff:ImageLength":"200","mime.type":"image/jpeg","gethttp.remote.source":"unsplash.it","Component3":"Cr com

Hortonworks does not use puppet agent with HDP 1.3 but we still use puppet locally on each node invoked by Ambari Agent.    Currently there are no plans to support chef or puppet in a way that IT can manage configuration of HDP clusters from their own scripts.   This article created by Hortonworks Support (Article: 000001181) on 2013-08-30 17:21 OS: n/a Type: Cluster_Admnistration

Oracle has recently changed the way they manage the process of license acceptance on JDK downloads.  That change is preventing the use of a wget and also preventing a download straight onto your disk.  This issue only occurs with gsInstaller.  HMC installation works fine.We suggest to download both the JDK 6u26 32 bit and 64 bit JDKs from Oracle and set them up on each node and point to those directories in the gsInstaller.properties.  http://www.oracle.com/technetwork/java/javase/downloads/jdk-6u26-download-400750.htmlIn gsInstaller.properties, please update the following config settings with the respective install locations:java32home=java64home= This article created by Hortonworks Support (Article: 000001180) on 2012-11-14 16:32 OS: n/a Type: Installation

SYMPTOMS:  Below is a sample query CREATE VIEW salary AS SELECT fname, (CAST (lname AS STRING)) from Employees; The result is: FAILED: IllegalArgumentException replace op boundaries of <ReplaceOp@@17,43:47='lname',<26>,1:43..@17,43:47='lname',<26>,1:43:"`employees`.`lname`"> overlap with previous <ReplaceOp@@13,36:36='(',<266>,1:36..@23,59:59=')',<276>,1:59:"`employees`.`lname`"> hive>   ROOT CAUSE:  Product defect    SOLUTION:  Filed bug BUG-8944 that has been fixed in HDP 1.3.3 This article created by Hortonworks Support (Article: 000001718) on 2013-12-23 13:05 OS: Linux Type: Developing_Code

HBase is just awesome. Yes I will start with that. HBase tuning like any other service within the ecosystem requires understanding of the configurations and the impact (good or bad) it may have on the service. I will share some simple quick hitters which you can check for to increase performance on your phoenix/hbase cluster What is your hfile locality?Do a simple check on ambari should show you the percentage. The metric is Regionserver.Server.percentFilesLocalWhat does this metric mean? More info here. Thanks to @Predrag Minovic it means "That's the ratio of HFiles associated with regions served by an RS having one replica stored on the local Data Node in HDFS. RS can access local files directly from a local disk if short-circuit is enabled. And If you, for example run a RS on a machine without DN, its locality will be zero. You can find more details here. And on HBase Web UI page you can find locality per table."So if your percentage is lower then 70.. time to run major compact

Apache Shiro design is intuitive and simple way to ensure the safety of the application.For more detail on Apache Shiro project go to http://shiro.apache.org/what-is-shiro.html Software design is generally based on user stories to achieve, that is, based on how users interact with the system to design the user interface or service API. For example, a user story will be displayed after a user logs on a button to view personal account information, if the user is not registered, it displays a registration button.This user story implies major application user needs to be accomplished. Even if the user is not a person here, but third-party systems, when coding is also interacting with the system as "user" to deal with.Apache Shiro reaction of this concept in their design by the intuitive notion exposed to developers so that Apache Shiro in almost all applications are easy to use.Outline:Shiro has three main top-level concept: Subject, SecurityManager, Realms. The following diagram describes

This article is first series of three articles, next coming articles with some code and mechanism present in latest version of HBase supporting HBase Replication. HBase ReplicationHbase Replication solution can solve the cluster security, data security, read and write separation, operation and maintenance, and the guest operating errors, and so the ease of management and configuration, provide powerful online applications support.Hbase replication currently used in the industry are rare, because there are many aspects, such as HDFS has multiple backup copies in a way to help security HBASE underlying data, and the relatively small number of companies in the cluster size. Another reason the data is not very high degree of importance, such as some logging system or as a second warehouse of historical data to split a large number of read requests. Such data lost to be present or back up at other places (database cluster). For such cases the Slave Replication cluster become dispensable, th

Update: apparently when you initiate a support case resolution capture, let's say for HBase service, it will pull HDFS namenode logs in addition to the HBase logs. You may be faced with the same issue and may have to apply the approach below to overcome timeouts. In SmartSense 1.3.0 this will no longer be an issue. Until then, this is a way to avoid capture time outs. Firstly, lets discuss the difference between capture for analysis and support case resolution. Analysis bundles do not collect service logs. For support cases, you're going to fetch configuration and logs. Then based on how much anonymization you will want to apply, large log files will take a long time to collect. This is especially prominent with HDFS namenode logs. They tend to be big and this is exactly the scenario we're trying to address. Firstly, increase the threshold for agent time out in Ambari. In my case it was 30min. Feel free to raise it up to 2hrs on the Ambari SmartSense Operations page. Then, we're going

OpenHAB - Build your smart home in no time! Welcome to http://www.openhab.org/ A vendor and technology agnostic open source automation software for your home. OpenHAB is a mature, open source home automation platform that runs on a variety of hardware and is protocol agnostic, meaning it can connect to nearly any home automation hardware on the market today. If you’ve been frustrated with the number of manufacturer specific apps you need to run just to control your lights, then I’ve got great news for you: OpenHAB is the solution you’ve been looking for – it’s the most flexible smart home hub you’ll ever find. Source Demo: Go to http://www.openhab.org/getting-started/downloads.html Download Runtime core and Demo files Extract Runtime core files in a directory called openHAB and extract Demo files under OpenHAB. See the following: Now, download smartphone app called openHAB in your smartphone. I am using iOS and once you launch the app then disable DEMO tab and e

Teradata's JDBC connector contains two jar files (tdgssconfig.jar and terajdbc4.jar) that must both be contained within the classpath. NiFi Database processors like ExecuteSQL or PutSQL use a connection pool such as DBCPConnectionPool which defines your JDBC connection to a database like Teradata. Follow the steps below to integrate Teradata JDBC connector into your DBCPConnectionPool:1) Download the Teradata connectors (tdgssconfig.jar and terajdbc4.jar) - you can download the Teradata v1.4.1 connector on http://hortonworks.com/downloads/2) Extract the jar files (tdgssconfig.jar and terajdbc4.jar) from hdp-connector-for-teradata-1.4.1.2.3.2.0-2950-distro.tar.gz and move these files to your NIFI_DIRECTORY/lib/*3) Restart NiFi4) Under your DBCPConnectionPool (Controller > Controller Services), Edit your existing DBCPConnectionPool (if your pool is active, disable it before editing) 5) Under the Configuration Controller Service > Properties, define the followingDatabase Connectio

Apache Kylin originIn today's era of big data, Hadoop has become the de facto standards, and a large number of tools one after another around the Hadoop platform to build, to address the needs of different scenarios.For example, Hadoop Hive is a data warehouse tools, data files stored on HDFS distributed file system can be mapped to a database table and provides SQL queries. Hive execution engine can be converted to SQL MapReduce task to run, ideally suited for data warehouse data analysis. Another example HBase is based on Hadoop, high availability, high performance, column-oriented, scalable distributed storage system, Hadoop HDFS architecture to provide high reliability of HBase underlying storage support.Although existing business analytical tools such as Tableau, etc. are here because of the lack of Hadoop-based distributed analysis engine. But they exist with significant limitations, such as difficult to extend horizontally, can not handle large scale data, but also the lack of s

su hdfshadoop fs -mkdir /udfhadoop fs -put urldetector-1.0-jar-with-dependencies.jar /udf/hadoop fs -put libs/url-detector-0.1.15.jar /udf/hadoop fs -chown -R hdfs /udfhadoop fs -chgrp -R hdfs /udfhadoop fs -chmod -R 775 /udfCreate Hadoop Directories and upload the two necessary libraries. CREATE FUNCTION urldetector as 'com.dataflowdeveloper.detection.URLDetector' USING JAR 'hdfs:///udf/urldetector-1.0-jar-with-dependencies.jar', JAR 'hdfs:///udf/url-detector-0.1.15.jar';Create Hive Function with those HDFS referenced JARsselect http_user_agent,urldetector(remote_host)asurls,remote_host from AccessLogs limit 100;Test the UDF via Hive QL@Description(name="urldetector", value="_FUNC_(string) - detectsurls")public final class URLDetector extends UDF{}Java Header for the UDFset hive.cli.print.header=true;add jar urldetector-1.0-jar-with-dependencies.jar;CREATE TEMPORARY FUNCTION urldetector as 'com.dataflowdeveloper.detection.URLDetector';select urldetector(description) from sample_07 lim

In general, you have following options when running R on Hortonworks Data Platform (HDP) -o RHadoop (rmr) - R program written in MapReduce paradigm. MapReduce is not a vendor specific API and any program written with MapReduce is portable across Hadoop distributions.https://github.com/RevolutionAnalytics/RHadoop/wiki/rmro Hadoop Streaming - R program written to make use of Hadoop Streaming but the program structure still aligns with MapReduce. Above benefit still applies.o RJDBC - This example does not require the R programs to be written using MapReduce and still remains 100% native R APIs without any third party packages.Here is a tutorial with a video, sample data and R script:http://hortonworks.com/hadoop-tutorial/using-revolution-r-enterprise-tutorial-hortonworks-sandbox/Using RJDBC, the R program can have Hadoop parallelize pre-processing and filtering. R submits a query to Hive or SparkSQL making use of distributed and parallel processing. Then uses existing R models, as is &amp

Objective The objective of this article is to provide an introduction to User-Defined Functions for spatial analysis in Hive.Credit This article is based on multiple ESRI resources referenced throughout the article. Thanks also to my colleague Artem Ervits for his contribution to the proof-of-concept that lead to this article.Spatial Types Spatial types are data types that store geometry data and they have associated functions or methods that are used to access and manipulate the data using Structured Query Language (SQL). Unlike Oracle, DB2, SQL Server, or PostGreSQL, Hive does not support yet a geometry native data type. The best approach is to store spatial data as text. That will not impact the use of UDFs available for Hive.ESRI Libraries The Esri Geometry API for Java includes geometry objects (e.g. points, lines, and polygons), spatial operations (e.g. intersects, buffer), and spatial indexing. By deploying the library (as a jar) within Hadoop, you are able to build custo

On Kerberos-secured clusters managed by Ambari versions prior to 2.0.0, the hdfs.headless.keytab file is deployed on all the nodes in the cluster. From Ambari version 2.0.0 onwards, this keytab file is not deployed by default on every node. If the Ranger HBase plugin is enabled and set up to write to HDFS, Ambari's RegionServer start up script tries to access the hdfs.headless.keytab file. If the host that the RegionServer is running on does not also have the HDFS client role assigned to it, the hdfs.headless.keytab file will be absent and the RegionServer will fail to start with an error such as this in Ambari: resource_management.core.exceptions.Fail: Execution of '/usr/bin/kinit -kt /etc/security/keytabs/hdfs.headless.keytab hdfs-examplecluster@KRBEXAMPLE.COM' returned 1. kinit: Key table file '/etc/security/keytabs/hdfs.headless.keytab' not found while getting initial credentials This problem is expected to be resolved with the release of Ambari 2.4.0, but until then, the work-

I have done data analysis for one of my project using below approach and hopefully it may help you understand underlying subject. Soon i'll post my project on data analysis and detail description on technology used Python(web scraping- data collection), Hadoop, Spark and R.Data analysis is a highly iterative and non-linear process, better reflected by a series of cyclic process, in which information is learned at each step, which then informs whether (and how) to refine, and redo, the step that was just performed, or whether (and how) to proceed to the next step.Setting the SceneData analysis is a study of subjective question and study even includes developing and executing a plan for collecting data, a data analysis presumes the data have already been collected. More specifically, a study includes the development of a hypothesis or question, the designing of the data collection process (or study protocol), the collection of the data, and the analysis and interpretation of the data.Act

I have worked with almost 20 to 25 applications. Whenever i start working first i have to understand each applications naming convention and i keep thinking why we all not follow single naming convention. As Hadoop is evolving rapidly therefore would like to share my naming convention so that may be if you come to my project will feel comfortable and so as I if you follow too.Database Names:If application serve to technology then database name would be like<APPID>_<TECHNOLOGY>_TBLS<APPID>_<TECHNOLOGY>_VIEWIf application serve to vendor then database name would be like<APPID>_<VENDORNAME>_TBLS<APPID>_<VENDORNAME>_VIEWIf database application further required to divide by module then database name would be like<APPLID>_<MODULE>_TBLS<APPLID>_<MODULE>_VIEWFact Table Names:TFXXX_<FREQUENCY>_<AGRT> Note: AGGRT is will not be there for the table stores lowest granularity table. It will be added only to aggregat

Short Description:Teragen and Terasort Performance testing on AWSArticleThis article should be used with extreme care. Do not use as benchmark. I performed this test to simply run a quick 1 Terabype teragen test on AWS to determine what type of performance I can get from mapreduce on AWS with VERY LITTLE configuration tweaking/tuningOn my github page here you will find the following: teragen scripthadoop,yarn,mapred,capacity scheduler configurations used during testingHardware: (Master & Datanode)1 Master, 3 Data nodesd2.4xlarge, 16vCPU, 122GB ram, (max) 12x2000 StorageTeraGen Results:1hrs, 6mins, 38secJob Counters:Terasort Results:1hrs, 34mins, 20secTeravalidate Results:25mins, 27sec

Two simple steps do direct access custom banana dashboards:1- Copy your_custom_dashboard.json file to folder below in banana server (you can export .json from your banana UI): /opt/lucidworks-hdpsearch/solr/server/solr-webapp/webapp/banana/app/dashboards/2- Point your browser to URL like below:http://sandbox.hortonworks.com:8983/solr/banana/index.html#/dashboard/file/your_custom_dashboard.json