Resources

The objective of this post is to briefly explain how to setup an unsecured NiFi cluster with NiFi 1.0.0 (a three nodes cluster with three embedded ZooKeeper instances). One really important change with this new version is the new paradigm around cluster installation. From the NiFi documentation, we can read:Starting with the NiFi 1.0 release, NiFi employs a Zero-Master Clustering paradigm. Each of the nodes in a NiFi cluster performs the same tasks on the data but each operates on a different set of data. Apache ZooKeeper elects one of the nodes as the Cluster Coordinator, and failover is handled automatically by ZooKeeper. All cluster nodes report heartbeat and status information to the Cluster Coordinator. The Cluster Coordinator is responsible for disconnecting and connecting nodes. As a DataFlow manager, you can interact with the NiFi cluster through the UI of any node in the cluster. Any change you make is replicated to all nodes in the cluster, allowing for multiple e

It is a very common operation to do prefix scan in HBase. For example, when reading HBase table from HBase, we may use the following table scan api:val prefixFilter = new PrefixFilter(prefix)val scan: Scan = new Scan()scan.setFilter(prefixFilter)However, the code above may appear to be very slow when scanning a large HBase table. The reason is: we need to set StartRow before using PrefixFilter. Without setting the start row properly, your HBase scan may have to begin with the very first region and waste lots of time to get to the first right place.The recommended way is to use setRowPrefixFilter(byte[] rowPrefix), from its source code below, we can see that it helps us to set up the start row before doing table scan.409 public Scan setRowPrefixFilter(byte[] rowPrefix) {410 if (rowPrefix == null) {411 setStartRow(HConstants.EMPTY_START_ROW);412 setStopRow(HConstants.EMPTY_END_ROW);413 } else {414 this.setStartRow(rowPrefix);415 this.setStopRow(calculateTheClos

This article is an update of one on http://hadooped.blogspot.com/2013/10/apache-oozie-part-13-oozie-ssh-action_30.html, authored by @Anagha Khanolkar.Below are the steps to setup Oozie workflow using ssh-action: Step 1. Create job.properties.Example:#************************************************* # job.properties#*************************************************nameNode=hdfs://<namenode-machine-fqdn>:8020jobTracker=<resource-manager-fqdn>:8050queueName=defaultoozie.libpath=${nameNode}/user/oozie/share/liboozie.use.system.libpath=trueoozie.wf.rerun.failnodes=trueoozieProjectRoot=${nameNode}/user/${user.name}appPath=${oozieProjectRoot}oozie.wf.application.path=${appPath}inputDir=${oozieProjectRoot}focusNodeLogin=<username>@<remote-host-where-you-have-your-shell-script(s)>shellScriptPath=~/uploadFile.shemailToAddress=<email-id> Step 2. Write workflow.xmlExample:<!--******************************************--><!--workflow.xml

First lets create a sample file in S3: In the AWS Console , Go to S3 and create a bucket “S3Demo” and pick your region. Upload the file manually by using the upload button (example file name used later in scala: S3HDPTEST.csv)In the HDP 2.4.0 Sandbox :Download the aws sdk for java https://aws.amazon.com/sdk-for-java/ Uploaded it to the hadoop directory. You should see the aws-java-sdk-1.10.65.jar in /usr/hdp/2.4.0.0-169/hadoop/[root@sandbox bin]# ll /usr/hdp/2.4.0.0-169/hadoop/total 242692-rw-r--r-- 1 root root 32380018 2016-03-31 22:02 aws-java-sdk-1.10.65.jardrwxr-xr-x 2 root root 4096 2016-02-29 18:05 bindrwxr-xr-x 2 root root 12288 2016-02-29 17:49 clientlrwxrwxrwx 1 root root 25 2016-03-31 21:08 conf -> /etc/hadoop/2.4.0.0-169/0drwxr-xr-x 2 root root 4096 2016-02-29 17:46 etc-rw-r--r-- 1 root root 17366 2016-02-10 06:44 hadoop-annotations-2.7.1.2.4.0.0-169.jarlrwxrwxrwx 1 root root 40 2016-02-29 17:46 hadoop-annotations.jar -> hadoop-annotati

Cluster upgrade is a well-planned maintenance activity. Before starting the upgrade, it is important to ensure that your cluster is ready for upgrade. Ambari has built-in pre-checks that are executed before the actual upgrade to check the overall health of your cluster and validate its readiness for upgrade. Let us look at the pre-checks, their purpose and how to remediate the issues, so that your cluster upgrade process is smooth. A bit of background first:BackgroundAmbari supports two kinds of stack (cluster) upgrades: Rolling Upgrade (RU) and Express Upgrade (EU). Pre-checks apply to both upgrade methods, although RU has a larger set of pre-checks since the upgrade does not involve any downtimeFirst up, the types of pre-checks: Cluster level checks: Example – newer version packages should be installed on all hosts in the clusterService level checks: Example – all service components should be in installed state (no failures)Host level checks: All hosts must be communicating with Amba

SynopsisBoth Pig and Spark (PySpark) excel at iterative data processing against weblogs data in text delimited format. Is one faster than the other? Pig on Tez shows to outperform PySparkPySpark using the Databricks CSV parser shows to outperform Pig on TezPig Apache Pig allows users to MapReduce transformations using a simple scripting language called Pig Latin. Pig translates the Pig Latin script into MapReduce so that it can be executed within YARN against a dataset stored in the Hadoop Distributed File System (HDFS) Apache Tez is a distributed data processing framework for building batch and interactive applications. Tez improves the MapReduce paradigm by dramatically improving its speed, while maintaining MapReduce’s ability to scale to petabytes of data. Pig on Tez excels at iterative data processing. It is fast, easy, and the optimal solution to implement when it comes to parsing weblogs.Spark General purpose in-memory distributed data processing engine API’s to write cod

Objective: The purpose of this tutorial is to walk you through the process of enabling the Elasticsearch interpreter for Zeppelin on the HDP 2.5 TP sandbox. As part of this process, we will install Elasticsearch and use Zeppelin to index and query data using Zeppelin and Elasticsearch.This is the first of two articles covering Elasticsearch on HDP. The second article covers pushing Twitter data to Elasticsearch using NiFi and provides a sample Zeppelin dashboard. You can find that article here: HCC Article Note: The Zeppelin Elasticserch interpreter is a community provided interpreter. It is not yet considered GA by Hortonworks and should only be used for development and testing purposes.Prerequisites: You should already have installed the Hortonworks Sandbox (HDP 2.5 Tech Preview). Note: While not required, I recommend using Vagrant to manage multiple versions of the Sandbox. Follow my tutorial here to set that up: HCC ArticleScope: This tutorial was tested using the following

Self Service Hadoop – well some starting pointsSay you want to get started with Big Data and concurrently want to start to empower your relatively savvy end users that have been in the frustrating desktop data management land for a long time. This very simple article will hopefully help with a few options.This diagram below gives a little perspective around some of the sources and mechanisms around ingesting, manipulating and using your data assets. You can see following the numbers from 1 to 6 that you have many options with working with your data. This article will concentrate on just showing a simple end user example of ingesting data, understanding it and using options for using the data from a self service approach. Really an approach to get started and help promote some of the value of a modern data architecture as your team matures.An end users want to self service some data from their desktop/server into HDFS and be able to query and understand that data from their existing

There are three main installation paths for creating a new Cloudera Manager deployment:   Installation Path A - Automated Installation by Cloudera Manager (Non-Production Mode). "This path is recommended for demonstration and proof-of-concept deployments, but is not recommended for production deployments because it's not intended to scale and may require database migration as your cluster grows"[1].   Installation Path B - Installation Using Cloudera Manager Parcels or Packages. This path requires you to first manually install and configure an external database for the Cloudera Manager Server, Cloudera Management services and any other CDH services that may reside on this database instance (such as Hue, Hive, Oozie etc.).   Installation Path C - Manual Installation Using Cloudera Manager Tarballs. In this path, you install the the Oracle JDK, Cloudera Manager server and Agent software using tarballs.   This tutorial covers the Path B installation of Cloudera Manager

IntroductionAmbari 2.4 introduces the Hue-to-Views Migration tool, which is specifically designed to migrate existing Hue artifacts to an Ambari View.Currently in Hue the hive queries are saved in ‘Saved Queries’ in Hue UI. These queries can be invoked by the user by going to Saved Queries in Hue UI. HUETOAMBARI_MIGRATION would help in migrating these saved Hive artifacts from Hue to Ambari Hive View. Once the migration is done, the Saved Queries can be viewed from Ambari Hive View. Currently HUETOAMBARI_MIGRATION would help in migrating the following artifacts from Hue to Ambari Views.Hive - Saved Queries, Query History Pig - Saved Scripts, Pig JobsThe view would use JDBC to connect to Hue database to obtain the artifacts. Once obtained, these would be stored in the Ambari View Server database.Today, the view migrates the scripts one user at a time. The user to be migrated would be specified during the migration.Note:- This view does not validate scripts. Non-working Hue scripts wil

Here is the steps to enable HTTPS for HBASE Master UI. 1. Add following properties in "Custom hbase-site" hbase.ssl.enabled=truehbase.http.policy=HTTPS_ONLY hadoop.ssl.enabled=true2. In HBASE, there is no direct option to add/configure the keystore files (JKS), it uses HADOOP configurations files. Lets say If HTTPS is already enabled for HDFS Web UI then your host would have already configured with JKS files in ssl-server.xml and ssl-client.xml files, so HBASE also uses same ssl-server.xml files. You can find more details to configure HTTPS for HDFS at https://community.hortonworks.com/articles/52875/enable-https-for-hdfs.html3. If you are planning to have different JKS files for HDFS and HBASE then you can copy the the ssl-server.xml to /etc/hbase/conf path and configure the JKS file. Restart HBASE server and server should come fine and should be accessable https://HOSTNAME:16010/ Note:Please note that there is a bug in AMBARI because of that HBASE quick always try to open with HTTP

Recently I had a client ask about how would we go about connecting a windows share to Nifi to HDFS, or if it was even possible. This is how you build a working proof of concept to demo the capabilities!You will need two Servers or Virtual machines. One for windows, one for Hadoop + Nifi. I personally elected to use these two The Sandbox http://hortonworks.com/products/hortonworks-sandbox/A windows VM running win 7 https://developer.microsoft.com/en-us/microsoft-edge/tools/vms/linux/You then need to install nifi on the sandbox, I find this repo to be the easiest to follow. https://github.com/abajwa-hw/ambari-nifi-serviceBe sure the servers can talk to each other directly, I personally used a bridged network connection in virtual box and looked up the IPs on my router's control panel.Next you need to setup a windows share of some format. This can be combined with active directory but I personally just enabled guest accounts and made an account called Nifi_Test. These instruct

SYNOPSIS Which is faster when analyzing data using Spark 1.6.1: HDP with HDFS for storage, or EMR? Testing shows that HDP using HDFS has performance gains over using EMR. HDP/HDFS outperforms EMR by 46% when tested against 1 full day of 37 GB Clickstream (Web) data.HDP/HDFSEMRTime Elapsed3 mins, 29 sec5 mins, 5 sec * See below at end of article for validation and screen prints showing Resource Manager logs HDP Hortonworks Data Platform (HDP) is the industry's only true secure, enterprise-ready open source Apache Hadoop distribution. The Hadoop Distributed File System (HDFS) is a Java-based distributed block storage file system that is used to store all data in HDP. EMR Amazon Elastic MapReduce (Amazon EMR) is a managed Hadoop framework to distribute and process vast amounts of data across dynamically scalable Amazon EC2 instances. S3 is an inexpensive object store that can theoretically scale out infinitely without the limitations inherent to a hierarchical block stora

In the recent weeks I have tested Hadoop on various IaaS providers in hope to find additional performance insights. BigStep blew away my expectation in terms of Hadoop performance on IaaS. I wanted to take the testing a step further. Lets quantify performance measures by adding nodes to a cluster. Even for a small 1TB data set, would 5 nodes perform far greater then 3? I have heard a few times when it comes to small datasets adding more nodes may not have a impact. So this led me to test a 3 node cluster vs a 5 node cluster using 1TB dataset. Does the extra 2 nodes increase performance with processing and IO? Lets find out.Started the testing with dfsio which is a distributed IO benchmark tool. Here are results:From 3 to 5 data nodes IO read performance increased approx. 36%From 3 to 5 data nodes IO write performance increased approx. 49%With 2 additional data nodes a performance IO throughput of 49%! Wish I had more boxes to play with. Can't image where this would take the m

New table region split/merge APINew API in HBase HDP 2.5 allows user to disable/enable automatic region splits and merges. From HBase shell you can run the following commands:Enable region splits hbase> splitormerge_switch 'SPLIT', trueDisable region splitshbase> splitormerge_switch 'SPLIT', falseEnable region mergeshbase> splitormerge_switch 'MERGE', trueDisable region mergeshbase> splitormerge_switch 'MERGE', falseCheck region split switch statushbase> splitormerge_enabled 'SPLIT'Check region merge switch statushbase> splitormerge_enabled 'MERGE'Usage in HBase hbck toolHBase hbck tool can automatically use this API during restore operation if the following command-line argument is specified: -disableSplitAndMerge or tool is run in repair mode. Disabling region splits and merges during repair or diagnostic runs improves tool's ability to diagnose and repair HBase cluster.Usage in table snapshots It is recommended now to disable both: region splits and merges before

Sharing source code I wrote to a prospect that wants to try Phoenix + .NET.It's a simple code that POST the query to Phoenix Server and parse result JSON and prints results on console:using System;using System.Net;using System.Collections.Generic;using Newtonsoft.Json;using System.Text; namespace ConsoleApplication1{ class PhoenixSample { public class FirstFrame { public String offset; public List<List<String>> rows; } public class Result { public FirstFrame firstFrame; public int updateCount; } public class PhoenixResult { public string response { get; set; } public List<Result> results { get; set; } } static void Main(string[] args) { try { string url = "http://192.168.56.203:8765"; string query = "select * from test"; var syncClient = new WebClient

Brandon Wilson has a great article that shows how to use the "CACHE TABLE" cmd in Tableau, however more recent drivers have come out and you can now connect directly to the thriftserver using a spark-sql driver. This is using HDP 2.5 and SimbaSparkOdbc.First pull up a Tableau connection and select the thriftServer. Additionally had to open the virtualbox port 10015. Next if you don't have the driver Tableau will jump you to a page where you can download a spark-sql driver and inside that package chose this driver.Once you establish a valid connection you will see Tableau flag the connects based on the driver. Below you will see the Hive connection from Brandon's article and now the new Spark connection.Next using the CACHE cmd enter the below into Tableau's initial SQL box.Finally check the storage of spark for the warehouse/crimes table in memory. Or any table of your chosing for that matter.Some visuals from Tableau.

Microsoft HDInsights provides the four different options under Cluster Type and they are: Hadoop, Hbase, Storm and Spark (Preview); Some of HDP components are not standard in the HDInsight distribution of choice. In this particular case, the customer was interested in Spark / Spark Notebooks Components and especially Zeppelin. When choosing the Spark Option, only these services will be provided under Ambari: HDFS, MR2, YARN, Tez, Hive, Pig, Sqoop, OOzie, Zookeeper, Ambari Metrics, Jupyter, Livy (Livy is the remote job submission server for Spark), and Spark; Zeppelin is missing from the HDI Spark distro at the time of writing this article; So in order to install Zeppelin When choosing 'Spark' as the HDI ‘Cluster Type' , and under 'New HDInsights Cluster' >> 'Optional Configuration’ >> 'Scripts Action' put the following URI as shown below: https://hdiconfigactions.blob.core.windows.net/linuxincubatorzeppelinv01/install-zeppelin-spark151-v01.sh Further services

Linkedin PostHave you heard of computing on traditional disk-based or flash-based technologies? We all use disks/flash storage in our laptops, desktop or servers sitting in the data center.What is Apache Ignite In-Memory Data Fabric ?It's high-performance, integrated and distributed in-memory platform for computing and transacting on large-scale data sets in real-time "orders of magnitude faster than possible with traditional disk-based or flash-based technologies."Fabric: In information technology, fabric is a synonym for the words framework or platform. You can view Ignite as a collection of independent, well-integrated, in-memory components geared to improve performance and scalability of your application.Source How Ignite fits with HDFS? (Ignite file system , IGFS)IGFS shakes hand with HDFS. Hadoop can run over IGFS in plug-n-play fashion and significantly reduce I/O and improve both, latency and throughput.Why do we need another layer on the top of HDFS? IGFS supports dual-mode. A

Atlas API : Create Trait TypeTraits are Atlas types that act like decorators to provide a mechanism for classification. Traits can be applied to any Atlas entities (instances of classes). An entity can have any number of traits. For example, a SECURITY_CLEARANCE trait could be defined with an attribute ‘security_level’. An Atlas entity (e.g. an instance of Person class) could be tagged with the SECURITY_CLEARANCE trait by providing a value for the trait’s ‘security_level’ attribute.In order to use a trait, you first need to define the trait type. You can do that by posting to the Types resource of the Atlas REST APIs.Types ResourceA type is the description of any representable item. In this case we are creating a trait type but the same resource is used to represent enums, structs and classes as well. /typesPOSTSubmits a types definition containing a type definition representing a trait.Request BodyTypes Definition application/json Response BodyTypes Response application/json

Spark SQL comes with a nice feature called: "JDBC to other Databases", but, it practice, it's JDBC federation feature."It can be used to create data frames from jdbc databases using scala/python, but it also works directly with Spark SQL Thrift server and allow us to query external JDBC tables seamless like other hive/spark tables."Example below using sandbox 2.3.2 and spark 1.5.1 TP (https://hortonworks.com/hadoop-tutorial/apache-spark-1-5-1-technical-preview-with-hdp-2-3/).This feature works with spark-submit, spark-shell, zeppelin, spark-sql client and spark sql thift server. In this post two examples: #1 using spark-sql thrift server, #2 using spark-shell.Example #1 using Spark SQL Thrift Server1- Run Spark SQL Thrift Server with mysql jdbc driver:sudo -u spark /usr/hdp/2.3.2.1-12/spark/sbin/start-thriftserver.sh --hiveconf hive.server2.thrift.port=10010 --jars "/usr/share/java/mysql-connector-java.jar"2- Open beeline and connect to Spark SQL Thrift Server:beeline -u "jdbc:hive2://

IntroductionWe know that parameter passing is valuable for pig script reuse. One lesser known understanding is that parameters do not simply pass variables to pig scripts but rather (and more fundamentally) they pass text that replaces placeholders in the script. This is a subtle but powerful difference: it means that we can dynamically pass code alternatives to the script at run-time. This allows us to build a script with the same larger purpose but whose logic, settings, schema, storage types, UDFs, etc. can be swapped at runtime. This results in significantly fewer yet more flexible scripts that need to be built and maintained across your projects, group or organization.In this article I will show techniques to leverage this approach through a few script examples. Keep in mind the goal is not the final examples themselves, but rather the possibilities for your own scripting.Key ideas and techniques are:parameters as text substitution-param whose parameter value holds a code sni

Here is an example of scheduling Oozie co-ordinator based on input data events. it starts Oozie workflow when input data is available.In this example coordinator will start at 2016-04-10, 6:00 GMT and will keep running till 2017-02-26, 23:25GMT (please note start and end time in xml file) start="2016-04-10T06:00Z" end="2017-02-26T23:25Z" timezone="GMT"Frequency is 1 day frequency="${coord:days(1)}"Below ETL function gives same value as start time which means coordinator will look for input data which has value same as start data in /user/root/output/YYYYMMDD format <instance>${coord:current(0)}</instance>Below are the working configuration files.coordinator.xml:<coordinator-app name="test" frequency="${coord:days(1)}" start="2016-04-10T06:00Z" end="2017-02-26T23:25Z" timezone="GMT" xmlns="uri:oozie:coordinator:0.2"> <datasets> <dataset name="inputdataset" frequency="${coord:days(1)}" initial-instance="2016-04-10T06:00Z" timezone

Use Case: Process a Media Feed, Store Everything, Run Sentiment Analysis on the Stream, and Act on a ConditionI have my GetTwitter processor looking at my twitter handle and the keyword Hadoop, something I tend to tweet frequently.I use an EvaluateJsonPath to pull out all the attributes I like (msg, user name, geo information, etc...). I use a AttributesToJSON processor to make a new JSON file from just my attributes for a smaller tweet. I store the raw JSON data in HDFS as well in a separate directory.Sending HTML Email is a bit tricky, you need to make sure you don't include extra text, so nothing in the message but RAW HTML as seen below and don't Attach Files or Include All Attributes in Message. Make sure you set the Content Type to text/html.For sentiment analysis I wanted to run something easy, so I use an ExecuteStreamCommand to run a Python 2.7 script that uses NLTK Vader SentimentIntensityAnalyzer. The NiFi part is easy, just a command and call a shell script. The

hdfs dfsadmin -setBalancerBandwidth 100000000 on all the DN and the client we ran the command below hdfs balancer -Dfs.defaultFS=hdfs://<NN_HOSTNAME>:8020 -Ddfs.balancer.movedWinWidth=5400000 -Ddfs.balancer.moverThreads=1000 -Ddfs.balancer.dispatcherThreads=200 -Ddfs.datanode.balance.max.concurrent.moves=5 -Ddfs.balance.bandwidthPerSec=100000000 -Ddfs.balancer.max-size-to-move=10737418240 -threshold 5 This will faster balance your HDFS data between datanodes and do this when the cluster is not heavily used. Hope this helps you.