Community Articles

In this article I will review the steps required to enrich and filter logs. It is assumed that the logs are landing one at a time as a stream into the nifi cluster. The steps involved Extract Attributes - IP and ActionExtract Attributes - IP and ActionCold Store non ip logs GeoEnrich the IP addressCold store local IP addressesRoute the remaining logs based on threat levelStore the low threat logs in HDFSPlace high threat logs into an external tableExtract IP Address and Action - ExtractText Processor This processor will evaluate each log and parse the information into attributes. To create a new attribute add a property and give it a name(soon to be attribute name) and a java-style regex command. As the processor runs it will evaluate the regex and create an attribute with the result. If there is no match it will be sent to the 'unmatched' result which is a simple way of filtering out different logs.GeoEnrichIP - GeoEnrichIP Processor This processor takes the ipaddr attribute gener

Sensor Reading with Apache NiFi 1.0.0There are many types of sensors, devices and meters that can be great sources of data. Some can push data, some can pull data, some provide APIs, some give you access to install software. How To Access SensorsOne option is to install MiNiFi on the device if you have root access. This will provide fast access to allow you to script and manage your local data. Another option for bigger devices is to install a full Java based NiFi node. It starts becoming harder once you have tens of thousands of devices. You can install an HDF Edge and communicate from this node to your HDF cluster via Site-to-Site protocol. From this Edge Node that acts as an accumulator for many devices (a good idea so that you don't send 10,000 network requests a second from each set of devices, keep as much traffic locally to save time, time-outs, networking and cloud costs). You can also now aggregate and send larger batches of data and also process some summaries

It is helpful to see values in the input fields of the wizard that actually worked! Ambari helps you at this point with some tooltips, but still you might doubt about the expected syntax of the values to provide. Sometimes you learn the hard way that the values provided are just not what Ambari expects them to be. Note: this only applies to the Kerberos wizard with the "use an existing Active Directory" option.1. (KDC host) The FQDN of your AD host2. (Realm name) The realm name, make sure you use UPPERCASE here, otherwise the install will fail !! In my case the realm name is the same the AD domain, but that is not required. If you already did submit a lowercase Realm, here is help to fix this.3. (LDAP uri) The use of secure ldap (ldaps) is required here4. (Container DN) Distinguished Name of the container (or 'OU' Organisational Unit in ldap lingo) that will hold the Ambari/HDP service accounts. This OU has to exist already in your AD.5. (Domains) Optional---

Running Spark Jobs Through Apache Beam on HDP 2.5 Yarn Cluster Using the Spark Runner with Apache Beam Apache Beam is still in incubator and not supported on HDP 2.5 or other platforms.sudo yum -y install git wget http://www.gtlib.gatech.edu/pub/apache/maven/maven-3/3.3.9/binaries/apache-maven-3.3.9-bin.tar.gz After you get Maven downloaded, move it to /opt/demo/maven or into your path. The maven download mirror will change, so grab a new URL from http://maven.apache.org/.Using Yum will give you an older Maven not supported and may interfere with something else. So I recommend getting a new Maven just for this build. Make sure you have Java 7 or greater, which you should have on an Apache machine. I am recommending Java 8 on your new HDP 2.5 nodes if possible.cd /opt/demo/ git clone https://github.com/apache/incubator-beam cd incubator-beam /opt/demo/maven/bin/mvn clean install -DskipTests If you want to run this on Spark 2.0 and not Spark 1.6.2, look here for changing environme

HDP 2.4.3.0 is released and available for immediate use. This release builds upon the last HDP 2.3.6.0 and HDP 2.4.2.0 releases, and adds over 200+ bugs from 2.3.6.0. Spark 1.6.2 is also included in this release.Please refer to the documentation site and the release notes for any late breaking information and details on how to start your installs/upgrades.This will be made available on Azure's HDInsight offering in due time.

Thanks Rob for an excellent article .. https://community.emc.com/community/products/isilon/blog/2016/08/16/configuring-ambari-hive-view-with-onefs

I often hear stories of wanting faster performance from Hadoop & spark without knowing basic statistics within ones environment. One of the first questions I ask is whether the hardware can perform at the level which is being expected. The software is still bound to the physics of the hardware.If your IO disk speed is 10MB per sec, Hadoop/Spark nor any other software will magically make that disk speed faster. Again we are bound to the physical limits of the hardware we choose. What makes Hadoop and other distributed processing engines amazing is its ability to add more "cheap" nodes to the cluster to increase performance. However we should be aware the maximum throughput per node. This will help level set expectations before committing to any SLA bound to performance.Typically I love to use the sysbench tool. SysBench is a modular & multi-threaded benchmark tool for evaluating OS parameters ie. CPU, ram, IO, and mutex. I use sysbench prior to installing any software outs

Hortonworks Data Platform 2.5 enhances your data access experience by redesigning and centralizing data access content, including providing you with content roadmaps for Apache Hive and Apache HBase. These content roadmaps help you find your way through the wider HDP and Apache libraries, with links to relevant sites and topics based on your task.Working with Hive? You might want to try the technical preview of Interactive and Sub-Second SQL Queries through LLAP. HDP 2.5 also includes improved security through row-level filtering and column masking. And by default, the Hive View now supports a JDBC connection.HBase users, you can now backup and restore datasets that run on production clusters by performing a single full backup and then capturing incremental changes to the dataset. HDP 2.5 also provides HBase Medium Object (MOB) Storage support.Using Apache Phoenix? HDP 2.5 supports Apache Phoenix version 4.7 including improved integration of HBase and Phoenix namespaces. And you can co

Objective: The purpose of this tutorial is to walk you through the process of using NiFi to pull data from Twitter and push it to Elasticsearch. I also show an example Zeppelin dashboard which queries the Twitter data in Elasticsearch. This is the second of two articles covering Elasticsearch on HDP. The first article covers manually creating Movie data in Zeppelin and pushing that data to Elasticsearch. You can find that article here: HCC Article Note: The Zeppelin Elasticserch interpreter is a community provided interpreter. It is not yet considered GA by Hortonworks and should only be used for development and testing purposes. Prerequisites: You should already have installed the Hortonworks Sandbox (HDP 2.5 Tech Preview). You should already have enabled the Elasticsearch interpreter in Zeppelin. See this article: HCC Article You should already have twitter access keys. You create your access keys here: Twitter Apps. Read more here: Twitter Docs. Note: While not required, I recomm

There’s a new Data Movement and Integration (DMI) guide for the HDP 2.5.0 release. The guide currently focuses on Falcon features and functionality around the definition and scheduling of jobs for the backup, mirror, transfer, and processing of data. It also includes some content related to Flume, Sqoop, and Oozie.The existing Falcon content was moved out of the Data Governance Guide and into the DMI guide. New content was also added to the DMI guide to reflect changes to the Falcon UI and new Falcon features. In addition, the content has gone through an initial reorganization in an effort to more closely reflect the user’s experience.Significant changes to Falcon content in the guide include: Rewrote Tasks based on an improved, redesigned UI.Added Content about new functionality, including: RDBMS backup capabilitySnapshot-based backupsHive data mirroringMirroring data between on-premise and cloud data with Azure and AWS S3Ability to edit cluster entities using safe modeNew entity depe

Ranger installation is not as simple as Adding Service on Ambari.Ranger also requires DB and Solr for audit.In this tutorial we will go through installing Ranger or default postgres DB installed during Ambari Server installation.If you did a vanilla Ambari-server Install (HDP 2.4) you should have postgres running.Verify Default Post gresql running on Ambari-ServerOn the Ambari-Server if you run "ps -eaf | grep ambari | grep postgres" you should be able to verify postgres db running.We can see the process id is 9595. Running "netstat -anp | grep 9595" gives you the port number postgres DB is running on. We can try connecting using psql command. In our case it is running on default installation port 5432.Now Let's create DB for ranger and give it permissions. If you have sudo you can log into postgres with postgres user. For reference please take a look at (https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.0/bk_security/content/configuring_postgresql_for_ranger.html)Confirm by

When you create a database or internal tables in hive cli then by default it creates with 777 permission.Even though if you have umask in hdfs then also it will be same permission. But now you can change it with the help of following steps.1.From the command line in the Ambari server node, edit the filevi /var/lib/ambari–server/resources/common–services/HIVE/0.12.0.2.0/package/scripts/hive.pySearch for hive_apps_whs_dir which should go to this block:params.HdfsResource(params.hive_apps_whs_dir,type=“directory”,action=“create_on_execute”,owner=params.hive_user,group=params.user_group,mode=0755)2. Modify the value for mode from 0777 to the desired permission, for example 0750.Save and close the file.3. Restart the Ambari server to propagate the change to all nodes in the cluster:ambari–server restart4. From the Ambari UI, restart HiveServer2 to apply the new permission to the warehouse directory. If multiple HiveServer2 instances are configured, any one instance can be restarted.hive>

One of the most gargantuan tasks of any cluster administrator is upgrading the bits since it is a tedious, risky, and inherently complex process that can take days. Ambari comes to the rescue with two features: Rolling Upgrade (RU), and Express Upgrade (EU), aimed upgrading the HDP cluster with a couple of clicks.For starters, Rolling Upgrade was first released in Ambari 2.0.0 (March 2015) and the latest incarnation (as of October 2015) in Ambari 2.1.2 provides added robustness and a couple of goodies.Express Upgrade is set to be released in Ambari 2.2.0 (ETA is Dec 16, 2015).So how does it work? What are the gotcha's? What do I need to know? Overview:Both upgrade mechanisms update the bits and configurations of your cluster. The main difference is that RU upgrades one node at a time, and maintains the cluster and all Hadoop jobs running, while EU stops all services, changes the version and configs, then starts all services in parallel. Therefore, RU is the prime candidate for environm

Enabling SMTP in Cloudbreak---------------------------1. The Profile file2. Bug in mailer.js and a workaround    2.1 The Problem    2.2 The Cause    2.3 Details    2.4 A Workaround3. Fix postfix config---1. The Profile file-------------------In the Profile file, set the following CLOUDBREAK_SMTP_* variables and set CBD_FORCE_START to enable starting Cloudbreak containers with amodified file docker-composer.yml (see section 2.4):  cloudbreak $ cd $CBD_ROOT  cloudbreak $ more Profile  export PUBLIC_IP=example.compute.amazonaws.com  export AWS_SECRET_ACCESS_KEY=***  export AWS_ACCESS_KEY_ID=***  export CBD_FORCE_START=true  export CLOUDBREAK_SMTP_SENDER_HOST="172.17.0.1"  export CLOUDBREAK_SMTP_SENDER_FROM="cloudbreak@compute.amazonaws.com"  export CLOUDBREAK_SMTP_AUTH=false  export CLOUDBREAK_SMTP_STARTTLS_ENABLE=false---2. Bug in mailer.js and a workaround------------------------------------See  https://github.com/sequenceiq/cloudbreak/issues/1492---2.1 The Problem---------------Can not

Apache Zeppelin with HDP 2.4.2In March, 2016 we delivered 2nd technical preview of Apache Zeppelin on HDP 2.4. Meanwhile we and the Zeppelin community continues to add new features. We now give you the final technical preview of Zeppelin, based on snapshot of Apache Zeppelin 0.6.0. The main features in this Zeppelin technical preview are: Ambari Managed InstallationZeppelin Livy integrationSecurity Execute jobs as authenticated userZeppelin authentication against LDAPNotebook AuthorizationPrerequisites:HDP 2.4.2 is installedThe cluster contains Spark 1.6 or 1.5Git is installed on the node running Ambari Server You can install git as ‘ sudo yum install git’Java 8 is installed on the node where Zeppelin is installedThis document provides instructions for :Setting up Zeppelin on HDP 2.4.2 with Spark 1.6 Ambari Managed InstallRunning Zeppelin against Spark on YARN with Livy interpreterAuthentication with ZeppelinAuthenticate users against LDAPAccess Control on NotebooksNote, while both A

Are you eager to see how the new LLAP Hive feature in Hortonworks Data Platform 2.5 Tech Preview optimizes queries? Using Hortonworks Cloud, you can easily stand up a cluster in Amazon Web Services configured with LLAP, load up some benchmark data and compare queries using LLAP and without LLAP.1. A Key Pair is required when logging into an AWS host. If you already have a key pair and its associated PEM file, skip to the next step. If this is the first time you are creating AWS instances in a region or you want to create a new Key Pair, follow the steps in the AWS KeyPair documentation. When you download the key file, make sure you save the file to a known location. You will not be able to log into your instance without it and AWS will not give you another opportunity to download it. If you lose the key file, you can terminate your existing instance and launch new ones.2. Launch a CloudController instance. The CloudController provides a web interface where you can quickly spin up a clu

One of the most highly anticipated features of Apache NiFi 1.0.0 is the introduction of Zero-Master Clustering. Previous versions of NiFi relied upon a single "Master Node" (more formally known as the NiFi Cluster Manager) to show the User Interface. If this node was lost, data continued to flow, but the application was unable to show the topology of the flow, or show any stats. Additionally, Site-to-Site communications continued to send data but could not obtain up-to-date information about cluster topology, which resulted in less efficient load balancing.Version 1.0.0 of NiFi addresses these issues by switching to a Zero-Master Clustering paradigm. This post will explore the approaches taken to ensure that NiFi provides high availability of the control plane without sacrificing the User Experience. After all, the User Experience is what has allowed NiFi to become the go-to solution for providing dataflow management to small organizations as well as the world's l

The objective of this post is to briefly explain how to setup an unsecured NiFi cluster with NiFi 1.0.0 (a three nodes cluster with three embedded ZooKeeper instances). One really important change with this new version is the new paradigm around cluster installation. From the NiFi documentation, we can read:Starting with the NiFi 1.0 release, NiFi employs a Zero-Master Clustering paradigm. Each of the nodes in a NiFi cluster performs the same tasks on the data but each operates on a different set of data. Apache ZooKeeper elects one of the nodes as the Cluster Coordinator, and failover is handled automatically by ZooKeeper. All cluster nodes report heartbeat and status information to the Cluster Coordinator. The Cluster Coordinator is responsible for disconnecting and connecting nodes. As a DataFlow manager, you can interact with the NiFi cluster through the UI of any node in the cluster. Any change you make is replicated to all nodes in the cluster, allowing for multiple e

It is a very common operation to do prefix scan in HBase. For example, when reading HBase table from HBase, we may use the following table scan api:val prefixFilter = new PrefixFilter(prefix)val scan: Scan = new Scan()scan.setFilter(prefixFilter)However, the code above may appear to be very slow when scanning a large HBase table. The reason is: we need to set StartRow before using PrefixFilter. Without setting the start row properly, your HBase scan may have to begin with the very first region and waste lots of time to get to the first right place.The recommended way is to use setRowPrefixFilter(byte[] rowPrefix), from its source code below, we can see that it helps us to set up the start row before doing table scan.409 public Scan setRowPrefixFilter(byte[] rowPrefix) {410 if (rowPrefix == null) {411 setStartRow(HConstants.EMPTY_START_ROW);412 setStopRow(HConstants.EMPTY_END_ROW);413 } else {414 this.setStartRow(rowPrefix);415 this.setStopRow(calculateTheClos

This article is an update of one on http://hadooped.blogspot.com/2013/10/apache-oozie-part-13-oozie-ssh-action_30.html, authored by @Anagha Khanolkar.Below are the steps to setup Oozie workflow using ssh-action: Step 1. Create job.properties.Example:#************************************************* # job.properties#*************************************************nameNode=hdfs://<namenode-machine-fqdn>:8020jobTracker=<resource-manager-fqdn>:8050queueName=defaultoozie.libpath=${nameNode}/user/oozie/share/liboozie.use.system.libpath=trueoozie.wf.rerun.failnodes=trueoozieProjectRoot=${nameNode}/user/${user.name}appPath=${oozieProjectRoot}oozie.wf.application.path=${appPath}inputDir=${oozieProjectRoot}focusNodeLogin=<username>@<remote-host-where-you-have-your-shell-script(s)>shellScriptPath=~/uploadFile.shemailToAddress=<email-id> Step 2. Write workflow.xmlExample:<!--******************************************--><!--workflow.xml

First lets create a sample file in S3: In the AWS Console , Go to S3 and create a bucket “S3Demo” and pick your region. Upload the file manually by using the upload button (example file name used later in scala: S3HDPTEST.csv)In the HDP 2.4.0 Sandbox :Download the aws sdk for java https://aws.amazon.com/sdk-for-java/ Uploaded it to the hadoop directory. You should see the aws-java-sdk-1.10.65.jar in /usr/hdp/2.4.0.0-169/hadoop/[root@sandbox bin]# ll /usr/hdp/2.4.0.0-169/hadoop/total 242692-rw-r--r-- 1 root root 32380018 2016-03-31 22:02 aws-java-sdk-1.10.65.jardrwxr-xr-x 2 root root 4096 2016-02-29 18:05 bindrwxr-xr-x 2 root root 12288 2016-02-29 17:49 clientlrwxrwxrwx 1 root root 25 2016-03-31 21:08 conf -> /etc/hadoop/2.4.0.0-169/0drwxr-xr-x 2 root root 4096 2016-02-29 17:46 etc-rw-r--r-- 1 root root 17366 2016-02-10 06:44 hadoop-annotations-2.7.1.2.4.0.0-169.jarlrwxrwxrwx 1 root root 40 2016-02-29 17:46 hadoop-annotations.jar -> hadoop-annotati

Cluster upgrade is a well-planned maintenance activity. Before starting the upgrade, it is important to ensure that your cluster is ready for upgrade. Ambari has built-in pre-checks that are executed before the actual upgrade to check the overall health of your cluster and validate its readiness for upgrade. Let us look at the pre-checks, their purpose and how to remediate the issues, so that your cluster upgrade process is smooth. A bit of background first:BackgroundAmbari supports two kinds of stack (cluster) upgrades: Rolling Upgrade (RU) and Express Upgrade (EU). Pre-checks apply to both upgrade methods, although RU has a larger set of pre-checks since the upgrade does not involve any downtimeFirst up, the types of pre-checks: Cluster level checks: Example – newer version packages should be installed on all hosts in the clusterService level checks: Example – all service components should be in installed state (no failures)Host level checks: All hosts must be communicating with Amba

SynopsisBoth Pig and Spark (PySpark) excel at iterative data processing against weblogs data in text delimited format. Is one faster than the other? Pig on Tez shows to outperform PySparkPySpark using the Databricks CSV parser shows to outperform Pig on TezPig Apache Pig allows users to MapReduce transformations using a simple scripting language called Pig Latin. Pig translates the Pig Latin script into MapReduce so that it can be executed within YARN against a dataset stored in the Hadoop Distributed File System (HDFS) Apache Tez is a distributed data processing framework for building batch and interactive applications. Tez improves the MapReduce paradigm by dramatically improving its speed, while maintaining MapReduce’s ability to scale to petabytes of data. Pig on Tez excels at iterative data processing. It is fast, easy, and the optimal solution to implement when it comes to parsing weblogs.Spark General purpose in-memory distributed data processing engine API’s to write cod

Objective: The purpose of this tutorial is to walk you through the process of enabling the Elasticsearch interpreter for Zeppelin on the HDP 2.5 TP sandbox. As part of this process, we will install Elasticsearch and use Zeppelin to index and query data using Zeppelin and Elasticsearch.This is the first of two articles covering Elasticsearch on HDP. The second article covers pushing Twitter data to Elasticsearch using NiFi and provides a sample Zeppelin dashboard. You can find that article here: HCC Article Note: The Zeppelin Elasticserch interpreter is a community provided interpreter. It is not yet considered GA by Hortonworks and should only be used for development and testing purposes.Prerequisites: You should already have installed the Hortonworks Sandbox (HDP 2.5 Tech Preview). Note: While not required, I recommend using Vagrant to manage multiple versions of the Sandbox. Follow my tutorial here to set that up: HCC ArticleScope: This tutorial was tested using the following

Self Service Hadoop – well some starting pointsSay you want to get started with Big Data and concurrently want to start to empower your relatively savvy end users that have been in the frustrating desktop data management land for a long time. This very simple article will hopefully help with a few options.This diagram below gives a little perspective around some of the sources and mechanisms around ingesting, manipulating and using your data assets. You can see following the numbers from 1 to 6 that you have many options with working with your data. This article will concentrate on just showing a simple end user example of ingesting data, understanding it and using options for using the data from a self service approach. Really an approach to get started and help promote some of the value of a modern data architecture as your team matures.An end users want to self service some data from their desktop/server into HDFS and be able to query and understand that data from their existing