Resources

Ambari 2.4.0.0 officially supports LogSearch Component [ Tech Preview] . To learn more about Log Search component please refer to link Ambari LogSearch . While LogSearch component does support pushing the logs to Kafka topic [ based on which real time log analytics can be performed ] this is official not supported in Ambari 2.4.0.0. This might get addressed in Ambari 2.5.0.0 probably.This article provides the details on how to configure LogSearch [ LogFeeder component ] to push to Kafka topic if there is a need to capture and perform real time analytics based on logs in your cluster.1.After installing LogSearch component from Ambari 2.4.0.0, go to the LogSearch component config screen and add below property under "Advanced logfeeder-properties" to property "logfeeder.config.files"{default_config_files},kafka-output.json2. Create the kafka-output.json file with below content under directory /etc/ambari-logsearch-logfeeder/conf/ on the nodes which has logfeeder [ ideally all the nodes i

ObjectivesThis tutorial will walk you the process of starting Atlas on the Hortonworks Sandbox for HDP 2.5. By default the service is disabled. However manually starting the service will fail unless you start the dependencies. Atlas depends on Ambari Infra (which provides Solr), Kafka and HBase. Atlas will start with just the Ambari Infra service running, you won't have proper functionality without Kafka and KBaseScopeThis has been testing on the following: VirtualBox 5.1.6Hortonworks Sandbox for HDP 2.5StepsStart Atlas ServiceGet your sandbox up and running and log into Ambari. Click on the Atlas service link. You should see something similar to this:Because Atlas is in maintenance mode, it will not automatically start. When you try to start it by going to Service Actions -> Start like this:You will see the following error:If you look at the error message provide you will see the problem is related to Solr:Client is connected to ZooKeeperUsing default ZkACLProviderUpdating cl

eu-hdfs-restart.pngDuring the process of Express Upgrade for a cluster that was deployed using Blueprint configured to be a HDFS HA, for a cluster with versions HDP 2.4.2 and Ambari 2.2.2, the NN failed to restart as shown below. 13debf893a605e8a88df18a7d8d214f571e05289; compiled by 'jenkins' on 2016-04-25T05:46Z\nSTARTUP_MSG: java = 1.8.0_60\n************************************************************/\n16/09/14 18:40:33 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT]\n16/09/14 18:40:33 INFO namenode.NameNode: createNameNode [-bootstrapStandby, -nonInteractive]\n16/09/14 18:40:34 WARN common.Util: Path /hadoopfs/fs1/hdfs/namenode should be specified as a URI in configuration files. Please update hdfs configuration.\n16/09/14 18:40:34 WARN common.Util: Path /hadoopfs/fs1/hdfs/namenode should be specified as a URI in configuration files. Please update hdfs configuration.\n16/09/14 18:40:35 INFO ipc.Client: Retrying connect to server: <IP ADDRESS>:80

This has come up a few times. You’ll sometimes notice that after a Banana deployment in SOLR that you can’t save your dashboards in Banana. To enable this, you have to create an index that stores these dashboards. In order to enable this, all you need to do is run the following statement which will create a banana-int index.sh ${SOLR_HOME}/bin/solr create_core -c banana-int -d ../server/solr-webapp/webapp/banana/resources/banana-int-solr-5.0/confThen restart SOLR...sh ${SOLR_HOME}/bin/solr restartThen you can...1) Save your dashboard:2) And access your saved dashboard:Happy searching!

Objective This tutorial is intended to describe the process of converting the Hortonworks HDP 2.5 Technical Preview sandbox, which is a VirtualBox virtual machine, into a Vagrant box.Overview Given the limited resources of a single virtual machine, it can be difficult to run and manage multiple demo environments within a single virtual machine. It is a common desire to add additional software components to the sandbox. Adding additional components often means having to manually manage port forwarding rules in VirtualBox or you have to add an additional network interface to the virtual machine to directly access it via an IP address using either host-only or bridged networking. Given the steps involved, it is common to do this once and try to reuse the same virtual machine. This may not always be ideal. Using Vagrant, you can quickly create a clean sandbox with a simple command like vagrant init HDP25TP && vagrant up. Vagrant makes it easy to connect to the virtual mach

Objectives: This article will walk you through the process of creating a dashboard in Kibana using Twitter data that was pushed to Elasticsearch via NiFi. The tutorial will also cover basics of Elasticsearch mappings and templates. Prerequisites: You should already have installed the Hortonworks Sandbox (HDP 2.5 Tech Preview). You should already have completed the NiFi + Twitter + Elasticsearch tutorial here: HCC Article Make sure your GetTwitter and PutElasticsearch processors in your NiFi data flow are stopped. NOTE: While not required, I highly recommend using Vagrant to manage multiple Virtualbox environments. You can read more about converting the HDP Sandbox Virtualbox virtual machine into a Vagrant box here: HCC Article Scope This tutorial was tested using the following environment and components: Mac OS X 10.11.6 HDP 2.5 Tech Preview on Hortonworks Sandbox Apache NiFi 1.0.0 (Read more here: Apache NiFi) Elasticsearch 2.3.5 and Elasticsearch 2.4.0 (Read more here: Elasticsear

The most recent release of Kafka 0.9 with it's comprehensive security implementation has reached an important milestone. In his blog post Kafka Security 101 Ismael from Confluent describes the security features part of the release very well.As a part II of the here published post about Kafka Security with Kerberos this post discussed a sample implementation of a Java Kafka producer with authentication. It is part of a mini series of posts discussing secure HDP clients, connecting services to a secured cluster, and kerberizing the HDP Sandbox (Download HDP Sandbox). In this effort at the end of this post we will also create a Kafka Servlet to publish messages to a secured broker.Kafka provides SSL and Kerberos authentication. Only Kerberos is discussed here.Kafka from now on supports four different communication protocols between Consumers, Producers, and Brokers. Each protocol considers different security aspects, while PLAINTEXT is the old insecure communication protocol.PLAINT

Why not trying a Kaggle Challenge (Titanic) !(This is a work in progress, I will update this article as soon as I get more free time. Hope you will enjoy it !) Let's create the corresponding database first : CREATE DATABASE IF NOT EXISTS kaggle_titanic; And then load the data : USE kaggle_titanic;DROP TABLE IF EXISTS train_tmp;ADD JAR csv-serde-1.1.2-0.11.0-all.jar;CREATE TABLE train_tmp ( PassengerId DOUBLE COMMENT 'regex : 999', Survived DOUBLE COMMENT 'regex : 9', Pclass INT COMMENT 'regex : 9', Name STRING COMMENT 'regex : _Zzz!_Zzzzz_!Zzzzzzzzz_!!Zzzzzz!!_Zzzzzz_Zzzzz!!', Sex STRING COMMENT 'regex : zzzzzz', Age DOUBLE COMMENT 'regex : 99!9', SibSp DOUBLE COMMENT 'regex : 9', Parch DOUBLE COMMENT 'regex : 9', Ticket STRING COMMENT 'regex : ZZZZZ!Z!Z!_9999999', Fare DOUBLE COMMENT 'regex : 999!9999', Cabin STRING COMMENT 'Z99_Z99_Z99_Z99', Embarked STRING COMMENT 'regex : Z')ROW FORMAT SERDE 'com.bizo.hive.serde.csv.CSVSerde'with serdeproperties ( "separatorChar" =

In this article I will review the steps required to enrich and filter logs. It is assumed that the logs are landing one at a time as a stream into the nifi cluster. The steps involved Extract Attributes - IP and ActionExtract Attributes - IP and ActionCold Store non ip logs GeoEnrich the IP addressCold store local IP addressesRoute the remaining logs based on threat levelStore the low threat logs in HDFSPlace high threat logs into an external tableExtract IP Address and Action - ExtractText Processor This processor will evaluate each log and parse the information into attributes. To create a new attribute add a property and give it a name(soon to be attribute name) and a java-style regex command. As the processor runs it will evaluate the regex and create an attribute with the result. If there is no match it will be sent to the 'unmatched' result which is a simple way of filtering out different logs.GeoEnrichIP - GeoEnrichIP Processor This processor takes the ipaddr attribute gener

Sensor Reading with Apache NiFi 1.0.0There are many types of sensors, devices and meters that can be great sources of data. Some can push data, some can pull data, some provide APIs, some give you access to install software. How To Access SensorsOne option is to install MiNiFi on the device if you have root access. This will provide fast access to allow you to script and manage your local data. Another option for bigger devices is to install a full Java based NiFi node. It starts becoming harder once you have tens of thousands of devices. You can install an HDF Edge and communicate from this node to your HDF cluster via Site-to-Site protocol. From this Edge Node that acts as an accumulator for many devices (a good idea so that you don't send 10,000 network requests a second from each set of devices, keep as much traffic locally to save time, time-outs, networking and cloud costs). You can also now aggregate and send larger batches of data and also process some summaries

It is helpful to see values in the input fields of the wizard that actually worked! Ambari helps you at this point with some tooltips, but still you might doubt about the expected syntax of the values to provide. Sometimes you learn the hard way that the values provided are just not what Ambari expects them to be. Note: this only applies to the Kerberos wizard with the "use an existing Active Directory" option.1. (KDC host) The FQDN of your AD host2. (Realm name) The realm name, make sure you use UPPERCASE here, otherwise the install will fail !! In my case the realm name is the same the AD domain, but that is not required. If you already did submit a lowercase Realm, here is help to fix this.3. (LDAP uri) The use of secure ldap (ldaps) is required here4. (Container DN) Distinguished Name of the container (or 'OU' Organisational Unit in ldap lingo) that will hold the Ambari/HDP service accounts. This OU has to exist already in your AD.5. (Domains) Optional---

Running Spark Jobs Through Apache Beam on HDP 2.5 Yarn Cluster Using the Spark Runner with Apache Beam Apache Beam is still in incubator and not supported on HDP 2.5 or other platforms.sudo yum -y install git wget http://www.gtlib.gatech.edu/pub/apache/maven/maven-3/3.3.9/binaries/apache-maven-3.3.9-bin.tar.gz After you get Maven downloaded, move it to /opt/demo/maven or into your path. The maven download mirror will change, so grab a new URL from http://maven.apache.org/.Using Yum will give you an older Maven not supported and may interfere with something else. So I recommend getting a new Maven just for this build. Make sure you have Java 7 or greater, which you should have on an Apache machine. I am recommending Java 8 on your new HDP 2.5 nodes if possible.cd /opt/demo/ git clone https://github.com/apache/incubator-beam cd incubator-beam /opt/demo/maven/bin/mvn clean install -DskipTests If you want to run this on Spark 2.0 and not Spark 1.6.2, look here for changing environme

HDP 2.4.3.0 is released and available for immediate use. This release builds upon the last HDP 2.3.6.0 and HDP 2.4.2.0 releases, and adds over 200+ bugs from 2.3.6.0. Spark 1.6.2 is also included in this release.Please refer to the documentation site and the release notes for any late breaking information and details on how to start your installs/upgrades.This will be made available on Azure's HDInsight offering in due time.

Thanks Rob for an excellent article .. https://community.emc.com/community/products/isilon/blog/2016/08/16/configuring-ambari-hive-view-with-onefs

I often hear stories of wanting faster performance from Hadoop & spark without knowing basic statistics within ones environment. One of the first questions I ask is whether the hardware can perform at the level which is being expected. The software is still bound to the physics of the hardware.If your IO disk speed is 10MB per sec, Hadoop/Spark nor any other software will magically make that disk speed faster. Again we are bound to the physical limits of the hardware we choose. What makes Hadoop and other distributed processing engines amazing is its ability to add more "cheap" nodes to the cluster to increase performance. However we should be aware the maximum throughput per node. This will help level set expectations before committing to any SLA bound to performance.Typically I love to use the sysbench tool. SysBench is a modular & multi-threaded benchmark tool for evaluating OS parameters ie. CPU, ram, IO, and mutex. I use sysbench prior to installing any software outs

Hortonworks Data Platform 2.5 enhances your data access experience by redesigning and centralizing data access content, including providing you with content roadmaps for Apache Hive and Apache HBase. These content roadmaps help you find your way through the wider HDP and Apache libraries, with links to relevant sites and topics based on your task.Working with Hive? You might want to try the technical preview of Interactive and Sub-Second SQL Queries through LLAP. HDP 2.5 also includes improved security through row-level filtering and column masking. And by default, the Hive View now supports a JDBC connection.HBase users, you can now backup and restore datasets that run on production clusters by performing a single full backup and then capturing incremental changes to the dataset. HDP 2.5 also provides HBase Medium Object (MOB) Storage support.Using Apache Phoenix? HDP 2.5 supports Apache Phoenix version 4.7 including improved integration of HBase and Phoenix namespaces. And you can co

Objective: The purpose of this tutorial is to walk you through the process of using NiFi to pull data from Twitter and push it to Elasticsearch. I also show an example Zeppelin dashboard which queries the Twitter data in Elasticsearch. This is the second of two articles covering Elasticsearch on HDP. The first article covers manually creating Movie data in Zeppelin and pushing that data to Elasticsearch. You can find that article here: HCC Article Note: The Zeppelin Elasticserch interpreter is a community provided interpreter. It is not yet considered GA by Hortonworks and should only be used for development and testing purposes. Prerequisites: You should already have installed the Hortonworks Sandbox (HDP 2.5 Tech Preview). You should already have enabled the Elasticsearch interpreter in Zeppelin. See this article: HCC Article You should already have twitter access keys. You create your access keys here: Twitter Apps. Read more here: Twitter Docs. Note: While not required, I recomm

There’s a new Data Movement and Integration (DMI) guide for the HDP 2.5.0 release. The guide currently focuses on Falcon features and functionality around the definition and scheduling of jobs for the backup, mirror, transfer, and processing of data. It also includes some content related to Flume, Sqoop, and Oozie.The existing Falcon content was moved out of the Data Governance Guide and into the DMI guide. New content was also added to the DMI guide to reflect changes to the Falcon UI and new Falcon features. In addition, the content has gone through an initial reorganization in an effort to more closely reflect the user’s experience.Significant changes to Falcon content in the guide include: Rewrote Tasks based on an improved, redesigned UI.Added Content about new functionality, including: RDBMS backup capabilitySnapshot-based backupsHive data mirroringMirroring data between on-premise and cloud data with Azure and AWS S3Ability to edit cluster entities using safe modeNew entity depe

Ranger installation is not as simple as Adding Service on Ambari.Ranger also requires DB and Solr for audit.In this tutorial we will go through installing Ranger or default postgres DB installed during Ambari Server installation.If you did a vanilla Ambari-server Install (HDP 2.4) you should have postgres running.Verify Default Post gresql running on Ambari-ServerOn the Ambari-Server if you run "ps -eaf | grep ambari | grep postgres" you should be able to verify postgres db running.We can see the process id is 9595. Running "netstat -anp | grep 9595" gives you the port number postgres DB is running on. We can try connecting using psql command. In our case it is running on default installation port 5432.Now Let's create DB for ranger and give it permissions. If you have sudo you can log into postgres with postgres user. For reference please take a look at (https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.0/bk_security/content/configuring_postgresql_for_ranger.html)Confirm by

When you create a database or internal tables in hive cli then by default it creates with 777 permission.Even though if you have umask in hdfs then also it will be same permission. But now you can change it with the help of following steps.1.From the command line in the Ambari server node, edit the filevi /var/lib/ambari–server/resources/common–services/HIVE/0.12.0.2.0/package/scripts/hive.pySearch for hive_apps_whs_dir which should go to this block:params.HdfsResource(params.hive_apps_whs_dir,type=“directory”,action=“create_on_execute”,owner=params.hive_user,group=params.user_group,mode=0755)2. Modify the value for mode from 0777 to the desired permission, for example 0750.Save and close the file.3. Restart the Ambari server to propagate the change to all nodes in the cluster:ambari–server restart4. From the Ambari UI, restart HiveServer2 to apply the new permission to the warehouse directory. If multiple HiveServer2 instances are configured, any one instance can be restarted.hive>

One of the most gargantuan tasks of any cluster administrator is upgrading the bits since it is a tedious, risky, and inherently complex process that can take days. Ambari comes to the rescue with two features: Rolling Upgrade (RU), and Express Upgrade (EU), aimed upgrading the HDP cluster with a couple of clicks.For starters, Rolling Upgrade was first released in Ambari 2.0.0 (March 2015) and the latest incarnation (as of October 2015) in Ambari 2.1.2 provides added robustness and a couple of goodies.Express Upgrade is set to be released in Ambari 2.2.0 (ETA is Dec 16, 2015).So how does it work? What are the gotcha's? What do I need to know? Overview:Both upgrade mechanisms update the bits and configurations of your cluster. The main difference is that RU upgrades one node at a time, and maintains the cluster and all Hadoop jobs running, while EU stops all services, changes the version and configs, then starts all services in parallel. Therefore, RU is the prime candidate for environm

Enabling SMTP in Cloudbreak---------------------------1. The Profile file2. Bug in mailer.js and a workaround    2.1 The Problem    2.2 The Cause    2.3 Details    2.4 A Workaround3. Fix postfix config---1. The Profile file-------------------In the Profile file, set the following CLOUDBREAK_SMTP_* variables and set CBD_FORCE_START to enable starting Cloudbreak containers with amodified file docker-composer.yml (see section 2.4):  cloudbreak $ cd $CBD_ROOT  cloudbreak $ more Profile  export PUBLIC_IP=example.compute.amazonaws.com  export AWS_SECRET_ACCESS_KEY=***  export AWS_ACCESS_KEY_ID=***  export CBD_FORCE_START=true  export CLOUDBREAK_SMTP_SENDER_HOST="172.17.0.1"  export CLOUDBREAK_SMTP_SENDER_FROM="cloudbreak@compute.amazonaws.com"  export CLOUDBREAK_SMTP_AUTH=false  export CLOUDBREAK_SMTP_STARTTLS_ENABLE=false---2. Bug in mailer.js and a workaround------------------------------------See  https://github.com/sequenceiq/cloudbreak/issues/1492---2.1 The Problem---------------Can not

Apache Zeppelin with HDP 2.4.2In March, 2016 we delivered 2nd technical preview of Apache Zeppelin on HDP 2.4. Meanwhile we and the Zeppelin community continues to add new features. We now give you the final technical preview of Zeppelin, based on snapshot of Apache Zeppelin 0.6.0. The main features in this Zeppelin technical preview are: Ambari Managed InstallationZeppelin Livy integrationSecurity Execute jobs as authenticated userZeppelin authentication against LDAPNotebook AuthorizationPrerequisites:HDP 2.4.2 is installedThe cluster contains Spark 1.6 or 1.5Git is installed on the node running Ambari Server You can install git as ‘ sudo yum install git’Java 8 is installed on the node where Zeppelin is installedThis document provides instructions for :Setting up Zeppelin on HDP 2.4.2 with Spark 1.6 Ambari Managed InstallRunning Zeppelin against Spark on YARN with Livy interpreterAuthentication with ZeppelinAuthenticate users against LDAPAccess Control on NotebooksNote, while both A

Are you eager to see how the new LLAP Hive feature in Hortonworks Data Platform 2.5 Tech Preview optimizes queries? Using Hortonworks Cloud, you can easily stand up a cluster in Amazon Web Services configured with LLAP, load up some benchmark data and compare queries using LLAP and without LLAP.1. A Key Pair is required when logging into an AWS host. If you already have a key pair and its associated PEM file, skip to the next step. If this is the first time you are creating AWS instances in a region or you want to create a new Key Pair, follow the steps in the AWS KeyPair documentation. When you download the key file, make sure you save the file to a known location. You will not be able to log into your instance without it and AWS will not give you another opportunity to download it. If you lose the key file, you can terminate your existing instance and launch new ones.2. Launch a CloudController instance. The CloudController provides a web interface where you can quickly spin up a clu

One of the most highly anticipated features of Apache NiFi 1.0.0 is the introduction of Zero-Master Clustering. Previous versions of NiFi relied upon a single "Master Node" (more formally known as the NiFi Cluster Manager) to show the User Interface. If this node was lost, data continued to flow, but the application was unable to show the topology of the flow, or show any stats. Additionally, Site-to-Site communications continued to send data but could not obtain up-to-date information about cluster topology, which resulted in less efficient load balancing.Version 1.0.0 of NiFi addresses these issues by switching to a Zero-Master Clustering paradigm. This post will explore the approaches taken to ensure that NiFi provides high availability of the control plane without sacrificing the User Experience. After all, the User Experience is what has allowed NiFi to become the go-to solution for providing dataflow management to small organizations as well as the world's l