Community Articles

This short post concentrates on solving most common issue found while publishing metadata to kafka topic for Atlas server over a secure(kerberized) cluster.Issue:With AtlasHook configured for Hive/Storm/Falcon, if you are seeing below stack trace in the logs of the corresponding component. This means, AtlasHook is not able to publish metadata to kafka for Atlas consumption. The reason for this failure could be Kafka topic to which the hook is trying to publish does not exist. ORKafka topic does not have proper access control lists(ACL) configured for the user.org.apache.kafka.common.KafkaException: Failed to construct kafka producer at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:335) at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:188) at org.apache.atlas.kafka.KafkaNotification.createProducer(KafkaNotification.java:312) at org.apache.atlas.kafka.KafkaNotification.sendInternal(KafkaNotif

This article assumes that, you have a cluster with more than one node(which is a requirement for enabling HA on Atlas). Also make sure that atlas is up and running on that cluster. Please refer to this documentation link for deploying cluster with Atlas enabled.Prerequisites for High Availability feature in AtlasThe following prerequisites must be met for setting up the High Availability feature. Ensure that you install Apache Zookeeper on a cluster of machines (a minimum of 3 servers is recommended for production).Select 2 or more physical machines to run the Atlas Web Service instances on. These machines define what we refer to as a 'server ensemble' for Atlas.Step1: Verify from Ambari UI that Atlas is up and running..Step2: Stop Atlas using AmbariStep3: Navigate to host page in Ambari where Atlas service is not installed and add one more atlas service.Step4: If Infra solr client is not installed on the host where we are trying to install another instance of Atlas, then Ambari would

Background:prior to HDP2.5 release , people used to look for a solution to sync the users from multiple organisation unit. but this was not possible, so in HDP2.5 this feature is released to support syncing the users from multiple OU.How to configure the multiple OU:OU can be configured in the same way as it was earlier , but if you want to configure multiple OU then it should semicolon(;) separated as follows:ou=Executives,DC=abc,DC=com;ou=Engineering,DC=abc,DC=comSample User setting:Sample Group setting:

In some application use cases, developers want to save Spark DataFrame table directly into Phoenix instead of saving into HBase as a intermediate step. In those case, we can use Apache Phoenix-Spark plugin package. The related api is very simple:df.save("org.apache.phoenix.spark", SaveMode.Overwrite, Map("table" -> "OUTPUT_TABLE", "zkUrl" -> "****:2181:/****"))However, we need to pay attention that in Apache Phoenix, all the column names by default are considered as uppercase unless you surround it with quotation marks "". Therefore, if you have specified lowercase column name in your Phoenix Schema, you have to do some column names transformation in Spark. The example code is as follows:val oldNames = df.columnsval newNames = oldNames.map(name => col(name).as("\"" + name + "\""))val df2 = df.select(newNames:_*)

Summary: Automation/AMI to install HDP 2.5.x with Nifi 1.1.0 on any cloud and deploy commonly used demos via Ambari blueprints Currently supported demos: Nifi-Twitter IoT (trucking) demo Zeppelin notebooks Vanilla HDF 2.1 (w/o any demos)Option 1: Deploy single node instances using AMIs 1. For deploying the above on single node setups on Amazon, AMI images are also available. To launch an instance using one of the AMIs, refer to steps below. A video that shows using these steps to launch the HDP 2.5.3 AMI is available here. Login into EC2 dashboard using your credentials Change your region to "N. California" Click 'Launch instance' Choose AMI: search for 081339556850 under Community AMIs (as shown in screenshot), select the desired AMI. For the HDP 2.5.x version of the AMI that has the demos pre-installed, select "HDP 2.5 Demo kit cluster"Choose instance type: select m4.2xlarge for HDP AMIs or m4.xlarge for HDF Configure instance: leave defaults Add storage: 100gb or la

GoalRestricting HiveCLI access to specific usersPrerequisite Access to admin account for AmbariProcedureOpen Hive Config from Ambari, you can use the following link to do sohttp://<ambari-server-hostname>:8080/#/main/services/HIVE/configs Locate the following code within "Advanced hive-env" if [ "$SERVICE" = "cli" ]; then if [ -z "$DEBUG" ]; then export HADOOP_OPTS="$HADOOP_OPTS -XX:NewRatio=12 -XX:MaxHeapFreeRatio=40 -XX:MinHeapFreeRatio=15 -XX:+UseNUMA -XX:+UseParallelGC -XX:-UseGCOverheadLimit" else export HADOOP_OPTS="$HADOOP_OPTS -XX:NewRatio=12 -XX:MaxHeapFreeRatio=40 -XX:MinHeapFreeRatio=15 -XX:-UseGCOverheadLimit" fi fiReplace the above code with the following declare -a users=(hdfs centos) if [ "$SERVICE" = "cli" ]; then for auser in ${users[@]}; do if [ "$auser" = "$USER" ]; then echo "User $USER is not authorized to use Hive" exit 1 fi done if [ -z "$DEBUG" ]; then export HADOOP_OPTS="$HADOOP_OPTS -XX:New

You may be in a bind if you need to install HDP on Azure with CentOS 6 or RHEL 6 and certain services (not everything). By following these steps below, you will be able to use ambari-server to install HDP on any of the supported Hortonworks/Azure VMs.1) Configure your VMs - use the same VNet for all VMsRun the next steps as root or sudo the commands:2) Update /etc/hosts on all your machines: vi /etc/hosts 172.1.1.0 master1.jd32j3j3kjdppojdf3349dsfeow0.dx.internal.cloudapp.net172.1.1.1 master2.jd32j3j3kjdppojdf3349dsfeow0.dx.internal.cloudapp.net172.1.1.2 master3.jd32j3j3kjdppojdf3349dsfeow0.dx.internal.cloudapp.net 172.1.1.3 worker1.jd32j3j3kjdppojdf3349dsfeow0.dx.internal.cloudapp.net172.1.1.4 worker2.jd32j3j3kjdppojdf3349dsfeow0.dx.internal.cloudapp.net172.1.1.5 worker3.jd32j3j3kjdppojdf3349dsfeow0.dx.internal.cloudapp.net* use the FQDN (find the fqdn by typing hostname -f). The ip address are internal and can be found on eth0 by typing ifconfig3) Edit /etc/sudoers.d/waagent so th

Highlights of integrating Apache NiFi with Apache Ambari/Ranger Article credits: @Ali Bajwa, @Bryan Bende, @jluniya, @Yolanda M. Davis, @brosander With the recently announced HDF 2.0, users are able to deploy an HDF cluster comprised of Apache NiFi, Apache Storm, Apache Kafka and other components. The mechanics of setting this up using Apache Ambari’s Install Wizard are outlined in the official documentation here and sample steps to automate the setup via Ambari blueprints are provided here. The goal of this article is to highlight some features NiFi administrators can leverage when using Ambari managed HDF 2.0 clusters vs using NiFi standalone The article is divided into sections on how the integration helps administrators with HDF: Deployment Configuration Monitoring Security Ease of Deployment Users have the choice of deploying NiFi through Ambari install wizard or operationalize via blueprints automation (For detailed steps, see links provided on above

SYNOPSISThis article will demonstrate using Spark (PySpark) with the S3A filesystem client to access data in S3.The code can also be found on GitHub @ https://github.com/mathewbk/spark_s3aThe source data in the S3 bucket is Omniture clickstream data (weblogs). The goal is to write PySpark code against the S3 data to RANK geographic locations by page view traffic - which areas generate the most traffic by page view counts.The S3A filesystem client (s3a://) is a replacement for the S3 Native (s3n://):It uses Amazon’s libraries to interact with S3Supports larger files Higher performanceSupports IAM role-based authenticationProduction stable since Hadoop 2.7 (per Apache website) The code below is a working example tested using HDP 2.4.0.0-169 (Hadoop version 2.7.1.2.4.0.0-169)Note that I am using the HiveContext and not the SQLContext. I am NOT querying or accessing any Hive tables, but I am using the RANK() function to perform a ranking. If you want to use the RANK() function on a Data

We just updated Hortonworks Data Cloud for AWS to Technical Preview. This new TP (version #1.6) includes few bug fixes and a new Command Line Interface to help with scripting + automating cluster creation.Checkout the Release Notes for more details:http://hortonworks.github.io/hdp-aws/releasenotes/index.htmlGet Started and Launch Hortonworks Data Cloud in your AWS environment:http://hortonworks.github.io/hdp-aws/index.htmlCheck it out. You can provide feedback and ask questions here at the Hortonworks Community Connection:http://hortonworks.github.io/hdp-aws/feedback/index.html#how-to-create-a-questionHave fun!

We'd like to announce the immediate availability of Ambari 2.4.1. Please refer to the product documentation for more details about this maintenance release. Ambari 2.4.1 Product Documentation http://docs.hortonworks.com/HDPDocuments/Ambari/Ambari-2.4.1.0/index.html Ambari 2.4.1 Upgrade Guide http://docs.hortonworks.com/HDPDocuments/Ambari-2.4.1.0/bk_ambari-upgrade/content/upgrading_ambari.html Ambari 2.4.1 Release Notes http://docs.hortonworks.com/HDPDocuments/Ambari-2.4.1.0/bk_ambari-release-notes/content/ch_relnotes-ambari-2.4.1.0.html

1 Challenges managing LAS filesSiloed datasetsWorking with disparate, complex datasets under a traditional analysis model limits innovation and does not allow for the speed required for unconventionalsLAS File VolumeA single well could have 10s or 100s of LAS files making it difficult to provide a consolidated view for analysisExtrapolating this volume out across 1000s of wells requires an automated approachManual QC processIdentifying out of range data is time consuming and challenging even for experienced geoscientists and petrophysicistsManagement and storage is expensiveWhat if cost could be reduced from $23/Gb to $.19/Gb; $55 GB could cost $1,200 or $10Delta is 1-2 orders of magnitudeDownload Sample Data SetThe wellbook concept is about a single view of an oil well and its history- something akin to a "Facebook Wall" for oil wells.This repo is built from data collected and made available by the North Dakota Industrial Commission.I used the wellindex.csv file to obtain a list of w

Hortonworks DataFlow 2.0 comes with the ability to configure TLS for Apache NiFi through Apache Ambari, this is implemented using the tls-toolkit in client/server mode.To demonstrate this functionality, lets set up a 3 node NiFi secured cluster through Ambari locally in Docker containers.First, install docker on your machine. Please note that the default docker-machine vm size is too small for this guide. You should have at least 8 gigs of ram, a few cpus, and 100 gigs of hard drive space allocated to the docker-machine you’re using.Build the Ambari stack: mkdir toolkit-demo-3/cd toolkit-demo-3/git clone https://github.com/brosander/dev-dockerfiles.gitdev-dockerfiles/ambari/server/centos6/buildStack.sh http://public-repo-1.hortonworks.com/ambari/centos6/2.x/updates/2.4.0.1/ambari.repoGenerate an ssh key for use talking to the gateway and Ambari: mkdir ambari-ssh-keysssh-keygen -t rsa -b 4096 -f ambari-ssh-keys/id_rsaRun the Ambari stack and install HDF mpack: wget -P mpack http://publi

Once you have Atlas up and running( see this for getting started. ) you will want to create your first tag and use it to tag data to begin exploring what Atlas can do. Lets start by creating a tag that will be called "PII" which we will later use it to tag Personally Identifiable data. Log into Atlas and select the + icon on the homepage. Enter "PII" for the tag name and click create. That's it! Now we have a tag to use. Now click on the search tab select "dsl" and from the drop down select "hive_table" hit enter and select the "customer" table.You should see the summary details for the customer table in hive. Select the schema tab. Here you see all columns available in the customer data table. Let's mark the "account_num" field as "PII". Next to "account_num" click the + icon and select "PII" from the drop down.Now the column has been tagged and both searchable from Atlas as well as configured to be administered through Hive for auditing and permissions. Click the "Tags" tab in Atlas

Env: HDP-2.3.4.0-3485Java 8Attached code contain:—pom.xml to manage all the dependencies— HiveClientSecure.java - oozie java action to be configured into workflow.xml— jaas.conf — oozie uses jaas configuration for kerberos login— log4j.properties - to capture logsjaas.conf: modify principal name and key tab location accordingly and place it to on each node on the cluster.I have placed it on /tmp/jaas/jaas.conf for testing purpose.Client {com.sun.security.auth.module.Krb5LoginModule requireduseKeyTab=trueuseTicketCache=trueprincipal="ambari-qa-hbase234@HWXBLR.COM"keyTab="/etc/security/keytabs/smokeuser.headless.keytab"debug="true"doNotPrompt=true;};workflow.xml:<workflow-app xmlns="uri:oozie:workflow:0.2" name="java-main-wf"> <start to="java-node"/> <a<action name="java-node"> <java> <job-tracker>${jobTracker}</job-tracker> <name-node>${nameNode}</name-node> <configuration>

You have a need of debugging, testing and operating a Hadoop cluster, especially when you run dangerous dfsadmin commands, try customized packages with changes of Hadoop/Spark source code, trying aggressive configuration values. You have a laptop and you have a production Hadoop cluster. You don't dare to operate the production cluster blindly, which is appreciated by your manager. You want to try something on a hadoop cluster and even you breaks it, no one blames you. You have several choices (perhaps you're using one of them now): psudo-distributed Hadoop cluster on a single machine, which is nontrivial to run HA, to use per-node configurations, to pause and launch multiple nodes, or to test HDFS balancer/mover etc. setting up a real cluster, which is complex and heavy to use, and in the first place you can afford a real cluster. building Ambari cluster using vbox/vmware virtual machines, nice try. But if you run 5 nodes cluster, you'll see your CPU is overloaded and memory is

Logical Disk EncryptionThe approach is to avoid using HDFS encryption and use Disk LUKS encryption for data at rest encryption requirement specially when using public cloud IAAS.To build manually encrypted volumes or drives ,use following steps on a d2*8xLarge instance flavor.lsblkcryptsetup --verbose --verify-passphrase luksFormat /dev/xvdbcryptsetup --verbose --verify-passphrase luksFormat /dev/xvdccryptsetup --verbose --verify-passphrase luksFormat /dev/xvddcryptsetup luksOpen /dev/xvdb vol1cryptsetup luksOpen /dev/xvdc vol2cryptsetup luksOpen /dev/xvdd vol3dd if=/dev/urandom of=/root/keyfile1 bs=1024 count=4chmod 0400 /root/keyfile1cryptsetup luksAddKey /dev/xvdb /root/keyfile1cryptsetup luksAddKey /dev/xvdc /root/keyfile1cryptsetup luksAddKey /dev/xvdd /root/keyfile1mkfs.ext4 /dev/mapper/vol1mkfs.ext4 /dev/mapper/vol2mkfs.ext4 /dev/mapper/vol3echo "/dev/mapper/vol1 /data/vol1 ext4 defaults,nofail,nodev 0 2" >> /etc/fstabecho "/dev/mapper/vol2 /data/vol2 ext4 defaults,nofail,

Ambari 2.4.0.0 officially supports LogSearch Component [ Tech Preview] . To learn more about Log Search component please refer to link Ambari LogSearch . While LogSearch component does support pushing the logs to Kafka topic [ based on which real time log analytics can be performed ] this is official not supported in Ambari 2.4.0.0. This might get addressed in Ambari 2.5.0.0 probably.This article provides the details on how to configure LogSearch [ LogFeeder component ] to push to Kafka topic if there is a need to capture and perform real time analytics based on logs in your cluster.1.After installing LogSearch component from Ambari 2.4.0.0, go to the LogSearch component config screen and add below property under "Advanced logfeeder-properties" to property "logfeeder.config.files"{default_config_files},kafka-output.json2. Create the kafka-output.json file with below content under directory /etc/ambari-logsearch-logfeeder/conf/ on the nodes which has logfeeder [ ideally all the nodes i

ObjectivesThis tutorial will walk you the process of starting Atlas on the Hortonworks Sandbox for HDP 2.5. By default the service is disabled. However manually starting the service will fail unless you start the dependencies. Atlas depends on Ambari Infra (which provides Solr), Kafka and HBase. Atlas will start with just the Ambari Infra service running, you won't have proper functionality without Kafka and KBaseScopeThis has been testing on the following: VirtualBox 5.1.6Hortonworks Sandbox for HDP 2.5StepsStart Atlas ServiceGet your sandbox up and running and log into Ambari. Click on the Atlas service link. You should see something similar to this:Because Atlas is in maintenance mode, it will not automatically start. When you try to start it by going to Service Actions -> Start like this:You will see the following error:If you look at the error message provide you will see the problem is related to Solr:Client is connected to ZooKeeperUsing default ZkACLProviderUpdating cl

eu-hdfs-restart.pngDuring the process of Express Upgrade for a cluster that was deployed using Blueprint configured to be a HDFS HA, for a cluster with versions HDP 2.4.2 and Ambari 2.2.2, the NN failed to restart as shown below. 13debf893a605e8a88df18a7d8d214f571e05289; compiled by 'jenkins' on 2016-04-25T05:46Z\nSTARTUP_MSG: java = 1.8.0_60\n************************************************************/\n16/09/14 18:40:33 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT]\n16/09/14 18:40:33 INFO namenode.NameNode: createNameNode [-bootstrapStandby, -nonInteractive]\n16/09/14 18:40:34 WARN common.Util: Path /hadoopfs/fs1/hdfs/namenode should be specified as a URI in configuration files. Please update hdfs configuration.\n16/09/14 18:40:34 WARN common.Util: Path /hadoopfs/fs1/hdfs/namenode should be specified as a URI in configuration files. Please update hdfs configuration.\n16/09/14 18:40:35 INFO ipc.Client: Retrying connect to server: <IP ADDRESS>:80

This has come up a few times. You’ll sometimes notice that after a Banana deployment in SOLR that you can’t save your dashboards in Banana. To enable this, you have to create an index that stores these dashboards. In order to enable this, all you need to do is run the following statement which will create a banana-int index.sh ${SOLR_HOME}/bin/solr create_core -c banana-int -d ../server/solr-webapp/webapp/banana/resources/banana-int-solr-5.0/confThen restart SOLR...sh ${SOLR_HOME}/bin/solr restartThen you can...1) Save your dashboard:2) And access your saved dashboard:Happy searching!

Objective This tutorial is intended to describe the process of converting the Hortonworks HDP 2.5 Technical Preview sandbox, which is a VirtualBox virtual machine, into a Vagrant box.Overview Given the limited resources of a single virtual machine, it can be difficult to run and manage multiple demo environments within a single virtual machine. It is a common desire to add additional software components to the sandbox. Adding additional components often means having to manually manage port forwarding rules in VirtualBox or you have to add an additional network interface to the virtual machine to directly access it via an IP address using either host-only or bridged networking. Given the steps involved, it is common to do this once and try to reuse the same virtual machine. This may not always be ideal. Using Vagrant, you can quickly create a clean sandbox with a simple command like vagrant init HDP25TP && vagrant up. Vagrant makes it easy to connect to the virtual mach

Objectives: This article will walk you through the process of creating a dashboard in Kibana using Twitter data that was pushed to Elasticsearch via NiFi. The tutorial will also cover basics of Elasticsearch mappings and templates. Prerequisites: You should already have installed the Hortonworks Sandbox (HDP 2.5 Tech Preview). You should already have completed the NiFi + Twitter + Elasticsearch tutorial here: HCC Article Make sure your GetTwitter and PutElasticsearch processors in your NiFi data flow are stopped. NOTE: While not required, I highly recommend using Vagrant to manage multiple Virtualbox environments. You can read more about converting the HDP Sandbox Virtualbox virtual machine into a Vagrant box here: HCC Article Scope This tutorial was tested using the following environment and components: Mac OS X 10.11.6 HDP 2.5 Tech Preview on Hortonworks Sandbox Apache NiFi 1.0.0 (Read more here: Apache NiFi) Elasticsearch 2.3.5 and Elasticsearch 2.4.0 (Read more here: Elasticsear

The most recent release of Kafka 0.9 with it's comprehensive security implementation has reached an important milestone. In his blog post Kafka Security 101 Ismael from Confluent describes the security features part of the release very well.As a part II of the here published post about Kafka Security with Kerberos this post discussed a sample implementation of a Java Kafka producer with authentication. It is part of a mini series of posts discussing secure HDP clients, connecting services to a secured cluster, and kerberizing the HDP Sandbox (Download HDP Sandbox). In this effort at the end of this post we will also create a Kafka Servlet to publish messages to a secured broker.Kafka provides SSL and Kerberos authentication. Only Kerberos is discussed here.Kafka from now on supports four different communication protocols between Consumers, Producers, and Brokers. Each protocol considers different security aspects, while PLAINTEXT is the old insecure communication protocol.PLAINT

Why not trying a Kaggle Challenge (Titanic) !(This is a work in progress, I will update this article as soon as I get more free time. Hope you will enjoy it !) Let's create the corresponding database first : CREATE DATABASE IF NOT EXISTS kaggle_titanic; And then load the data : USE kaggle_titanic;DROP TABLE IF EXISTS train_tmp;ADD JAR csv-serde-1.1.2-0.11.0-all.jar;CREATE TABLE train_tmp ( PassengerId DOUBLE COMMENT 'regex : 999', Survived DOUBLE COMMENT 'regex : 9', Pclass INT COMMENT 'regex : 9', Name STRING COMMENT 'regex : _Zzz!_Zzzzz_!Zzzzzzzzz_!!Zzzzzz!!_Zzzzzz_Zzzzz!!', Sex STRING COMMENT 'regex : zzzzzz', Age DOUBLE COMMENT 'regex : 99!9', SibSp DOUBLE COMMENT 'regex : 9', Parch DOUBLE COMMENT 'regex : 9', Ticket STRING COMMENT 'regex : ZZZZZ!Z!Z!_9999999', Fare DOUBLE COMMENT 'regex : 999!9999', Cabin STRING COMMENT 'Z99_Z99_Z99_Z99', Embarked STRING COMMENT 'regex : Z')ROW FORMAT SERDE 'com.bizo.hive.serde.csv.CSVSerde'with serdeproperties ( "separatorChar" =