Resources

In the previous article, we showed how to enable SSL and set up identity mappings for Apache Nifi on the previously installed HDF 2.x or 3.0 cluster. Here, we will build on the same cluster and show how to install Apache Ranger and setup the Ranger Nifi plugin. For simplicity, we will assume this is a demo environment where there is no requirement to enable SSL for Ranger. If instead you would like to use secured Ranger with NiFi, follow steps here SummaryAt a high level, Apache Ranger provides a centralized platform to define, administer and manage security policies consistently across Hadoop components. In the case of HDF, it enables the administrator to create/manage authorization policies for Kafka, Storm and Nifi from the same web interface (or REST APIs).To achieve this, the high level steps we will follow are: Ranger install prerequisitesRanger installUpdate Nifi Ranger repoTest Ranger pluginCreate Ranger uses and policiesTest Nifi access as nifiadmin userThe official documentat

hcc-58591.zipHive RegexSerDe can be used to extract columns from the input file using regular expressions. It's used only to deserialize data, while data serialization is not supported (and obviously not needed). The initial motivation to create such a SerDe was to process Apache web logs. There are two classes available: org.apache.hadoop.hive.contrib.serde2.RegexSerDe, introduced in Hive-0.4 by HIVE-662, and org.apache.hadoop.hive.serde2.RegexSerDe, a built-in class introduced in Hive-0.10 by HIVE-1719The former is kept to facilitate easier migration for legacy apps, while the letter is recommended for the new apps. The SerDe works by matching columns in the table definition with regex groups defined and captured by the regular expression. A regex group is defined by parenthesis "(...)" inside the regex. Note that this is one of common mistakes by beginners who spend time creating great regular expressions but displace or fail to mark regex groups. The new, built-in version suppo

Our stream analytics documentation received substantial updates for HDP 2.5.0.The Apache Storm Component Guide was reorganized to focus on administration and development workflow: installing and configuring Storm, developing Storm applications, moving data into and out of Storm, and managing topologies. New content includes the following topics:Installing Storm using Ambari.Configuring Storm for a production environment.Implementing windowing computations on data streams.Implementing state management in core Storm.Configuring and using the HDFS spout to ingest data from HDFS.Monitoring and debugging Apache Storm topologies through the use of dynamic log levels, topology event logging, distributed log search, and dynamic worker profiling.The Storm application development chapter differentiates more clearly between core Storm and Trident, and the Kafka spout subsection describes performance settings and tradeoffs.The Storm Ambari view is documented in the Apache Ambari Views Guide, and t

The new Apache Zeppelin Component Guide has instructions for installing, configuring, and using the web-based Zeppelin notebook for interactive data exploration and visualization.The Apache Spark Component Guide was reorganized to focus on administration and development workflows: installing and configuring Spark, developing Spark applications, using SparkR, and tuning Spark. For HDP 2.5, the book includes the following new topics:Installing the Spark 2.0 technical preview alongside Spark 1.6.2, using Ambari.Configuring Spark for wire encryption.The Apache Solr Search Installation Guide now contains instructions for installing HDP Search using Ambari.

This article "Perform Data Analysis using SAP Vora on SAP Hana data - Part 4" is continuation of "Load Demo data in SAP Vora Using Eclipse HANA Modelling tools - Part 3"Log back in to SAP Cloud Appliance Library - the free service to manage your SAP solutions in the public cloud. You should have HANA and Vora instances up and running: Open Apache Zeppelin web UI click on Connect in your SAP HANA Vora instance in CAL, and then pick Open a link for Application: Zeppelin In Zeppelin click Create new note, name the notebook as you like ex "interact with hana data"In the first cell lets define our imports and HANA connection information import sqlContext.implicits._import scala.collection.mutable.Mapval HANA_HOSTNAME = "xxx.xxx.xx.xxx"val HANA_INSTANCE = "00"val HANA_SCHEMA = "CODEJAMMER"val HANA_USERNAME = "CODEJAMMER"val HANA_PASSWORD = "CodeJam2016"4. In the next cell we will use sqlContext that was automatically created by Zeppelin to Connect to HANA using "com.sap.spark.hana" sqlConte

This article "Load Demo data in SAP Vora Using Eclipse HANA Modelling tools - Part 3" is continuation of "Configure SAP Vora HDP Ambari - Part 2"You will need to download Eclipse Neon - Eclipse IDE for Java Developers to connect to the SAP HANA that we setup in Part 1 . After you setup eclipse we will need to configure Eclipse to install HANA Modelling tools that will allow us to connect to SAP HANA and execute sql scripts to setup demo data that we will use from SAP Vora.Eclipse Setup Procedure Open the Eclipse IDE.In the main menu, choose HelpInstall New Software .Depending on the Eclipse version you have installed, enter one of the following URLs in the Work with field: For Eclipse Neon (4.6), add URL: https://tools.hana.ondemand.com/neonSelect SAP HANA Tools (the whole feature group).noteIn case you need to develop with SAPUI5, install also SAP HANA Cloud Platform ToolsUI development toolkit for HTML5 (Developer Edition) .Choose Next.On the next wizard page, you get an overview

This article "Configure SAP Vora HDP Ambari - Part 2" is continuation of "Getting started with SAP Hana and Vora with HDP using Apache Zeppelin for Data Analysis - Part 1 Intro"Log back in to SAP Cloud Appliance Library - the free service to manage your SAP solutions in the public cloud. You should have HANA and Vora instances up and running:Open Apache Ambari web UI click on Connect in your SAP HANA Vora instance in CAL, and then pick Open a link for Application: Ambari. The port of Ambari web UI has been preconfigured for you in the SAP HANA Vora, developer edition, in CAL. As well its port has been opened as one of the default Access Points. As you might remember it translates into the appropriate inbound rule in the corresponding AWS’s security group.Log into Ambari web UI using the user admin and the master password you defined during process of the creation of the instance in CAL.You can see that (1) all services, including SAP HANA Vora components, are running, that (2) there a

SAP HANA Vora is an in-memory processing engine that runs on a Hadoop cluster and is tightly integrated with Spark. It is designed for handling big data. SAP HANA Vora makes available OLAP-style capabilities on Hadoop, provides deeper integration with SAP HANA, enabling high-performance enterprise analytics, and delivers contextual insights by combining corporate data in SAP HANA with big data stored in Hadoop systems. In this multi-part guide , I will show you how to spin up an SAP HANA instance in AWS and a Vora + HDP installation on 2nd node. We will utilize Apache Zeppelin to interact with SAP HANA using a Vora interpreter . In this scenario you will be able to join data from other various data sources like HDFS and RDBMs to join to Hana data. This is a Federated Query approach to multiple data sources. "Federation" tier to act as a single point of access to data from multiple sources. For details on the concepts of Data Federation see "Virtual Integration of Hadoop with Exter

This short post concentrates on solving most common issue found while publishing metadata to kafka topic for Atlas server over a secure(kerberized) cluster.Issue:With AtlasHook configured for Hive/Storm/Falcon, if you are seeing below stack trace in the logs of the corresponding component. This means, AtlasHook is not able to publish metadata to kafka for Atlas consumption. The reason for this failure could be Kafka topic to which the hook is trying to publish does not exist. ORKafka topic does not have proper access control lists(ACL) configured for the user.org.apache.kafka.common.KafkaException: Failed to construct kafka producer at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:335) at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:188) at org.apache.atlas.kafka.KafkaNotification.createProducer(KafkaNotification.java:312) at org.apache.atlas.kafka.KafkaNotification.sendInternal(KafkaNotif

This article assumes that, you have a cluster with more than one node(which is a requirement for enabling HA on Atlas). Also make sure that atlas is up and running on that cluster. Please refer to this documentation link for deploying cluster with Atlas enabled.Prerequisites for High Availability feature in AtlasThe following prerequisites must be met for setting up the High Availability feature. Ensure that you install Apache Zookeeper on a cluster of machines (a minimum of 3 servers is recommended for production).Select 2 or more physical machines to run the Atlas Web Service instances on. These machines define what we refer to as a 'server ensemble' for Atlas.Step1: Verify from Ambari UI that Atlas is up and running..Step2: Stop Atlas using AmbariStep3: Navigate to host page in Ambari where Atlas service is not installed and add one more atlas service.Step4: If Infra solr client is not installed on the host where we are trying to install another instance of Atlas, then Ambari would

Background:prior to HDP2.5 release , people used to look for a solution to sync the users from multiple organisation unit. but this was not possible, so in HDP2.5 this feature is released to support syncing the users from multiple OU.How to configure the multiple OU:OU can be configured in the same way as it was earlier , but if you want to configure multiple OU then it should semicolon(;) separated as follows:ou=Executives,DC=abc,DC=com;ou=Engineering,DC=abc,DC=comSample User setting:Sample Group setting:

In some application use cases, developers want to save Spark DataFrame table directly into Phoenix instead of saving into HBase as a intermediate step. In those case, we can use Apache Phoenix-Spark plugin package. The related api is very simple:df.save("org.apache.phoenix.spark", SaveMode.Overwrite, Map("table" -> "OUTPUT_TABLE", "zkUrl" -> "****:2181:/****"))However, we need to pay attention that in Apache Phoenix, all the column names by default are considered as uppercase unless you surround it with quotation marks "". Therefore, if you have specified lowercase column name in your Phoenix Schema, you have to do some column names transformation in Spark. The example code is as follows:val oldNames = df.columnsval newNames = oldNames.map(name => col(name).as("\"" + name + "\""))val df2 = df.select(newNames:_*)

Summary: Automation/AMI to install HDP 2.5.x with Nifi 1.1.0 on any cloud and deploy commonly used demos via Ambari blueprints Currently supported demos: Nifi-Twitter IoT (trucking) demo Zeppelin notebooks Vanilla HDF 2.1 (w/o any demos)Option 1: Deploy single node instances using AMIs 1. For deploying the above on single node setups on Amazon, AMI images are also available. To launch an instance using one of the AMIs, refer to steps below. A video that shows using these steps to launch the HDP 2.5.3 AMI is available here. Login into EC2 dashboard using your credentials Change your region to "N. California" Click 'Launch instance' Choose AMI: search for 081339556850 under Community AMIs (as shown in screenshot), select the desired AMI. For the HDP 2.5.x version of the AMI that has the demos pre-installed, select "HDP 2.5 Demo kit cluster"Choose instance type: select m4.2xlarge for HDP AMIs or m4.xlarge for HDF Configure instance: leave defaults Add storage: 100gb or la

GoalRestricting HiveCLI access to specific usersPrerequisite Access to admin account for AmbariProcedureOpen Hive Config from Ambari, you can use the following link to do sohttp://<ambari-server-hostname>:8080/#/main/services/HIVE/configs Locate the following code within "Advanced hive-env" if [ "$SERVICE" = "cli" ]; then if [ -z "$DEBUG" ]; then export HADOOP_OPTS="$HADOOP_OPTS -XX:NewRatio=12 -XX:MaxHeapFreeRatio=40 -XX:MinHeapFreeRatio=15 -XX:+UseNUMA -XX:+UseParallelGC -XX:-UseGCOverheadLimit" else export HADOOP_OPTS="$HADOOP_OPTS -XX:NewRatio=12 -XX:MaxHeapFreeRatio=40 -XX:MinHeapFreeRatio=15 -XX:-UseGCOverheadLimit" fi fiReplace the above code with the following declare -a users=(hdfs centos) if [ "$SERVICE" = "cli" ]; then for auser in ${users[@]}; do if [ "$auser" = "$USER" ]; then echo "User $USER is not authorized to use Hive" exit 1 fi done if [ -z "$DEBUG" ]; then export HADOOP_OPTS="$HADOOP_OPTS -XX:New

You may be in a bind if you need to install HDP on Azure with CentOS 6 or RHEL 6 and certain services (not everything). By following these steps below, you will be able to use ambari-server to install HDP on any of the supported Hortonworks/Azure VMs.1) Configure your VMs - use the same VNet for all VMsRun the next steps as root or sudo the commands:2) Update /etc/hosts on all your machines: vi /etc/hosts 172.1.1.0 master1.jd32j3j3kjdppojdf3349dsfeow0.dx.internal.cloudapp.net172.1.1.1 master2.jd32j3j3kjdppojdf3349dsfeow0.dx.internal.cloudapp.net172.1.1.2 master3.jd32j3j3kjdppojdf3349dsfeow0.dx.internal.cloudapp.net 172.1.1.3 worker1.jd32j3j3kjdppojdf3349dsfeow0.dx.internal.cloudapp.net172.1.1.4 worker2.jd32j3j3kjdppojdf3349dsfeow0.dx.internal.cloudapp.net172.1.1.5 worker3.jd32j3j3kjdppojdf3349dsfeow0.dx.internal.cloudapp.net* use the FQDN (find the fqdn by typing hostname -f). The ip address are internal and can be found on eth0 by typing ifconfig3) Edit /etc/sudoers.d/waagent so th

Highlights of integrating Apache NiFi with Apache Ambari/Ranger Article credits: @Ali Bajwa, @Bryan Bende, @jluniya, @Yolanda M. Davis, @brosander With the recently announced HDF 2.0, users are able to deploy an HDF cluster comprised of Apache NiFi, Apache Storm, Apache Kafka and other components. The mechanics of setting this up using Apache Ambari’s Install Wizard are outlined in the official documentation here and sample steps to automate the setup via Ambari blueprints are provided here. The goal of this article is to highlight some features NiFi administrators can leverage when using Ambari managed HDF 2.0 clusters vs using NiFi standalone The article is divided into sections on how the integration helps administrators with HDF: Deployment Configuration Monitoring Security Ease of Deployment Users have the choice of deploying NiFi through Ambari install wizard or operationalize via blueprints automation (For detailed steps, see links provided on above

SYNOPSISThis article will demonstrate using Spark (PySpark) with the S3A filesystem client to access data in S3.The code can also be found on GitHub @ https://github.com/mathewbk/spark_s3aThe source data in the S3 bucket is Omniture clickstream data (weblogs). The goal is to write PySpark code against the S3 data to RANK geographic locations by page view traffic - which areas generate the most traffic by page view counts.The S3A filesystem client (s3a://) is a replacement for the S3 Native (s3n://):It uses Amazon’s libraries to interact with S3Supports larger files Higher performanceSupports IAM role-based authenticationProduction stable since Hadoop 2.7 (per Apache website) The code below is a working example tested using HDP 2.4.0.0-169 (Hadoop version 2.7.1.2.4.0.0-169)Note that I am using the HiveContext and not the SQLContext. I am NOT querying or accessing any Hive tables, but I am using the RANK() function to perform a ranking. If you want to use the RANK() function on a Data

We just updated Hortonworks Data Cloud for AWS to Technical Preview. This new TP (version #1.6) includes few bug fixes and a new Command Line Interface to help with scripting + automating cluster creation.Checkout the Release Notes for more details:http://hortonworks.github.io/hdp-aws/releasenotes/index.htmlGet Started and Launch Hortonworks Data Cloud in your AWS environment:http://hortonworks.github.io/hdp-aws/index.htmlCheck it out. You can provide feedback and ask questions here at the Hortonworks Community Connection:http://hortonworks.github.io/hdp-aws/feedback/index.html#how-to-create-a-questionHave fun!

We'd like to announce the immediate availability of Ambari 2.4.1. Please refer to the product documentation for more details about this maintenance release. Ambari 2.4.1 Product Documentation http://docs.hortonworks.com/HDPDocuments/Ambari/Ambari-2.4.1.0/index.html Ambari 2.4.1 Upgrade Guide http://docs.hortonworks.com/HDPDocuments/Ambari-2.4.1.0/bk_ambari-upgrade/content/upgrading_ambari.html Ambari 2.4.1 Release Notes http://docs.hortonworks.com/HDPDocuments/Ambari-2.4.1.0/bk_ambari-release-notes/content/ch_relnotes-ambari-2.4.1.0.html

1 Challenges managing LAS filesSiloed datasetsWorking with disparate, complex datasets under a traditional analysis model limits innovation and does not allow for the speed required for unconventionalsLAS File VolumeA single well could have 10s or 100s of LAS files making it difficult to provide a consolidated view for analysisExtrapolating this volume out across 1000s of wells requires an automated approachManual QC processIdentifying out of range data is time consuming and challenging even for experienced geoscientists and petrophysicistsManagement and storage is expensiveWhat if cost could be reduced from $23/Gb to $.19/Gb; $55 GB could cost $1,200 or $10Delta is 1-2 orders of magnitudeDownload Sample Data SetThe wellbook concept is about a single view of an oil well and its history- something akin to a "Facebook Wall" for oil wells.This repo is built from data collected and made available by the North Dakota Industrial Commission.I used the wellindex.csv file to obtain a list of w

Hortonworks DataFlow 2.0 comes with the ability to configure TLS for Apache NiFi through Apache Ambari, this is implemented using the tls-toolkit in client/server mode.To demonstrate this functionality, lets set up a 3 node NiFi secured cluster through Ambari locally in Docker containers.First, install docker on your machine. Please note that the default docker-machine vm size is too small for this guide. You should have at least 8 gigs of ram, a few cpus, and 100 gigs of hard drive space allocated to the docker-machine you’re using.Build the Ambari stack: mkdir toolkit-demo-3/cd toolkit-demo-3/git clone https://github.com/brosander/dev-dockerfiles.gitdev-dockerfiles/ambari/server/centos6/buildStack.sh http://public-repo-1.hortonworks.com/ambari/centos6/2.x/updates/2.4.0.1/ambari.repoGenerate an ssh key for use talking to the gateway and Ambari: mkdir ambari-ssh-keysssh-keygen -t rsa -b 4096 -f ambari-ssh-keys/id_rsaRun the Ambari stack and install HDF mpack: wget -P mpack http://publi

Once you have Atlas up and running( see this for getting started. ) you will want to create your first tag and use it to tag data to begin exploring what Atlas can do. Lets start by creating a tag that will be called "PII" which we will later use it to tag Personally Identifiable data. Log into Atlas and select the + icon on the homepage. Enter "PII" for the tag name and click create. That's it! Now we have a tag to use. Now click on the search tab select "dsl" and from the drop down select "hive_table" hit enter and select the "customer" table.You should see the summary details for the customer table in hive. Select the schema tab. Here you see all columns available in the customer data table. Let's mark the "account_num" field as "PII". Next to "account_num" click the + icon and select "PII" from the drop down.Now the column has been tagged and both searchable from Atlas as well as configured to be administered through Hive for auditing and permissions. Click the "Tags" tab in Atlas

Env: HDP-2.3.4.0-3485Java 8Attached code contain:—pom.xml to manage all the dependencies— HiveClientSecure.java - oozie java action to be configured into workflow.xml— jaas.conf — oozie uses jaas configuration for kerberos login— log4j.properties - to capture logsjaas.conf: modify principal name and key tab location accordingly and place it to on each node on the cluster.I have placed it on /tmp/jaas/jaas.conf for testing purpose.Client {com.sun.security.auth.module.Krb5LoginModule requireduseKeyTab=trueuseTicketCache=trueprincipal="ambari-qa-hbase234@HWXBLR.COM"keyTab="/etc/security/keytabs/smokeuser.headless.keytab"debug="true"doNotPrompt=true;};workflow.xml:<workflow-app xmlns="uri:oozie:workflow:0.2" name="java-main-wf"> <start to="java-node"/> <a<action name="java-node"> <java> <job-tracker>${jobTracker}</job-tracker> <name-node>${nameNode}</name-node> <configuration>

You have a need of debugging, testing and operating a Hadoop cluster, especially when you run dangerous dfsadmin commands, try customized packages with changes of Hadoop/Spark source code, trying aggressive configuration values. You have a laptop and you have a production Hadoop cluster. You don't dare to operate the production cluster blindly, which is appreciated by your manager. You want to try something on a hadoop cluster and even you breaks it, no one blames you. You have several choices (perhaps you're using one of them now): psudo-distributed Hadoop cluster on a single machine, which is nontrivial to run HA, to use per-node configurations, to pause and launch multiple nodes, or to test HDFS balancer/mover etc. setting up a real cluster, which is complex and heavy to use, and in the first place you can afford a real cluster. building Ambari cluster using vbox/vmware virtual machines, nice try. But if you run 5 nodes cluster, you'll see your CPU is overloaded and memory is

Logical Disk EncryptionThe approach is to avoid using HDFS encryption and use Disk LUKS encryption for data at rest encryption requirement specially when using public cloud IAAS.To build manually encrypted volumes or drives ,use following steps on a d2*8xLarge instance flavor.lsblkcryptsetup --verbose --verify-passphrase luksFormat /dev/xvdbcryptsetup --verbose --verify-passphrase luksFormat /dev/xvdccryptsetup --verbose --verify-passphrase luksFormat /dev/xvddcryptsetup luksOpen /dev/xvdb vol1cryptsetup luksOpen /dev/xvdc vol2cryptsetup luksOpen /dev/xvdd vol3dd if=/dev/urandom of=/root/keyfile1 bs=1024 count=4chmod 0400 /root/keyfile1cryptsetup luksAddKey /dev/xvdb /root/keyfile1cryptsetup luksAddKey /dev/xvdc /root/keyfile1cryptsetup luksAddKey /dev/xvdd /root/keyfile1mkfs.ext4 /dev/mapper/vol1mkfs.ext4 /dev/mapper/vol2mkfs.ext4 /dev/mapper/vol3echo "/dev/mapper/vol1 /data/vol1 ext4 defaults,nofail,nodev 0 2" >> /etc/fstabecho "/dev/mapper/vol2 /data/vol2 ext4 defaults,nofail,