Community Articles

Objective Given the limited resources available in a virtualized sandbox, you may choose to turn specific services on or off. You may choose to enable or disable security, such as Kerberos. Depending on your scenario, you may have a need to switch between these configurations frequently. For reproducible demos, you likely do not want to make these changes between one demo and the next. If you are like me, you may want to have different copies of HDP sandboxes to cover different demo scenarios. With VirtualBox or VMWare sandboxes, you can easily import or clone a sandbox to have multiple, distinct copies. Each copy is unique with no sharing of configuration or data. However, this approach is not quite as intuitive when using the Docker sandbox. If you tried to create multiple containers on a Docker image thinking they would be separate copies, you likely have found they are not completely separate! This tutorial will guide you through the process of using a single sandbox image, with

Do the UpgradePreparation ● For each target cluster find out as much details as possible by reviewing the following: ● Hardware configuration, operating systems and network topology ● Current deployment, cluster topology, configuration of each component ● Security configuration ● User access and management ● Current data ingestion process (if applicable) ● Current running applications and clients connecting into the cluster ● Find the applicable Upgrade Guide from docs.hortonworks.com ● Select validation applications ● Prepare an Upgrade Log to keep track of any upgrade issues and their workarounds ● [Optional but recommended] Prepare one or more Lab (virtual) clusters and install the current HDP stack and Ambari. Use these clusters for mock upgrades and rollbacks to troubleshoot any upgrade issues.Upgrading Clusters ● To upgrade a single cluster use the Upgrade Procedure given below. ● [Optional but recommended] Mock (lab) cluster upgrade: Attempt an upgrade on a Lab

Plan and Assess This is a purely planning step. The expected deliverable is an Upgrade Plan. Gather all details about existing environment to plan for the upgrade path and associated upgrade tasks. 1) Determine Upgrade Path Based on the current and target version of the HDP stack, and whether Ambari is used or not, select the supported upgrade guide from Hortonworks documentation site. Identify key requirement if Namenode HA or other HA needs to be disabled or Security needs to be disabled. Current version: ● HDP Stack version ● Ambari version (if Ambari is used) ● OS Version Target version: ● HDP Stack version ● Ambari version (if Ambari is used) Below are some useful links HDP Stacks Managed by Different Ambari Versions: https://docs.hortonworks.com/HDPDocuments/Ambari-2.4.0.0/bk_ambari-installation/content/determine_stack_compatibility.html Upgrading to Ambari 2.4: https://docs.hortonworks.com/HDPDocuments/Ambari-2.4.0.1/bk_ambari-upgrade/c

With NiFi 1.00 I ingested a lot of image data from drones, mostly to get metadata like geo. I also ingested a resized version of the image in case I wanted to use it. I found a use for it.I am pulling this out very simply with Spring for a simple HTML page. So I wrote a quick Java program to pull out fields I stored in Phoenix (from the metadata) and I wanted to display the image. I could have streamed it out of HDFS using HDFS libraries to read the file and then stream it. sql = "select datekey, fileName, gPSAltitude, gPSLatitude, gPSLongitude, orientation,geolat,geolong,inception from dronedata1 order by datekey asc";out.append(STATIC_HEADER);PreparedStatement ps = connection.prepareStatement(sql);ResultSet res = ps.executeQuery();while (res.next()) { try { out.append("<br><br>\n<table width=100%><tr><td valign=top><img src=\""); out.append("http://tspanndev10.field.hortonworks.com:50070/webhdfs/v1/drone/"). append(res.getSt

In the previous articles, we showed how to deploy an HDF 2.x/3.0 cluster, enable SSL for Nifi and setup the Ranger Nifi plugin. Here we will build on the same cluster and show how to enable kerberos using Active Directory.Summary To achieve this, the high level steps we will follow are: Setup certificate trust for HDF nodesRun Ambari security wizardCreate Ranger policy for nifiadmin userDelete certificateLogin to Nifi using AD principal credentialsPre-requisites You have correctly setup AD as described here Active Directory setup with domain: CLOUD.HORTONWORKS.COMAD already preconfigured with LDAPSCertificate (.crt) used to enable LDAPS is availableOU created where HDF principals will be createdhadoop user has permission to write principals to above OUnifiadmin user created in AD (optionally synced over to Ranger) Test to ensure you can access AD over LDAPS using hadoopadmin user succeeds:ldapsearch -H ldaps://sme-security-ad03.cloud.hortonworks.com:636 -D hadoopadmin@cloud.hortonworks

1. Acquire an EDI File (GetFile, GetFTP, GetHTTP, GetSFTP, Fetch...) 2. Install open source nifi-edireader on NIFI 1.0.0 Download https://github.com/BerryWorksSoftware/edireader Maven Install Berry Works EDIReader Download https://github.com/mrcsparker/nifi-edireader-bundle Maven packge nifi-edireader (must be Maven 3.3 or newer - may have to download and install separately from standard linux package) cp nifi-edireader-nar/target/nifi-edireader-nar-0.0.1.nar to your NIFI/lib Restart NiFi Service 3. Add EdiXML Processor and connect from EDI File input 4. Add extra processing, conversion or routing (TransformXML with XSLT or EvaluateXPATH) to convert to JSON 5. Land to HDFS (PutHDFS) 6. Used the Web Form linked below to generate a test EDI file. ISA*00* *00* *ZZ*SENDER ID *ZZ*RECEIVER ID *010101*0101*U*00401*000000001*0*T*!GS*IN*SENDER ID*APP RECEIVER*01010101*01010101*1*X*004010ST*810*0001BIG*20021208*00001**A999N1*ST*Timothy Spann*9*122334455N3*115 xxx av

We recently had some issues with Ambari and Kafka alerts. It all started when in HDP 2.3.4, every time we changed the kafka listener port from 6667 to any other port, Ambari would complain and give us an error saying that it couldn’t reach port 6667 even though the broker service is actually running in another port. Here’s the exact error: “Connection failed: [Errno 111] Connection refused to sandbox.hortonworks.com:6667” This can be quite annoying especially if you have multiple brokers and they’re all reporting CRITICAL and you just can’t seem to get rid of it. To cut the long story short, here are the steps we did to get rid of the problem. In the section below, we’ll jump into some troubleshooting tips: 1. Get the ID of the kafka broker > curl -u admin:admin -H 'X-Requested-By: ambari' -X GET "http://localhost:8080/api/v1/clusters/Sandbox/alert_definitions" 2. Get the definitions and save it locally > curl -u admin:admin -H 'X-Requested-By: ambari' -X GET "

If you've received the error exitCode=7 after enabling Kerberos, you are hitting this Jira bug. Notice the bug outlines the issue but does not outline a solution. The good news is the solution is simple, as I'll document below.Problem:If you've enabled Kerberos through Ambari, you'll get through around 90-95% of the last step "Start and Test Services" and then receive the error:16/09/26 23:42:49 INFO mapreduce.Job: Running job: job_1474928865338_002216/09/26 23:42:55 INFO mapreduce.Job: Job job_1474928865338_0022 running in uber mode : false16/09/26 23:42:55 INFO mapreduce.Job: map 0% reduce 0%16/09/26 23:42:55 INFO mapreduce.Job: Job job_1474928865338_0022 failed with state FAILED due to: Application application_1474928865338_0022 failed 2 times due to AM Container for appattempt_1474928865338_0022_000002 exited with exitCode: 7For more detailed output, check application tracking page:http://master2.fqdn.com:8088/cluster/app/application_1474928865338_0022Then, click on links to lo

Introduction This article will address some of the most least documented issues you may find while importing data using oozie-sqoop-hcatalog and respective solutions. Heart beat, follow by read-timed-out with thrift server SYMPTOM While trying to execute an oozie workflow, on the stdout of the oozie sqoop job you can see: hive.metastore (HiveMetaStoreClient.java:open(382)) - Trying to connect to metastore with URI thrift://host:port Heart beat Heart beat Heart beat ERROR [main] hive.log (MetaStoreUtils.java:logAndThrowMetaException(1221)) - Got exception: org.apache.thrift.transport.TTransportException java.net.SocketTimeoutException: Read timed outorg.apache.thrift.transport.TTransportException: java.net.SocketTimeoutException: Read timed out ROOT CAUSE Missing hive-site.xml for the sqoop action caused the errorRESOLUTION Save the hive-site.xml on the hdfs and reference from the sqoop action file tag: <file>${nameNode}/user/${wf:user()}/${appRoot}/env/${cluster}/hiv

Most user access control is accomplished on the group level. One of the example mentioned in Ranger release is to control each doctor only able to view the patients of his/her own. However most example provided up-to-date are mostly static filter condition that will not change for individual user. In this tutorial, we would demonstrate how to create a row level filter that applies to a group but takes different effect on the user level by leveraging UDF. The use case is as following: a patient table contains patient information as well as the id of the doctor treating the patient (this is a simplified example, in really world the relation may be a many-to-many relation, but it could a handled by the same approach). We also have a doctor reference table where the doctor login credential could be found. We will use the login credential to identify the doctor and only show the patient that associated with the doctor.To setup the environment, first download the HDP2.5 Sandbox form Hortonwo

I ran the same flow myself and examined the AVRO file in HDFS using AVRO Cli.Even though I didn't specify SNAPPY compression, it was there in the file.java -jar avro-tools-1.8.0.jar getmeta 23568764174290.avrolog4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.lib.MutableMetricsFactory).log4j:WARN Please initialize the log4j system properly.log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.avro.codec snappyavro.schema {"type":"record","name":"people","doc":"Schema generated by Kite","fields":[{"name":"id","type":"long","doc":"Type inferred from '2'"},{"name":"first_name","type":"string","doc":"Type inferred from 'Gregory'"},{"name":"last_name","type":"string","doc":"Type inferred from 'Vasquez'"},{"name":"email","type":"string","doc":"Type inferred from 'gvasquez1@pcworld.com'"},{"name":"gender","type":"string","doc":"Type inferred from 'Male'"},{"name":"ip_address","type":"string","doc":"Type inferred from '32.8.254.252'"},{"n

I had a few hours in the morning before the conference schedule kicked in, so I decided to write a little HDF 2.0 flow to grab all the tweets about Strata Hadoop conference. First up, I used GetTwitter to read tweets and filtered on these terms:strata,stratahadoop,strataconf,NIFI,FutureOfData,ApacheNiFi,Hortonworks,Hadoop,ApacheHive,HBase,ApacheSpark,ApacheTez,MachineLearning,ApachePhoenix,ApacheCalcite,ApacheStorm,ApacheAtlas,ApacheKnox,Apache Ranger, HDFS, Apache Pig, Accumulo, Apache Flume, Sqoop, Apache FalconInput:InvokeHttp: I used this to download the first image URL from tweets.GetTwitter: This is our primary source of data and the most important. You must have a twitter account, a twitter developer account and create a twitter application. Then you can access the keywords and hashtags above. So far I've ingested 14,211 tweets into Phoenix. This included many times I've shut it down for testing and moving things around. I've had this run live as I've added pieces. I d

In many cases, it's sufficient or even desirable to use self-signed, or company-internal signed certificates to provide TLS encryption, for example between your users' browsers and Ambari and also between Ambari and an LDAP service (LDAPS). If you're not careful with your management of the certificates for each component you're working with, it's possible that you can end up with a broken Ambari views installation. In such cases, when you try to load Ambari views, you might instead see a stack trace which on the surface may appear cryptic, with an error code such as "RA040". If you do see javax.net.ssl.SSLHandshakeException in the trace somewhere, it's likely that your problem lies with the custom trust store in Ambari. Consider this scenario: 1. You set up HTTPS for Ambari using a private CA - e.g. your corporate IT team to sign your certificates so that IT-managed browsers which already have imported the CA certificates do not generate warnings. You used "ambari-server setup-s

There are many situations in which running benchmarks for certain workloads on Apache Phoenix can provide meaningful insight into an installation. Commonly, such a benchmark is very useful to understand the baseline characteristics of a new installation of Apache Phoenix. Alternatively, the ability to re-run the same benchmark after changing a configuration property change is extremely useful in understanding the effect of that change. Many approaches exist to test systems that have a SQL interface, many of them focused on a specific type of workload. The following approaches aim to describe a few benchmarks which users can run on their own and tweak to a workload which makes sense for their cluster.Apache JMeter Automation Apache JMeter is a tool which was initially designed to test web applications; however, it also has the ability to execute SQL queries against some JDBC Driver. JMeter allows us to define queries to execute against a Phoenix table. An instance of JMeter can

One key feature in Apache Atlas is the ability to track data lineage in your Data Lake visually. This allows you to very quickly understand the lifecycle of your data and answer questions about where the data originated from and how it relates to other data in the Data Lake. To Illustrate this we will use our own twitter data to perform sentiment analytics on our tweets in Hive and see how this is reflected in Apache Atlas. By now you should have a working sandbox or HDP environment up and running with Atlas enabled. If not please take a look at the following tutorial to help get you started: Getting Started with Atlas in HDP 2.5First we need to gather the data sets we will use in this tutorial. Log into twitterClick on your account settings at the top rightSelect "Your Twitter Data" from the list on the left side of your screen. Now enter your password and in the "Other Data" section at the bottom select "Twitter Archive" This might take a little time but you will get a link to downlo

The Security Guide was extensively updated to reflect new Ranger and Knox features for HDP 2.5. This new content includes: Ranger-Atlas integration using tag-based services and policies.Tags and policy evaluation.Atlas Ranger Authorization plugin.Ranger-Hive row-filtering and column-masking.Migration script to convert from Audit-to-DB to Audit-to-Solr.Ranger Reports page improvements.Knox SSO.Ranger Usersync enhancements.Ranger High Availability (added to Apache Hadoop High Availability guide).Updates to reflect the new Ambari Infra Solr instance.The Data Governance guide was completely rewritten for HDP 2.5 to reflect new Apache Atlas capabilities, along with a new Atlas UI. This new content includes: Atlas-Ranger integration and Tag synch. Installation and Configuration.Configuring Atlas Security.Configuring Atlas High Availability.Searching and viewing Atlas Assets.Working with Atlas tags.Enabling and Managing the Atlas Business Taxonomy (Technical Preview).Updated REST API resou

HDP 2.3 Index Life Cycle(Client events are in orange)-Columns Names in SYSTEM.CATALOG table:-Query:-“select TABLE_NAME,DATA_TABLE_NAME,INDEX_TYPE,INDEX_STATE,INDEX_DISABLE_TIMESTAMP from system.catalog where INDEX_TYPE is not null;”TABLE_NAME:-Name of the indexDATA_TABLE_NAME:-Name of the parent data table of indexINDEX_TYPE:-GLOBAL(1)LOCAL(2)INDEX_STATE:-Index States (Abbreviation stored in SYSTEM.CATALOG, states in bold are available to client too):-BUILDING("b") USABLE("e")UNUSABLE("d")ACTIVE("a")INACTIVE("i")DISABLE("x")REBUILD("r")(Below paragraph is from phoenix site)DISABLE will cause the no further index maintenance to be performed on the index and it will no longer be considered for use in queries.REBUILD will completely rebuild the index and upon completion will enable the index to be used in queries again. BUILDING will partially rebuild the index from the last disabled timestamp and upon completion.INACTIVE/UNUSABLE will cause the index to no longer be considere

In the previous article, we showed how to enable SSL and set up identity mappings for Apache Nifi on the previously installed HDF 2.x or 3.0 cluster. Here, we will build on the same cluster and show how to install Apache Ranger and setup the Ranger Nifi plugin. For simplicity, we will assume this is a demo environment where there is no requirement to enable SSL for Ranger. If instead you would like to use secured Ranger with NiFi, follow steps here SummaryAt a high level, Apache Ranger provides a centralized platform to define, administer and manage security policies consistently across Hadoop components. In the case of HDF, it enables the administrator to create/manage authorization policies for Kafka, Storm and Nifi from the same web interface (or REST APIs).To achieve this, the high level steps we will follow are: Ranger install prerequisitesRanger installUpdate Nifi Ranger repoTest Ranger pluginCreate Ranger uses and policiesTest Nifi access as nifiadmin userThe official documentat

hcc-58591.zipHive RegexSerDe can be used to extract columns from the input file using regular expressions. It's used only to deserialize data, while data serialization is not supported (and obviously not needed). The initial motivation to create such a SerDe was to process Apache web logs. There are two classes available: org.apache.hadoop.hive.contrib.serde2.RegexSerDe, introduced in Hive-0.4 by HIVE-662, and org.apache.hadoop.hive.serde2.RegexSerDe, a built-in class introduced in Hive-0.10 by HIVE-1719The former is kept to facilitate easier migration for legacy apps, while the letter is recommended for the new apps. The SerDe works by matching columns in the table definition with regex groups defined and captured by the regular expression. A regex group is defined by parenthesis "(...)" inside the regex. Note that this is one of common mistakes by beginners who spend time creating great regular expressions but displace or fail to mark regex groups. The new, built-in version suppo

Our stream analytics documentation received substantial updates for HDP 2.5.0.The Apache Storm Component Guide was reorganized to focus on administration and development workflow: installing and configuring Storm, developing Storm applications, moving data into and out of Storm, and managing topologies. New content includes the following topics:Installing Storm using Ambari.Configuring Storm for a production environment.Implementing windowing computations on data streams.Implementing state management in core Storm.Configuring and using the HDFS spout to ingest data from HDFS.Monitoring and debugging Apache Storm topologies through the use of dynamic log levels, topology event logging, distributed log search, and dynamic worker profiling.The Storm application development chapter differentiates more clearly between core Storm and Trident, and the Kafka spout subsection describes performance settings and tradeoffs.The Storm Ambari view is documented in the Apache Ambari Views Guide, and t

The new Apache Zeppelin Component Guide has instructions for installing, configuring, and using the web-based Zeppelin notebook for interactive data exploration and visualization.The Apache Spark Component Guide was reorganized to focus on administration and development workflows: installing and configuring Spark, developing Spark applications, using SparkR, and tuning Spark. For HDP 2.5, the book includes the following new topics:Installing the Spark 2.0 technical preview alongside Spark 1.6.2, using Ambari.Configuring Spark for wire encryption.The Apache Solr Search Installation Guide now contains instructions for installing HDP Search using Ambari.

This article "Perform Data Analysis using SAP Vora on SAP Hana data - Part 4" is continuation of "Load Demo data in SAP Vora Using Eclipse HANA Modelling tools - Part 3"Log back in to SAP Cloud Appliance Library - the free service to manage your SAP solutions in the public cloud. You should have HANA and Vora instances up and running: Open Apache Zeppelin web UI click on Connect in your SAP HANA Vora instance in CAL, and then pick Open a link for Application: Zeppelin In Zeppelin click Create new note, name the notebook as you like ex "interact with hana data"In the first cell lets define our imports and HANA connection information import sqlContext.implicits._import scala.collection.mutable.Mapval HANA_HOSTNAME = "xxx.xxx.xx.xxx"val HANA_INSTANCE = "00"val HANA_SCHEMA = "CODEJAMMER"val HANA_USERNAME = "CODEJAMMER"val HANA_PASSWORD = "CodeJam2016"4. In the next cell we will use sqlContext that was automatically created by Zeppelin to Connect to HANA using "com.sap.spark.hana" sqlConte

This article "Load Demo data in SAP Vora Using Eclipse HANA Modelling tools - Part 3" is continuation of "Configure SAP Vora HDP Ambari - Part 2"You will need to download Eclipse Neon - Eclipse IDE for Java Developers to connect to the SAP HANA that we setup in Part 1 . After you setup eclipse we will need to configure Eclipse to install HANA Modelling tools that will allow us to connect to SAP HANA and execute sql scripts to setup demo data that we will use from SAP Vora.Eclipse Setup Procedure Open the Eclipse IDE.In the main menu, choose HelpInstall New Software .Depending on the Eclipse version you have installed, enter one of the following URLs in the Work with field: For Eclipse Neon (4.6), add URL: https://tools.hana.ondemand.com/neonSelect SAP HANA Tools (the whole feature group).noteIn case you need to develop with SAPUI5, install also SAP HANA Cloud Platform ToolsUI development toolkit for HTML5 (Developer Edition) .Choose Next.On the next wizard page, you get an overview

This article "Configure SAP Vora HDP Ambari - Part 2" is continuation of "Getting started with SAP Hana and Vora with HDP using Apache Zeppelin for Data Analysis - Part 1 Intro"Log back in to SAP Cloud Appliance Library - the free service to manage your SAP solutions in the public cloud. You should have HANA and Vora instances up and running:Open Apache Ambari web UI click on Connect in your SAP HANA Vora instance in CAL, and then pick Open a link for Application: Ambari. The port of Ambari web UI has been preconfigured for you in the SAP HANA Vora, developer edition, in CAL. As well its port has been opened as one of the default Access Points. As you might remember it translates into the appropriate inbound rule in the corresponding AWS’s security group.Log into Ambari web UI using the user admin and the master password you defined during process of the creation of the instance in CAL.You can see that (1) all services, including SAP HANA Vora components, are running, that (2) there a

SAP HANA Vora is an in-memory processing engine that runs on a Hadoop cluster and is tightly integrated with Spark. It is designed for handling big data. SAP HANA Vora makes available OLAP-style capabilities on Hadoop, provides deeper integration with SAP HANA, enabling high-performance enterprise analytics, and delivers contextual insights by combining corporate data in SAP HANA with big data stored in Hadoop systems. In this multi-part guide , I will show you how to spin up an SAP HANA instance in AWS and a Vora + HDP installation on 2nd node. We will utilize Apache Zeppelin to interact with SAP HANA using a Vora interpreter . In this scenario you will be able to join data from other various data sources like HDFS and RDBMs to join to Hana data. This is a Federated Query approach to multiple data sources. "Federation" tier to act as a single point of access to data from multiple sources. For details on the concepts of Data Federation see "Virtual Integration of Hadoop with Exter