Resources

In a Kerberized HDP cluster, I use FreeIPA as LDAP and run HDP 2.3.2 including Storm, Kafka and Ranger. After creating a storm topology as a normal user, I keep getting the following runtime error: 2015-12-30 18:31:53.274 o.a.c.ConnectionState [ERROR] Authentication failed ... 2015-12-30 18:31:53.286 o.a.z.ClientCnxn [WARN] SASL configuration failed: javax.security.auth.login.LoginException: No password provided Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it. ... 2015-12-30 18:31:53.328 b.s.util [ERROR] Async loop died! java.lang.RuntimeException: java.lang.RuntimeException: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode ... This problem comes from Kafka and Ranger. As a normal failure recovery mechanism, Kafka will keep creating the topic if it cannot find the designated topic mentioned in the Kafka producer. In a kerberized environment, this CREATE request will be sent to Ranger for approval. How

This is part 2 of the fault tolerance tutorial. Part 1 can be found here: https://community.hortonworks.com/articles/8607/ho...--------------------------------------------------------------------------------------------------------------------------------------Multi-Cluster Fault Tolerance (One cluster in each zone)(All Clusters Active all the time) Multiple clusters can be setup across multiple zones and all are active at the same time. This option requires some external process to keep flows in sync across all clusters. Lets take a look at what this setup might look like if we had 3 zones all running a NiFi cluster. We can see right away that there is a lot more involved with the S2S configuration between the sources systems and these multiple zones. Towards the beginning of this guide we showed what a typical NiFi S2S setup might look like on a source NiFi. With this configuration the sources would need to be setup differently. Rather than delivering data to the Nodes of a sin

Using NiFi to provide fault tolerance across data centers (The Multi-Zone approach) The purpose of this document is to show how to take a NiFi cluster and create some additional fault tolerance using multiple zones/regions. You can think of these zones/regions as physically separated data centers. (They may not even be geographically co-located) The document will cover a few different configurations including one large cluster that spans across multiple zones, two independent clusters (each located in a different zone) with only one zone active at a time, and finally multiple clusters (each located in a different zone) all active at the same time. There are pros and cons to each configuration that will be covered. Depending on your deployment model, one configuration may be better suited then another.A NiFi cluster consists of a single NiFi Cluster Manager (NCM) and 1 to N number of NiFi Nodes. NiFi does not set a limit on the number of Nodes that can exist in a cluster. While a NiF

I’m going to show you a neat way to work with CSV files and Apache Hive. Usually, you’d have to do some preparatory work on CSV data before you can consume it with Hive but I’d like to show you a built-in SerDe (Serializer/Deseriazlier) for Hive that will make it a lot more convenient to work with CSV. This work was merged in Hive 0.14 and there’s no additional steps necessary to work with CSV from Hive.Suppose you have a CSV file with the following entries id first_name last_name email gender ip_address 1 James Coleman jcoleman0@cam.ac.uk Male 136.90.241.52 2 Lillian Lawrence llawrence1@statcounter.com Female 101.177.15.130 3 Theresa Hall thall2@sohu.com Female 114.123.153.64 4 Samuel Tucker stucker3@sun.com Male 89.60.227.31 5 Emily Dixon edixon4@surveymonkey.com Female 119.92.21.19to consume it from within Hive, you’ll need to upload it to hdfshdfs dfs -put sample.csv /tmp/serdes/ now all it takes is to create a table schema on top of the filedrop table if exists sample;create

In Ambari the Hiveserver2 is shown as "green", but there is an alert indicating problem with the Hiverserver2 process.This is due to a known bug in HDP2.2.3/2.2.4, that at Hive startup time no Kerberos ticket will be grabbed. To get rid of that alert, login to the Hiveserver node, become user hive and execute a "kinit -kt /etc/security/keytabs/hive.service.keytab hive/<hiveserver>@<REALM>" (if you keytab is in that default directory) to ensure the hive user got a valid Kerberos ticket.Afterwards you can restart Hiveserver in Ambari and the alert will disappear (Thanks to @dprichici for highlighting this)Regards, Gerd

Part 2 Linkedin Post Extending Blog 2 to look for starwars tweet Searching for Yoda, Love & Hate Let's see Tweets/Data with word YODA related to Starwars tweets Keyword : LOVE in STARWARS Source Giphy Word Hate Happy Hadooping!!!

Some basic charts are already included in Zeppelin. Visualizations are not limited to SparkSQL's query, any output from any language backend can be recognized and visualized. This sufficient for a Data Analyst, but if you were a Data Scientist, the built in tools within Zeppelin just don't cut. If you are a Python programmer and have been working with data in IPython, you must definitely be well versed in matplotlib library. You can install matplotlib in your python shell as long as Zeppelin is still using the same version of python. Once you do that the Pyspark interpreter will be able to import matplotlib libraries and you will be able to create graphs and charts in the Zeppelin interface itself. Make sure to add the following code at the end of your script and modify it as needed:def show(graph): img = StringIO.StringIO() graph.savefig(img, format='svg') img.seek(0) print "%html <div style='width:1000px'>" + img.buf + "</div>"show(plt)

Follow steps below to run this sample spark-shell code to index a Spark Data Frame into a Solr Cloud:1- Install java 8 in your sandbox 2.3.2yum install all java-1.8.0-openjdk*2- Define java 8 as your default java, using alternativesalternatives --config java##choose /usr/lib/jvm/jre-1.8.0-openjdk.x86_64/bin/java3- Logoff / Logon againexitssh root@sandbox.hortonworks.com4- Change java version in spark, in spark-env.shvi /usr/hdp/current/spark-client/conf/spark-env.sh##change JAVA_HOME like below:export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.65-0.b17.el6_7.x86_645- Download and build spark-solrgit clone https://github.com/LucidWorks/spark-solr.gitcd spark-solrmvn clean install -DskipTests 6- Create a sample text file and write into hdfsecho "1,guilherme" > your_text_fileecho "2,isabela" >> your_text_filehadoop fs -put your_text_file /tmp/7- Start sandbox solr cloudcd./start_solr.sh8- Create a solr collection/opt/lucidworks-hdpsearch/solr/bin/solr create -c testsparksolr

Original postA web-based notebook that enables interactive data analytics. You can make beautiful data-driven, interactive and collaborative documents with SQL, Scala and more.In few words " It's really cool tool to interact with Data"HDFS, Hive, Spark, Kylin, Flink This is from HDP latest SandboxContinue to Blog 3 on NiFiLet's analyze Starwars dataHive DemoTable definition and Top 10 users based on tweet countTop 10 users who used the word "love" in #starwarsWord hate used in #starwarsWord yoda used in #starwarsYou can see the Tweet sent by my id in Zeppelin output. Spark I used this for the sentiment analysis. Replace %hive with %sql (Assuming that you have setup the Zeppelin correctly)LinksZeppelinHortonworks and ZeppelinHappy Hadooping!!!

If you go to port 4200 on your HDP sandbox http://sandbox.hortonworks.com:4200/

OverviewThe objective of this article is to introduce the ESRI Spatial Framework for Hadoop and demonstrate how to use it with HDP 2.3.2. The following is a high-level view of how we will accomplish this: set up the ESRI Spatial Framwork for Hadoop on the HDP 2.3.2 sandboxETL data from the Open Baltimore Data PortalExecute simple geospatial queries to correlate crime data in Baltimore with spatial neighborhood dataPrior to continuing the tutorial, please ensure you have downloaded and started the HDP Sandbox.ESRI Spatial Framework for HadoopThe ESRI Spatial Framework for Hadoop is a collection of Hive UDFs that allow users to perform complex spatial analysis directly in Hive. The framework has built-in support for representing geometric shapes (point, polygon, etc.) as well as functions that operate on these shapes. For example, users can perform a binary test for overlap between a pair of polygons or compute the geometry of the intersection. This framework provides a powerful metho

When running Solr in a clustered mode (SolrCloud), it has a runtime dependency on a ZooKeeper, where it stores configs, coordinates leader election, tracks replicas allocation, etc. All-in-all, there's a whole tree of ZK nodes created with sub-nodes. Deploying SolrCloud into a Hadoop cluster usually means re-using the centralized ZK quorum already maintained by HDP. Unfortunately, if not explicitly taken care of, SolrCloud will happily dump all its ZK content in ZK root, which really complicates things for an admin down the line.If you need to clean up your ZK first, take a look at this how-to.Solution is to put all SolrCloud ZK entries under its own ZK node (e.g. /solr). Here's how one does it:su - zookeepercd /usr/hdp/current/zookeeper-client/bin/ # point it a ZK quorum (or just a single ZK server is ok, e.g. localhost)./zkCli.sh -server lake02:2181,lake03:2181,lake04:2181 # in zk shell now# note the empty brackets are _required_create /solr [] # verify the zk node has been created,

Your Ambari throwing a weird error and you need to troubleshoot? No problem....just install this view and browse this forum from Ambari itself!1. Install the view: wget https://github.com/hortonworks-gallery/ambari-iframe-view/raw/master/samples/answerhub-view-1.0-SNAPSHOT.jar -P /var/lib/ambari-server/resources/views/2. Then restart Ambari:#on sandboxservice ambari restart#on non-sandboxservice ambari-server restart3. Now open the view:

Update: added a GitBook linkThe unofficial little black book of Kerberos, created and maintained by a HWX engineer, Steve Loughran. Lots of questions that you were afraid to ask. Many advanced customers found it a very useful guide, especially if one needs to develop solutions and code for a Kerberized cluster.I felt this guide needed much more exposure than it had so far. All credit goes to @stevel@hortonworks.comhttps://github.com/steveloughran/kerberos_and_hadoopClick on the Contents links..or..Enjoy as a GitBook, readable online, on mobile and exportable as e.g. a PDF:https://steveloughran.gitbooks.io/kerberos_and_hadoop/content/

When submitting issues in the Sandbox forums, make sure to provide the following information: Describe the problem: Provide stack traces, screen shots and error messages. For instance, "Receiving error 'permission denied' when trying to access file in the File Browser View"Steps to reproduce the issue: Here just provide steps of how you reproduce the behavior you are seeing. What have you done to analyze the issue: For instance, reinstalled Sandbox, cleared cache on my browser etc..Version: Get the Sandbox Version sandbox-version

HDP 2.2 and Isilon OneFS 7.2.0.4 are now officially certified by Hortonworks and EMC Isilon and ready for Hadoop deployment. Hortonworks and EMC Isilon have a close engineering relation that started in September of 2014, to ensure that Hortonworks Data Platform (HDP) is integrated with the Isilon OneFS filesystem. The HDP test suite employed to certify the Isilon OneFS filesystem has over 10,000 individual tests. The gives assurance to customers that both EMC and Hortonworks have done the due diligence to ensure that all Hadoop workloads work on this integrated platform, with security and operational ease.HDP with Isilon reference architectureIsilon OneFS has implemented the HDFS API as an over the wire protocol consistent with its multi-protocol support for NFS, SMB and others. This is different from implementations of Hadoop Compatible File Systems (HCFS) in that OneFS mimics the HDFS behavior for the subset of features that it supports. The deployment model for HDP with Isil

Environment Best practices dictate that, where possible, a Hadoop cluster should be maintained behind a firewall to minimize any potential security vulnerabilities that may arise from exposed ports and web interfaces. A common approach to enabling user access in this situation is to open up SSH into a set of gateway/edge nodes. This ensures that users must authenticate prior to accessing any pieces of the Hadoop ecosystem and implicitly encrypts all data sent between the client and the cluster. This is a common setup for vanilla cloud-based installations. The problem with this setup is that, by default, all access is limited to the CLI on the gateway machines. Users outside of the cluster firewall cannot access valuable features such as web UIs, and JDBC/ODBC connections. There are a few options to securely enable these capabilities:Enable Kerberos+SPNEGO and Knox. Then open up the appropriate ports in the firewall. Implement firewall rules to expose specific ports and ho

Connections among nodes in Hadoop cluster should not be restricted.Many ports used within the cluster by various components are dynamic and are not even known until install occurs.If you want to set firewall rules for external access to the cluster, and you want to know all the ports to restrict see the following: For HDP 2.3: http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.2/bk_HDP_Reference_Guide/content/reference_chap2.htmlFor Ambari 2.1.2: http://docs.hortonworks.com/HDPDocuments/Ambari-2.1.2.1/bk_ambari_reference_guide/content/ch_configuring_network_port_numbers.htmlFor HDP 2.2: http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.2.9/bk_HDP_Reference_Guide/content/reference_chap2.html For Ambari 2.1: http://docs.hortonworks.com/HDPDocuments/Ambari-2.1.1.0/bk_ambari_reference_guide/content/_default_network_port_numbers_-_ambari.htmlIf you employ Kerberos which is a must for truly secure clusters for authentication, kerberos already identifies users, services and machines.I fo

Sentiment Analysis of Live Twitter Stream Using Apache Spark ============================================================This application analyzes live tweets and predicts if they are positive, or negative. The application works by connecting to the Twitter stream, and applying a model built offline using Spark's machine learning library (Mllib) to classify the tweet's sentiment. Using the instructions on this page, you will be able to build the model on HDP Sandbox and then apply it to a live twitter stream.Prerequisites-------------* Download the HDP 2.3 Sandbox from [here](http://hortonworks.com/products/hortonworks-sandbox/#install)* Start the Sandbox, and add its IP address into your local machine's /etc/hosts file:```bash$ sudo echo "172.16.139.139 sandbox.hortonworks.com" >> /etc/hosts```* Log into the sandbox, and clone this repository: ```bash$ ssh root@sandbox.hortonworks.com$ cd$ git clone https://github.com/DhruvKumar/spark-twitter-sentiment```* Download the labeled

Some interesting talks for Hadoop Summit 2016 (EMEA). Please vote: the deadline is Dec 15 and you can vote for as many talks as you wantHadoop governance, security, deployment, operations1. Rolling & Express Upgrades of the Stack in Ambarihttps://hadoopsummit.uservoice.com/forums/332061-hadoop-governance-security-deployment-and-operat/suggestions/10848390-rolling-express-upgrades-of-the-stack-in-ambariData science0. Analyzing Hollywood with Spark and Hadoop:https://hadoopsummit.uservoice.com/forums/332055-data-science-applications-for-hadoop/suggestions/10846959-analyzing-hollywood-with-spark-and-hadoop1. Magellan (how to scale geospatial joins using Magellan):https://hadoopsummit.uservoice.com/forums/332079-the-future-of-apache-hadoop/suggestions/10848510-magellan-spark-as-a-geospatial-analytics-engine2. Online Learning (how to do machine learning online and fast while being accurate):https://hadoopsummit.uservoice.com/forums/332082-hadoop-and-the-internet-of-things/suggestions/1

Problem:When trying to distcp to AWS, this error is reported:2015-12-03 09:50:01,132 FATAL [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster java.util.ServiceConfigurationError: org.apache.hadoop.fs.FileSystem: Provider org.apache.hadoop.fs.s3a.S3AFileSystem could not be instantiated at java.util.ServiceLoader.fail(ServiceLoader.java:224) and also:Caused by: java.lang.ClassNotFoundException: com.amazonaws.event.ProgressListener Solution:In ambari, under yarn, configs please add: /usr/hdp/2.3.2.0-2950/hadoop/* and /usr/hdp/2.3.2.0-2950/hadoop/lib* to the yarn.application.classpath Then under mapred, configs, do the same for mapreduce.application.classpath restart all affected services and it should work.

Caused by: java.lang.OutOfMemoryError: Java heap spaceThis particular case is related to "Reducer tasks of hive job fails with Out Of Memory error during shuffle fetcher stage"Fix: Increase hive.tez.container.size if it is set too low.tez.runtime.shuffle.memory.limit.percent from default value 0.7 Changed to 0.4 Decrease tez.runtime.shuffle.fetch.buffer.percent from default 0.25 to .15 if needed. (Different values were tested between the range of 0.25 to 0.10)

The Hadoop Ecosystem Tablehttps://hadoopecosystemtable.github.io/

Use caseThere are 2 groups Analytics and DW. We want to split the cluster resources between these 2 groups. User - neeraj belongs to Analytics group.User - dwuser belongs to DW groupUser neeraj is not allowed to use Default and dwuser queue. Be default, all the jobs submitted by user neeraj must go to it's assigned queue. User dwuser is not allowed to use Default and Analytics queue. By default, all the jobs submitted by user dwuser must go to it's assigned queue. Environment HDP 2.3 (Hortonworks Data Platform) and Ambari 2.1This tutorial completely independent of Hadoop distribution. Yarn is must i,e Hadoop 2.x I will be using Capacity Scheduler view to configure queues. yarn.scheduler.capacity.maximum-am-resource-percent=0.2 yarn.scheduler.capacity.maximum-applications=10000 yarn.scheduler.capacity.node-locality-delay=40 yarn.scheduler.capacity.queue-mappings=u:neeraj:Analytics,u:dwuser:DW yarn.scheduler.capacity.queue-mappings-override.enable=true yarn.scheduler.capacity.root.acces

yum install expect*#!/usr/bin/expect spawn ambari-server sync-ldap --existingexpect "Enter Ambari Admin login:"send "admin\r"expect "Enter Ambari Admin password:"send "admin\r"expect eof