Community Articles

1OverviewTraditionally enterprises have been dealing with data flows or data movement within their data centers. But as the world has become more flattened and global presence of companies has become a norm, enterprises are faced with the challenge of collecting and connecting data from their global footprint. This problem was daunting NSA a decade ago and they came up with a solution for this using a product which was later named as Apache Nifi. Apache nifi is a easy to use, powerful, and reliable system to process and distribute data. Within Nifi, as you will see, I will be able to build a global data flow with minimal to no Coding. You can learn the details about Nifi from Apache Nifi website. This is one of most well documented Apache projects. The focus of this article to just look at one specific feature within Nifi that I believe no other software product does it as well as Nifi. And this feature is “site to site” protocol data transfer. 2Business use caseOne of the classi

Below are steps to replace disk in slave nodes or to perform maintenance of slave nodes servers.1. Decommission the Datanode and all services running on it (i.e. NodeManager, HBase RegionServer, Datanode etc). Refer below docs for the same.https://docs.hortonworks.com/HDPDocuments/Ambari-2.4.1.0/bk_ambari-user-guide/content/decommissioning_masters_and_slaves.html https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.0/bk_administration/content/ch_slave_nodes.html 2. Replace the disks or perform any other tasks for server maintenance.3. Recommission the node.4. Start all services components on the node.5. Run Fsck utility for HDFS to ensure that HDFS is in healthy state. FSCK reports usually show a few over replicated blocks after a datanode is recommissioned which would automatically be fixed over time.

This is part 2 of the Dataflow Optimization tutorial. Part 1 can be found here: NiFi/HDF Dataflow Optimization part 1-----------------------------------------------------------------------------------------------------------------------------------------How do I prevent my system/processors from becoming overwhelmed?Another aspect of optimization is preventing the dataflows from overwhelming the underlying system and affecting NiFi software stability and/or performance. If allowed to run unconstrained, it is more likely to see a severe performance hit. You are more likely to see severe performance hit if dataflows are unbounded and the content repo fills 100%For example, if you have 50MB for the content_repository partition and data normally is delivered via an input port in groups of 10MB. What would happen if the data source was down for a few hours and then came back online with a backlog of 75MB? Once the content_repository filled, the input port would start generating errors wh

NiFi/HDF Dataflow OptimizationWho is This Guide For?This guide is for Dataflow Managers (DFM) very comfortable with NiFi and a good knowledge of the underlying system on which NiFi is running. This guide also assumes that the user has applied the recommend changes in the Configuration Best Practices section of the Admin Guide https://nifi.apache.org/docs.html and applied appropriate changes recommended in the setup best practices here: https://community.hortonworks.com/content/kbentry/7882/hdfnifi-best-practices-for-setting-up-a-high-perfo.html. This guide is not intended to be an exhaustive manual for Dataflow Optimization, but more of points to consider when optimizing their own dataflow.What is meant by dataflow optimization?Dataflow Optimization isn’t an exact science with hard and fast rules that will always apply. But more of a balance between system resources (memory, network, disk space, disk speed and CPU), the number of files and size of those files, the types of processor

There are new Data Visualization, Data Explorer tabs that come part of Hive view which are really nice. I found this by accident so thought there may be others who are probably not aware of this either.Environment details: Should work in vanilla cluster/sandbox as well but in my case the env is as below (setup using steps here😞 Kerborized HDP 2.3.2 w/ Ranger installed Secure Ambari 2.1.2 (authenticating to users in IPA LDAP)The sample salary data from sandbox has also been imported and was used for visualization1. After logging to Ambari 2.1.2, open the Hive view (in this case I had to create a new instance of the view configured for kerberos). 2. Click the Data Visualization tab on the right and drag/drop description and salary fields onto x, y fields3. Select from the various chart options to change the chart 4. Click transpose button:5. Navigate to Data Explorer tab to explore your data using the different fieldsSo all in all: visualizations look good, are responsive (at least on

Symptoms  "Permission denied" errors can present in a variety of use cases and from nearly any application that utilizes CDH.   For example, when attempting to start the jobtracker using this command   service hadoop-0.20-mapreduce-jobtracker start    You may see this error, or one similar   org.apache.hadoop.security.AccessControlException: Permission denied: user=mapred, access=WRITE, inode="/":hdfs:supergroup:drwxr-xr-xat org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:224)at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:204)at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:149)at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:4891)at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:4873)at org.apache.hadoop.hdfs.server.namenode.FSNamesy

Please refer to below forked project for details of benchmark commands ,configuration and results, was able to achieve 3 Million records/sec Read , 1.5 Million records/sec and 5 msec 99.9 Percentile for End-to-end Latency.https://gist.github.com/saurabhmishra/6ed428be4c00003dd926

NiFi can be deployed to run as any non-root user. By default NiFi runs as user "nifi", but this can be modified.For instructions, https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html then go to 'Admin Guide' then go to 'Bootstrap Properties’. In there it talks about 'run.as'

**** This article only pertains to HDF 1.x or Apache NiFi 0.x baseline versions. HDF 2.x+ and Apache NiFi 1.x went through a major infrastructure change in the NiFi core.Converting a Standalone NiFi instance into a NiFi Cluster The purpose of this article is to cover the process of taking standalone NiFi instance and converting it into a NiFi cluster. Often times as a NiFi user we start simple by creating a new dataflow on a single installed instance of NiFi. We may find ourselves doing this to save on resources during the build and test phase or because at the time a single server or VM provide all the resources required. Later we want to add in the fault tolerance that a NiFi cluster provides or resources become constrained as our dataflow grows requiring additional resources that a NiFi cluster also provides. It is possible to take all that work done on that single Instance and convert it for use in NiFi Cluster. Your single NiFi instance could easily become one of the

In event there is an error while executing any oozie action during a workflow processing, those root cause errors are wrapped within an Oozie error and some times you have to dig a little deeper to find the root cause of the issue. Once you look in the right place (screen/log) the error gives a direct indication of what the issue is but that error is buried a couple of layers below the Oozie UI. Follow these steps to get more details on Oozie issues - 1) Open Oozie UI 2) Locate the job instance that failed and double click on it 3) Locate the step that failed and double click on it 4) From the Action pop-up, (which also has the cryptic error message in Error Message) click on the small magnifying glass on the right hand side of the Console URL. 5) From the next screen, click on the link that reads "logs" and view the details. (TBD - Add Screen snapshot)

Groovy UDF exampleCan be compiled at run timeCurrently only works in "hive" shell, does not work in beeline<code>su guesthive paste the following code into the hive shellthis will use Groovy String replace function to replace all instances of lower case 'e' with 'E'<code>compile `import org.apache.hadoop.hive.ql.exec.UDF \;import org.apache.hadoop.io.Text \;public class Replace extends UDF { public Text evaluate(Text s){ if (s == null) return null \; return new Text(s.toString().replace('e', 'E')) \; }} ` AS GROOVY NAMED Replace.groovy; now create a temporary function to leverage the Groovy UDF<code>CREATE TEMPORARY FUNCTION Replace as 'Replace'; now you can use the function in your SQL<code>SELECT Replace(description) FROM sample_08 limit 5; full example<code>hive> compile `import org.apache.hadoop.hive.ql.exec.UDF \; > import org.apache.hadoop.io.Text \; > public class Replace extends UDF { > public Text evaluate(Text s){

In a Kerberized HDP cluster, I use FreeIPA as LDAP and run HDP 2.3.2 including Storm, Kafka and Ranger. After creating a storm topology as a normal user, I keep getting the following runtime error: 2015-12-30 18:31:53.274 o.a.c.ConnectionState [ERROR] Authentication failed ... 2015-12-30 18:31:53.286 o.a.z.ClientCnxn [WARN] SASL configuration failed: javax.security.auth.login.LoginException: No password provided Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it. ... 2015-12-30 18:31:53.328 b.s.util [ERROR] Async loop died! java.lang.RuntimeException: java.lang.RuntimeException: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode ... This problem comes from Kafka and Ranger. As a normal failure recovery mechanism, Kafka will keep creating the topic if it cannot find the designated topic mentioned in the Kafka producer. In a kerberized environment, this CREATE request will be sent to Ranger for approval. How

This is part 2 of the fault tolerance tutorial. Part 1 can be found here: https://community.hortonworks.com/articles/8607/ho...--------------------------------------------------------------------------------------------------------------------------------------Multi-Cluster Fault Tolerance (One cluster in each zone)(All Clusters Active all the time) Multiple clusters can be setup across multiple zones and all are active at the same time. This option requires some external process to keep flows in sync across all clusters. Lets take a look at what this setup might look like if we had 3 zones all running a NiFi cluster. We can see right away that there is a lot more involved with the S2S configuration between the sources systems and these multiple zones. Towards the beginning of this guide we showed what a typical NiFi S2S setup might look like on a source NiFi. With this configuration the sources would need to be setup differently. Rather than delivering data to the Nodes of a sin

Using NiFi to provide fault tolerance across data centers (The Multi-Zone approach) The purpose of this document is to show how to take a NiFi cluster and create some additional fault tolerance using multiple zones/regions. You can think of these zones/regions as physically separated data centers. (They may not even be geographically co-located) The document will cover a few different configurations including one large cluster that spans across multiple zones, two independent clusters (each located in a different zone) with only one zone active at a time, and finally multiple clusters (each located in a different zone) all active at the same time. There are pros and cons to each configuration that will be covered. Depending on your deployment model, one configuration may be better suited then another.A NiFi cluster consists of a single NiFi Cluster Manager (NCM) and 1 to N number of NiFi Nodes. NiFi does not set a limit on the number of Nodes that can exist in a cluster. While a NiF

I’m going to show you a neat way to work with CSV files and Apache Hive. Usually, you’d have to do some preparatory work on CSV data before you can consume it with Hive but I’d like to show you a built-in SerDe (Serializer/Deseriazlier) for Hive that will make it a lot more convenient to work with CSV. This work was merged in Hive 0.14 and there’s no additional steps necessary to work with CSV from Hive.Suppose you have a CSV file with the following entries id first_name last_name email gender ip_address 1 James Coleman jcoleman0@cam.ac.uk Male 136.90.241.52 2 Lillian Lawrence llawrence1@statcounter.com Female 101.177.15.130 3 Theresa Hall thall2@sohu.com Female 114.123.153.64 4 Samuel Tucker stucker3@sun.com Male 89.60.227.31 5 Emily Dixon edixon4@surveymonkey.com Female 119.92.21.19to consume it from within Hive, you’ll need to upload it to hdfshdfs dfs -put sample.csv /tmp/serdes/ now all it takes is to create a table schema on top of the filedrop table if exists sample;create

In Ambari the Hiveserver2 is shown as "green", but there is an alert indicating problem with the Hiverserver2 process.This is due to a known bug in HDP2.2.3/2.2.4, that at Hive startup time no Kerberos ticket will be grabbed. To get rid of that alert, login to the Hiveserver node, become user hive and execute a "kinit -kt /etc/security/keytabs/hive.service.keytab hive/<hiveserver>@<REALM>" (if you keytab is in that default directory) to ensure the hive user got a valid Kerberos ticket.Afterwards you can restart Hiveserver in Ambari and the alert will disappear (Thanks to @dprichici for highlighting this)Regards, Gerd

Part 2 Linkedin Post Extending Blog 2 to look for starwars tweet Searching for Yoda, Love & Hate Let's see Tweets/Data with word YODA related to Starwars tweets Keyword : LOVE in STARWARS Source Giphy Word Hate Happy Hadooping!!!

Some basic charts are already included in Zeppelin. Visualizations are not limited to SparkSQL's query, any output from any language backend can be recognized and visualized. This sufficient for a Data Analyst, but if you were a Data Scientist, the built in tools within Zeppelin just don't cut. If you are a Python programmer and have been working with data in IPython, you must definitely be well versed in matplotlib library. You can install matplotlib in your python shell as long as Zeppelin is still using the same version of python. Once you do that the Pyspark interpreter will be able to import matplotlib libraries and you will be able to create graphs and charts in the Zeppelin interface itself. Make sure to add the following code at the end of your script and modify it as needed:def show(graph): img = StringIO.StringIO() graph.savefig(img, format='svg') img.seek(0) print "%html <div style='width:1000px'>" + img.buf + "</div>"show(plt)

Follow steps below to run this sample spark-shell code to index a Spark Data Frame into a Solr Cloud:1- Install java 8 in your sandbox 2.3.2yum install all java-1.8.0-openjdk*2- Define java 8 as your default java, using alternativesalternatives --config java##choose /usr/lib/jvm/jre-1.8.0-openjdk.x86_64/bin/java3- Logoff / Logon againexitssh root@sandbox.hortonworks.com4- Change java version in spark, in spark-env.shvi /usr/hdp/current/spark-client/conf/spark-env.sh##change JAVA_HOME like below:export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.65-0.b17.el6_7.x86_645- Download and build spark-solrgit clone https://github.com/LucidWorks/spark-solr.gitcd spark-solrmvn clean install -DskipTests 6- Create a sample text file and write into hdfsecho "1,guilherme" > your_text_fileecho "2,isabela" >> your_text_filehadoop fs -put your_text_file /tmp/7- Start sandbox solr cloudcd./start_solr.sh8- Create a solr collection/opt/lucidworks-hdpsearch/solr/bin/solr create -c testsparksolr

Original postA web-based notebook that enables interactive data analytics. You can make beautiful data-driven, interactive and collaborative documents with SQL, Scala and more.In few words " It's really cool tool to interact with Data"HDFS, Hive, Spark, Kylin, Flink This is from HDP latest SandboxContinue to Blog 3 on NiFiLet's analyze Starwars dataHive DemoTable definition and Top 10 users based on tweet countTop 10 users who used the word "love" in #starwarsWord hate used in #starwarsWord yoda used in #starwarsYou can see the Tweet sent by my id in Zeppelin output. Spark I used this for the sentiment analysis. Replace %hive with %sql (Assuming that you have setup the Zeppelin correctly)LinksZeppelinHortonworks and ZeppelinHappy Hadooping!!!

If you go to port 4200 on your HDP sandbox http://sandbox.hortonworks.com:4200/

OverviewThe objective of this article is to introduce the ESRI Spatial Framework for Hadoop and demonstrate how to use it with HDP 2.3.2. The following is a high-level view of how we will accomplish this: set up the ESRI Spatial Framwork for Hadoop on the HDP 2.3.2 sandboxETL data from the Open Baltimore Data PortalExecute simple geospatial queries to correlate crime data in Baltimore with spatial neighborhood dataPrior to continuing the tutorial, please ensure you have downloaded and started the HDP Sandbox.ESRI Spatial Framework for HadoopThe ESRI Spatial Framework for Hadoop is a collection of Hive UDFs that allow users to perform complex spatial analysis directly in Hive. The framework has built-in support for representing geometric shapes (point, polygon, etc.) as well as functions that operate on these shapes. For example, users can perform a binary test for overlap between a pair of polygons or compute the geometry of the intersection. This framework provides a powerful metho

When running Solr in a clustered mode (SolrCloud), it has a runtime dependency on a ZooKeeper, where it stores configs, coordinates leader election, tracks replicas allocation, etc. All-in-all, there's a whole tree of ZK nodes created with sub-nodes. Deploying SolrCloud into a Hadoop cluster usually means re-using the centralized ZK quorum already maintained by HDP. Unfortunately, if not explicitly taken care of, SolrCloud will happily dump all its ZK content in ZK root, which really complicates things for an admin down the line.If you need to clean up your ZK first, take a look at this how-to.Solution is to put all SolrCloud ZK entries under its own ZK node (e.g. /solr). Here's how one does it:su - zookeepercd /usr/hdp/current/zookeeper-client/bin/ # point it a ZK quorum (or just a single ZK server is ok, e.g. localhost)./zkCli.sh -server lake02:2181,lake03:2181,lake04:2181 # in zk shell now# note the empty brackets are _required_create /solr [] # verify the zk node has been created,

Your Ambari throwing a weird error and you need to troubleshoot? No problem....just install this view and browse this forum from Ambari itself!1. Install the view: wget https://github.com/hortonworks-gallery/ambari-iframe-view/raw/master/samples/answerhub-view-1.0-SNAPSHOT.jar -P /var/lib/ambari-server/resources/views/2. Then restart Ambari:#on sandboxservice ambari restart#on non-sandboxservice ambari-server restart3. Now open the view:

Update: added a GitBook linkThe unofficial little black book of Kerberos, created and maintained by a HWX engineer, Steve Loughran. Lots of questions that you were afraid to ask. Many advanced customers found it a very useful guide, especially if one needs to develop solutions and code for a Kerberized cluster.I felt this guide needed much more exposure than it had so far. All credit goes to @stevel@hortonworks.comhttps://github.com/steveloughran/kerberos_and_hadoopClick on the Contents links..or..Enjoy as a GitBook, readable online, on mobile and exportable as e.g. a PDF:https://steveloughran.gitbooks.io/kerberos_and_hadoop/content/