Community Articles

Data is the becoming the new precious resource. As the world produces more and more data, business units find increasingly more ways to monetize that data. This means that data that used to be retained for a short time or not at all, is now being persisted long term. This data is being gathered from more and more sources and not necessarily from within the organization that uses it. It is also increasingly being generated by machines, meaning that the volume, velocity, and variety of the data proliferate at an overwhelming rate. There are now lots of tools that enable an organization to address the challenges imposed by the proliferation of data. However, many organizations have been focused on dealing with volume and velocity while not focusing on the challenges created by the lack or inconsistency of structure. In order to truly unlock the power of all that data, an organization must first apply a consistent set of guidelines for governance of the data. Getting value from new data so

IntroductionThis article is not meant to show how to install or create a “Hello World” Nifi data flow, but how to resolve a data filtering problem with NiFi providing two approaches, using a filter list as a file on the disk, which could be static or dynamic, and a list stored in a distributed cache populated from the same file. The amount of data used was minimal and simplistic and no performance difference can be perceived, however, at scale, where memory is available, a caching implementation should perform better.This article assumes some familiarity with NiFi, knowing what a processor or a queue is and how to set the basic configurations for a processor or a queue, also how to visualize the data at various steps throughout the flow, starting and stopping processors.Since you are somehow familiar with Nifi, you probably know how to install it and start it, however, I will provide a quick refresher below.Pre-requisitesFor this demo, I used the latest version of Nifi available at the

Purpose of this article: When you install HDP for dev/test environment, you would repeat same commands to set up your host OS. To save time, created a BASH script which helps to set up the host OS (Ubuntu only) and docker image (CentOS). What this script does: Install packages on Ubuntu host OS Set up docker, such as creating image and spawning containers [Optional] Set up a local repository for HDP (not Ambari) with Apache2 What this script does NOT: As of this writing, this does not install HDP Please use Ambari Blueprint if you would like to automate HDP installation as well This step is NOT for production environment but would be useful to test HA components Host set up steps: Install Ubuntu 14.x LTS on your VirtualBox/VMware/Azure/AWS. It should be easy to deploy Ubuntu VM if you use Azure or AWS. If you are using VirtualBox/VMWare, you might want to backup Ubuntu installed VM as a template, so that later you can clone. Login to Ubuntu and become root (sudo -

How to setup cross realm trust between two MIT KDC – In this post, we will see how to setup cross realm trust between two MIT KDC. We can access and copy data from one cluster to another if the cross realm trust is setup correctly..In our example, we have 2 clusters with same HDP version(2.4.2.0) and Ambari version(2.2.2.0).Cluster 1:172.26.68.47 hwx-1.hwx.com  hwx-1172.26.68.46 hwx-2.hwx.com  hwx-2172.26.68.45 hwx-3.hwx.com  hwx-3Cluster 2:172.26.68.48 support-1.support.com support-1172.26.68.49 support-2.support.com support-2172.26.68.50 support-3.support.com support-3.Below are the steps:.Step 1: Make sure both the clusters are kerberized with MIT KDC. You can use below automated script for configuring Kerberos on HDP.https://community.hortonworks.com/articles/29203/automated-kerberos-installation-and-configuration.html.Step 2: Please configure /etc/hosts file on both the clusters to have Ip <-> hostname mappings.Example:On both clusters /etc/hosts file should look like

Apache ZooKeeper is a “high-performance coordination service for distributed applications.” Most users do not use ZooKeeper directly; however, most users are also hard-pressed to deploy a Hadoop-based architecture that doesn’t rely on ZooKeeper in some way. With its prevalence in the data-center, resource management within ZooKeeper is paramount to ensure that the various applications and services relying on ZooKeeper are able to access it in a timely manner. To this end, one of ZooKeeper’s protection mechanisms is known as “max client connections” or “ maxClientCnxns”. maxClientCnxns refers to a configuration property that can be added to the zoo.cfg configuration file. This property limits the number of active connections from a host, specified by IP address, to a single ZooKeeper server. By default, this limit is 60 active connections; one host is not allowed to have more than 60 active connections open to one ZooKeeper server. Changes to this property in the zoo.cfg file requir

Kafka from 0.9 onwards started support SASL_PLAINTEXT ( authentication and non-encrypted) for communication b/w brokers and consumer/produce r with broker. To know more about SASL, please refer to this link. Maven Dependency Add below maven dependency to your pom.xml <dependency><br> <groupId>org.apache.kafka</groupId><br> <artifactId>kafka-clients</artifactId><br> <version>0.10.0.0</version><br></dependency> 2. Kerberos Setup Configure JAAS configuration file with contents as belowKafkaClient { com.sun.security.auth.module.Krb5LoginModule required useTicketCache=true principal="user@EXAMPLE.COM" useKeyTab=true serviceName="kafka" keyTab="/etc/security/keytabs/user.headless.keytab";};Above configuration is set to use key tab and ticket cache. The Kakfa Client Producer use above info to get TGT and authenticates with Kafka broker.Note:a] Make sure the /etc/krb5.conf has realms mapping for "EXAMPLE.COM" an

Overview ORC provides many benefits, including support for compression. However, there are times when you need to store “raw” data in HDFS, but want to take advantage of compression. One such use case is with Hive external tables. Using PIG, you can LOAD the contents of uncompressed data from an HDFS directory, and then STORE that data in a compressed format into another HDFS directory. This approach is particularly useful if you are already using PIG for part of your ETL processing.Scope This tutorial has been tested with the following configuration: Mac OS X 10.11.5 VirtualBox 5.1.2 HDP 2.5 Tech Preview on Hortonworks SandboxPrerequisites The following should be completed before following this tutorial: VirtualBox for Mac OS X installed VirtualBox Link Hortonworks HDP 2.5 Technical Preview Sandbox configured in VirtualBox Sandbox Link Steps1. Connect to the Sandbox If you are using Vagrant to spin up your sandbox, then you can simply connect using: $ vagrant ssh If

With Apache NiFi 1.0 you can now act as a simple SMTP server (though it is recommended to sit behind a real SMTP MTA and just get mail forwards). It makes for an easy way to ingest mail, headers and attachments.The first thing you will notice is the awesome new UI, which is much cleaner and a joy to use.'First add a processor, ListenSMTP, this will be your mail gateway/SMTP server. As you can see there's also processors for extracting attachments and headers from Email.You need to make sure you set Listening Port, SMTP hostname and Max. # of Connections.The entire flow for mail processing is pretty simple, but easy to follow. We listen for SMTP over TCP Port (I chose 2025, but with Root access you could run on 25). I send the original flow file right to HDFS. I extract the attachments and put them in a separate HDFS directory and finally pull out the email headers and also send them to an HDFS file. I have a little test flow in the bottom to read a file and send email to o

1. Recover the lease for the fileWhen you do "hdfs dfs -cat file1" from the command line, you get the exception saying that it "Cannot obtain block length for LocatedBlock". Usually this means the file is still in being-written state, i.e., it has not been closed yet, and the reader cannot successfully identify its current length by communicating with corresponding DataNodes.Suppose you're pretty sure the writer client is dead, killed, or lost connection to the servers. You're wondering what else you can do other than waiting.hdfs debug recoverLease -path <path-of-the-file> [-retries <retry-times>] This command will ask the NameNode to try to recover the lease for the file, and based on the NameNode log you may track to detailed DataNodes to understand the states of the replicas. The command may successfully close the file if there are still healthy replicas. Otherwise we can get more internal details about the file/block state. Please refer to https://community.hortonwo

The Apache Nifi community recently released the beta version of Apache Nifi 1.0.0. This version comes with significant updates, which include a UI refresh, transition to zero master clustering, added multi-tenant authorization, and templates that are now deterministically ordered allowing for version controlled templates! The beta also boasts nine new processors, bringing the total to 165. The full list of release notes can be found here. Below I will do a very basic walkthrough of some of the UI changes from 0.7.0 to the 1.0.0 beta: Original Apache Nifi 0.7.0 Flow New Apache Nifi 1.0.0 Beta Outside of the more modernized look and feel, there are some key UI Changes: (1) Apache Nifi 1.0.0 beta now includes a status bar showing statistics of the overall flow, including bytes in and out, number of started and stopped processors, processors in error, last refresh, etc.(2) The beta also has new collapsable navigation and operation panes (3) There's a new drop down menu where you c

Test Conducted on:1) HDP2.3.42) Data Size : 1.4 GB2) Cluster is ideal and not running any other jobs.Conclusion:Observed that Zlib is doing more compression than SNAPPY but SNAPPY jobs are completing quicker than ZLib.Results are here:Execution Time in Sec:Disk Usage Comparison:

Just a few months ago, Apache Storm announced release 1.0 for the distribution. The bullet points below summarize the new features available. For more detailed descriptions, you can go to this link to read the full release notes:http://storm.apache.org/2016/04/12/storm100-released.htmlApache Storm 1.0 ReleaseApache Storm 1.0 is *up to 16 times faster than previous versions, with latency reduced up to 60%.”Pacemaker – Heartbeat ServerPacemaker is an optional Storm daemon designed to process heartbeats from workers. (overcomes scaling problems of Zookeeper)Distributed Cache APIFiles in the distributed cache can be updated at any time from the command line, without the need to redeploy a topology. HA NimbusMultiple instances of the Nimbus service run in a cluster and perform leader election when a Nimbus node failsNative Streaming Window APIStorm has support for sliding and tumbling windows based on time duration and/or event count.Automatic BackpressureStorm now has an automatic backpres

Hortonworks and SAS have partnered to create two new Apache NiFi processors. These processors allow data/events to be streamed between Hortonworks DataFlow (HDF) and SAS Event Stream Processing.Why does this matter? HDF powered by Apache NiFi, Kafka and Storm, is an integrated system for real-time dataflow management and streaming analytics on-premise or in the cloud. SAS Event Stream Processing is a real-time, low-latency, high-throughput event processing solution that can deploy SAS machine learning models. By integrating these technologies, organizations now have the option of deploying their SAS models in real-time within the Hortonworks platform. This offers flexible deployment options for your streaming analytics projects, while providing powerful analytics from SAS. How does this integration work? There are two new processors that can be added to NiFi: ListenESP: This processor initiates a listener within NiFi that receives events from the SAS Event Stream Processi

To enable the SSL/HTTPS for Ambari, have to input signed certificate and Privatekey file. Generally many organizations would have JavaKeyStore/JKS file from which they would have generated a CSR and got the signed certificate from CA. In this case they may not have separate Privatekey file with them. In that scenario below steps can be followed to enable the SSL. Here I am providing the steps to enable the SSL using self signed certificate.1. Create Java key store: keytool -genkey -keyalg RSA -alias ambari -keystore keystore.jks -validity 360 -keysize 2048 2. Export cert to server.cer filekeytool -export -alias ambari -file server.cer -keystore keystore.jks Note: In case of CA signing, CSR can be exported from JKS file and get that signed from appropriate CA.3. Convert .cer file from DER format to PEM formatopenssl x509 -in server.cer -inform der -outform pem -out server.pem 3. Export Privatekey to PKCS12 keytool -v -importkeystore -srckeystore keystore.jks -srcalias ambari -des

UPDATE: I'm happy to report that my patch for PIG-4931 was accepted and merged to trunk.I was browsing through Apache Pig Jiras and stumbled on Jira https://issues.apache.org/jira/browse/PIG-4931 requiring to document Pig "IN" operator. Turns out Pig had IN operator since days of 0.12 and no one had a chance to document it yet. The associated JIRA is https://issues.apache.org/jira/browse/PIG-3269. In this short article I will go over the IN operator and until I'm able to submit a patch to close out the ticket this should serve as its documentation. Now, IN operator in Pig works like in SQL. You provide a list of fields and it will return just those rows. It is a lot more useful than for example a = LOAD '1.txt' USING PigStorage(',') AS (i:int); b = FILTER a BY (i == 1) OR (i == 22) OR (i == 333) OR (i == 4444) OR (i == 55555); You can rewrite the same statement as a = LOAD '1.txt' USING PigStorage(',') AS (i:int); b = FILTER a BY i IN (1,22,333,4444,55555); The best thing

Working with customers I get this question quite often on how to size their cluster. Apart from master nodes, one question that often comes up is the size of data nodes. Hardware vendors now offer disks up to 8TB capacity that can offer customers up to 96 TB of storage per node assuming twelve spindles per node. Even if you go with 12x4TB disks, that is still a whopping 48 TB of storage per node. For most cases, I have always recommended my customers 12x2TB disks over last three years and I continue to do so, given bandwidth remains very expensive, and as we'll see below, it is a very important component when you are sizing a cluster and deciding between high and low density data nodes.The calculations I am sharing here were done for a customer when they told me that re-replication of blocks when a node fails takes a very long time. This customer had 12x4TB disks on each node.So rather than preferring one opinion over the other, let's do some maths and then decide what works for your u

Hive Streaming Compaction This is the second part of the Hive Streaming Article series. In this article we will review the issues around compacting Hive Streaming files. One of the results of ingesting data through Hive streaming is the creation of many small 'Delta' files. Left uncompacted you could run the risk of running into NameNode capacity problems. Fortunately, compaction functionality is part of Hive Streaming. The remainder of this Article reviews design considerations as well as commands necessary to enable and control compaction for your Hive tables. Hive Compaction Design considerations The Compaction process has a set of cleaner processes running in the background during the ingest process looking for opportunities to compact the delta files based on the rules you specify. The first thing to keep in mind is that there are two forms of Compaction; ‘minor’ and ‘major’. A ‘minor’ compaction will just consolidate the delta files. This approach does not have to worr

Starting Assumption: Your Nifi data flow similar to the TwitterSolr.xml template on the Nifi website(Nifi Templates). Here is the template workflow (MergeContent is optional) :In order to add sentiment analytics, we need to alter the data flow by updating the flowfile (the actual data being passed in Nifi) with attributes that represent the sentiment score of the tweet in question.Script Components: 1)Importing the JSON and accessing tweet text2)Running a sentiment analyzer on the tweet text and getting sentiment scores3)Appending sentiment scores to JSON and writing back to the flowfile***Note***: For the actual script itself I used Groovy (a Java-compatible scripting language), but you can use other scripting languages such as Lua, Python, Ruby, or ECMAScript (Script Code). For the sentiment analytics portion, I used a Java port of the Python sentiment analytics module vaderSentiment, but there are a plethora of open-source sentiment analytics APIs such as Python’s NLTK module, Stan

Note - For this tutorial I assume that you already have HDP cluster kerberized. If you want to configure automated kerberos then please refer to https://community.hortonworks.com/articles/29203/automated-kerberos-installation-and-configuration.html..First you need to configure your Ambari Server for Kerberos configuration, without configuring Ambari for Kerberos, views will not work..Use below commands to configure Ambari for Kerberos:.1 Login to kadmin and create principal for Ambari Server Note - Please replace your REALM in REALM.COMaddprinc -randkey ambari-server@REALM.COM.2 Extract principal created in above step to keytab filexst -k ambari.server.keytab ambari-server@REALM.COM.3 Above command should generate keytab file in current working directory, copy keytab file to /etc/security/keytabs/ locationcp ambari.server.keytab /etc/security/keytabs/ambari.server.keytabNote - Please make sure that, user running ambari server daemon should have read access to ambari.server.keytab.4 Sto

TipsRead each processor directions carefully, inputs and outputs vary greatly from attributes, JSON, JSONPath statements and other entries. JSONPath supports a radically different syntax than NIFI Expression Language.Know your Expression language, there are a lot of useful functions like UUID() and nextint().When you are doing a lot of streaming and testing data, you will find that logs grow huge on your Centos sandbox. I built up gigabytes of logs in Hive, Hadoop and NiFi in just a few days of operations. Check your /var/log every few days and watch the Ambari screens for data usage. When your data gets too large, things can start failing or slowing down due to a lack of temporary space, swap space and space for logs.Delete Test Files For Ever! hdfs dfs -rm -f -skipTrash /twitter/*.jsonClean up junk files.cd /du -hsx * | sort -rh | head -10 ) PutHiveQL requires hiveql.args.1.value and hiveql.args.1.type for all fields. If you are using EvaluateJSONPath, you cannot set your types

Clients normally want a development environment for SQL. They often have Eclipse-based SQL development tools already ( Teradata SQL Editor, Eclipse Data tools platform ). Hue and command line are not always an option.To connect to HDP2.3 ( should work for HDP2.2) with Eclipse:1. Install the Eclipse Data Tools Platform- Download Eclipse from eclipse.org ( for example Luna ) - Select Help->Install new Software- "Work with" the official update location for your release ( Luna )- Install all plugins under "Database Development" and restart Eclipse2. Create an Hive JDBC driver- Open the Data Source Explorer View ( Windows->Show View->Others->Type Data Source Explorer)- Under Database Connections select "New" - Select "Generic JDBC Driver". You should see three tabs ( Name, Jars , Properties )- Create a new Driver with the plus button and give name "Hive"- On Jars add the following jars from your HDP installationFrom HDP_INSTALLATION/hive/lib ( hive-jdbc.jar and commons-loggingx

There is situation when unfortunately and unknowingly you delete /hdp/apps/2.3.4.0-3485 either with skipTrash or without skipTrash then you will be in trouble and other services will be impacted. You will not be able to run hive,mapreduce or sqoop command, You will get following error.Case 1: If you deleted it without skipTrash then it is very easy to recover: [root@m1 ranger-hdfs-plugin]# hadoop fs -rmr /hdp/apps/2.3.4.0-3485rmr: DEPRECATED: Please use ‘rm -r’ instead.16/07/28 01:59:22 INFO fs.TrashPolicyDefault: Namenode trash configuration: Deletion interval = 360 minutes, Emptier interval = 0 minutes.Moved: 'hdfs://HDPTSTHA/hdp/apps/2.3.4.0' to trash at: hdfs://HDPTSTHA/user/hdfs/.Trash/CurrentIn this case it would be very easy to recover it as after deleting it goes to your current dir and you can recover it from there.hadoop fs -put hdfs://HDPTSTHA/user/hdfs/.Trash/Current//hdp/apps/2.3.4.0 /hdp/apps/ Case 2: If you deleted it with -skipTrash then you need to execute following st

I have seen an issue with Application Timeline Server (ATS). Actually Application Timeline Server (ATS) uses a LevelDB database which is stored in the location specified by yarn.timeline-service.leveldb-timeline-store.path in yarn-site.xml.All metadata store in *.sst files under specified location.Due to this we may face an space issue.But It is not good practice to delete *.sst files directly. An *.sst file is a sorted table of key/value entries sorted by key and key/value entries are partitioned into different *.sst files by key instead of timestamp, such that there’s actually no old *.sst file to delete.But to solve the space of the leveldb storage, you can enable TTL (time to live). Once it is enabled, the timeline entities out of ttl will be discarded and you can set ttl to a smaller number than the default to give a timeline entity shorter lifetime.<property> <description>Enable age off of timeline store data.</description> <name>yarn.timeline-service.ttl

I was reviewing some posts related to Pig, and found the following question interesting:https://community.hortonworks.com/questions/47720/apache-pig-guarantee-that-all-the-value-in-a-colum.html#answer-47767I wanted to share an alternative solution using Pentaho Data Integration (PDI), an open source ETL tool, that provides visual mapreduce capabilities. PDI is YARN ready, so when you configure PDI to use your HDP cluster (or sandbox) and run the attached job, it will run as a YARN application.The following image is your Mapper.Above, you see the main transformation. It reads input, which you configure in the Pentaho MapReduce Job (seen below). The transformation follows a pattern, which is to immediately split the delimited file into individual fields. Next, I use a Java Expression to determine if a field is numeric. If not, the we set the value of the field as the String, null. Next, to prepare for MapReduce output, we concatenate the fields back together as a single value and pass th

IntroductionThis article is a complement to Geo-spatial Queries with Hive using ESRI Geometry and Spatial Framework for Hadoop and includes a few more findings, mainly documenting the differences between ST_Geometry functions supported in Hive and those in commercial spatial packages for Oracle, SQL Server or Netezza.The Hive UDF's are modeled after existing implementations of ST_Geometry. Some functions exist only in Hive’s implementation, a few behave different or don’t exist. Additional ST_Geometry Functions in HiveST_GeomFromJSON - Create a geometry from Esri JSONST_AsJSON - Return Esri JSON representation of geometryST_GeomFromGeoJSON - Create a geometry from GeoJSONST_AsGeoJSON - Return GeoJSON representation of geometry (requires geometry-api-java version 1.1)ST_PointZ - constructor for 3D points from direct positionST_SetSRID - set the spatial-reference ID, for a geometry constructed without itST_GeodesicLengthWGS84 - geodesic length in meters rather than in anglesST_Bin, ST_Bi