Community Articles

Pre-requisitesHortonworks Data Platform 2.5 on CentOS 7.2Python distribution that comes with HDP 2.5 - Python 2.7.5Download and install pip#wget https://bootstrap.pypa.io/get-pip.pyInstall add-on package#pip install requestsStart Python CLI (default version)#pythonImport pre-reqs>>>import requests>>>import json>>>import sysEnvironment VariablesSet Ambari domain variable to the IP address or FQDN of your Ambari node.>>>AMBARI_DOMAIN = '127.0.0.1'Set Ambari port, Ambari user and password variables to match your specifics.>>>AMBARI_PORT = '8080'>>>AMBARI_USER_ID = 'admin'>>>AMBARI_USER_PW = 'admin'Set the following variable to the IP address or FQDN of your ResourceManager node.>>>RM_DOMAIN = '127.0.0.1'Set Resource Manager port variable>>>RM_PORT = '8088'Ambari REST API Call ExamplesLet's find Cluster Name, Cluster Version, Stack and Stack Version:>>>restAPI='/api/v1/clusters'>>>url="http:/

Drones / unmanned aerial vehicle (UAV) Ingest We have several options on ingesting data from Drones depending on the model, location, networking and needs. MQTT is a popular option if you are on the same WIFI or network. See my article on MQTT https://community.hortonworks.com/articles/72420/ingesting-remote-sensor-feeds-into-apache-phoenix.html. Many UAVs will let you push up to a cloud MQTT server and asynchronously consume this data from your enterprise HDF system. They can also be access via TCP, UDP, REST and offline download. For my example I am parsing a directory full of downloaded BEBOP Parrot 2 Drone images. NIFI with Four Parallel Flows from the Ingested File Step 1: GetFile from directory of images. Step 2: PutHDFS Store FlowFile Raw images to HDFS. Step 3: Update Attribute add an attribute (API KEY) to pass to Microsoft Cognitive Services. Step 4: ReplaceText to convert body to a JSON message with image Step 5: InvokeHTTP to call https://api.projectoxford.ai/vision/v1.0/

During debugging a problem with delete topic,I dig into Kafka code to know how delete command works, this the sequence of event occurred during command execution1. TopicCommand issues topic deletion /usr/hdp/current/kafka-broker/bin/kafka-run-class.sh kafka.admin.TopicCommand --zookeeper rkk3.hdp.local:2181 --delete --topic sample 2. which create a new admin path /admin/delete_topics/<topic> 3. The controller listens for child changes on /admin/delete_topic and starts topic deletion for the respective topics 4. The controller has a background thread that handles topic deletion. A topic's deletion can be started only by the onPartitionDeletion callback on the controller.5. Once DeleteTopicsThread invoked it looks for topicsToBeDeleted and for each topic it deregister partition change listener on the deleted topic, This is to prevent the partition change listener firing before the new topic listener when a deleted topic gets auto created. 6. Controller will remove this replica from

This article is a follow-up on my original article about the visualization of a cluster and its services/components ( https://community.hortonworks.com/articles/2010/visualizing-hdp-cluster-service-allocation.html). In the first part I am particularly focusing on a new feature that enables users to build and plan a cluster by using a drag-n-drop Web UI.Build a ClusterUntil now visualizing a cluster and its service allocation either meant exporting the information from Ambari or writing a JSON file that outlines the details of the nodes. Planning and deploying a cluster should be easier, right?I'll introduce, Build a Cluster 🙂This simple Web UI is based on different drag-n-drop functionalities and allows the creation of a new cluster by simply dragging Hadoop components from the elements list to the indivdual nodes.Lets go over the different features....User InterfaceThe UI is divided into two sections Elements & Settings and Cluster:Elements & Settings (left): Contain the avai

BackgroundTungsten became the default in Spark 1.5 and can be enabled in earlier versions by setting spark.sql.tungsten.enabled to true (or disabled in later versions by setting this to false). Even without Tungsten, Spark SQL uses a columnar storage format with Kryo serialization to minimize storage cost.GoalThe goal of Project Tungsten is to improve Spark execution by optimizing Spark jobs for CPU and memory efficiency (as opposed to network and disk I/O which are considered fast enough).ScopeTungsten focuses on the hardware architecture of the platform Spark runs on, including but not limited to JVM, LLVM, GPU, NVRAM, etc. Optimization FeaturesOff-Heap Memory Management using binary in-memory data representation aka Tungsten row format and managing memory explicitly,Cache Locality which is about cache-aware computations with cache-aware layout for high cache hit rates,Whole-Stage Code Generation (aka CodeGen).Design ImprovementsTungsten includes specialized in-memory data structures

ENV: HDP 2.4.2STEP 1: Setting up MySQL SSL# Create clean environmentshell> rm -rf newcertsshell> mkdir newcerts && cd newcerts# Create CA certificateshell> openssl genrsa 2048 > ca-key.pemshell> openssl req -new -x509 -nodes -days 3600 \ -key ca-key.pem -out ca.pem# Create server certificate, remove passphrase, and sign it# server-cert.pem = public key, server-key.pem = private keyshell> openssl req -newkey rsa:2048 -days 3600 \ -nodes -keyout server-key.pem -out server-req.pemshell> openssl rsa -in server-key.pem -out server-key.pemshell> openssl x509 -req -in server-req.pem -days 3600 \ -CA ca.pem -CAkey ca-key.pem -set_serial 01 -out server-cert.pem# Create client certificate, remove passphrase, and sign it# client-cert.pem = public key, client-key.pem = private keyshell> openssl req -newkey rsa:2048 -days 3600 \ -nodes -keyout client-key.pem -out client-req.pemshell> openssl rsa -in client-key.pem -out client-key

while debugging kafka producer slowness, I observed following steps Kafka producer take while producing a single record to kafka broker.ENV: HDP 2.52 node kafka cluster, single producer producing a record to 'testtopic' which has 2 partitions with 2 replicasKafka Producer start with the configured settings it start adding matrices sensors.update cluster metadata version which includes cluster information like broker nodes and partitions, assign version id to this cluster metadata version.Updated cluster metadata version 1 to Cluster(nodes = [Node(-2, rkk2, 6667), Node(-1, rkk1, 6667)], partitions = [])Set up and start Kafka producer I/O thread aka Sender thread.Request metadata update for topic testtopic.Producer's NetworkClient metadata request to one of the broker which consist of api_key,api_version,correlation_id and client_id.Sending metadata request ClientRequest(expectResponse=true, callback=null, request=RequestSend(header={api_key=3,api_version=0,correlation_id=0,client_id=pro

To send data to my firewall protected internal Hadoop cluster, I have my remote Raspberry Pi 3 with attached Sense-Hat use a cron job to send MQTT messages to a cloud hosted MQTT Broker. NiFi then subscribes to that queue and pulls down the messages asynchronously. I use JSON as my packaging format as that is very easy to work with and process in NiFi and elsewhere.The Sense-Hat connects to your non-Pi Zero and provides a number of easy to access sensors including ones for temperature, humidity, X,Y, Z, Roll and Pitch. It also has a nice light display that can be used to display graphical messages. For data capture needs it also has standard Linux reports like CPU temperature that we can also grab. You could also grab memory usage and disk space. Since the API to access Sense-Hat is Python, it makes sense to keep my access program in Python. So I have a small Python program that reads and formats the Sense Hat sensor values, puts them into a JSON document and sends them up

Yarn exposes cluster application REST API to get the application stats.With the Applications API, you can obtain a collection of resources, each of which represents an application. When you run a GET operation on this resource, you obtain a collection of Application Objects. In this example, we will analyze application stats using Spark-SQL.Env: HDP 2.5Spark Version: Spark-2Data Preparation: get the apps.json while invoking REST API using curl commandcurl -v -X GET -H "Content-Type: application/json" http://`hostname`:8088/ws/v1/cluster/apps > /tmp/apps.jsonnext step is to invoke spark shell and execute the sequence of steps as following[spark@rkk1 ~]$ spark-shell SPARK_MAJOR_VERSION is set to 2, using Spark2Setting default log level to "WARN".To adjust logging level use sc.setLogLevel(newLevel).16/12/17 16:01:52 WARN SparkContext: Use an existing SparkContext, some configuration may not take effect.Spark context Web UI available at http://172.26.81.127:4040Spark context available a

Many times we find that the operations (like start/stop/restart ...etc) are failing from ambari UI. In such cases if we want to troubleshoot what Ambari UI did to perform that operation or How the commands were executed. Then we can manually execute those same operations from the individual host with the help of "/var/lib/ambari-agent/data/command-xxx.json" file.- When we perform any operation from ambari UI like (Starting / Stopping Datanode) then we will notice that ambari shows the operation progress in the UI. There we can see basically the following two files.Example: stderr: /var/lib/ambari-agent/data/errors-952.txtstdout: /var/lib/ambari-agent/data/output-952.txt- Apart from the above files there is one more important file which ambari agent uses to execute the instructions/commands that are sent by the AmbariServer. We can find that specific file in the ambari-agent's "/var/lib/ambari-agent/data/command-xxx.json" file. /var/lib/ambari-agent/data/command-952.json - Here the "c

[Related Article On Ambari Server Tuning : https://community.hortonworks.com/articles/131670/ambari-server-performance-tuning-troubleshooting-c.html ]-The jcmd utility comes with the JDK and is present inside the "$JAVA_HOME/bin". It is used to send diagnostic command requests to the JVM, where these requests are useful for controlling Java Flight Recordings, troubleshoot, and diagnose JVM and Java Applications. Following are the conditions for using this utility. - 1. It must be used on the same machine where the JVM is running - 2. Only a user who own the JVM process can connect to is using this utility. This utility can help us in getting many details about the JVM process. Some of the most useful information's are as following: Syntax: jcmd $PID $ARGUMENT Example1: Classes taking the most memory are listed at the top, and classes are listed in a descending order./usr/jdk64/jdk1.8.0_60/bin/jcmd $PID GC.class_histogram > /tmp/22421_ClassHistogram.txt Example2: Generate Hea

Origin: http://henning.kropponline.de/2015/05/19/hivesink-for-flume/With the most recent release of HDP (v2.2.4) Hive Streaming is shipped as technical preview. It can for example be used with Storm to ingest streaming data collected from Kafka as demonstrated here. But it also still has some serious limitations and in case of Storm a major bug. Nevertheless Hive Streaming is likely to become the tool of choice when it comes to streamline data ingestion to Hadoop. So it is worth to explore already today.Flume's upcoming release 1.6 will contain a HiveSink capable of leveraging Hive Streaming for data ingestion. In the following post we will use it as a replacement for the HDFS sink used in a previous post here. Other then replacing the HDFS sink with a HiveSink none of the previous setup will change, except for Hive table schema which needs to be adjusted as part of the requirements that currently exist around Hive Streaming. So let's get started by looking into these restrict

IntroductionMany organizations have come to rely on Hadoop for dealing with the ever-increasing quantities of data that they gather. Today, it is clear what problems Hadoop can solve, however, cloud is still not the first choice for Hadoop deployment. Pros and cons for Hadoop in the cloud have been shared across multiple blogs and books, but the question is always coming-up in discussions with enterprises considering Hadoop in the Cloud. Thus, I thought it would be useful to collate together a few pros and cons, as well as mention a pragmatic approach to consider a hybrid cloud for organizations that have made significant investments on-prem. For organizations going to Hadoop for the first time, Cloud is probably a better bet especially if they don't have a lot of IT expertise and a great stream of revenue exists and needs to be exploited immediately.Pro CloudLack of space. You don’t have space to keep racks of physical servers, along with the necessary power and cooling.Flexibility. I

As part of this article we will see that sometimes we get the following very common error while accessing some services (in the following case while running very basic "hdfs" commands).While running the command we see the error as following:[root@kjss1 ~]# su - hdfs[hdfs@kjss1 ~]$ hdfs dfs -ls /user/admin16/12/16 12:28:27 WARN ipc.Client: Exception encountered while connecting to the server : 16/12/16 12:28:27 WARN retry.RetryInvocationHandler: Exception while invoking GetUserMappingsProtocolClientSideTranslatorPB.getGroupsForUser over kjss2.example.com/10.20.169.238:8020. Not retrying because failovers (15) exceeded maximum allowed (15)java.io.IOException: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host is: "kjss1.example.com/172.26.69.237"; destination host is: "kjss2.example.com":8020; at org.apa

- In this article we will see that in general how ambari makes use of the Python "urllib2" and "json" (simplejson) module to query and parse the API data. - We will need to install the "simplejson" python module on our system. - Now we can simply write following kind of python script "/tmp/NameNodeMetricsDetails.py"import timeimport jsonimport urllib2import base64import pprint # Ambari Admin credentials.USERNAME = "admin"PASSWORD = "admin"# NameNode metrics.query = "http://kjss1:8080/api/v1/clusters/JoyCluster/services/HDFS/components/NAMENODE"connection_timeout = 30.0def get_json_beans(query, connection_timeout): response = None try: raw = "%s:%s" % (USERNAME, PASSWORD) auth = 'Basic %s' % base64.b64encode(raw).strip() request = urllib2.Request(query) request.add_unredirected_header('Authorization', auth) response = urllib2.urlopen(request, timeout=connection_timeout) data = response.read() data_dict = json.loads(data) return data_dict finally: if response is not None

HDFS Snapshots are read-only point-in-time copies of the file system. Snapshots can be taken on a subtree of the file system or the entire file system.Snapshots are very efficient because they only copy data that are changed. We can restore the data to any previous snapshot. Some common use cases of snapshots are Data backup and Disaster recovery.HDFS Snapshot Extension:Falcon will support HDFS snapshot-based replication through HDFS Snapshot extension. Using this feature,create and manage snapshots on source/target directories.Mirror data from source to target for disaster recovery using these snapshots.Perform retention on the snapshots created on source and target.Snapshot replication will only work from single source directory to single target directory.For snapshot to work, we expect users to do the following­Both source and target clusters must have a version of Hadoop 2.7.0 or higher.The user submitting and scheduling the falcon extension should have permissions on both source a

Introduction Recently worked with use case which required heavy xml processing. Instead of writing complex custom code end up achieved everything easily with NiFi. I thought this will be useful of someone interested for XML processing in NiFi. The document in general covers the following. Base64 Encoding and Decoding of XML message.Character Set conversion from UTF to Ascii ISO-8859-1XML validation against the XSD.Split the XML into smaller chunks.Transform XML to JSON.Extract the content and outputs into unique files based on content. This is very generic XML processing flow which can be leveraged across many business use cases which process xml data. Apache NiFi Flow In the sample demo scenario, External system sends the Base64 encoded XML data in file format which is read through GetFile processor. Next Base64EncodeContent processor decoded the Base64 content.Incoming data in UTF-8 format with leading BOM bytes which gets converted to the ISO-8859-1 format using the ConvertCha

I have a Phoenix PutSQL that worked in the past, but not today. From the error box, not much help. Thought maybe Hbase or Phoenix was done. Turned out one field was missing. How to investigate this issue? Digging through the logs I quickly found it: Caused by: java.sql.SQLException: ERROR 218 (23018): Constraint violation. DRONEDATA1.DATEKEY may not be null That would be nice in the error listed in NiFi.If something seems like an odd error, dig into the logs for more detailed descriptions.Also the Summary Screen is pretty awesome and will let you find a lot of information for debugging and fixing problems.The Connections tab will let you sort by Queue Size and you can see if things are backed up.If you click the system diagnostics link you will see two tabs of useful information. The first has memory usage and garbage collection.The System tab has your usage for FlowFile and Content Repositories, processors and process load.For more details, check out the following reference

1. WASB --> https://blogs.msdn.microsoft.com/cindygross/2015/02/04/understanding-wasb-and-hadoop-storage-in-azure/WASB is a storage model which allows storage of data in Blobs within storage accounts/ containers in Azure cloud. 2. DASH --> http://sequenceiq.com/cloudbreak-deployer/latest/azure_pre_prov/ --> This link describes a few scale-out related limits of WASB, and proposes DASH as the solution. DASH is not supported as a storage option, and there are scalability limitations on the number of storage accounts. To quote, "When WASB is used as a Hadoop filesystem the files are full-value blobs in a storage account. It means better performance compared to the data disks and the WASB filesystem can be configured very easily but Azure storage accounts have their own limitations as well. There is a space limitation for TB per storage account (500 TB) as well but the real bottleneck is the total request rate that is only 20000 IOPS where Azure will start to throw errors when

Prior to Version 1.6.2-rc.8 version of Cloudbreak, provisioning of HDP Clusters on Azure was by Public IP only. This article describes provisioning of a cluster based on Private IP. While creating a Network within Cloudbreak, there are options to 1. Create a New Network and a new subnet2. Use an existing subnet in an existing virtual networkUnder the Option 2 "Use an existing subnet in an existing virtual network" There are 2 options added , a) Don't create public IPsb) Don't create new firewall rulesUpon selecting these options ( and using Private IP "export PUBLIC_IP=xx.xx.xx.xx" in the Profile file for Cloudbreak, ), Cloudbreak can install clusters using Private IP only in Azure cloud. A screen shot of this feature is attached here --> networkoptions.png

A usual query that comes up is on using snappy and other compression codes when loading data from hdfs using the nifi puthdfs component. A common error that users come across is "java.lang.UnsatisfiedLinkError". This error occurs because snappy and other compression codecs, which are part of the native linux binaries and which the java libraries for these codecs utilize to do the actual compression, are not in the path of the JVM. Since the JVM cannot find them the bindings fails and you get a java.lang.UnsatisfiedLinkError.Follow the following steps to resolve this issue.1. copy the native folder containing the compression libraries, from one of your hadoop nodes. cd /usr/hdp/x.x.x.x-xx/hadoop/lib/tar -cf ~/native.tar native/2. scp the native.tar from your hadoop node to your NiFi node, untar to a location of your choice. in my case i use /home/myuser/hadoop/ cd ~ mkdir hadoop cd hadoop tar -cf /path/to/native.tar3. go to your nifi folder , and open conf/bootstr

To upgrade JDK across all HDP services on all nodes using Ambari, the following instructions have proven to be helpful. This requires downtime on ALL HDP components.a. Use the 'Stop all' from the Actions button off the main Ambari dashboard to shut down all HDP components. b. Using Clush/Pdsh/Cluster scripting tool of choice, confirm that ALL Java processes are fully stopped across ALL cluster nodes (excluding Ambari), killing any orphaned processes that Ambari was unable to stop. c. Run the following from Ambari master node (as root) to stop the Ambari-agent and Ambari-server processes. service ambari-agent stop service ambari-server stop d. Run the following from Ambari-server master node (as root): ambari-server setup (and follow the prompts, example provided below) e.g.: [root@sandbox conf]# ambari-server setup Using python /usr/bin/python2.6 Setup ambari-server Checking SELinux... SELinux status is 'disabled' Customize user account for ambari-server daemon [y/n] (n)? n Adjusting a

Ambari REST APIs to add a component in kerberized cluster. I tried and tested these in Ambari 2.4.1 versionHere is an example to install SECONDARY_NAMENODE1. API to Add a Componentcurl -s -H "X-Requested-By:ambari" --user admin:admin -i -X POST -d '{"host_components":[{"HostRoles":{"component_name":"SECONDARY_NAMENODE"}}]}' http://AMBARIHOST:8080/api/v1/clusters/CLUSTERNAME/hosts?Hosts/host_name=COMPONENTHOST2. API to install the Above added component.curl -s -H "X-Requested-By:ambari" --user admin:admin -i -X PUT -d '{"RequestInfo":{"context":"Install SECONDARY_NAMENODE","operation_level":{"level":"HOST_COMPONENT","cluster_name":"test","host_name":"COMPONENTHOST","service_name":"HDFS"}},"Body":{"HostRoles":{"state":"INSTALLED"}}}' http://AMBARIHOST:8080/api/v1/clusters/CLUSTERNAME/hosts/COMPONENTHOST/host_components/SECONDARY_NAMENODE3. API to provide kerberos credentials.curl -s -H "X-Requested-By:ambari" --user admin:admin -i -X POST -d '{ "Credential" : { "principal" : "admin/admi

Status, Icons, and Color:The Apache Nifi 1.0 release brought its users a welcome refresh of the UI with emphasis on a flatter more modern design that included muting the color palette. While these updates have been well received the community around Apache Nifi requested that we "bring some color back". Well we listened and in Apache Nifi 1.1 we have introduced more color in the UI for easier recognition of status changes and necessary user actions!One area (identified by the community) where color makes sense are the icons. In many cases in the Nifi UI icons provide valuable feedback worthy of drawing the attention from the user. However not all icons meet this requirement. To differentiate it was decided that wherever icons are reporting status or require the user to take an action then the icon should receive color and if an icon invokes an action in the UI then it should not.Let's take a look at the canvas:At first glance you will notice the red bulletins and yellow warning icons "

If you wish to reference a file in S3 from a pig script you might do something like this:set fs.s3n.awsSecretAccessKey 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxx';set fs.s3n.awsAccessKeyId 'xxxxxxxxxxxxxxxxxxxxx';A = load 's3n://<bucket>/<path-to-file>' USING TextLoader;If you're on HDP 2.2.6, you'll likely see this error:Error: java.io.Exception, no filesystem for scheme: s3nThe following steps resolve this issue:In core-site.xml add:<property><name>fs.s3n.impl</name><value>org.apache.hadoop.fs.s3native.NativeS3FileSystem</value><description>The FileSystem for s3n: (Native S3) uris.</description></property>Then add to the MR2 and/or TEZ class path(s):/usr/hdp/${hdp.version}/hadoop-mapreduce/*These configs ensure 2 things:That the worker YARN containers spawned by pig have access to the hadoop-aws.jar fileThat the worker YARN containers know which class implements the file system type identified by "s3n://"References: Apache Mailing Lis