Community Articles

Distributed System concepts are derived from the concepts working on a single Operating System (machine) , hence the motivation would be to understand the aspects on an Operating System which will help us understand the bigger and complex architecture of distributed system.Main Idea OS concepts Distributed System concept resource CPU , RAM , Network YARNfilesystem NTFS , ext3 HDFS process java , perl, python processSPARK , MR databasemysql NoSqlauthenticationPAM module KnoxauthorizationNSS module Ranger Holistically Security is based on foundation of Authentication Authorization Audit Securely exchanging data during the above mentioned process is based on the concept of cryptography. Symmetric-keyAsymmetric-keyHashing Symmetric-key Same key is used to encrypt and decrypted the message . Example : AES 2. Asymmetric-key

Objectives Everyday Hortonworks customers are taking advantage of the flexibility and elasticity that cloud platforms provide. For many of these customers, Cloudbreak is used to manage their HDP clusters and to provide autoscaling capability. Cloudbreak's autoscaling features are tied to Ambari Alerts. Ambari ships with a set of alerts out of the box. However, you may want to enable an autoscaling policy based on an alert that Ambari doesn't provide out of the box. The good news is Ambari supports creating custom alerts. Custom alerts created in Ambari are visible to Cloudbreak and usable with Cloudbreak autoscaling policies. A common desire with autoscaling is to scale the cluster based on memory used, cores used, or perhaps the number of running applications. You can use the YARN ResourceManager JMX data to determine these values. For example, you may have a typical cluster with 5 Node Managers. You also know that sometimes your cluster usage will spike and you want to increase the n

This is part 2 of a miniseries. Part 1 can be found here.In part 1 of this miniseries, I showed how a simple line count of a text file can be done using NiFi's record processors. The QueryRecord processor that I used creates a new FlowFile for each query result, which is nice if you are only interested in the result. But what if I want to add the line count as an attribute to the original FlowFile?QueryRecord sends the query result(s) to a relationship that is named after the custom attribute defining the query. The original FlowFile is routed to the relationship named "original". I want to match these two together and I am going to use the Wait/Notify processor pair.The Wait processor holds a FlowFile until it receives a signal, which is marked by the presence of a defined value in a distributed cache. The Notify processor can set such a value when it receives a FlowFile. From the FlowFile that is processed by the Notify processor, attributes can be transferred to the FlowFile that is

Objective This is the second of a two article series on the ValidateRecord processor. The first walks you through a NiFI flow that converts a CVS file into JSON format and validates the data against a given schema. This article discusses the effects of enabling/disabling the "Strict Type Checking" property of the ValidateRecord processor. Note: The ValidateRecord processor was introduced in NiFi 1.4.0.Environment This tutorial was tested using the following environment and components: Mac OS X 10.11.6 Apache NiFi 1.4.0Strict Type Checking Property A useful property of the ValidateRecord processor is "Strict Type Checking". If the incoming data has a Record where a field is not of the correct type, this property determines how to handle the Record. If set to "true", the Record will be considered invalid. If set to "false", the Record will be considered valid. To demonstrate both cases, we need to ingest data that can distinguish between different types (which our CSV data from t

The HDP document present details steps of configuring SSL for each service component. However, more than often the services and corresponding applications work in the composite security environment, such as authentication with wire encryption, saying Kerberos + SSL. The blog in below link provided a solution for Kafka after being ssl secured. The solution is to join the certificate of Kafka brokers in JDK truststore. https://community.hortonworks.com/articles/55154/kafka-ssl-kerberos-cheat-sheet-settingsconsole-com.html Here I want to intensify a bit further, we will talk about the solution that places Kafka brokers' certificates outside of JDK truststore, and methods to testify. And for Kafka client Java application, how it works with ssl configuration, as well as that for Spark streaming successfully connecting ssl-secured Kafka brokers. Before those topics, we need to specify how to conduct ssl configuration for Kafka broker. The environment is HDP 2.6.3.0 with FreeIPA kerberized, S

Apache NiFi supports several authentication mechanisms and provides a pluggable authentication model, allowing anyone to implement a custom identity provider. By default, NiFi always supports two-way SSL as an authentication mechanism, but also provides the ability to use LDAP or Kerberos. This post will demonstrate how to use Kerberos for authentication. Setting up a KDC In order to do anything with Kerberos, the first thing we need to do is setup a KDC. For this post we are going to use the *docker-kdc* project which makes it really easy to run a KDC on any environment.I'm working on OSX and had previously installed docker-machine. The *docker-kdc* project was originally created to work with boot2docker, so to work around that we can do the following:Create a boot2docker machine: docker-machine create --driver virtualbox boot2docker eval "$(docker-machine env boot2docker)" Create /usr/local/bin/boot2docker with the contents provided here https://gist.github.com/sttts/b9c03604be3da01b

Hello, I'm still seeing some people struggling to run their own mapreduce applications using a command line. For those who are not java developers, here is some quick guidance. Let's create a new directory and put our new java extension within it. import org.apache.hadoop.conf.*;import org.apache.hadoop.io.*;import org.apache.hadoop.mapreduce.*;import org.apache.hadoop.fs.Path;import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;import org.apache.hadoop.util.GenericOptionsParser;public class WordCount { public static class TokenizerMapper extends Mapper<Object, Text, Text, IntWritable>{ private final static IntWritable one = new IntWritable(1); private Text word = new Text(); public void map(Object key, Text value, Context context ) throws IOException, InterruptedException { StringTokenizer itr = new StringTokenizer(value.toString()); while (itr.hasMoreTo

In this article we will focus on configuring the IBM Bluemix IoT Cloud to except simulated IIoT messages. We will configure the Bluemix IoT cloud in gateway mode so that devices can be auto registered when IIoT messages are sent from the device. Prerequisites IBM Bluemix IoT account. You can create an account here Hortonworks Data Flow (HDF) 2.x or Apache Nifi 1.1.0 Setup The Gateway Definition on Bluemix IoT Cloud Login to your Bluemix IoT account and navigate to your IoT Dashboard, which will look like this: Click on the IoT icon (which looks like a microchip), which will navigate us to the Devices section. This is where you can define all the devices that you want to communicate with the Bluemix IoT cloud. We will define our gateway device here. Click on "Add Device" Click "Create device type". You will have the option to select a gateway or device type. Each device must have a type. In our case we are going to create a gateway device type called nifi-gateway and click next W

IntroductionIn this article I demonstrate how to use NiFi to manipulate with data records structured in columns, by showing how to perform the following three ETL operations in one flow against a dataset: Remove one or more columns (fields) from the datasetFilter out rows based on one or more field valuesChange field values (e.g convert all nulls to empty characters)These operations are based strongly on use of regular expressions in native NiFi processors. I assume you have basic knowledge of regular expressions and point out the key regular expression pattern needed to work with data structured in columns (e.g to operate on the 3rd , 5th and 6th columns of a delimited file). For transformations more complex than shown here, you use the same flow and regex pattern but more complex expressions inside of the regex pattern. Note that if NiFi starts to feel limited as an ETL tool (extreme complexity of transformations or volumes), consider pig with UDFs or 3rd party ETL tools. Also k

Assumptions:-You have a running HDP cluster with Sqoop installed-Basic knowledge of Sqoop and its parametersIngesting SAP HANA data with SqoopTo ingest SAP HANA data, all you need is a JDBC driver. To the HDP platform, HANA is just another database - drop the JDBC driver in and you can plug & play.1. Download the JDBC driver. This driver is not publicly available - it is only available to customers using the SAP HANA product. Find it on their members-only website and download it.2. Drop the JDBC driver into Sqoop's lib directory. For me, this is located at /usr/hdp/current/sqoop-client/lib3. Execute a Sqoop import. This command has many variations and many command-line parameters, but the following is one such example.sqoop import --connect "jdbc:sap://<HANA_SERVER>:30015" --driver com.sap.db.jdbc.Driver --username <YOUR_USERNAME> --password <PASSWORD> --table "<TABLE_NAME>" --target-dir=/path/to/hdfs/dir -m 1 -- --schema "<YOUR_SCHEMA_NAME>"The '-m 1'

IntroductionBy integrating with LDAP, username/password authentication can be enabled in NiFi. This tutorial provides step by step instructions to setup NiFi - LDAP Authentication via Ambari (Using Knox Demo Ldap Server)Prerequisite1) Assuming you already have HDF-2.x Installed on your VM/Server, Ambari, NiFi is up and running with out security. If not, I would recommend "Ease of Deployment" section of this article to install it [You can also follow this article for Automated installation of HDF cluster or refer hortonworks.com for detailed steps]Setting up Demo LDAP Server1) As HDF and HDP cannot co-exist on a single node, lets download knox zip file from apache for this tutorial for easily setting up an ldap server. Execute below steps for the same after establishing ssh connectivity to the VM/Server (name of my host is node1):# ssh node1# mkdir /opt/knox/# cd /opt/knox/# wget http://mirror.cogentco.com/pub/apache/knox/0.11.0/knox-0.11.0.zip# unzip knox-0.11.0.zip# /opt/knox/knox-0.

IntroductionThis is part 1 of a series of articles on Data Enrichment with NiFi: Part 1: Data flow enrichment with LookupRecord and SimpleKV Lookup Service is available herePart 2: Data flow enrichment with LookupAttribute and SimpleKV Lookup Service is available herePart 3: Data flow enrichment with LookupRecord and MongoDB Lookup Service is available hereEnrichment is a common use case when working on data ingestion or flow management. Enrichment is getting data from external source (database, file, API, etc) to add more details, context or information to data being ingested. It's common that data contains references (ex. IDs) rather than the actual information. These references can be used to query another data source (known as reference table in the relational world) to get other attributes of an entity (location, name, etc). Often, the enrichment is done in batch using the join operation. However, doing the enrichment on data streams in realtime is more interesting.Motivation Enri

Trying to Understand Spark from Map Reduce Perspective . If we look at the history of ETL we started with SQL on RDBMS => Excel => Map Reduce => Pig/Hive => Spark .Its just like evolution of Programming Language from C => C++ => Java . In each stage of evolution the problems of priors were tackled and new features were added . Same holds true for the Evolution of ETL(Extract Transform and Load ) .When thinking of a problem its very important to understand what is the level of abstraction we are trying to find the solution in. For Example : reading keys from a file and doing a lookup from RDBMS we think in terms of the logical flow of events rather than how File buffer is being maintained by JAVA or Operating System.In Big Data Map Reduce makes us think at very low level , Pig/Hive Makes us think at a higher abstraction level of ETL and later translates the them into lower level Map Reduce.Spark Abstraction works a higher abstraction similar to PIG/Hive and in

Abstract: Nimbus metrics are critical to operations as well as development teams for monitoring the performance and stability of Storm applications / topology. Usually most production environments have a metrics / operations monitoring systems including solr, elasticsearch, tsdbs etc. This post shows you; how you can use Collectd to forward these metrics over to your desired metrics environment and alert on them. Solution: Collectd is a standard metrics collection tool that can be run natively on linux operating systems. It's capable of capturing a wide variety of metrics, you can find more information on Collectd here: https://collectd.org/ So to capture Storm nimbus metrics, here's a collectd plugin that needs to be complied and built: https://github.com/srotya/storm-collectd (using Maven). Simply run: mvn clean package assembly:single In addition, you will need to install collectd and ensure that it has Java plugin capability. Here's a great post on how to do that: http:/

Problem:If you have an AD/LDAP environment and using HDP with Ranger, it's critical to review the case in which usernames and group ids are stored in your Directory Services environment. Ranger authorization is case sensitive therefore if the username / group id doesn't match the one returned from Directory (AD/LDAP) authorization will be deniedSolution:To solve this problem Ranger offers 2 parameters that can be set via Ambari. This should ideally be done at install time to avoid the need to re-sync all users. Ranger usersync properties for case conversion are: ranger.usersync.ldap.username.caseconversionranger.usersync.ldap.groupname.caseconversionYou can set these properties to lower or upper; this will make sure that Ranger will store the usernames and groups in the above specified format in it's local database therefore when users login their authorization parameter will match correctly.

SAM’s main goal is to let developers, operators and business analysts build, deploy, manage, monitor streaming applications easily in minutes. In this article, an aspect of SAM that deals with its extensibility will be elaborated upon. Most of the streaming use cases are already covered by the built in SAM processors like Join, Split, Rules, Aggregate, Normalization, etc. However, from our interaction with customers and other users it was realized that apart from providing an easy to use visual interface for the users to build new streaming applications, there exists custom logic that cannot be easily represented using existing processors. Extensibility, thus was always an important consideration while designing SAM. How can one hook their custom processing logic in SAM using Custom Processors(CPs) and User Defined Functions(UDFs)? This is what this article will attempt to explain.CPs and UDFs let you define new functions and custom processing in a way that prevents different people

Apache NiFi provides several processors for receiving data over a network connection, such as ListenTCP, ListenSyslog, and ListenUDP. These processors are implemented using a common approach, and a shared framework for receiving and processing messages.When one of these processors is scheduled to run, a separate thread is started to listen for incoming network connections. When a connection is accepted, another thread is created to read from that connection, we will call this the channel reader (the original thread continues listening for additional connections). The channel reader starts reading data from the connection as fast as possible, placing each received message into a blocking queue. The blocking queue is a shared reference with the processor. Each time the processor executes it will poll the queue for one or more messages and write them to a flow file. The overall setup looks like the following:There are several competing activities taking place: The data producer is writin

When running Apache NiFi behind a proxy there are a couple of key items to follow during deployment. 1) NiFi is comprised of a number of web applications (web ui, web api, documentation, custom ui's, data viewers, etc). So the mapping needs to be configured for the root path. That way all context paths are passed through accordingly. For instance, if only the /nifi context path was mapped, the custom ui for UpdateAttribute will not work since it's available at /update-attribute-ui-<version>. 2) NiFi's REST API will generate URI's for each component on the graph. Since requests are coming through a proxy, certain elements of the URI's being generated need to be overridden. Without overriding the users will be able to view the dataflow on the canvas but will be unable to in modify existing components. Requests will be attempting to call back directly to NiFi, not through the proxy. The elements of the URI can be overridden by adding the following HTTP headers when the proxy gener

If you have not attended a DataWorksSummit, I highly recommend it. It is an amazing event held at three locations a year and is a great community experience. The content is deep and highly technical and you will learn about the current state of the art and what is coming next. It's not just Big Data, but AI, Streaming, Microservices, Containers, Cloud and many other topics that startups and enterprises alike need to know.My topic was a simple talk on using Apache NiFi to ingest and transform various data types.There is a small group forming around my quickly released Inception V3 TensorFlow Apache NiFi Processor, I encourage you to try it and provide feedback, pull requests, bug reports, documentation, unit tests, examples and more. The Java API for TensorFlow is new so this is really basic. Thanks to @Simon Elliston Ball for a major cleanup on it.https://github.com/tspannhw/nifi-tensorflow-processorWhat do we want to do?MiniFi ingests camera images and sensor data Run TensorFl

HDFS per-user Metrics aren't emitted by default. Kindly exercise caution before enabling them and make sure to refer to the details of client and service port numbers.To be able to use the HDFS - Users dashboard in your Grafana instance as well as to view metrics for HDFS per user, you will need to add these custom properties to your configuration.Step-by-step guidePresumption for this guide:This is a HA environment with dfs.internal.nameservices=nnha and dfs.ha.namenodes.nnha=nn1,nn2 in Ambari, HDFS > Configs > Advanced > Custom hdfs-site1. In Ambari, HDFS > Configs > Advanced > Custom hdfs-site - Add the following properties.dfs.namenode.servicerpc-address.<dfs.internal.nameservices>.nn1=<namenodehost1>:8050dfs.namenode.servicerpc-address.<dfs.internal.nameservices>.nn2=<namenodehost2>:8050ipc.8020.callqueue.impl=org.apache.hadoop.ipc.FairCallQueueipc.8020.backoff.enable=trueipc.8020.scheduler.impl=org.apache.hadoop.ipc.DecayRpcScheduleripc

DescriptionLearn how to consume real-time data from the Satori RTM platform using NiFi.BackgroundSatori is a cloud-based live platform that provides a publish-subscribe messaging service called RTM, and also makes available a set of free real-time data feeds as part of their Open Data Channels initiative:https://www.satori.com/docs/using-satori/overviewhttps://www.satori.com/opendata/channelsThis article steps through how to consume from Satori's Open Data Channels in NiFi, using a custom NiFi processor. Note - the article assumes you already have a working version of NiFi up and running.Link to code on github: https://github.com/laurencedaluz/nifi-satori-bundleInstalling the custom processorTo create the required nar file, simply clone and build the following repo with maven:git clone https://github.com/laurencedaluz/nifi-satori-bundle.gitcd nifi-satori-bundlemvn clean installThis will make the following .nar file under the nifi-satori-bundle-nar/target/ directory:nifi-satori-bundle-n

Log Search is a log analysis / monitoring tool which is shipped with Ambari. Log Search has 2 components: Log Search Portal (server + web) and Log Feeder. The second one is responsible to monitor specific log files and ship the parsed log lines into Solr. To define which files should be monitored and how the parsing should work for those files, you will need input config descriptors (list of the input config descriptors are located in /etc/ambari-logsearch-logfeeder/conf/logfeeder.properties, all of them are defined in logfeeder.config.files configuration property, on ambari managed Log Search service, those can be found in logfeeder-properties/logfeeder.config.files configuration entry)If you have a specific custom service (see: https://cwiki.apache.org/confluence/display/AMBARI/Custom+Services), to support that inside Log Search application you will need a *-logsearch-conf.xml (* can be a custom name, Ambari will generate input.config-*.json file based on the name inside /etc/ambari-

Understanding scale issues in AMS (Why does it hapen)The Metrics Collector component is the central daemon that receives metrics from ALL the service sinks and monitors that sends metrics. The collector uses HBase as its store and phoenix as the data accessor layer.In a high level, the metrics collector performs 2 operations related to scale in a continuos basis. Handle raw writes - A raw write is a bunch of metric data points received from services written onto HBase through phoenix. There is no read or aggregation involved. Periodically aggregate data - AMS aggregates data across cluster and across time. Cluster Aggregator - Computing the min,max,avg and sum of memory across all hosts is done by a cluster aggregator. This is called a 'TimelineClusterAggregatorSecond' which runs every 2 mins. In every run it reads the entire last 2 mins data and calculates aggregates and writes back. The read is expensive since it has to read non-aggregated data, while the write volume is smaller s

Working with airbnb's SupersetThis is a very cool open source Analytics platform based on some cool Python.I installed this on a CentOS 7 edge node.sudo yum upgrade python-setuptoolssudo yum install gcc libffi-devel python-devel python-pippython-wheel openssl-devel libsasl2-devel openldap-develpip install virtualenvvirtualenv venv. ./venv/bin/activatepip install --upgrade setuptools pippip install mysqlclient pip install pyhivepip install supersetfabmanager create-admin --app superset2017-01-27 18:15:37,864:INFO:flask_appbuilder.security.sqla.manager:Created PermissionView: menu access on Query Search2017-01-2718:15:37,885:INFO:flask_appbuilder.security.sqla.manager:Added Permission menuaccess on Query Search to role AdminRecognized Database Authentications.2017-01-2718:15:37,907:INFO:flask_appbuilder.security.sqla.manager:Added user adminAdmin User admin created.superset db upgradesuperset load_examplessuperset initsuperset runserver -p 8088The main things you will need are PythonBrow

Installing the Alarm Fatigue Demo via Cloudbreak: There are multiple ways to deploy the Alarm Fatigue Demo via Cloudbreak. Below are four options: 1) Deploy via the Cloudbreak UI a) Login to https://cbdtest.field.hortonworks.comb) Select your credentials – if you credentials don’t exist, create them under “Manage Credentials”c) Once your credentials are selected, click “Create Cluster”d) Make-up a cluster name and choose the Availability Zone (SE) and then click “Setup Network and Security"e) “fieldcloud-openstack-network” should be selected and click “Choose Blueprint”f) Select the Blueprint called “alarm_fatigue_v2”Host Group 1 (Select Ambari Server, alarm-fatigue-demo and pre-install-java8)Host Group 2 (select pre-install-java8)Host Group 3 (select pre-install-java8)g) Click on “Review and Launch”e) Click on “Create and start cluster” (After clicking, the deployment via Cloudbreak will likely take 30-50 minutes, go get a coffee) 2) Deploy via Bash Script (specifying configur