Community Articles

Why Hortonworks Schema Registry?Both Apache NiFi and Hortonworks Streaming Analytics Manager (SAM) are extremely powerful tools in the data-in-motion space. To make them truly enterprise ready, Hortonworks DataFlow (HDF), which is a cutting edge data-in-motion platform, offers a set of enterprise foundational services. Apache Ambari is one example, which provides cluster management and provisioning capabilities; Apache Ranger is another example, which provides a centralized place for users to manage policies/permissions across HDF and HDP components. In HDF 3.0, Hortonworks Schema Registry (SR) became a GA offering as part of the foundational services.SR offers a centralized place to store and manage reusable schemas. This becomes extremely handy when you have data sharing the same schema either generated by different producers, or being delivered to different consumers. Think of data coming from different ATM machines, different trucks, different producers, or being delivered to diffe

For now, it is not possible to encrypt the data during its transmission from Sqoop.This is a known limitation, Jira SQOOP-917 is already in place for the feature.

sqoop-validate.pdf

With HDF 1.1.2.1 release, HDF supports accessing Kerberos enabled kafka topics. For a stanalone NiFi node, following instructions can be used.1. new jass file should be created with following entries.Client { com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true keyTab="./conf/nifi.keytab" useTicketCache=false principal="nifi@EXAMPLE.COM”; };KafkaClient { com.sun.security.auth.module.Krb5LoginModule required useTicketCache=false renewTicket=true serviceName="kafka" useKeyTab=true keyTab="./conf/nifi.keytab" principal="nifi@EXAMPLE.COM";};*** Both the KafkaClient and Client configs should use the same principal and key tab ***2. bootstrap.conf file should be changed to include following line.java.arg.15=-Djava.security.auth.login.config=/<path>/zookeeper-jaas.confRestart NiFi node after changes to bootstrap.conf.3. GetKafka/PutKafka processors should be modified to include a new property security.protocol and value for this property should be set

Use Case :1 . ADDING a member to another group and being able to manage them internally without having to deal with outside or additional products.2 . To be able to easily determine what members reside in what groups instead of having to scroll down page after page to see what members are in what groups especially when you have hundreds of users to keep track of. 3 . To easily administer various groups without having the hassle of creating more and more Active Directory/LDAP associations and having to submit change control requests to other departments for something we should be able to administer on our own. Ranger User Sync Process supports reading users and group information from one of the following sources: Unix Text file - CSV or JSON formatLADP/ADCSV Format :If the filename does not end with .json, each line in the file will be treated as a delimiter separated fields of the following format. Default delimiter is a comma; this can be changed using configuration shown above.user-1

A region is decided to be split when store file size goes above hbase.hregion.max.filesize or according to defined region split policy.At this point this region is divided into two by region server.Region server creates two reference files for these two daughter regions.These reference files are stored in a new directory called splits under parent directory.Exactly at this point, parent region is marked as closed or offline so no client tries to read or write to it.Now region server creates two new directories in splits directory for these daughter regions.If steps till 6 are completed successfully, Region server moves both daughter region directories under table directory.The META table is now informed of the creation of two new regions, along with an update in the entry of parent region that it has now been split and is offline. (OFFLINE=true , SPLIT=true)The reference files are very small files containing only the key at which the split happened and also whether it represents top h

Apache Storm is a distributed real-time computation system for processing large volumes of high-velocity data. Debugging a distributed system is inherently hard since there are many moving parts spread across a large number of hosts in a cluster. Tracing failures to a particular component or a host in the system is hard and requires one to collect and analyze a lot of logs and debug/trace processes running in the distributed cluster. In the past the support for debugging topologies in an Apache storm cluster was very minimal. Users would have to first turn on debug logs, restart the workers, redeploy the topologies and then collect logs from different worker nodes and analyze them. If they wanted to take a jstack dump or profile the workers, they would have to login to the worker hosts and run the commands manually which was painful. If one wanted to inspect the tuples flowing through the topology pipeline, they would have to add special “debug” bolts to log the tuples to some log a

IntroductionFor NLP, mostly I want to do two things, Entity Recognition (people, facility, organizations, locations, products, events, art, language, groups, dates, time, percent, money, quantity, ordinal and cardinal)Sentiment AnalysisSo basically what is it and why don't people like it. These two features are very useful as part of a real-time streaming processing of social, email, logs and semistructured document data. I can use both of these in Twitter ingest via Apache NiFi or Apache Spark. Don't confuse text entity recognition with image recognition that we looked at with TensorFlow previously. You can certainly add that to your flow as well, but that is working with images and not text.My debate with sentiment analysis is do you give numbers, really general terms like Neutral, Negative or Positive. Or do you get more detailed like Stanford CoreNLP which has multiple of each.There are a lot of libraries available for NLP and Sentiment Analysis. The first two decisions

The PutSolrContentStream processor in Apache NiFi makes use of Solr's ContentStreamUpdateRequest which means it can stream arbitrary data to Solr. Typically this processor is used to insert JSON documents, but it can be used to stream any kind of data. The following tutorial shows how to use NiFi to stream data to Solr's Extracting Request Handler.Setup SolrDownload the latest version of Solr (6.0.0 for writing this) and extract the distributionStart Solr with the cloud example: ./bin/solr start -e cloud -nopromptVerify Solr is running by going to http://localhost:8983/solr in your browserSetup NiFiDownload the latest version of NiFi (0.6.1) and extract the distributionStart NiFi: ./bin/nifi.sh startVerify NiFi is running by going to http://localhost:8080/nifi in your browserCreate a directory under the NiFi home for listening for new files:cd nifi-0.6.1mkdir datamkdir data/inputCreate the NiFi FlowCreate a simple flow of GetFile -> PutSolrContentStream -> LogAttribute:The GetFil

Cloudbreak is a cloud agnostic Hadoop As A Service solution. It is easy to install, only one command in the command line, and an other one to start the service itself. During startup Cloudbreak Deployer tries to figure out the underlay infrastructure and sets required environment variables with competent default values, but some cases it isn't enough. In this article i collected all of the possible customizations, default values and some hint what does the variable configure in Profile file. ADDRESS_RESOLVING_TIMEOUT 120000 - DNS lookup timeout for internal service discoveryAWS_ACCESS_KEY_ID "" - Access key of the AWS accountAWS_ROLE_NAME cbreak-deployer - Name of the AWS role for the `cbd aws [generate-rol, show role]` commandsAWS_SECRET_ACCESS_KEY "" - Secret access key of the AWS accountAZURE_SUBSCRIPTION_ID - Azure subscription ID for interactive login in Web UIAZURE_TENANT_ID - Azure tenant ID for interactive login in Web UICAPTURE_CRON_EXPRESSION - SmartSense

The MiniFi flow executes two scripts: one to call TensorFlow Python that captures an OpenCV Raspberry Pi Camera and runs Inception on it. That message is formatted as JSON and sent on. The second script reads GPS values from a USB GSP sensor and outputs JSON. Get File reads the Pi Camera image produced by the ClassifyImages process. Cleanup logs is a standalone timed script that cleans up old logs on the Raspberry Pi.Using InferredAvroSchema I created a schema for the GPS unit and stored it in the Hortonworks Schema Registry.This is the provenance event for a typical GPS message sent. You can see what shell script we ran and from what host.In Apache NiFi we process the message, routing to the correct place, setting a schema and querying it for a latitude. Then we convert the AVRO record to ORC to save as a Hive table.MiniFi requires we change the NiFi created template to a configuration file via the command-line MiniFi Toolkit.minifi-toolkit-0.2.0/bin/config.sh transform

Technology: CSV, AVRO, Hortonworks Schema Registry, Apache NiFi, Streaming Analytics Manager, Kafka, Hadoop HDFSThis is a simple example of reading CSV data, using it's schema to convert it to AVRO, sending it via Kafka to SAM. SAM then reads it and stores it to HDFS. This is a simple flow, but a start to setting up any level of complex flow.A number of questions have come up on how to setup this basic flow.Gotchas: Must send AVRO, beware of nulls, set your schema, put your schema name somewhere, create your kafka topic and make sure you have permissions.Building Sample CSV Datacurl "http://api.mockaroo.com/api/ANID?count=1000&key=AKEY" > "simple.csv"****You need to sign up for mockaroo and create your own schema. You will then need to add the key and account id to the above link.Import this schema to your account<strong>{ "id": 76671, "num_rows": 1000, "file_format": "csv", "name": "simple", "include_header": true, "columns": [ { "name": "first",

haveibeenpwned has downloadable files that contains about 320 million password hashes that have been involved in known data breaches.This site has a search feature that allows you to check whether a password exists in the list of known breached passwords. From a security perspective, entering passwords into a public website is a very bad idea. Thankfully, the downloadable files make it possible to perform this analysis offline.Fast random access of a dataset that contains hundreds of millions of records is a great fit for HBase. Queries execute in a few milliseconds. In the example below, we'll load the data into HBase. We'll then use a few lines of Python to convert passwords into a SHA-1 hash and query HBase to see if they exist in the pwned list.On a cluster node, download the files:wget https://downloads.pwnedpasswords.com/passwords/pwned-passwords-1.0.txt.7zwget https://downloads.pwnedpasswords.com/passwords/pwned-passwords-update-1.txt.7zwget https://downloads.pwnedpasswords.com/

This past year Hadoop celebrated it 10th birthday. At Hadoop Summit we talked about how far Hadoop has come from its origins as a Google whitepaper to its initial implementation at Yahoo to where it is today, powering more than half of fortune 500 companies and being one, if not the most disruptive technologies of this century so far. In that time, we’ve seen platform improvements such as the elimination of the single point of failure in the Name Node, YARN opened up the platform for mixed workloads and the proliferation of new distributed computing frameworks beyond MapReduce and advances in hardware has increased speed and computing density of worker nodes.For those of you not in the know, when you write to HDFS, Hadoop’s distributed file system, your documents and data files are broken into blocks. For each of those blocks 2 additional copies are created and all the blocks are distributed across the data nodes in the cluster. There is a scheme to how the blocks are scattered but it’

PROBLEM:After upgrade from IOP (v 4.2.5) to HDP (v 2.6.x) in a kerberized setup, HSI start fails with following in HSI start log :WARN impl.LlapZookeeperRegistryImpl: The cluster is not started yet (InvalidACL); will retryERROR impl.LlapZookeeperRegistryImpl: Unable to start curator PathChildrenCacheorg.apache.zookeeper.KeeperException$InvalidACLException: KeeperErrorCode = InvalidACL for /llap-sasl/user-hive at org.apache.zookeeper.KeeperException.create(KeeperException.java:121) ~[zookeeper-3.4.6.2.6.2.0-173.jar:3.4.6-173--1] at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) ~[zookeeper-3.4.6.2.6.2.0-173.jar:3.4.6-173--1] at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783) ~[zookeeper-3.4.6.2.6.2.0-173.jar:3.4.6-173--1] at org.apache.curator.utils.ZKPaths.mkdirs(ZKPaths.java:232) ~[curator-client-2.7.1.jar:?] at org.apache.curator.utils.EnsurePath$InitialHelper$1.call(EnsurePath.java:148) ~[curator-client-2.7.1.jar:?] at org.apache.curator.RetryLoop.cal

PROBLEM:After upgrade from IOP (v 4.2.0) to HDP (v 2.6.x) in a kerberized setup, HSI install fails with Configuration parameter 'hive.llap.daemon.service.principal' missing in dictionary.Traceback (most recent call last): File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_server_interactive.py", line 616, in <module> HiveServerInteractive().execute() File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 329, in execute method(env) File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_server_interactive.py", line 75, in install import params File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/params.py", line 27, in <module> from params_linux import * File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/params_linux.py", line 697, in <module> hive_llap_principal = (config['configuratio

PROBLEM:After upgrade from IOP (v 4.2.0) to HDP (v 2.6.x) in a kerberized setup, HSI install fails with Configuration parameter 'hive.llap.daemon.service.principal' missing in dictionary.Traceback (most recent call last): File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_server_interactive.py", line 616, in <module> HiveServerInteractive().execute() File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 329, in execute method(env) File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_server_interactive.py", line 75, in install import params File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/params.py", line 27, in <module> from params_linux import * File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/params_linux.py", line 697, in <module> hive_llap_principal = (config['configuratio

In case of a Blueprint based deployment Ambari stores deployment tasks in topology tables: topology_request, topology_logical_request, topology_host_request, topology_host_task, topology_logical_task.In case of some errors during deployment task creation these tables could be left in an inconsistent state, which has been fixed in 2.5.0 (AMBARI-19929). Ambari 2.5.0 introduced (removed in 2.5.2) two DB consistency checks related to topology tables:1. Check topology_request, topology_logical_request tables are consistent, by comparing the row counts returned:select count(tpr.id) from topology_request tpr;select count(DISTINCT tpr.id) from topology_request tpr join topology_logical_request tlr on tpr.id = tlr.request_id; In case row counts are different it means there’s a missing row in topology_logical_request table. To fix this a new row has to be inserted in topology_logical_request table: get next available: next_topology_logical_request_id = select max(id) + 1 from topology_l

Various open-source tools and applications are available to do performance, scalability and reliability testing for various popular HDP components. Here is a list of some of the popular ones:HDFSTestDFSIOMeasure the I/O performance of HDFS in your cluster. Source code for the tool can be found here.NameNode BenchmarkApplies load on the namenode by performing continuous read, write, rename and delete operations on small files. Source code for this tool can be found here.Synthetic Load GeneratorThe synthetic load generator (SLG) is a tool for testing NameNode behavior under different client loads. The user can generate different mixes of read, write, and list requests by specifying the probabilities of read and write. The user controls the intensity of the load by adjusting parameters for the number of worker threads and the delay between operations. More information on the tool can be found here.YARN/MRTeraSortMeasure performance by measuring time to sort 1TB of data. The test runs in t

Cloudbreak is now available directly from the Azure Marketplace. The goal of this article is to help you get started: quickly launch Cloudbreak from the Azure Marketplace, log in to the Cloudbreak UI and create a Cloudbreak credential using the interactive method. Let's get started!UpdateThis article was written in 2017. As of today, Azure Marketplace no longer includes the latest version of Cloudbreak. Fo the latest version use this quickstart.OverviewWe will be performing the following steps:Meet the Prerequisites.Launch Cloudbreak.Log in to the Cloudbreak UI.Create a Cloudbreak credential using the interactive method. Meet the Prerequisites 1.In order to launch Cloubdreak on the Azure Marketplace, you must have a Microsoft Azure account. If you don't have an account, you can set it up at https://azure.microsoft.com.2.In order to provision clusters on Azure, Cloudbreak must be able to assume a sufficient Azure role ("Owner" or "Contributor") via Cloudbreak credential:Your account mus

Sometimes you want to trigger events with a click on a special touchpad or device mounted somewhere. This could be in a factory, on a door or at your desk.For me it's a small device on my desk that I can use to trigger events. I have this running every 15 seconds looking for gestures. I could put it in an infinite loop, but Python and RPI could leak some memory. We are very constrained, so I am trying to keep it a little more minimal. I keep my batch duration to 15 seconds and my run schedule to 15 seconds.Let's Build A Simple NiFi Flow to Receive The JSON Data and React To ItIn my RouteOnContent, I just look for the word "center". I have thought of many options for running sql, doing a backup, etc.. Build The MiniFi Flow in NiFiDownloaded minifi 0.2.0 and minifi 0.2.0 toolkit (you can use newer, but make sure you install the same version on the device you are going to move your config.yml to).minifi-toolkit-0.2.0/bin/config.sh transform minififlick.xml config.yml Then SCP t

Many organizations still ask the question, “Can I run BI (Business Intelligence) workloads on Hadoop?” These workloads range from short, low-latency ad-hoc queries to canned or operational reporting. The primary concerns center around user experience. Will a query take too long to return an answer? How quickly can I change my mind with a report and drill down other dimensional attributes? For almost 20 years vendors have engineered highly customized solutions to solve these problems. Many times these solutions require fine-tuned appliances that tightly integrate hardware and software in order to squeeze out every last drop of performance. The challenges with these solutions are mainly around cost and maintenance. These solutions become cost-prohibitive at scale and require large teams to manage and operate. The ideal solution is one that affordably scales but retains the same performance advantages as your appliance. Your analysts should not see the difference between the costly a

Very well know way to handle JSON is to use JSON SerDe which originated from hcatalog. There is another interesting rather unconventional method to handle JSON data in HIVE. json_tuple and LATERAL VIEW. Table here only has one column that loads JSON data as a single string. json_tuple() is a User defined Table Function ( UDTF ) introduced in Hive 0.7. It takes a set of names (keys) and a JSON string, and returns a tuple of values using one function.In the words of ASF "A lateral view first applies the UDTF to each row of base table and then joins resulting output rows to the input rows to form a virtual table having the supplied table alias".Consider below is sample data line:{"user":{"location":"","id":1514008171,"name":"Auzzie Jet","screenname":"metalheadgrunge","geoenabled":false},"tweetmessage":"Anthrax - Black - Sunglasses hell yah\n http://t.co/qCNjba57Dm","createddate":"2013-06-20T12:08:44","geolocation":null}Table creation:CREATE EXTERNAL table tweets{ tweet STRING }LOAD dat

In the following article, Ambari Views Server is the Standalone Ambari Server used for hosting Views and Ambari Server is the Operational Ambari Server which manages a Hadoop ClusterBefore Ambari 2.4, when Ambari Views Servers are setup, the only way to configure views was to use ‘Custom Configuration’. In this method details had to be manually entered for several fields. For example, to configure Files View, all the entries as described in Ambari Views Documentaion for 2.2 had to be made.This is made easier in Ambari 2.4 with ‘Remote Cluster Configuration’ . The remote cluster configuration is used to set up an Ambari Views Server to host views for a remote Ambari server, regardless of the cluster configuration.Once a remote cluster configuration is set, all it takes to create a new view is to select the 'Remote Cluster' option in 'Cluster configuration' and select a cluster name in create view wizard. How to define a new remote cluster configuration?Note: If remote Ambari server is e

Previous article demonstrated how to use nifi rest api to create a template. The objective for this article is to demonstrate how to use NiFi rest api to change a work flow on a secured cluster real time without service disruption or downtime.Use CaseA process group named CICD is a active data flow (data is flowing through it) and a new processor will be added at the end of a flow without application disruption or downtime. Comments in the picture above.StepsCreate Access TokenCreate Client IDFetch Process Group attributes (process group which the changes will be applied to)Create LogAttribute ProcessorFetch the UUID of the UpdateAttribute processor Why? Data will flow from this existing processor into newly added LogAttribute Processor Link LogAttribute & UpdateAttribute via success relationshipUpdate LogAttribute to terminate on success relationshipSteps 1 & 2 & 3These steps are described here so no reason to copy and paste.Step 4 - Create LogAttribute Processorpayload