Community Articles

NiFi Restricted components are those processors, controller services, or reporting tasks that have the ability to run user-defined code or access/alter localhost filesystem data.-The NiFi User guide explains this as follows:-----------------------------------------Restricted components will be marked with a  icon next to their name. These are components that can be used to execute arbitrary unsanitized code provided by the operator through the NiFi REST API/UI or can be used to obtain or alter data on the NiFi host system using the NiFi OS credentials. These components could be used by an otherwise authorized NiFi user to go beyond the intended use of the application, escalate privilege, or could expose data about the internals of the NiFi process or the host system. All of these capabilities should be considered privileged, and admins should be aware of these capabilities and explicitly enable them for a subset of trusted users. Before a user is allowed to create and modify

Update:See here for a Docker on YARN sandbox solution:https://community.hortonworks.com/articles/232540/docker-on-yarn-sandbox.htmlOverviewThis guide has been tested with and without Kerberos on HDP 3.0.1.YARN offers a DNS service backed by Zookeeper for service discovery, but that can be challenging to setup. For a quickstart scenario, I will use docker swarm and an overlay network instead. If your environment is a single host, the networking is even simpler. This configuration is not recommended for production.I will use pssh to run commands in parallel across the cluster based on a hostlist file and a workerlist file. The hostlist file should contain every host in the cluster, and the workerlist file should include every node except for the one chosen to be the docker swarm master node.PrerequisitesInstall HDP 3.0.1 with or without KerberosInstall Docker on every host in the cluster#pssh -i -h hostlist -l cloudbreak -x "-i ~/cloudbreak.pem -o 'StrictHostKeyChecking no'" "echo hostna

Hadoop archives is one of the methodology which is followed to reduce the load on the Namenode by archiving the files and referring all the archives as a single file via har reader.Testing: To understand the behavior of the HAR, we try following example. 1. Create test folders harSourceFolder2 : Where the initial set of small files are stored. Ex. (In HDFS ) /tmp/harSourceFolder2 harDestinationFolder2 : Where the final archived files are stored. Ex. (In HDFS) /tmp/harDestinationFolder2 2. Ingest small files in the source folder.sudo -u hdfs hadoop fs -copyFromLocal /tmp/SampleTest1.txt /tmp/harSourceFolder2NOTE: thid command shows one file "SampleTest1", however in our example we used five files with index extending till 5 (SampleTest5.txt)3. Capture fsck report across the "/" and NN report after small files are ingested.sudo -u hdfs hdfs fsck / -files > ./fsckWhenFilesCreated.txt143 files and directories, 48 blocks = 191 total filesystem object(s).4. Execute hadoop archive c

Introduction This article continues where part 2 left off. It describes how to enable two new features that make the HDFS NameNode more responsive to high RPC request loads. Audience This article is for Apache Hadoop administrators who are familiar with HDFS and its components. If you are using Ambari you should know how to manage services and configurations with Ambari. It is assumed that you have read part 1 and part 2 of the article. RPC Congestion Control Warning: You must first enable the service RPC port (described in part 1) and restart services so the service RPC setting is effective. Failure to do so will break DataNode-NameNode communication. RPC Congestion Control is a relatively new feature added by the Apache Hadoop Community to help Hadoop Services respond more predictably under high load (see Apache Hadoop Jira HADOOP-10597). In part 2, we discussed how RPC queue overflow can cause request timeouts, and eventually job failures. This problem can be mitigated if the NameNo

Introduction Before being able to predict one's Beast Mode Quotient, we must find an easy way to ingest data from multiple different sources, in this case flat files and APIs. Luckily for everyone in the world, HDF, specifically Nifi is designed to handle multiple data inputs and outputs extremely efficiently, as I will detail in this tutorial. A few notes about the choices of technologies selected for this tutorial and series of article: Target - MySQL: I selected MySQL because this is what my BMQ app is going to use, but it could be any target depending on your use-case. Sources - Fitbit, Strava, MyFitnessPal: There are a bunch of options out there to track health. Fitbit and Strava are the only ones offering a comprehensive and public API (looking at you Garmin, and Apple Health; both have no public APIs, and make it not easy to extract data via file, either having to parse .fit files or only exporting the full set of data); as for MyFitnessPal, I could have used Fitbit API (afte

Usually hive treats \n as newline character and everytime we ingest the data it will split single line into multiple lines depending on no of \n in the text.For eg. Step1: Create a table and try ingesting the data using insert into table command.Step2: List the contents of table. We will observe that single line with '\n' in between is split in two lines.WorkaroundThere are two ways to ingest data containing newline character1. Using escape character 2. Using load data inpath commandSolution1: When we use escape character "\" with "\n", hive will ignore \n as newline but will treat it as string. (abcd\ndefg)querying the tableSolution 2: Using "load data inpath" command.Step1: Upload the file on HDFSStep2: Load data from file to table

Recently, I have observed there are lot of questions in HCC on - how to reset Atlas and Is there a way to delete the registered types in Atlas. So, I thought of sharing this article with the community to clarify these queries.To give some context, Atlas uses HBase as its default datastore when its managed by Ambari. So, basically it uses two Hbase tables to store all its metadata.'atlas_titan' : stores all the metadata from various sources.'ATLAS_ENTITY_AUDIT_EVENTS': stores the audit information of the entities in AtlasThe above two table names can be changed using two properties "atlas.graph.storage.hbase.table" and "atlas.audit.hbase.tablename" respectively in the atlas-application.properties.Now, coming back to the actual question - how to wipe-out the metadata from the Atlas? Follow below steps to achieve the same.Stop Atlas via Ambari.In hbase terminal, to disable hbase table, run this command.disable 'atlas_titan'In hbase terminal, to drop hbase table, run this command.drop 'atl

IntroductionThe purpose of this article is to use Apache NiFi to ingest a huge csv file (few million rows) into a Kafka topic.The tricky point when dealing with big file (millions of rows) is that processing the file by NiFi on row-level will generate flow files for every row, this can produce java memory error. As a work around, we need to limit the generated number of rows by splitting the rows on multiple stages. In this article, the file was around 7 million rows, and 6 dividing stages (100K, 10K, 1K, 100, 10 then 1) were used to limit number of generated flow files.The other trick is to increase the “Splits” queue’s “Back Pressure Data Size Threshold” to an adequate size for handling file size, by default it’s 1GB. In this article, 2GB is used. Assumptions & DesignA standalone NiFi and Kafka instance is to be used for this exercise.The following NiFi flow will be used to split the workload of the multi-million row csv file to be ingested by dividing the ingestion into multi-s

In preparation for my talk at the Philadelphia Open Source Conference(https://phillyopensource.splashthat.com/), Apache Deep Learning 201, I wanted to have some good images for running various Apache MXNet GluonCV Deep Learning Algorithms for Computer Vision. See: https://gluon-cv.mxnet.io/Using Apache open source tools - Apache NiFi 1.8 and Apache MXNet 1.3 with GluonCV I can easily ingest live traffic camera images and run Object Detection, Semantic Segmentation and Instance Segmentation.Code: https://github.com/tspannhw/ApacheDeepLearning201It's so easy, I am running multiple on the data to see which gives me the results I like. I am like YOLO which is a one of my old favorites. yolonifitraffic.pydemo_mask_rcnn_nyc.pydemo_deeplab_nyc.pySo we can find the cars and people in these webcams. Use cases can be around traffic optimization, public safety and advertisement optimization. Due to licensing, I thought it better not to show Traffic Camera data here.To industrialize a

With the arrival of the Hive3Streaming processors, performance has never been better between NiFi and Hive3. Below, we'll be taking a look at the PutHive3Streaming (separate from the PutHiveStreaming processor) processor and how it can fit into a basic Change Data Capture workflow. We will be performing only inserts on the source Hive table and then carrying over those inserts into a destination Hive table. Updates are also supported through this process with the addition of a 'last_updated_datetime' timestamp, but that is out of scope for this article. This is meant to simulate copying data from one HDP cluster to another - we're using the same cluster as the source and destination here however.Here is the completed flow. Take note that most of this flow is just keeping track of the latest ID that we've seen, so that we can pull that back out of HDFS periodically and query Hive for records that were added beyond that latest ID. These last two processors, SelectHive3QL and PutHive3Stre

Druid Kafka Integration Service + Hive This tutorial will focus on was of ingesting raw streaming data into druid and exposing the data in hive for high performance analytics. For Reference, this tutorial was built with CentOS 7, HDP 3.0.1 w/ LLAP enabled and Kafka installed and running This tutorial will be divided into 6 sections: Ambari Configuration Kafka Setup Setup Data Stream Generating and Running the Supervisor Spec Hive Integration Create Streaming Druid Data Sources through HiveKafka Integration Service With Kafka Ingestion Service, Druid can guarantee that all records received will the processed and stored. That’s in contrast to the older push method of Tranquility. If an event was received after the window period with tranquility, the record was dropped. This meant that a secondary batch process needed to be run to true up the data. With Kafka Ingestion Service, if an event is received late, a segment will be created for it regardless of when it was received. This is h

Below are some FAQ's which helps you to quickly identify some important info for DPS-DLM deployment.Pre-req's for DPS and DLMDB version- postgres 9.3 to 9.6OS - RHEL 7.0 and aboveAmbari - 2.6.2HDP 2.6.5Distcp should work b/w source and target clusters.Beacon user should be created in AD there is no choice of using custom user in this DLM 1.1 releasethe onboarded service user for your application should exists in AD and needsto be resolved(id <username> on both source and target clusters .docker versionRequired ports needs to be open b/w source and target clusters and also to access DPS UIwhere to install DLM and DPS software components?DLM engine needs to be installed as m-pack on both clusters using Ambari serverDLM app is dockerized container needs to be installed on DPS hostWhich URL needs to be given to register cluster in DPS UIthe Ambari URL integrated with knox http://<>:8443I'm unable to see the DLM icon in DPS UI after enabling DLM component in DPSUser needs to be

Introduction Knowing how and when to leverage cloud infrastructure and when to use on premises equipment is at the heart of today's mature enterprise data strategies. I decided to dive deeper into the subject, and, like I did with the Personality Detection data science and engineering platform I built, I wanted to use an example to illustrate how to approach how to architect a hybrid cloud infrastructure. These types of endeavors can be quite theoretical (read hard to follow and potentially boring), so I decided to attach this hybrid cloud platform demonstration to something very concrete: my own health. I signed up for a marathon that will take place March 29, and I am starting to train November 5th. I plan to use my training, sleep, nutrition and general health data to evaluate and eventually predict my perceived level of fatigue that I coined under the term: "Beast Mode Quotient" or BMQ. This article is an introduction to the architecture and data flows I will put in place. It

My Environment details:HDP Version - 2.4.2.0Ambari - 2.2.2.0Mysql Version - 5.1.34.Hive shell was taking lot of time to load and when I checked Hive Metastore logs, I found below exception:com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'OPTION SQL_SELECT_LIMIT=DEFAULT' at line 1 .Complete StackTrace:NestedThrowablesStackTrace: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'OPTION SQL_SELECT_LIMIT=DEFAULT' at line 1 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.re

In this article we will see how to add sample users and enable password based authentication for the Zeppelin UI.By default when we access Zeppelin we see that we are able to access it as "anonymous" user. (means users are not challenged to provide credentials).1. Login to Ambari UI and then navigate to :Ambari UI --> Zeppelin Notebook --> Configs --> Advanced --> Advanced zeppelin-shiro-ini2. Then add the users inside the "[users]" section as following:[users]# List of users with their password allowed to access Zeppelin.# To use a different strategy (LDAP / Database / ...) check the shiro doc at http://shiro.apache.org/configuration.html#Configuration-INISectionsadmin = adminuser1=user1pwduser2=user2pwd3. Also edit the "[urls]" section and add "authBasic" as following to tell which all URL patterns needs to be protected:[urls]# anon means the access is anonymous.# authcBasic means Basic Auth Security# To enfore security, comment the line below and uncomment the next one /

This article is designed to extend my articles Twitter Sentiment using Spark Core NLP in Apache Zeppelin and Connecting Solr to Spark - Apache Zeppelin Notebook I have included the complete notebook on my Github site, which can be found on my GitHub site. Step 1 - Follow the tutorial in the provide articles above, and establish an Apache Solr collection called "tweets" Step 2 - Verify the version of Apache Spark being used, and visit the Solr-Spark connector site. The key is to match the version of Spark the version of the Solr-Spark connector. In the example below, the version of Spark is 2.2.0, and the connector version is 3.4.4%spark2scsc.version</p><p>Step 3 - Include the Solr-Spark dependency in Zeppelin. Important note: This needs to be run beforethe Spark Context has been initialized</p><pre>%dep z.load("com.lucidworks.spark:spark-solr:jar:3.4.4")//Must be used before SparkInterpreter (%spark2) initialized//Hint: put this paragraph before any S

I've just stared the third edition of the fastai course with Jeremy Howard. There’s been a lot of buzz around fast.ai and how the non-profit is making deep learning accessible to hundreds of thousands of developers. The latest mention was in the Economist: https://www.economist.com/business/2018/10/27/new-schemes-teach-the-masses-to-build-ai. For me the most exciting part of the course is learning how to get cutting edge results (in the 90%+ accuracy range) with just a few lines of code using fastai library methods that have best practices baked in.Below, I'll present a few lines of code that allow you to quickly classify different breeds of cats and dogs. You may recall that distinguishing between cats and dogs was big just a few years ago but now it's too easy. Thus, we’re using the Oxford-IIIT Pet Dataset http://www.robots.ox.ac.uk/~vgg/data/pets/.The code example references the latest fastai library v1.0.x built on top of PyTorch. See the github repo for more details https://github

Objectives HDPSearch 4.0 was recently announced (Blog) which upgrades Solr from 6.6 to 7.4. The HPDSearch 4.0 Ambari management pack will install HDPSearch 3.0 for HDP 2.6 and HDPSearch 4.0 for HDP 3.0. HDP 3.0 is required for HDPSearch 4.0 because the HDFS and Hive libraries have been updated for Hadoop 3.1. Using Cloudbreak 2.8 Tech Preview (TP) you can install an HDP 3.0 cluster that includes HDPSearch 4.0 using Cloudbreak's management pack extensions. Cloudbreak 2.8 is a Tech Preview release and is not suitable for production usage. Similarly CB 2.8 TP doesn't officially support deploying HDP 3.0 clusters. The intent is to become familiar with the process for when Cloudbreak 2.9 is released. This tutorial is designed to walk you through the process of deploying an HDP 3.0 cluster which includes HDPSearch 4.0 components on AWS using a custom Ambri blueprint. Prerequisites You should already have an installed version of Cloudbreak 2.8. You can find the documentation on Cloudbreak h

In the previous article, we showed how to deploy a cluster running HDF 2.x or 3.x. Here we will look into enabling SSL for Apache Nifi on the cluster setup previously and optionally setup identity mappings. This approach also sets up users/authorizations using Nifi's file-based authorizer (as opposed to Ranger based authorizer).Tested with HDF 2.x, 3.0, 3.21. Configure Nifi for SSL There are 2 options for configuring SSL for Apache Nifi via Ambari:i). Use Nifi CA to generate self-signed certificates (good for quick start/demos)ii). Use existing certificates (used in production envs)Option i) - Use Nifi Certificate Authority (CA) to generate self-signed certificates:Assuming Nifi CA is already installed (via Ambari when you installed NiFi), you can make the below config changes in Ambari under Nifi > Configs > “Advanced nifi-ambari-ssl-config” and click Save to commit the changes: a) Enable SSL? Check boxb) Clients need to authenticate? Check boxc) NiFi CA Token - Set this to long

This article is designed to extend the great work by @Ali Bajwa: Sample HDF/NiFi flow to Push Tweets into Solr/Banana, HDFS/Hive and my article Connecting Solr to Spark - Apache Zeppelin Notebook I have included the complete notebook on my Github site, which can be found on my Github site. Step 1 - Follow Ali's tutorial to establish an Apache Solr collection called "tweets" Step 2 - Verify the version of Apache Spark being used, and visit the Solr-Spark connector site. The key is to match the version of Spark the version of the Solr-Spark connector. In the example below, the version of Spark is 2.2.0, and the connector version is 3.4.4 %spark2scsc.versionres0: org.apache.spark.SparkContext = org.apache.spark.SparkContext@617d134ares1: String = 2.2.0.2.6.4.0-91 Step 3 - Include the Solr-Spark dependency in Zeppelin. Important note: This needs to be run before the Spark Context has been initialized. %depz.load("com.lucidworks.spark:spark-solr:jar:3.4.4")//Must be used before SparkI

The idea of this article is to help admins detect artifacts(files/folders) in the cluster that are older than certain days. Also, in certain cases, there may be empty directories that are lying in the cluster which are not used and hence contribute to the small file issue. Hence, we have the attached script which performs, 1. Identifies files older than X days. 2. Identifies folders older than X days. 3. Deletes empty folders.Script ExecutionScript name is "findAll.sh" which expects 2 parameters which is 1. Age of the artifact (file/folder) in terms of days. 2. Actual location of the artifact (file/folder) in HDFS. Based upon the type of artifact and kind of operation you would have to choose one of the three operations. NOTE: 1. Please make sure the user running the script has permissions to execute the command on the artifacts that is passed as parameter to script. 2. Also, running this script once may take some time based upon the size/hierarchy of the folders. But once

Objectives The release of Cloudbreak 2.7 enables you to deploy Hortonworks Data Flow (HDF) clusters. Currently there are two HDF cluster types supported: Flow Management (NiFi) and Messaging Management (Kafka). Cloudbreak expects HDF clusters to be deployed with security (LDAP, SSL). However, for testing purposes, many people would like to deploy a cluster without having to go through the steps of setting up SSL, LDAP, etc. Therefore, we'll need to modify the default HDF Flow Management blueprint to loosen the security configuration. This is not recommended for production use cases. This tutorial is designed to walk you through the process of of deploying an HDF 3.1 Flow Management Cluster on AWS using Cloudbreak 2.7 using a custom blueprint. Prerequisites You should already have an installed version of Cloudbreak 2.7. You can find the documentation on Cloudbreak here: Cloudbreak Documentation You can find an article that walks you through installing a local version of Cloudbreak wit

When you install HDP cluster and if you select 'MySQL' database to store ambari or hive metastore or Ranger or oozie databases. The important aspect is to setup MYSQL replication and below step will illustrate how to setup MYSQL database replication.Mysql Master-Master-Active-Passive Replication : ================================================ step 1- Configure Master-Slave Replication ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ On server A: ------------1) Shutdown the mysql server process if its running. #mysqladmin -u root shutdown -p2) Edit my.cnf file with following values:# cat /etc/my.cnf [mysqld] datadir=/hadoop/mysql socket=/var/lib/mysql/mysql.sock user=mysql # Disabling symbolic-links is recommended to prevent assorted security risks symbolic-links=0 # New parameters for Ranger partitioned tables innodb_file_per_table=1 innodb_buffer_pool_size=4G innodb_flush_log_at_trx_commit=2 innodb_flush_method=O_DIRECTlog_bin=mysql-bin binlog_format=ROW server_id=10 innodb_support_xa=

Configure Livy in Ambari Until https://github.com/jupyter-incubator/sparkmagic/issues/285 is fixed, set livy.server.csrf_protection.enabled ==> false in Ambari under Spark Config - Advanced livy-confInstall Sparkmagic Details see https://github.com/jupyter-incubator/sparkmagic Install Jupyter, if you don't already have it: $ sudo -H pip install jupyter notebook ipython Install Sparkmagic: $ sudo -H pip install sparkmagic Install Kernels: $ pip show sparkmagic # check path, e.g /usr/local/lib/python2.7/site-packages $ cd /usr/local/lib/python2.7/site-packages $ jupyter-kernelspec install --user sparkmagic/kernels/sparkkernel $ jupyter-kernelspec install --user sparkmagic/kernels/pysparkkernel Install Sparkmagic widgets $ sudo -H jupyter nbextension enable --py --sys-prefix widgetsnbextensionCreate local Configuration The configuration file is a json file stored under ~/.sparkmagic/config.json To avoid timeouts connecting to HDP 2.5 it is important to add "livy_server_

Motivation:When Knox is configured for perimeter security, the end users need to depend heavily on cURL tool or the browser to access the Hadoop services exposed via Knox. Similarly, Hive queries can be submitted by using WebHCat (Templeton) service via Knox. User can also set various parameters required for Hive job to run correctly.cURL Command Syntax:Here's the cURL command syntax which can be used to submit a Hive Job via Knox:$curl -ivk -u <username>:<password> -d <Hive parameters> [-d ...] https://<knox-server-FQDN>:8443/gateway/<topology>templeton/v1/hive"Complete list of Hive parameters can be found in WebHCat cURL Command Reference.The most important Hive parameters are: Hive Query : -d execute="<Hive-Query>" ORHive Program : -d file="/hdfs/path/to/hive/program"Specifies a Hive query string using 'execute' OR HDFS file name of Hive program to run using 'file'. It is mandatory to provide either "execute" OR "file" option. Hive Configuration :