Community Articles

Note: HDP 2.5 is used, but can be updated for HDP 2.6 and beyond.Boot an Amazon Linux instance with at least 16GB of RAMExecute the following. It will take a while ## install dockersudo yum update -ysudo yum install -y docker## fix docker for importing the large sandbox imagesed -i.backup 's/\(^OPTIONS=.*\)"$/\1 --storage-opt=dm.basesize=20G"/' /etc/sysconfig/docker## start dockersudo service docker start## confirm docker is workingsudo usermod -a -G docker ec2-userdocker info## download docker imagecurl -O http://hortonassets.s3.amazonaws.com/2.5/HDP_2.5_docker.tar.gz## load docker imagedocker load -i HDP_2.5_docker.tar.gz## confirm image is availabledocker images## get sandbox docker startup scriptcurl -O https://raw.githubusercontent.com/hortonworks/tutorials/hdp-2.5/tutorials/hortonworks/hortonworks-sandbox-hdp2.5-guide/start_sandbox.sh## start sandboxbash start_sandbox.sh## configure to start at bootecho "bash /root/start_sandbox.sh" >> /etc/rc.local## Print the URL for acce

Automating Social Media: Sending Tweets with Apache NiFi: Building A Smart BotThis is the new processor, PutTwitterProcessor, it will use Twitter4J to send tweets from a parameter you set.This is an example tweet and my reply to it. It's a regular tweet.This is the result of a run. I put the tweet id and datetime stamp in the result attribute.Another example:You need to create a Twitter application and obtain the Consumer Key, Consumer Secret, Access Token and Access secret. If you have a lat and long in your data you can send that. It's optional. The message takes expression language and lets you build a message.JUnit Test of my ProcessorJUnit Run ResultsMost of the code is using the very easy to use Twitter4J library.So what can I do with this? In my flow I am ingesting Tweets and with a streaming SQL query I can retweet ones where there's no existing retweet and the followers of the tweet are more than 1,000. I can also feed all this information to a Hive table and

How to set up a Ranger enabled cluster using Ambari-blueprint.In order to setup a cluster using Ambari blueprints we will need a minimum three node cluster.Lets say this cluster contains three hosts as below:secure-ranger-a.comsecure-ranger-b.comsecure-ranger-c.comPre-requisites: To setup a Ranger enabled cluster we need to install ambari-server on one of the hosts which can be the called the gateway node and ambari-agents should be installed on all the nodes including the gateway node.All the ambari-agents should be configured to point to the ambari-server on the gateway node.Note : The operating system used for this setup is Cent-OS.Lets consider the gateway node to be simple-ranger-a.com, to install ambari-server on this host we need to add ambari repository to the OS repositories.To update the OS repository we add ambari.repo file to /etc/yum.repos.d/ directory.Next we do a refresh of the OS repositories using command yum clean allThen we run the command yum install -y ambari-serve

As an extension to the article mentioned here we are using custom Ambari alerts to monitor the current state of the Journal Node edits health. With the default monitoring that is present in the Ambari, we would not be alerted about the failure of edits that may happen in the one of the JN quorum. In typical HDFS HA env, there are three Journal node daemons that are deployed. If any one of the daemons fails to maintain the edits, then we are at risk of failovers and eventual cluster outage if another journal node hits similar issue as other journal node (Because, if quorum of edits are not maintained, then NN fails to be up). Hence, we need to have necessary alerting mechanism put in place for such failures. Journal Nodes may not get updated due to various reasons such as,1. Disk getting full.2. Corrupt Permissions.3. Exhausted HDFS handlers in JN host, etc..Attached are the artifacts, which contains,1. alerts-test.json 2. jn_edits_tracker.pyjn_edits_tracker.py have preconfigured value

To create a simple Java 8 application to use to extract text from PDFs and then identify people's names, I have create a simple data application. This can be used as part of a larger Data Processing Pipeline or HDF flow by calling it via REST, Command Line or converting it to a NiFi Processor.<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd"> <modelVersion>4.0.0</modelVersion> <groupId>com.dataflowdeveloper</groupId> <artifactId>categorizer</artifactId> <packaging>jar</packaging> <version>1.0</version> <name>categorizer</name> <url>http://maven.apache.org</url> <dependencies> <dependency> <groupId>junit</groupId> <artifactId>junit</artifactId> <version>3.8.1</version>

FlowWe can remove Sentiment if you don't want to install my custom processor:This is what they look like:I grab some fields I like:These are fields I want to save:This is a simple version of the flow to just ingest tweets, run sentiment analysis and store in directory as clean JSON.You can drop the sentiment analysis and do it later. You can also run a python script for that.We could make this simpler and just have GetTwitter then PutFile. This will store the RAW Twitter JSON file which is a very sparse nested JSON file. if you want the raw data, that is an option. It's a pain to work with that format and not perfect for analytics. I flatten it and just grab what I have seen as the core attributes, you can add more or drop some of them easily.This is a simple version that could be used for Art or Personal Projects or anyone who wants to store their own tweets and related items.Get Your Twitter ID: https://tweeterid.com/Documentation: https://developer.twitter.com/en/docs/

Tracking Air Quality with HDP and HDF: Part 1 - Apache NiFi IngestPart 2: Plan Data Storage. Store to Apache Hive, Apache Druid and Apache HBase.Part 3: Query and Visualize Data with Apache Zeppelin and SupersetThere was an Air Quality alert a few days ago near me and I was curious how I could keep track of this important environmental information. So NiFi! This data is different from weather data, but makes a lot of sense for analytics to add in data from Weather, Social and locally captured cameras. It's very easy to ingest these JSON and Camera Images via Apache NiFi. In the next section we will analyze the datasets and determine how we can aggregate and accumulate massive quantities of this data for tracking air quality in various areas over time and use that as a dimension with other relevant data like weather.We are tracking contaminants and particles in the air.These include: pm25, pm10 - atmospheric particulate matterso2 - sulfur dioxideno2 - nitrogen dioxideo3

Summary:While automating setup of Hortoniabank demo, we needed to automate the task of associating Atlas tags to HDP entities like HDFS, Hive, HBase, Kafka using the names of entities (rather than their guids in Atlas). One option is to use Atlas APIs to find the entity you are looking for using qualifiedName attribute and then use the guid to associates tag to it. For components like Hive that already have Atlas hook, the Atlas entities for Hive tables will automatically be created when the table is created. For these, we have just provided the API calls to associate the tags with the entity.For others like Kafka, HDFS, Hbase etc that do not have an Atlas hook (as of HDP 2.6.x), you will need to create the entity first. For these, we have provided both the API call to create the entity and the call to associate the tags with the entity.Code samples:The below code examples assume the tags have already been created. these can be created either manually via Atlas UI or using the API. Her

Short Description:This Tutorial describes how to use DeleteHBaseRow processor to delete one or more row keys in HBase table.Thanks alot @Ed Berezitsky helping out to write this article.Introduction:NiFi 1.6 introduced DeleteHbaseRow processor, based on the row key values provided to this Processor, deletes those row key/s in Hbase table. This processor does deleteall operation on the row key that we intended to delete from the hbase table.This Processor works with row key values are presented as FlowFile Content (or) FlowFile attributes.We can select Row ID Location in this processor one of the below ways, by default this property configured to have FlowFile content as the value.1. FlowFile content -Get the row key(s) from the flowfile content.2. FlowFile attributes -Get the row key from an expression language statement. 2.1. Having one row key value associated with the flowfile.2.2. Having more than one row key values associated with the flowfile.DeleteHBaseRow Processor Params:Clien

Converting CSV Files to Apache Hive Tables with Apache ORC FilesI received some CSV files of data to load into Apache Hive. There are many ways to do this, but I wanted to see how easy it was to do in Apache NiFi with zero code.I read CSV files from a directory of files. Then I can Convert the CSV to AVRO directly with ConvertRecord.I will need a schema, so I use the below settings for InferAvroSchema. if ever file is different, you will need to do this every time.CSV ReaderI use the Jackson CSV parser which works very well. The first line of the CSV is a header. It can figure out the fields from the header.Once I have an Apache AVRO file it's easy to convert to Apache ORC and then store in HDFS.Template:csvprocess.xml

Assume you have a MySQL database server running on mysqlserver.example.com. Summary of the steps MySQL set up Set up a grafana database on mysqlserver.example.com Create a MySQL user called grafana, assign and flush privileges Edit the grafana.ini configuration file to enable use of MySQL (Edit the Advanced ams-grafana-ini section in the Ambari Metrics Config tab in Ambari) Start the Grafana Starting Grafana creates the grafana tables. The session table however does not get created and the grafana docs say to create it for MySQL and postgres - so create the session table as per the grafana docs http://docs.grafana.org/installation/configuration/ Start Grafana - and all should be well Detailed steps On MySQL server (be sure to change the password): mysql> create database grafana;Query OK, 1 row affected (0.01 sec)mysql> GRANT USAGE ON `grafana`.* to 'grafana'@'mysqlserver.example.com' identified by 'grafanamysqlpasswd';Query OK, 0 rows affected (0.00 se

DataWorks Summit (DWS) is the industry’s Premier Big Data Community Event in Europe and the US. The last DWS was in Berlin, Germany, on April 18th and 19th. This was the 6th year occurence in Europe and this year there was over 1200 attendees from 51 different countries, 77 breakouts in 8 tracks, 8 Birds-of-a-Feather sessions and 7 Meetups. I had the opportunity to attend as a speaker this year, where I gave a talk on “Best practices and lessons learnt from Running Apache NiFi”. It was a joint talk with the Big Data squad team from Renault, a French car manufacturer. The presentation recording will be available on the DWS website. In the meantime, I’ll share with you the 3 key takeaways from our talk.NiFi is an accelerator for your Big Data projectsIf you worked on any data project, you already know how hard it is to get data into your platform to start “the real work”. This is particularly important in Big Data projects where companies aim to ingest a variety of data sources ranging f

Issue:This issue has been observed on an HDP cluster running NiFi service with HDF 3.0.1 managed by Ambari 2.6.1.5.HDF 3.0.1 is supported only with Ambari 2.5.1 version. Therefore, in situation where you upgrade Ambari from 2.5.x to 2.6.y HDF services like NiFi won't start & complain the error shared in the below stack trace.To handle this, you have to upgrade HDF mpack from 3.0.1 to 3.0.2 after the Ambari upgrade is complete. This additional activity is required to get the HDF service working with Ambari 2.6.y. Also, this step is independent of the HDF upgrade that you would have planned separatelystderr:Traceback (most recent call last): File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi.py", line 309, in <module> Master().execute() File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 375, in execute method(env) File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi.py

ETL With Lookups with Apache HBase and Apache NiFi(Microservices Style ETL)When we are ingesting tabular / record-oriented data, we often want to enrich the data by replacing ids with descriptions or visa-versa. There are many transformations that may need to happen before the data is in a happy state. When you are denormalizing your data in Hadoop and usually building very wide tables you often want descriptions or other data to enhance it's usability. Only one call to get everything you need is nice, especially when you have 100 trillion records.We are utilizing a lot of things built already (https://community.hortonworks.com/articles/146198/data-flow-enrichment-with-nifi-part-3-lookuprecord.html). Make sure you read Abdelkrim's first 3 lookup articles. I added some fields to his generated data for testing. I want to do my lookups against HBase which is a great NoSQL store for lookup tables and generate datasets.First I created an HBase Table to use for lookups.Create HBase

Hive Insert Query OptimizationSome business users deeply analyze their data profile, especially skewness across partitions.There are many other tuning parameters to optimize inserts like tez parallelism, manually changing reduce tasks (not recommended), setting reduce tasks etc.This article focuses on insert query tuning to give more control over handling partitions with no need to tweak any of these properties.Consider a scenario where we are inserting 3 million records across 200 partitions. The file format is ORC and it is further bucketed and sorted for quicker retrieval during select. Apart from writing, bucketing/sorting is going to add additional work for the reducer.Target Tablecreate table if not exists final.final_data_2 (creation_timestamp bigint,creator string,deletion_timestamp bigint,deletor string,subject string,predicate string,object string,language_code string )partitioned by(range_partition bigint)CLUSTERED BY(creation_timestamp) SORTED BY(creation_timestamp) INTO 8

SANDBOX VERSION AFFECTEDHDP 2.6.0.3 Sandbox as identified below.# wget https://downloads-hortonworks.akamaized.net/sandbox-hdp-2.6/HDP_2.6_docker_05_05_2017_15_01_40.tar.gz# md5sum HDP_2.6_docker_05_05_2017_15_01_40.tar.gz886845a5e2fc28f773c59dace548e516 HDP_2.6_docker_05_05_2017_15_01_40.tar.gzISSUEWhen using classic Hive CLI after a while the following error surfaces.[root@sandbox demos]# hivelog4j:WARN No such property [maxFileSize] in org.apache.log4j.DailyRollingFileAppender.Logging initialized using configuration in file:/etc/hive/2.6.0.3-8/0/hive-log4j.propertiesException in thread "main" java.lang.RuntimeException: java.lang.IllegalArgumentException: java.net.UnknownHostException: sandbox at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:547) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.

ProblemWhile using LLAP, (can happen anytime, even though it was working a second ago), you might see a MetaException complaining about incorrect MySQL server syntax. Here is the the error output with 10 retries by default, so this could get lengthy.0: jdbc:hive2://xlautomation-1.h.c:10500/defa> analyze table L3_MONTHLY_dw_dimRisk compute statistics for columns;Getting log thread is interrupted, since query is done!Error: Error while compiling statement: FAILED: RuntimeException Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient (state=42000,code=40000)org.apache.hive.service.cli.HiveSQLException: Error while compiling statement: FAILED: RuntimeException Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:277) at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:263) at org.apache.hive.jdbc.HiveStatement.runAsyncOnServer(HiveStatement.java:303) at org.apa

The feature of being able to download client configs after cluster deployment was initially added to Ambari in 1.7.0 version. This feature helps Hadoop admin in following known use cases:This can be helpful if Hadoop admin or some other developer wants to setup a client host that is not being managed by ambari. Ambari as of now (2.5.0 version) does not support multiple cluster and if user environment has multiple clusters with same set of client hosts that needs to be configured to interact with all clusters, then these hosts cannot be added to any one particular ambari cluster and needs to be managed outside of the ambari. In this scenario being able to download all client configs can be a helpful feature to set up these client hosts.Further this feature can also be helpful if user wants to create automation scripts for hadoop application running on an ambari managed cluster without using any existing node of the ambari cluster for running automation code. This will require to do set

We needed to create a data lake of all the companies data, the first set of data was from SQL Server. So using Apache NiFi 1.1.x I ingested it into Hive / ORC. A few of the smaller constantly changing tables need to stay in SQL Server so we need to be able to join tables in Hive with tables in SQL Server. Fortunately, Microsoft provides a very cool extension to SQL Server called Polybase that let's us build external tables pointing to Hadoop. Once those tables are referenced they act like regular tables. So now all the companies data including other data sources loaded into Hadoop Hive ORC tables can be queried and joined. And it's fast! Step 1: Apache NiFi MagicQueryDatabaseTable: One for each table picking a sequence id primary key the tables have. Could also do timestamp.ConvertAVROtoORC: Point to /etc/hive/conf/hive-site.xmlPutHDFS: Store in a separate HDFS, write this down as we need it for polybase.ReplaceText (GenerateHiveDDL): Builds create Hive table st

Why should you care about Taxonomy? Taxonomy is the missing link in information management projects. By organizing all corporate data, the usability of knowledge increases considerably!In this tutorial we will explore how Apache Atlas enables you to apply a Taxonomy to the data in you HDP data lake. Log into Apache Atlas.If you don't see a screen that looks like this (With Taxonomy available) You need to enable Taxonomy.To enable Taxonomy Go to Ambari and click on Atlas in the Services menu.Then select Configs and Advanced:Scroll to the bottom and in the Custom application-properties add a property atlas.feature.taxonomy.enable and set to true.Now restart Atlas.You should now see Taxonomy in the Atlas UI.In the Taxonomy tab click on the "..." elipse by Catalog and select "Create Sub-Term".I will create a term for Tagging Twitter related Assets used in the previous tutorials. You are free to create your own terms how you see fit or follow along. Click the Catalog drop down and select th

Vision Thing Part 3: Image AnalyticsOpen Source Computer Vision with TensorFlow, Apache MiniFi, Apache NiFi, OpenCV, Apache Tika and PythonIn preparation for this talk, I am releasing some articles detailing how to work with images. In this oneFor Linux machines I recommend building OpenCV yourself and installing the Python connector.sudo yum install -y https://centos7.iuscommunity.org/ius-release.rpmsudo yum update -ysudo yum groupinstall 'Development Tools' -ysudo yum install cmake git pkgconfig -ysudo yum install libpng-devel libjpeg-turbo-devel jasper-devel openexr-devel libtiff-devel libwebp-devel -ysudo yum install libdc1394-devel libv4l-devel gstreamer-plugins-base-devel -ysudo yum install gtk2-devel -ysudo yum install tbb-devel eigen3-devel -ysudo yum install -y python36u python36u-libs python36u-devel python36u-pip -ypip3.6 install numpycd ~git clone https://github.com/Itseez/opencv.gitcd opencvgit checkout 3.1.0git clone https://github.com/Itseez/opencv_contrib.gitcd opencv_

On HDP 2.4, some services may have corrupted jar and tar.gz files on HDFS. The specific files I have seen broken are as follows: hive.tar.gzmapreduce.tar.gzhadoop-streaming.jarpig.tar.gzspark-hdp-assembly.jarsqoop.tar.gztez.tar.gzAll of these are found in the /hdp/apps/<hdp-version> directory. On my install, they all had zero size (reported as 0.1 kB on HDFS File View). This led to errors in a variety of services, including the following: gzip: /foo/bar/yarn/local/filecache/11_tmp/tmp_mapreduce.tar.gz: unexpected end of filetar: This does not look like a tar archive Error: Could not find or load main class org.apache.spark.deploy.yarn.ExecutorLauncher There may be other errors, but those are the ones I personally experienced. This is fairly easy to fix. Each corrupt file has a healthy version on the local file system. The healthy version must be copied from the local system to HDFS, replacing the corrupt version. For example, to update Tez, perform the following:$ hdfs dfs -rm /h

Problem Description Atlas uses Solr to store Lineage meta information and uses Zookeeper for co-ordination and store/maintain configuration. Due to heavy load on Zookeeper on larger cluster we need to increase timeout for ZK session for some services from default. One such config is for Ambari infra (Solr) zookeeper time on Atlas side. ERROR: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper vb-hortonwork.com:2181/infra-solr within 15000 ms java.util.concurrent.TimeoutException: Could not connect to ZooKeeper vb-hortonwork.com:2181/infra-solr within 15000 ms RESOLUTION: We can increase the session timeout from default 15000 by adding below properties in custom application-properties in atlas -> configatlas.graph.index.search.solr.zookeeper-connect-timeout=60000 atlas.graph.index.search.solr.zookeeper-session-timeout=60000

ENV: HDP 2.5Please follow following steps to configure1. under Ambari-> Knox -> Advanced topologies 2. add the following snippet in advanced topologies <provider> <role>ha</role> <name>HaProvider</name> <enabled>true</enabled> <param> <name>HIVE</name> <value>maxFailoverAttempts=3;failoverSleep=1000;enabled=true;zookeeperEnsemble=rkk3.hdp.local:2181,rkk2.hdp.local:2181,rkk1.hdp.local:2181;zookeeperNamespace=hiveserver2</value></param> </provider> comment out the url from hive service tag<service> <role>HIVE</role> <!-- <url>http://{{hive_server_host}}:{{hive_http_port}}/{{hive_http_path}}</url> --> </service> 3. Restart KNOX4. Open Beeline and connect to HS2beelineBeeline version 1.2.1000.2.5.0.0-1133 by Apache Hivebeeline> !connect jdbc:hive2://rkk1.hdp.local:8443/;ssl=true;sslTrustStore=/var/lib/knox/data-2.5.0.0-1133/security/keystores/gateway.jks;tr

If you’re using Apache Kafka, you know it persists all the messages on disk as a distributed commit log. You might sometimes want to take advantage of that and reprocess some of the messages. Assuming that you want to reprocess all the messages currently stored on your brokers and you set auto.offset.reset to smallest, you can just delete your consumers’ data from Zookeeper. After restarting, your consumers should start from the beginning. But what if you forgot or didn’t want to set auto.offset.reset in you consumers to smallest? Then you can manually set the offsets for each partition for your consumers to the smallest currently available offset.List all topicskafka-topics --list --zookeeper localhost:2181 Get Offsets for the topickafka-run-class kafka.tools.GetOffsetShell --broker-list localhost:9092 -topic vital_signs --time -1 Set the offset manually$ ZOOKEEPER_HOME/zkCli$ ls /consumers/flasfka/offsets/vital_signs$ set /consumers/flasfka/offsets/vital_signs/0 11492949 Verify the n