Resources

Starting Assumption: Your Nifi data flow similar to the TwitterSolr.xml template on the Nifi website(Nifi Templates). Here is the template workflow (MergeContent is optional) :In order to add sentiment analytics, we need to alter the data flow by updating the flowfile (the actual data being passed in Nifi) with attributes that represent the sentiment score of the tweet in question.Script Components: 1)Importing the JSON and accessing tweet text2)Running a sentiment analyzer on the tweet text and getting sentiment scores3)Appending sentiment scores to JSON and writing back to the flowfile***Note***: For the actual script itself I used Groovy (a Java-compatible scripting language), but you can use other scripting languages such as Lua, Python, Ruby, or ECMAScript (Script Code). For the sentiment analytics portion, I used a Java port of the Python sentiment analytics module vaderSentiment, but there are a plethora of open-source sentiment analytics APIs such as Python’s NLTK module, Stan

Note - For this tutorial I assume that you already have HDP cluster kerberized. If you want to configure automated kerberos then please refer to https://community.hortonworks.com/articles/29203/automated-kerberos-installation-and-configuration.html..First you need to configure your Ambari Server for Kerberos configuration, without configuring Ambari for Kerberos, views will not work..Use below commands to configure Ambari for Kerberos:.1 Login to kadmin and create principal for Ambari Server Note - Please replace your REALM in REALM.COMaddprinc -randkey ambari-server@REALM.COM.2 Extract principal created in above step to keytab filexst -k ambari.server.keytab ambari-server@REALM.COM.3 Above command should generate keytab file in current working directory, copy keytab file to /etc/security/keytabs/ locationcp ambari.server.keytab /etc/security/keytabs/ambari.server.keytabNote - Please make sure that, user running ambari server daemon should have read access to ambari.server.keytab.4 Sto

TipsRead each processor directions carefully, inputs and outputs vary greatly from attributes, JSON, JSONPath statements and other entries. JSONPath supports a radically different syntax than NIFI Expression Language.Know your Expression language, there are a lot of useful functions like UUID() and nextint().When you are doing a lot of streaming and testing data, you will find that logs grow huge on your Centos sandbox. I built up gigabytes of logs in Hive, Hadoop and NiFi in just a few days of operations. Check your /var/log every few days and watch the Ambari screens for data usage. When your data gets too large, things can start failing or slowing down due to a lack of temporary space, swap space and space for logs.Delete Test Files For Ever! hdfs dfs -rm -f -skipTrash /twitter/*.jsonClean up junk files.cd /du -hsx * | sort -rh | head -10 ) PutHiveQL requires hiveql.args.1.value and hiveql.args.1.type for all fields. If you are using EvaluateJSONPath, you cannot set your types

Clients normally want a development environment for SQL. They often have Eclipse-based SQL development tools already ( Teradata SQL Editor, Eclipse Data tools platform ). Hue and command line are not always an option.To connect to HDP2.3 ( should work for HDP2.2) with Eclipse:1. Install the Eclipse Data Tools Platform- Download Eclipse from eclipse.org ( for example Luna ) - Select Help->Install new Software- "Work with" the official update location for your release ( Luna )- Install all plugins under "Database Development" and restart Eclipse2. Create an Hive JDBC driver- Open the Data Source Explorer View ( Windows->Show View->Others->Type Data Source Explorer)- Under Database Connections select "New" - Select "Generic JDBC Driver". You should see three tabs ( Name, Jars , Properties )- Create a new Driver with the plus button and give name "Hive"- On Jars add the following jars from your HDP installationFrom HDP_INSTALLATION/hive/lib ( hive-jdbc.jar and commons-loggingx

There is situation when unfortunately and unknowingly you delete /hdp/apps/2.3.4.0-3485 either with skipTrash or without skipTrash then you will be in trouble and other services will be impacted. You will not be able to run hive,mapreduce or sqoop command, You will get following error.Case 1: If you deleted it without skipTrash then it is very easy to recover: [root@m1 ranger-hdfs-plugin]# hadoop fs -rmr /hdp/apps/2.3.4.0-3485rmr: DEPRECATED: Please use ‘rm -r’ instead.16/07/28 01:59:22 INFO fs.TrashPolicyDefault: Namenode trash configuration: Deletion interval = 360 minutes, Emptier interval = 0 minutes.Moved: 'hdfs://HDPTSTHA/hdp/apps/2.3.4.0' to trash at: hdfs://HDPTSTHA/user/hdfs/.Trash/CurrentIn this case it would be very easy to recover it as after deleting it goes to your current dir and you can recover it from there.hadoop fs -put hdfs://HDPTSTHA/user/hdfs/.Trash/Current//hdp/apps/2.3.4.0 /hdp/apps/ Case 2: If you deleted it with -skipTrash then you need to execute following st

I have seen an issue with Application Timeline Server (ATS). Actually Application Timeline Server (ATS) uses a LevelDB database which is stored in the location specified by yarn.timeline-service.leveldb-timeline-store.path in yarn-site.xml.All metadata store in *.sst files under specified location.Due to this we may face an space issue.But It is not good practice to delete *.sst files directly. An *.sst file is a sorted table of key/value entries sorted by key and key/value entries are partitioned into different *.sst files by key instead of timestamp, such that there’s actually no old *.sst file to delete.But to solve the space of the leveldb storage, you can enable TTL (time to live). Once it is enabled, the timeline entities out of ttl will be discarded and you can set ttl to a smaller number than the default to give a timeline entity shorter lifetime.<property> <description>Enable age off of timeline store data.</description> <name>yarn.timeline-service.ttl

I was reviewing some posts related to Pig, and found the following question interesting:https://community.hortonworks.com/questions/47720/apache-pig-guarantee-that-all-the-value-in-a-colum.html#answer-47767I wanted to share an alternative solution using Pentaho Data Integration (PDI), an open source ETL tool, that provides visual mapreduce capabilities. PDI is YARN ready, so when you configure PDI to use your HDP cluster (or sandbox) and run the attached job, it will run as a YARN application.The following image is your Mapper.Above, you see the main transformation. It reads input, which you configure in the Pentaho MapReduce Job (seen below). The transformation follows a pattern, which is to immediately split the delimited file into individual fields. Next, I use a Java Expression to determine if a field is numeric. If not, the we set the value of the field as the String, null. Next, to prepare for MapReduce output, we concatenate the fields back together as a single value and pass th

IntroductionThis article is a complement to Geo-spatial Queries with Hive using ESRI Geometry and Spatial Framework for Hadoop and includes a few more findings, mainly documenting the differences between ST_Geometry functions supported in Hive and those in commercial spatial packages for Oracle, SQL Server or Netezza.The Hive UDF's are modeled after existing implementations of ST_Geometry. Some functions exist only in Hive’s implementation, a few behave different or don’t exist. Additional ST_Geometry Functions in HiveST_GeomFromJSON - Create a geometry from Esri JSONST_AsJSON - Return Esri JSON representation of geometryST_GeomFromGeoJSON - Create a geometry from GeoJSONST_AsGeoJSON - Return GeoJSON representation of geometry (requires geometry-api-java version 1.1)ST_PointZ - constructor for 3D points from direct positionST_SetSRID - set the spatial-reference ID, for a geometry constructed without itST_GeodesicLengthWGS84 - geodesic length in meters rather than in anglesST_Bin, ST_Bi

There are a lot of excellent talks from the summit.Deep LearningApache Spark with Machine Learning like TensorFlowDistributed Deep Learning on Hadoop Clusters (Yahoo)Apache SparkBig Data Heterogeneous Mixture Learning on Spark Integrating Apache Spark and NiFi for Data Lakes (ThinkBig)OperationsZero Downtime App Deployment Using Hadoop (Hortonworks)Debugging YARN Cluster in Production (Hortonworks)Yahoo's Experience Running Pig on Tez at ScaleThe DAP Where Yarn HBase Kafka and Spark go to Production (Cask)Extend Governance in Hadoop with Atlas Ecosystem (Hortonworks)Cost and Resource Tracking for Hadoop (Yahoo)Managing Hadoop HBase and Storm Clusters at Yahoo ScaleOperating and Supporting Apache HBase Best Practices and Improvements (Hortonworks)Scheduling Policies in Yarn (Slides)Future of DataArun Murthy, Hortonworks - Hadoop Summit 2016 San Jose - #HS16SJ - #theCUBEThe Future of Hadoop : An Enterprise View (Slides)StreamingThe Future of Storm (Hortonworks)Streaming ETL for All : Emb

We have noticed production job failures where customer upgraded their hive from .14 (HDP 2.1) to the latest version(>1.2.x) and resulted in critical jobs failing (not to mention the severity 1 case). This is due to the changes in the reserved words between the source and target hive versions.For example, Word 'date' is not a reserved word in Hive.14 but inHive 1.2.1 it is. Same is the case with REGEXP and RLIKE.Here are the reserved keywords which hive.support.sql11.reserved.keywords support. https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-ReservedKeywordsThere are two ways if the user still would like to use those reserved keywords as identifiers:1).use quoted identifiers,2).set hive.support.sql11.reserved.keywords=false.This would be the best option though code change is required. To illustrate option 1,For example,With word "user" being a keyword, We can use it as an identifier like the below SQL.SELECT createddate, `user`.screenname FROM twi

1. Fetch the latest FS Image from the Active NameNode:Look at the (NameNode directories) property in Ambari and copy the latest image to a node with free disk space and memory. (Ex: fsimage_0000000001138083674)2. Load the FS Image:On the node where you copied the FS Image. Run the below commands:export HADOOP_OPTS="-Xms16000m -Xmx16000m $HADOOP_OPTS" nohup hdfs oiv -i fsimage_0000000001138083674 -o fsimage_0000000001138083674.txt &Above command will make the FS Image available on a web server (temporary). 3. Create "ls -R" report from the FS Image:nohup hdfs dfs -ls -R webhdfs://127.0.0.1:5978/ > /data/home/hdfs/lsrreport.txt &This could take some time.Copy the data from /data/home/hdfs/lsrreport.txt to hdfs /user/hdfs/lsr/lsrreport.txt4. Analyze the ls-R output:Create required table, load data, create view and analyze:hive> add jar /usr/hdp/2.3.2.0-2950/hive/lib/hive-contrib.jar; hive> CREATE EXTERNAL TABLE lsr (permissions STRING, replication STRING, owner STRING, ow

As the Apache Hadoop project completed a decade, I will start with a fun fact. Hadoop is not an acronym- it was named after a toy elephant belonging to one of the creator’s son and now, is the fastest growing segment of the data management and analytics market! The data management and analytics market is projected to grow from $69.6B in 2015 to $132B in 2020 and the Hadoop segment is forecasted to grow 2.7 times faster than the data management and analytics market (according to a recent analyst report).Though Apache Hadoop started as a single system for batch workloads, Apache Hadoop is becoming the multi-use data platform for batch, interactive, online and streaming workloads. At the core of the Hadoop eco-system, we have Apache Hadoop File System (HDFS) to reliably store data and Apache Hadoop YARN (Yet Another Resource Negotiator) to reconcile the way applications use Hadoop system resources. While this post is focused on Apache HDFS, I will recommend Vinod Kumar Vavilapalli’s blog

You may run into slow Hadoop service start on your OS X development laptop. You can check this by opening up your service logs and looking for large (5-10 second) gaps between successive log entries at startup.DiagnosisIt often manifests as test failures for MiniDFSCluster-based tests that use short timeouts (<10 seconds).Here is an example from a NameNode log file with a 5 second stall at startup.2016-07-25 14:57:37,982 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties2016-07-25 14:57:43,060 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).Another 5 second stall during NameNode startup.2016-07-25 14:57:48,790 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true2016-07-25 14:57:53,914 INFO org.apache.hadoop.util.GSet: Computing capacity for map INodeMapResolutionIf you see this behavior you are likely running into an OS X bug. The fix is to put all your entr

This tutorial has been successfully tried on HDP-2.4.2.0 and Ambari 2.2.2.0.I have my HDP Cluster Kerberized with Namenode HA..Please follow below steps for Configuring File View on Kerberized HDP Cluster..Step 1 - Please configure your Ambari Server for Kerberos with the steps mentioned in below article. Please follow steps 1 to 5.https://community.hortonworks.com/articles/40635/configure-tez-view-for-kerberized-hdp-cluster.html.Step 2 - Please add below properties to core-site.xml via Ambari UI and restart required services..Note - If you are running Ambari Server as root user then add below propertieshadoop.proxyuser.root.groups=*hadoop.proxyuser.root.hosts=*.If you are running Ambari server as non-root user then please add below properties in core-site.xmlhadoop.proxyuser.<ambari-server-user>.groups=*hadoop.proxyuser.<ambari-server-user>.hosts=*Please replace <ambari-server-user> with user running Ambari Server in above example..I'm assuming that your ambari serve

Using the GetHTTP Processor we grab random images from the DigitalOcean's Unsplash.it free image site. I give it a random file name so we can save it uniquely in HDFS.The Entire Data Flow from GetHTTP to Final HDFS storage of image and it's metadata as JSON.ExtractMediaMetaData ProcessorThe final results:hdfs dfs -cat /mediametadata/random1469112881039.json{"Number of Components":"3","Resolution Units":"none","Image Height":"200pixels","File Name":"apache-tika-3181704319795384377.tmp","Data Precision":"8 bits","File Modified Date":"Thu Jul 21 14:54:43 UTC 2016","tiff:BitsPerSample":"8","Compression Type":"Progressive,Huffman","X-Parsed-By":"org.apache.tika.parser.DefaultParser,org.apache.tika.parser.jpeg.JpegParser","Component 1":"Y component: Quantization table 0, Sampling factors 2 horiz/2vert","Component 2":"Cb component: Quantization table 1,Sampling factors 1 horiz/1 vert","tiff:ImageLength":"200","mime.type":"image/jpeg","gethttp.remote.source":"unsplash.it","Component3":"Cr com

Hortonworks does not use puppet agent with HDP 1.3 but we still use puppet locally on each node invoked by Ambari Agent.    Currently there are no plans to support chef or puppet in a way that IT can manage configuration of HDP clusters from their own scripts.   This article created by Hortonworks Support (Article: 000001181) on 2013-08-30 17:21 OS: n/a Type: Cluster_Admnistration

Oracle has recently changed the way they manage the process of license acceptance on JDK downloads.  That change is preventing the use of a wget and also preventing a download straight onto your disk.  This issue only occurs with gsInstaller.  HMC installation works fine.We suggest to download both the JDK 6u26 32 bit and 64 bit JDKs from Oracle and set them up on each node and point to those directories in the gsInstaller.properties.  http://www.oracle.com/technetwork/java/javase/downloads/jdk-6u26-download-400750.htmlIn gsInstaller.properties, please update the following config settings with the respective install locations:java32home=java64home= This article created by Hortonworks Support (Article: 000001180) on 2012-11-14 16:32 OS: n/a Type: Installation

SYMPTOMS:  Below is a sample query CREATE VIEW salary AS SELECT fname, (CAST (lname AS STRING)) from Employees; The result is: FAILED: IllegalArgumentException replace op boundaries of <ReplaceOp@@17,43:47='lname',<26>,1:43..@17,43:47='lname',<26>,1:43:"`employees`.`lname`"> overlap with previous <ReplaceOp@@13,36:36='(',<266>,1:36..@23,59:59=')',<276>,1:59:"`employees`.`lname`"> hive>   ROOT CAUSE:  Product defect    SOLUTION:  Filed bug BUG-8944 that has been fixed in HDP 1.3.3 This article created by Hortonworks Support (Article: 000001718) on 2013-12-23 13:05 OS: Linux Type: Developing_Code

HBase is just awesome. Yes I will start with that. HBase tuning like any other service within the ecosystem requires understanding of the configurations and the impact (good or bad) it may have on the service. I will share some simple quick hitters which you can check for to increase performance on your phoenix/hbase cluster What is your hfile locality?Do a simple check on ambari should show you the percentage. The metric is Regionserver.Server.percentFilesLocalWhat does this metric mean? More info here. Thanks to @Predrag Minovic it means "That's the ratio of HFiles associated with regions served by an RS having one replica stored on the local Data Node in HDFS. RS can access local files directly from a local disk if short-circuit is enabled. And If you, for example run a RS on a machine without DN, its locality will be zero. You can find more details here. And on HBase Web UI page you can find locality per table."So if your percentage is lower then 70.. time to run major compact

Apache Shiro design is intuitive and simple way to ensure the safety of the application.For more detail on Apache Shiro project go to http://shiro.apache.org/what-is-shiro.html Software design is generally based on user stories to achieve, that is, based on how users interact with the system to design the user interface or service API. For example, a user story will be displayed after a user logs on a button to view personal account information, if the user is not registered, it displays a registration button.This user story implies major application user needs to be accomplished. Even if the user is not a person here, but third-party systems, when coding is also interacting with the system as "user" to deal with.Apache Shiro reaction of this concept in their design by the intuitive notion exposed to developers so that Apache Shiro in almost all applications are easy to use.Outline:Shiro has three main top-level concept: Subject, SecurityManager, Realms. The following diagram describes

This article is first series of three articles, next coming articles with some code and mechanism present in latest version of HBase supporting HBase Replication. HBase ReplicationHbase Replication solution can solve the cluster security, data security, read and write separation, operation and maintenance, and the guest operating errors, and so the ease of management and configuration, provide powerful online applications support.Hbase replication currently used in the industry are rare, because there are many aspects, such as HDFS has multiple backup copies in a way to help security HBASE underlying data, and the relatively small number of companies in the cluster size. Another reason the data is not very high degree of importance, such as some logging system or as a second warehouse of historical data to split a large number of read requests. Such data lost to be present or back up at other places (database cluster). For such cases the Slave Replication cluster become dispensable, th

Update: apparently when you initiate a support case resolution capture, let's say for HBase service, it will pull HDFS namenode logs in addition to the HBase logs. You may be faced with the same issue and may have to apply the approach below to overcome timeouts. In SmartSense 1.3.0 this will no longer be an issue. Until then, this is a way to avoid capture time outs. Firstly, lets discuss the difference between capture for analysis and support case resolution. Analysis bundles do not collect service logs. For support cases, you're going to fetch configuration and logs. Then based on how much anonymization you will want to apply, large log files will take a long time to collect. This is especially prominent with HDFS namenode logs. They tend to be big and this is exactly the scenario we're trying to address. Firstly, increase the threshold for agent time out in Ambari. In my case it was 30min. Feel free to raise it up to 2hrs on the Ambari SmartSense Operations page. Then, we're going

OpenHAB - Build your smart home in no time! Welcome to http://www.openhab.org/ A vendor and technology agnostic open source automation software for your home. OpenHAB is a mature, open source home automation platform that runs on a variety of hardware and is protocol agnostic, meaning it can connect to nearly any home automation hardware on the market today. If you’ve been frustrated with the number of manufacturer specific apps you need to run just to control your lights, then I’ve got great news for you: OpenHAB is the solution you’ve been looking for – it’s the most flexible smart home hub you’ll ever find. Source Demo: Go to http://www.openhab.org/getting-started/downloads.html Download Runtime core and Demo files Extract Runtime core files in a directory called openHAB and extract Demo files under OpenHAB. See the following: Now, download smartphone app called openHAB in your smartphone. I am using iOS and once you launch the app then disable DEMO tab and e

Teradata's JDBC connector contains two jar files (tdgssconfig.jar and terajdbc4.jar) that must both be contained within the classpath. NiFi Database processors like ExecuteSQL or PutSQL use a connection pool such as DBCPConnectionPool which defines your JDBC connection to a database like Teradata. Follow the steps below to integrate Teradata JDBC connector into your DBCPConnectionPool:1) Download the Teradata connectors (tdgssconfig.jar and terajdbc4.jar) - you can download the Teradata v1.4.1 connector on http://hortonworks.com/downloads/2) Extract the jar files (tdgssconfig.jar and terajdbc4.jar) from hdp-connector-for-teradata-1.4.1.2.3.2.0-2950-distro.tar.gz and move these files to your NIFI_DIRECTORY/lib/*3) Restart NiFi4) Under your DBCPConnectionPool (Controller > Controller Services), Edit your existing DBCPConnectionPool (if your pool is active, disable it before editing) 5) Under the Configuration Controller Service > Properties, define the followingDatabase Connectio

Apache Kylin originIn today's era of big data, Hadoop has become the de facto standards, and a large number of tools one after another around the Hadoop platform to build, to address the needs of different scenarios.For example, Hadoop Hive is a data warehouse tools, data files stored on HDFS distributed file system can be mapped to a database table and provides SQL queries. Hive execution engine can be converted to SQL MapReduce task to run, ideally suited for data warehouse data analysis. Another example HBase is based on Hadoop, high availability, high performance, column-oriented, scalable distributed storage system, Hadoop HDFS architecture to provide high reliability of HBase underlying storage support.Although existing business analytical tools such as Tableau, etc. are here because of the lack of Hadoop-based distributed analysis engine. But they exist with significant limitations, such as difficult to extend horizontally, can not handle large scale data, but also the lack of s