Community Articles

Where in the world is Tim Spann, The NiFi Guy?I recommend the BU-353-S4 USB GPS, it works well with Raspberry PIs and is very affordable. Connecting this to a RPIWZ, I can run this on a small battery and bring this everywhere for location tracking. Put it on your delivery truck, chasis, train, plane, drone, robot and more. I'll track myself so I can own the data. What do these GPS Fields Mean?EPS - Error Estimate in Meter/SecondEPX = Estimated Longitude Error in MetersEPV = Estimated Vertical Error in MetersEPT = Estimated Timestamp Error Speed = Speed !!!Climb = Climb (Positive) or Sink (Negative) rate in meters per second of upwards or downwards movement.Track = Course over ground in degrees from True NorthMode = NMEA mode; values are 0 - NA, 1 - No Fix, 2D and 3D.My question is if you already have an estimate of the error, do something about it!!!Python Walk ThroughFirst install the utilities you need for GPS and Python. We also install NTP to get as accurate time as possib

Here are the steps to get pyspark working on SHCa) add the following into Ambari -> Spark -> Configs -> Advanced spark-env -> spark-env template export SPARK_CLASSPATH=/usr/hdp/current/hbase-client/lib/hbase-common.jar:/usr/hdp/current/hbase-client/lib/hbase-client.jar:/usr/hdp/current/hbase-client/lib/hbase-server.jar:/usr/hdp/current/hbase-client/lib/hbase-protocol.jar:/usr/hdp/current/hbase-client/lib/guava-12.0.1.jar:/usr/hdp/current/hbase-client/lib/htrace-core-3.1.0-incubating.jar:/usr/hdp/2.5.3.0-37/spark/conf/b) kinit as e.g. hbase userc) run$ pyspark --packages com.hortonworks:shc:1.0.0-1.6-s_2.10 --repositories http://repo.hortonworks.com/content/groups/public/d) call each line separately:from pyspark.sql import Rowdata = range(0,255)rdd = sc.parallelize(data).map(lambda i : Row(name=i,age=i))import jsoncat = json.dumps({"table":{"namespace":"default", "name":"dk", "tableCoder":"PrimitiveType"},"rowkey":"key","columns":{"name":{"cf":"rowkey", "col":"key", "type":"

One of the search use cases that I’ve been introduced to would require the ability to index text such as scanned text in png files. I set out to figure out how to do this with SOLR. I came across a couple pretty good blog posts, but as usual, you have to put together what you learn from multiple sources before you can get things to work correctly (or at least that’s what usually happens for me). So I thought I would put together the steps I took to get it to work.I used HDP Sandbox 2.3.Step-by-step guideInstall dependencies - this will provide you support for processing pngs, jpegs, and tiffsyum install autoconf automake libtoolyum install libpng-develyum install libjpeg-develyum install libtiff-develyum install zlib-develDownload Leptonica, an image processing librarywget http://www.leptonica.org/source/leptonica-1.69.tar.gzDownload Tesseract, an Optical Character Recognition enginewget http://tesseract-ocr.googlecode.com/files/tesseract-ocr-3.02.02.tar.gzEnsure proper variables and p

To get an idea of the write performance of a Spark cluster i've created a Spark version of the standard TestDFSIO tool, which measures the I/O performance of HDFS in your cluster. Lies, damn lies and benchmarks, so the goal of this tool is providing a sanity check of your Spark setup, focusing on the HDFS writing performance, not on the compute performance. Think the tool can be improved? Feel free to submit a pull request or raise a Github issueGetting the Spark Jar Download the Spark Jar from here: https://github.com/wardbekker/benchmark/releases/download/v0.1/benchmark-1.0-SNAPSHOT-jar-with-dependencies.jar It's build for Spark 1.6.2 / Scala 2.10.5Or build from from source $ git clone https://github.com/wardbekker/benchmark $ cd benchmark && mvn clean packageSubmit args explains <file/partitions> : should ideally be equal to recommended spark.default.parallelism (cores x instances). <bytes_per_file> : should fit in memory: for example: 90000000. &l

IntroductionRecently I was asked how to Monitor and alert while MonitorDiskUsage or MonitorMemory Reporting Task WARNINGS are generated when it exceeds a predefined threshold, I am trying to capture the steps to implement the same.Prerequisites To test this, Make sure HDF-2.x version of NiFi is up an running.You Already have a MonitorDiskUsage/MonitorMemory Reporting task created and configured, set a lower threshold to force generate the alert while testing. 3. Make a note of the Reporting Task uuid to be monitored:Creating a Flow to Monitor Reporting Task. Drop a GenerateFlowFile to execute every 5mins, with 1 byte file to trigger the flow. 2. Drop an UpdateAttribute processor to the canvas with below configuration:NIFI_HOST : <Your NiFi FQDN>NIFI_PORT : <Your NiFi http port>REPORTING-TASK-UUID : <Your MonitorDiskUsage or MonitorMemory controller service uuid>3. Connect Success relationship of GenerateFlowFile to UpdateAttribute.4. Drop a InvokeHTTP proce

What kind of text or unstructured data are you collecting within your company? Are you able to fully utilize this data to enhance predictive models, build reports/visualizations, and detect emerging trends found within the text? Hadoop enables distributed, low-cost storage for this growing amount of unstructured data. In this post, I'll show one way to analyze unstructured data using Apache Spark. Spark is advantageous for text analytics because it provides a platform for scalable, distributed computing. When it comes to text analytics, you have a few option for analyzing text. I like to categorize these techniques like this: Text Mining (i.e. Text clustering, data-driven topics) Categorization (i.e. Tagging unstructured data into categories and sub-categories; hierarchies; taxonomies) Entity Extraction (i.e. Extracting patterns such as phrases, addresses, product codes, phone numbers, etc.) Sentiment Analysis (i.e. Tagging positive, negative, or neutral with varying levels of sentime

Scenario: Trying to add new columns to an already partitioned Hive table. Problem: The newly added columns will show up as null values on the data present in existing partitions. Solution: One of the workaround can be copying/moving the data in a temporary location,dropping the partition, adding back the data and then adding back the partition. It works and the new column picks up the values. But for big tables this is not a viable solution. Best approach: We need to construct the alter statement to add columns with CASCADE option as follows: ALTER TABLE default.test_table ADD columns (column1 string,column2 string) CASCADE; From the Hive documentation: “ALTER TABLE CHANGE COLUMN with CASCADE command changes the columns of a table's metadata, and cascades the same change to all the partition metadata. RESTRICT is the default, limiting column change only to table metadata.” I found out that this option is not at all in wide use and can help who face this situation. Thanks

Issue:User has a valid kerberos ticket, when he's trying to connect to Hive CLI - the session throws the below error[karthick@mrt1 ~]$ hive WARNING: Use "yarn jar" to launch YARN applications. Logging initialized using configuration in file:/etc/hive/2.4.3.0-227/0/hive-log4j.properties Exception in thread "main" java.lang.RuntimeException: org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown. No cluster diagnostics found. at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:544) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:680) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:624) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.RunJar.run(RunJar.java:221) at org

Ambari version 2.5 provides a new Upgrade History page. This is useful in many ways such as:Determine the history of upgrades that your cluster has undergone (and any issues).Collect any specific logs from within the web UI without the need to login to the respective host.Determine time taken for upgrade, which gives an indication of the time required for the next upgrade.In short, Upgrade planning and troubleshooting are two aspects addressed through this pageHere is how the ‘Upgrade History’ page looks:Besides the above UI, you could also retrieve the entire Upgrade output through a couple of Ambari API calls

In this article I will be explaining about creating oozie custom action node to clone a git repository to a required file path.Git repo location and file path will be taken as input while running the workflow.And we will use ambari's workflow manager view to create a workflow using newly created action node. If you are new to workflow manager, then this would be a good starting point.Details about custom action nodes could be found in oozie docs here.We need to follow some prerequisites before we jump to workflow manager view and start using custom action nodes.Step 1 : Implementing Oozie custom action handler : For this article, I will be creating a custom action which clones a git repository . This implementation extends ActionExecutor class (provided by Oozie) and overrides the required methods. This implementation follows Oozie documentation and implements all required methods.import java.io.File;import org.apache.oozie.ErrorCode;import org.apache.oozie.action.ActionExecutor;import

Sometimes when running Hive on Tez queries, such as "select * from table" large output files are created that may swamp your local disk. Error: Error while processing statement: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, vertexName=Map 1, vertexId=vertex_1444941691373_0130_1_00, diagnostics=[Task failed, taskId=task_1444941691373_0130_1_00_000007, diagnostics=[TaskAttempt 1 failed, info=[Error: Failure while running task:org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for output/attempt_1444941691373_0130_1_00_000007_1_10003_0/file.outat org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:402)......], ....Vertex did not succeed due to OTHER_VERTEX_FAILURE, failedTasks:0 killedTasks:106, Vertex vertex_1444941691373_0130_1_01 [Map 3] killed/failed due to:null]DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVerti

HORTONWORKS : SCRIPT HOW TO DISABLE THP REDHAT 7 ?#!/bin/bash### BEGIN INIT INFO# Provides: disable-transparent-hugepages# Required-Start: $local_fs# Required-Stop: # Author: Amine Hallam# Default-Start: 2 3 4 5# Default-Stop: 0 1 6# Short-Description: Disable Linux transparent huge pages# Description: Disable Linux transparent huge pages, to improve # database performance.### END INIT INFOcase $1 in start) if [ -d /sys/kernel/mm/transparent_hugepage ]; then thp_path=/sys/kernel/mm/transparent_hugepage elif [ -d /sys/kernel/mm/redhat_transparent_hugepage ]; then thp_path=/sys/kernel/mm/redhat_transparent_hugepage else return 0 fi echo 'never' > ${thp_path}/enabled echo 'never' > ${thp_path}/defrag re='^[0-1]+$' if [[ $(cat ${thp_path}/khugepaged/defrag) =~ $re ]] then # RHEL 7 echo 0 > ${thp_path}/khugepaged/defrag else # RHEL 6 echo 'no' > ${thp_path}/khugepaged/defrag fi unset re unset thp_path

Mindwave NeuroskyThe Mindwave Neurosky is a headset that allows you to record your brainwaves using EEG technology. In this article we show you how to ingest these brainwaves with NIFIMindwave Neurosky driver installation for OSX Sierra Download and install the latest driver from http://download.neurosky.com/public/Products/MindWave%20headset/RF%20driver%20for%20Mac/MindWaveDriver5.1.pkg After the driver is installed, download and install the latest MindWave Manager from http://download.neurosky.com/public/Products/MindWave%20headset/RF%20driver%20for%20Mac/MindWave%20Manager4.0.4.zip Launch the MindWave Manager, navigate to "Pairing" section and click the "Search for MindWave", then follow the instructions to pair the headset.Install NIFI on OSX Sierra with Homebrew Install Homebrew from the terminal: /usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)" Install NIFI (time of writing v1.1.2): brew install nifiImport NIFI Flow Template An

The Spark Component Guide and Command Line Installation Guide were updated to reflect new Spark features. Here are links to several of the latest features:Support for Spark 2, documented in several topics including: Installing Spark Using AmbariInstalling and Configuring Apache Spark 2 (manual installation)Running SparkConfiguring Spark2 for Wire EncryptionAutomating Spark Jobs with Oozie Spark ActionUsing Livy with Spark Versions 1 and 2Livy API information, in Submitting Spark Applications Through LivyEnabling Spark SQL user impersonation for the Spark Thrift Server (doAs support), in Configuring the Spark Thrift ServerThe Zeppelin Component Guide was updated with additional details and examples for configuring Zeppelin with LDAP/AD and Kerberos security; see Configuring Zeppelin Security. In addition, the documentation for interpreters and user impersonation was extended. Portions of this information that apply to HDP 2.5 were also added to the Security chapter in the HDP 2.

Please consider the following for this install;IBM Power servers are on centos 7The install is performed by using a non-root userthere is no access to internet or proxy to remote repos, we installed a local repos------------------------------------------------------------prerequisite-------------------------------------------------------------#Check the Maximum Open File Descriptors#The recommended maximum number of open file descriptors is 10000, or more. To check the current value set for the maximum number of open file descriptors, execute the following shell commands on each host:ulimit -Snulimit -Hn#If the output is not greater than 10000, run the following command to set it to a suitable default:ulimit -n 10000#SElinuxsudo setenforce 0sudo sh -c 'sudo sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config'sudo sh -c 'sudo sed -i 's/SELINUX=permissive/SELINUX=disabled/g' /etc/selinux/config'sudo umask 0022sudo echo umask 0022 >> /etc/profile---------------------

GOALTo change the default log location for HUE from /var/log/hue to elsewhere.StepsDefault HUE service writes the log files to /var/log/hue directory. This is somehow enforced in the code which states this The ``log_dir`` will replace the %LOG_DIR% in log.conf. If not specified, we look for the DESTKOP_LOG_DIR environment variable, and then default to the DEFAULT_LOG_DIR.However, trying to set either variables i.e. DESKTOP_LOG_DIR and DEFAULT_LOG_DIR don't seem to work. We can however change the %LOG_DIR% with an absolute path within /etc/hue/conf/log.conf. Here is an output of the log.conf replaced with absolute path for the log directory.#args=('%LOG_DIR%/access.log', 'a', 1000000, 3)args=('/opt/log/hue/access.log', 'a', 1000000, 3)--#args=('%LOG_DIR%/error.log', 'a', 1000000, 3)args=('/opt/log/hue/error.log', 'a', 1000000, 3)--#args=('%LOG_DIR%/%PROC_NAME%.log', 'a', 1000000, 3)args=('/opt/log/hue/%PROC_NAME%.log', 'a', 1000000, 3)--#args=('%LOG_DIR%/shell_output.log', 'a', 1000000

ErrorException in thread "main" java.lang.ArrayIndexOutOfBoundsException: 110 at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.planReadPartialDataStreams(RecordReaderImpl.java:914) at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.readPartialDataStreams(RecordReaderImpl.java:958) at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.readStripe(RecordReaderImpl.java:793) at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.advanceStripe(RecordReaderImpl.java:986) at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.advanceToNextRow(RecordReaderImpl.java:1019) at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.<init>(RecordReaderImpl.java:205) at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.rowsOptions(ReaderImpl.java:598) at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.rows(ReaderImpl.java:585) at org.apache.hadoop.hive.ql.io.orc.FileDump.printMetaDataImpl(FileDump.java:291) at org.apache.hadoop.hive.ql.io.orc.FileDump.printMetaData(FileDump.java:261) at org.apache.hadoop.hi

When running a distcp process from HDFS to AWS S3, credentials are required to authenticate to the S3 bucket. Passing these into the S3A URI would leak secret values into application logs. Storing these secrets in core-site.xml is also not ideal because this means any user with hdfs CLI access can access the S3 bucket to which these AWS credentials are tied.The Hadoop Credential API can be used to manage access to S3 in a more fine-grained way.The first step is to create a local JCEKS file in which to store the AWS Access Key and AWS Secret Key values:hadoop credential create fs.s3a.access.key -provider localjceks://file/path/to/aws.jceks<enter Access Key value at prompt>hadoop credential create fs.s3a.secret.key -provider localjceks://file/path/to/aws.jceks<enter Secret Key value at prompt>We'll then copy this JCEKS file to HDFS with the appropriate permissions.hdfs dfs -put /path/to/aws.jceks /user/admin/hdfs dfs -chown admin:admin /user/admin/aws.jcekshdfs dfs -chmod 400

Problem: There's a known bug in Ambari 2.4 and 2.5 that causes "ambari-server upgrade" to fail if the agent RPM is not upgraded first.E.g. stack trace:Using python /usr/bin/python Setup ambari-server Traceback (most recent call last): File "/usr/sbin/ambari-server.py", line 33, in from ambari_server.dbConfiguration import DATABASE_NAMES, LINUX_DBMS_KEYS_LIST File "/usr/lib/python2.6/site-packages/ambari_server/dbConfiguration.py", line 28, in from ambari_server.serverConfiguration import decrypt_password_for_alias, get_ambari_properties, get_is_secure, \ File "/usr/lib/python2.6/site-packages/ambari_server/serverConfiguration.py", line 36, in from ambari_commons.os_utils import run_os_command, search_file, set_file_permissions, parse_log4j_file ImportError: cannot import name parse_log4j_fileCause: This occurs because os_utils.py and other python files inside of /usr/lib/ambari-agent/lib/ambari_commons are upgraded by the agent's RPM and are used by the server's scripts to find which d

I am sharing this again with the configurations and prerequisites which were missing from above example.This was done using HDP version 2.3.4 on both source and target cluster.Pre-requisites :Source and target clusters should be up and running with required services. HDP stack services required on the clusters are HDFS, Yarn, Falcon, Hive, Oozie, Pig, Tez and Zookeeper as seen below:Staging and working directories should be present on hdfs with falcon users:To do that run below commands on both source and target clusters as falcon user.[falcon@src1 ~]$ hdfs dfs -mkdir /apps/falcon/staging /apps/falcon/working[falcon@src1 ~]$ hdfs dfs -chmod 777 /apps/falcon/stagingCreate source and target cluster entity using falcon CLI as below : For source cluster:falcon entity -type cluster -submit -file source-cluster.xmlsource-cluster.xml :<?xml version="1.0" encoding="UTF-8" standalone="yes"?><cluster name="source" description="primary" colo="primary" xmlns="uri:falcon:cluster:0.1">&

If I approached you on the street and challenged you to name a state capital building from a flashcard image how many capital buildings out of the 50 do you think you could recognize? That would be pretty difficult right? We humans tend to have a narrow scope of knowledge focused around our intimate interactions with day to day life. I would have no problem identifying Atlanta Georgia’s capital building for example because I live in Atlanta and am exposed to it almost daily. If presented with a image of Kansas's capital in Topeka however I would be stumped since I have never been there or even seen a picture of that capital building. The point is experience drives our ability to recognize images. Most millennials tend to also share these “experiences” to social media, blogs, text, etc. Google is dominate at crunching these digital “experiences” from its users and artfully marrying those impressions against validated datasets. Google Vision is a rest API hosted and managed by Google tha

Problem description :When upgrading to HDP 2.5.3 , some queries to Hive were failing Especially when executing Insert Overwrite queries. with the following error :ERROR : Failed with exception java.util.ConcurrentModificationException org.apache.hadoop.hive.ql.metadata.HiveException: java.util.ConcurrentModificationException at org.apache.hadoop.hive.ql.metadata.Hive.moveFile(Hive.java:2883) at ..... etcCaused by: java.util.ConcurrentModificationException Workaround:"Add SORT BY 0" to your query.INSERT OVERWRITE TABLE AAA SELECT A , B , C FROM BBBB SORT BY 0 ;More investigations to follow up ...

Problem summary: Users can end up in a situation where the SQL Engine for all Ambari tables is set to MyISAM instead of InnoDB which is required for Ambari. There are few ways to land up in this situation. Prior to MySQL version 5.5.5 the default engine was MyISAM. It could also be a configured default as a global default in my.cnf which affected Ambari versions before 2.5. AMBARI-18951 addressed this issue by making InnoDB default for new databases (2.5+ Ambari deployments) as well as throwing an explicit error preventing Ambari upgrade if the wrong engine is set in order to avoid landing up in an intermediate non-upgraded state.Following is an sample error that would be encountered on an upgrade path from 2.2.2 to 2.4.2ERROR [main] DBAccessorImpl:830 - Error executing query: ALTER TABLE hostcomponentdesiredstate ADD CONSTRAINT hstcmpnntdesiredstatecmpnntnme FOREIGN KEY (component_name, service_name, cluster_id) REFERENCES servicecomponentdesiredstate (component_name, service_name, cl

Setting up GPU-enabled Tensorflow to work with ZeppelinSometimes we want to do some quick Deep Learning prototyping using TensorFlow. We also want to take advantage of Spark for data pre-processing, scaling, feature extraction while keeping it all in the same place for demo. This step-by-step guide will go through process of setting up those tools to work with each other.My setup: AWS GPU-enabled instance (any g2) Ubuntu 14.04HDP 2.5This tutorial has been partly based on this script: https://gist.github.com/erikbern/78ba519b97b440e10640, but has been heavily updated and modified.First, let’s install and set up some pre-reqs:sudo apt-get update sudo apt-get upgrade -y # choose maintainers versionsudo apt-get install -y build-essential python-pip python-dev git python-numpy swig python-dev default-jdk zip zlib1g-devThen, disable nouveau and update initramfsecho -e "blacklist nouveau\nblacklist lbm-nouveau\noptions nouveau modeset=0\nalias nouveau off\nalias lbm-nouveau off\n" | sudo tee

Hive is having an improvised explain plan with the hive.explain.user set to true. Now the indentation and arrows clearly depict the execution flow. Retrieving the join order needed a lot of back and forth references in the old style which was made as easy as following the arrows now, even with the details of mappers and reducers. Let us take a look at one sample explain plan and retrieve the join order Plan optimized by CBO.Vertex dependency in root stageMap 2 <- Map 1 (BROADCAST_EDGE), Map 4 (BROADCAST_EDGE), Map 5 (BROADCAST_EDGE), Map 6 (BROADCAST_EDGE), Map 7 (BROADCAST_EDGE), Map 8 (BROADCAST_EDGE), Map 9 (BROADCAST_EDGE)Reducer 3 <- Map 2 (SIMPLE_EDGE)Stage-0 Fetch Operator limit:-1 Stage-1 Reducer 3 vectorized, llap File Output Operator [FS_101] Group By Operator [GBY_100] (rows=1 width=736) Output:["_col0","_col1","_col2","_col3"],aggregations:["min(VALUE._col0)","min(VALUE._col1)","min(VALUE._col2)","min(VALUE._col3)"] <-Map 2