Community Articles

SYMPTOM:Oozie sqoop action fails with below error while inserting data into Hive.20217 [Thread-30] INFO org.apache.sqoop.hive.HiveImport - Sorry ! hive-shell is disabled use 'Beeline' or 'Hive View' instead. Please contact cluster administrators for further information 20218 [main] ERROR org.apache.sqoop.tool.ImportTool - Encountered IOException running import job: java.io.IOException: Hive exited with status 1 at org.apache.sqoop.hive.HiveImport.executeExternalHiveScript(HiveImport.java:389) at org.apache.sqoop.hive.HiveImport.executeScript(HiveImport.java:342) at org.apache.sqoop.hive.HiveImport.importTable(HiveImport.java:246) at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:524) at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:615) at org.apache.sqoop.tool.JobTool.execJob(JobTool.java:243) at org.apache.sqoop.tool.JobTool.run(JobTool.java:298) at org.apache.sqoop.Sqoop.run(Sqoop.java:147) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at org.apac

SYMPTOM When running Hive queries from Resource Manager log the following error is displayed:2017-02-07 15:08:32,140 ERROR impl.MetricsSinkAdapter (MetricsSinkAdapter.java:publishMetricsFromQueue(148)) - Got sink exception, retry in 4600ms org.apache.hadoop.metrics2.MetricsException: Failed to putMetrics at org.apache.hadoop.metrics2.sink.timeline.HadoopTimelineMetricsSink.putMetrics(HadoopTimelineMetricsSink.java:216) at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.consume(MetricsSinkAdapter.java:186) at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.consume(MetricsSinkAdapter.java:43) at org.apache.hadoop.metrics2.impl.SinkQueue.consumeAll(SinkQueue.java:87) at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.publishMetricsFromQueue(MetricsSinkAdapter.java:134) at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter$1.run(MetricsSinkAdapter.java:88) Caused by: java.net.UnknownHostException: http at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketIm

IntroductionUsing NiFi, data can be exposed in such a way that a receiver can pull from it by adding an Output Port to the root process group. For Spark, we will use this same mechanism - we will use the Site-to-Site protocol to pull data from NiFi's Output Ports. In this tutorial we learn to capture NiFi app log from the Sandbox and parse it using Java regex and ingest it to Phoenix via Spark or Directly using NiFi PutSql Processor.Prerequisites1) Assuming you already have latest version of NiFi-1.x/HDF-2.x downloaded as zip file (HDF and HDP cannot be managed by Ambari on same nodes as of now) on to your HW Sandbox Version 2.5, else execute below after ssh connectivity to sandbox is established:# mkdir /opt/HDF-2.1.1# cd /opt/HDF-2.1.1 # wget http://public-repo-1.hortonworks.com/HDF/2.1.1.0/nifi-1.1.0.2.1.1.0-2-bin.tar.gz # tar -xvf nifi-1.1.0.2.1.1.0-2-bin.tar.gz 2) Spark, Zeppelin, YARN and HDFS are Installed on your VM and started. 3) Hbase is Installed with phoeix Query Server.

Article Goals Setup Hive ODBC on Ubuntu 14.x.x with/without Kerberos Setup Hive ODBC on CentOS 6.x with/without Kerberos Enabling Debug logging useful for debugging connectivity issues Notes/Requirements Article assumes that you have a setup of Ubuntu/CentOS server Access to Ubuntu/CentOS repository Have libsasl2-modules-gssapi-mit installedHave "gcc" installed, required for compilation when building from sourceHave "make" installed, required for compilation when building from source Verify if you have a working Ubuntu repository by typing "sudo apt-get update" and see if this returns successfully Within CentOS as a non superuser you can run "sudo yum repolist" wherein the "base" and "epel" repositories should get listed with thousands as value in "status" field Here are the examples of working repository on either systemsThis node should be able to talk to the HiveServer2 via hostname/IP addressEnsure that firewall is disabled (ufw status for Ubuntu) & (service iptables/ip

Part 1: https://community.hortonworks.com/articles/82964/getting-started-with-apache-ambari-workflow-design.htmlPart 2: https://community.hortonworks.com/articles/82967/apache-ambari-workflow-designer-view-for-apache-oo.htmlPart 3: https://community.hortonworks.com/articles/82988/apache-ambari-workflow-designer-view-for-apache-oo-1.htmlPart 4: https://community.hortonworks.com/articles/83051/apache-ambari-workflow-designer-view-for-apache-oo-2.htmlPart 5: https://community.hortonworks.com/articles/83361/apache-ambari-workflow-manager-view-for-apache-ooz.htmlPart 6: https://community.hortonworks.com/articles/83787/apache-ambari-workflow-manager-view-for-apache-ooz-1.htmlPart 7: https://community.hortonworks.com/articles/84071/apache-ambari-workflow-manager-view-for-apache-ooz-2.htmlPart 9: https://community.hortonworks.com/articles/85091/apache-ambari-workflow-manager-view-for-apache-ooz-4.htmlPart 10: https://community.hortonworks.com/articles/85354/apache-ambari-workflow-manager-view-

Imagine we have a use case of exchanging data with an external party via AWS S3 buckets: we want to push these messages into our internal Kafka cluster, after enriching each message with additional metadata, in an event-driven fashion. In AWS, this is supported by associating notifications with an S3 bucket.These destinations can make use of a few different destinations, namely, SQS, SNS, and Lambda. We'll focus on the SQS approach, and will make use of NiFi's GetSQS processor.To configure this in AWS, navigate to the S3 bucket and then to the Properties tab, and scroll down to Advanced settings > Events. You'll need to create an SQS queue for this purpose.With this configured, a new SQS message will appear any time an object is created within our S3 bucket.We need to configure some IAM policies in order for our NiFi data flow to be authorized to read from the S3 bucket and to read from the SQS queue. We will authenticate from NiFi using the Access Key and Secret Key associated with

- We can see that ambari has it's image files and web contents present inside the "/usr/lib/ambari-server/web/" directory. This directory contains all the static stuff that are needed by the UI..- Suppose we want to make changes to ambari UI logo. That can be accessed from the URL:http://localhost:8080/img/logo.pnghttp://localhost:8080/img/logo-white.png- In this example we will try changing the "logo-white.png". So in order to do that we will need to get our own logo likehttp://test.example.com/jboss/wp-content/uploads/2015/09/MM-Banner-logo.pngNow we want to use the above image as ambari "logo-white.png". So in order to do that in need to do the following:# mkdir /tmp/images# cd /tmp/images# wget http://test.example.com/jboss/wp-content/uploads/2015/09/MM-Banner-logo.png# mv MM-Banner-logo.png logo-white.png# gzip logo-white.png- We have converted out images in compressed format. We can see the file as following, which we will need to move inside the "/usr/lib/ambari-server/web/img"

Reference:a) Githubb) AMBARI-132051) Install SNMP on a sandbox or test environmentyum install net-snmp net-snmp-utils net-snmp-libs –y2) Modify the script /etc/snmp/snmptrapd.conf to include "disableAuthorization yes"# Example configuration file for snmptrapd## No traps are handled by default, you must edit this file!## authCommunity log,execute,net public# traphandle SNMPv2-MIB::coldStart /usr/bin/bin/my_great_script colddisableAuthorization yes3) Copy APACHE-AMBARI-MIB.txt file from the above github or jira referencea) cp APACHE-AMBARI-MIB.txt /usr/share/snmp/mibs b) chmod 777 /usr/share/snmp/mibs/APACHE-AMBARI-MIB.txt 4) Copy snmp_mib_script.sh script from the above github or jira reference and set appropriate permission to execute the scriptcp snmp_mib_script.sh /tmp/snmp_mib_script.sh5) Run the below commandnohup snmptrapd -m ALL -A -n -Lf /tmp/traps.log &6) Invoke a test trap to ensure that the snmptrapd is logging appropriately to '/tmp/traps.log'snmptrap -v 2c -c publi

SYMPTOM:same hive insert query is failing with following different exception intermittently.grep 'Failed to move' hive.log ERROR [main]: metadata.Hive (Hive.java:copyFiles(2652)) - Failed to move: java.util.ConcurrentModificationException ERROR [main]: metadata.Hive (Hive.java:copyFiles(2652)) - Failed to move: java.util.ConcurrentModificationException ERROR [main]: metadata.Hive (Hive.java:copyFiles(2652)) - Failed to move: java.util.ConcurrentModificationException ERROR [main]: metadata.Hive (Hive.java:copyFiles(2652)) - Failed to move: java.util.NoSuchElementException ERROR [main]: metadata.Hive (Hive.java:copyFiles(2652)) - Failed to move: java.util.NoSuchElementException ERROR [main]: metadata.Hive (Hive.java:copyFiles(2652)) - Failed to move: java.lang.ArrayIndexOutOfBoundsException: -3 ROOT CAUSE:earlier it was observed that if query execution time is small but the Move Task which copies the part file to destination directory is actually taking too long to complete if the destin

bug (HIVE-15355) is being worked on by Hive engineering team at HoortonworksYou can use the following workaround :1- use "SORT BY 0" at the end of the query which will force to use one reducer , please use this only if you have a small query.2- try to use set hive.mv.files.thread=0; before running the query.If you have any question regarding the above, please let me know.

To begin log into ambari and from the Views section select workflow manager. Now select create new workflow. Give your workflow a name and then click on the line that connects the start and end nodes to add a step.Select the email step to add it to your flow and then select it and click the settings(gear) icon. Add your custom settings for all required fields and click save. Now we have a workflow capable of sending an email. That was easy and no XML needed to be modified(big reason many people have never used Oozie) Click Submit and provide a path (that doesn't already exist) for the workflow to be saved. Now go to the Dashboard and find your submitted workflow. You can click run from the dashboard to run the flow or you can select "run on submit" in the step before saving and submitting the flow.

The document lists the steps to be performed to enable users on windows workstation to access the HDP cluster hosted on a different realm. Windows Workstation (realm: WIN.EXAMPLE.COM) HDP (realm: HDP.EXAMPLE.COM) Accessing WebUI components for Namenode, YARN, MapReduce, Oozie etc. Install and Setup MIT KerberosInstall FirefoxEnable MIT Kerberos on FirefoxGet Kerberos Ticket using MIT Kerberos UtilityOpen NN, RM UIs using FirefoxInstall and Setup MIT Kerberos For 64-bit machines (includes both 32 and 64 bit libraries)http://web.mit.edu/kerberos/dist/kfw/4.0/kfw-4.0.1-amd64.msi For 32-bit machines (includes only 32 bit libraries) http://web.mit.edu/kerberos/dist/kfw/4.0/kfw-4.0.1-i386.msi Default location of configuration file in Windows machine is "C:\Program Files\MIT\Kerberos" directory. Note: This is a hidden folder. Copy the krb5.conf file (from the HDP KDC) to above mentioned location and rename krb5.conf to krb5.ini. Configure the following environment properties for MIT K

The new year brings new innovation and collaborative efforts. Various teams from the Apache community have been working hard for the last eighteen months to bring the EZ button to Apache Hadoop technology and Data Lake. In the coming months, we will publish a series of blogs introducing our Data Lake 3.0 architecture and highlighting our innovations within Apache Hadoop core and its related technologies. The WhatYou probably heard of the Deep Learning powered cucumber sorter from a Japanese farmer Makoto Koike! In their cucumber farm, Makoto’s mother spends up to eight hours per day classifying cucumbers into different classes. Makoto is a trained embedded systems designer but not a trained “Machine Learning” engineer. He leveraged TensorFlow, a deep learning framework, with minor configurations to automate his mom’s complex art of cucumber sorting so that they can focus more on cucumber farming instead.This simple, yet powerful example mirrors the trip we have embarked

ISSUE: We did see that Knox gateway servers failing to return Load balancer url after submitting a WebHDFS commands. It could be seen that the logs in the server as below,2017-02-15 20:15:51,050 DEBUG hadoop.gateway (UrlRewriteProcessor.java:rewrite(157)) - Rewrote URL: http://<Gateway-server-hostname>:50075/webhdfs/v1/user/<username>/test2?op=CREATE&user.name=<username>&namenoderpcaddress=<NN-server-hostname>:8020&overwrite=false, direction: OUT via explicit rule: WEBHDFS/webhdfs/outbound/namenode/headers/location to URL: https://<Gateway-server-hostname>:8443/gateway/production/webhdfs/data/v1/webhdfs/v1/user/<username>/test2?_=AAAACAAAABAAAACwXrr5dePzjWo4CD7w6g—lwAqK25Z-yUGo9MJf3qOHlOPn-oZzMWN3qF17Me78ia7H00bVqhPCLVCZNEbeoRY9Sct1cEkfqtmuqyWnj5LI68GDrc7iKr9loQheBkXuceCE4nf-9zXLqE-m8CVtdQQyxMSQnxcZMaAIPesoLoWDWOVXAgGLzWeVMs—uafWTPORGv5KRqok61gSU_KtCt4_9Igcoa1RrpnFEDtusyUHD9osMP612VJdu4ggzJJaVtLdg_btQhxw20CAUSE: It is seen that co

Part 1: https://community.hortonworks.com/articles/82964/getting-started-with-apache-ambari-workflow-design.htmlPart 2: https://community.hortonworks.com/articles/82967/apache-ambari-workflow-designer-view-for-apache-oo.htmlPart 3: https://community.hortonworks.com/articles/82988/apache-ambari-workflow-designer-view-for-apache-oo-1.htmlPart 4: https://community.hortonworks.com/articles/83051/apache-ambari-workflow-designer-view-for-apache-oo-2.htmlPart 5: https://community.hortonworks.com/articles/83361/apache-ambari-workflow-manager-view-for-apache-ooz.htmlPart 7: https://community.hortonworks.com/articles/84071/apache-ambari-workflow-manager-view-for-apache-ooz-2.htmlPart 8: https://community.hortonworks.com/articles/84394/apache-ambari-workflow-manager-view-for-apache-ooz-3.htmlPart 9: https://community.hortonworks.com/articles/85091/apache-ambari-workflow-manager-view-for-apache-ooz-4.htmlPart 10: https://community.hortonworks.com/articles/85354/apache-ambari-workflow-manager-view-

Build flow in HDF/Nifi to push tweets to HDPIn this tutorial, we will learn how to use HDF to create a simple event processing flow by:Install HDF/Nifi on sandbox using the Ambari serviceSetup Solr/Banana/Hive tableImport/Instantiate a prebuilt Nifi templateVerify tweets got pushed to HDFS, Hive using Ambari viewsVisualize tweets in Solr using Banana dashboardExplore provenance features of NifiChange log9/30: Automation script to deploy HDP clusters (on any cloud) with this demo already setup, is available here9/15: Updated: Demo Ambari service for Nifi updated to support HDP 2.5 sandbox and Nifi 1.0. Steps to manually install demo artifacts remains unchanged (but below Nifi screenshots need to be updated)ReferencesFor a primer on HDF, you can refer to the materials here to get a basic backgroundThanks to @bbende@hortonworks.com for his earlier blog post that helped make this tutorial possible Pre-RequisitesThe lab is designed for the HDP Sandbox VM. To run on Azure sandbox, Azure spec

Open NLP Example Apache NiFi ProcessorI wanted to be able to add NLP processing to my dataflow without calling to Apache Spark jobs or other disconnected ways. A custom processor let's me write fast Java 8 microservices to process functionality in my stream in a concise way. All the source code for this processor is available with the Apache license in github. So I wrote one.See the attached generated HTML documentation for the processor.If you would like to use this processor.git clone https://github.com/tspannhw/nifi-nlp-processormvn packagecp nifi-nlp-nar/target/nifi-nlp-nar-1.0.nar /usr/hdf/current/nifi/lib/You can also download a prebuilt NAR from github.Then restart NIFI via Ambari and you can start using it.This has been tested for HDF 2.x NiFi.Add the NLP Processor.Then set the properties, you need to set sentence that you want parsed. You can use expression language to grab a field from an attribute like I am doing to grab the Tweet.Send it a sentence, say from Twitter a

HBase supports several different compression algorithms which can be enabled on a ColumnFamily. Data block encoding attempts to limit duplication of information in keys, taking advantage of some of the fundamental designs and patterns of HBase, such as sorted row keys and the schema of a given table. Compressors reduce the size of large, opaque byte arrays in cells, and can significantly reduce the storage space needed to store uncompressed data. Compressors and data block encoding can be used together on the same ColumnFamily. Changes Take Effect Upon Compaction. If you change compression or encoding for a ColumnFamily, the changes take effect during compaction. Some codecs take advantage of capabilities built into Java, such as GZip compression. Others rely on native libraries. Native libraries may be available as part of Hadoop, such as LZ4. In this case, HBase only needs access to the appropriate shared library. Other codecs, such as Google Snappy, need to

ProblemI log in to Ambari as numeric username (or partially numeric) i.e. 123user, 12345678.When accessing Hive View the following comes upHere is the full stack of the error:Service 'userhome' check failed:java.lang.IllegalArgumentException: Invalid value: "12345678" does not belong to the domain ^[A-Za-z_][A-Za-z0-9._-]*[$]?$ at org.apache.hadoop.hdfs.web.resources.StringParam$Domain.parse(StringParam.java:53) at org.apache.hadoop.hdfs.web.resources.StringParam.<init>(StringParam.java:25) at org.apache.hadoop.hdfs.web.resources.UserParam.<init>(UserParam.java:68) at org.apache.hadoop.hdfs.web.resources.UserParam.<init>(UserParam.java:75) at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getHomeDirectory(WebHdfsFileSystem.java:387) at org.apache.ambari.view.utils.hdfs.HdfsApi$7.run(HdfsApi.java:187) at org.apache.ambari.view.utils.hdfs.HdfsApi$7.run(HdfsApi.java:185) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subje

OverviewI have been running a similar program on Raspberry Pi devices with TensorFlow. Now that MXNet has entered Apache incubation, it has become incredibly interesting to me. With the backing of Apache and Amazon, this library cannot be ignored. So I tried in on the same Raspberry Pi 3B that I was using for TensorFlow. For this example, we are grabbing images from the standard Raspberry Pi Camera and running live image analysis on it with MXNet using the Inception pre-built model from the MXNet Model Zoo. This is the nearly the same as the TensorFlow example. What I noticed is a bit faster execution and smoother process. For accuracy, I have not run enough tests for weighing the two libraries out, but that is something I will look at doing for large number of images. Training both with my camera and images I am interested in would be very helpful. Some use cases I am thinking of are: Security Camera, Water Leak Detection, Evil Cat Sensing, Engine Vibration and self

EnvironmentHDP 2.5.xAmbari 2.4.xProblemI need to use anaconda for %livy.pyspark. Now, it is using the default python2.6 %livy.pysparkimport sysprint(sys.path)-----------------------------------------['/var/hadoop/b/yarn/local/usercache/<user>/appcache/application_1483612761447_100542/container_e11_1483612761447_100542_01_000001/tmp', u'/var/hadoop/b/yarn/local/usercache/<user>/appcache/application_1483612761447_100542/spark-0e25c417-e8c0-4b60-b167-789dc3293bd7/userFiles-a1d6eced-0de1-4e3d-9bc1-f5aea925915d/py4j-0.9-src.zip', u'/var/hadoop/b/yarn/local/usercache/<user>/appcache/application_1483612761447_100542/spark-0e25c417-e8c0-4b60-b167-789dc3293bd7/userFiles-a1d6eced-0de1-4e3d-9bc1-f5aea925915d/pyspark.zip', u'/var/hadoop/b/yarn/local/usercache/<user>/appcache/application_1483612761447_100542/spark-0e25c417-e8c0-4b60-b167-789dc3293bd7/userFiles-a1d6eced-0de1-4e3d-9bc1-f5aea925915d', '/var/hadoop/b/yarn/local/usercache/<user>/appcache/application_1483612

This is a second in the series of articles on WFM.Part 1: https://community.hortonworks.com/articles/82964/getting-started-with-apache-ambari-workflow-design.html Part 3: https://community.hortonworks.com/articles/82988/apache-ambari-workflow-designer-view-for-apache-oo-1.html Part 4: https://community.hortonworks.com/articles/83051/apache-ambari-workflow-designer-view-for-apache-oo-2.htmlPart 5: https://community.hortonworks.com/articles/83361/apache-ambari-workflow-manager-view-for-apache-ooz.htmlPart 6: https://community.hortonworks.com/articles/83787/apache-ambari-workflow-manager-view-for-apache-ooz-1.htmlPart 7: https://community.hortonworks.com/articles/84071/apache-ambari-workflow-manager-view-for-apache-ooz-2.htmlPart 8: https://community.hortonworks.com/articles/84394/apache-ambari-workflow-manager-view-for-apache-ooz-3.htmlPart 9: https://community.hortonworks.com/articles/85091/apache-ambari-workflow-manager-view-for-apache-ooz-4.htmlPart 10: https://community.hortonworks.c

Using Stanford CoreNLP in Your Big Data PipelinesCoreNLP OverviewThe latest version of Stanford CoreNLP includes a server that you can run and access via REST API. CoreNLP adds a lot of features, but the one most interesting to me is Sentiment Analysis. Installation and Setup (http://stanfordnlp.github.io/CoreNLP/corenlp-server.html)Download a recent full deployment (http://nlp.stanford.edu/software/stanford-corenlp-full-2016-10-31.zip) This is big, it has models and all the JARS and server code.Run the ServerGiving the JVM Four Gigs of RAM to run makes it run nice. Port 9000 works for me.java -mx4g -cp "*" edu.stanford.nlp.pipeline.StanfordCoreNLPServer -port 9000 -timeout 15000 Running the Serverstanford-corenlp-full-2016-10-31 git:(master) ✗ java -mx4g -cp "*" edu.stanford.nlp.pipeline.StanfordCoreNLPServer -port 9000 -timeout 15000[main] INFO CoreNLP - --- StanfordCoreNLPServer#main() called ---[main] INFO CoreNLP - setting default constituency parser[main] INFO CoreNLP -

AD admins may be busy and you may happen to know the ambari admin principal for enabling Kerberos. How would you go about adding a principal for AD with this information and add it to your kerberos keytab? Below is one way to do it. Thanks to @Robert Levas for collaborating with me on this. 1. Create LDIF file ad_user.ldif. (Make sure there are no spaces at the ends of each of these lines) dn: CN=HTTP/loadbalancerhost,OU=dav,OU=hortonworks,DC=HOST,DC=COMchangetype: addobjectClass: topobjectClass: personobjectClass: organizationalPersonobjectClass: userdistinguishedName: CN=HTTP/loadbalancerhost,OU=dav,OU=hortonworks,DC=HOST,DC=COMcn: HTTP/loadbalancerhostuserAccountControl: 514accountExpires: 0userPrincipalName: HTTP/loadbalancerhost@HOST.COMservicePrincipalName: HTTP/loadbalancerhostdn: CN=HTTP/loadbalancerhost,OU=dav,OU=hortonworks,DC=host,DC=comchangetype: modifyreplace: unicodePwdunicodePwd::IgBoAGEAZABvAG8AcABSAG8AYwBrAHMAMQAyADMAIQAiAA==dn: CN=HTTP/loadbalancerhost,OU=dav,OU=hort

BACKGROUND The modern Data Warehouse contains a heterogenous mix of data: delimited text files, data in Hadoop (HDFS/Hive), relational databases, NoSQL databases, Parquet, Avro, JSON, Geospatial data, and more. ETL (Extract-Transform-Load) is a process used to integrate these disparate data types and create a unified view of the data. Apache Spark is a fast general purpose distributed computation engine for fault-tolerant parallel data processing. Spark is an excellent choice for ETL: Works with a myriad of data sources: files, RDBMS's, NoSQL, Parquet, Avro, JSON, XML, and many more. Write your ETL code using Java, Scala, or Python. In-memory computing for fast data processing. API's to easily create schemas for your data and perform SQL computations. Distributed computing and fault-tolerance is built into the framework and abstracted from the end-user. This article will demonstrate how easy it is to use Spark with the Python API (PySpark) for ETL processing to join data from text fi