Resources

OverviewR is currently supported through livy interpreter by running %livy.sparkrEnvironmentI did test the below solution withHDP 2.5.0.0-1245 and Ambari 2.4.0.1HDP 2.5.3.0-37 and Ambari 2.4.2.0Steps To Follow1. Install R on all YARN worker nodes$ yum install R-devel libcurl-devel openssl-devel2. Confirm R works fine[root@dkhdp251 zeppelin]# R -e "print(1+1)"R version 3.3.2 (2016-10-31) -- "Sincere Pumpkin Patch"Copyright (C) 2016 The R Foundation for Statistical ComputingPlatform: x86_64-redhat-linux-gnu (64-bit)R is free software and comes with ABSOLUTELY NO WARRANTY.You are welcome to redistribute it under certain conditions.Type 'license()' or 'licence()' for distribution details. Natural language support but running in an English localeR is a collaborative project with many contributors.Type 'contributors()' for more information and'citation()' on how to cite R or R packages in publications.Type 'demo()' for some demos, 'help()' for on-line help, or'help.start()' for an HTML brow

Part 1: https://community.hortonworks.com/articles/82964/getting-started-with-apache-ambari-workflow-design.htmlPart 2: https://community.hortonworks.com/articles/82967/apache-ambari-workflow-designer-view-for-apache-oo.htmlPart 3: https://community.hortonworks.com/articles/82988/apache-ambari-workflow-designer-view-for-apache-oo-1.htmlPart 4: https://community.hortonworks.com/articles/83051/apache-ambari-workflow-designer-view-for-apache-oo-2.htmlPart 5: https://community.hortonworks.com/articles/83361/apache-ambari-workflow-manager-view-for-apache-ooz.htmlPart 6: https://community.hortonworks.com/articles/83787/apache-ambari-workflow-manager-view-for-apache-ooz-1.htmlPart 7: https://community.hortonworks.com/articles/84071/apache-ambari-workflow-manager-view-for-apache-ooz-2.htmlPart 8: https://community.hortonworks.com/articles/84394/apache-ambari-workflow-manager-view-for-apache-ooz-3.htmlPart 9: https://community.hortonworks.com/articles/85091/apache-ambari-wor

There is a two part process before any access to NiFi UI is possible:1. Authentication:By default NiFi will use a user/server's SSL certificate when provided in the connection to authenticate. When NO user/server certificate is presented, NiFi will then look for a Kerberos TGT (If Spnego has been configured in NiFi). Finally, if neither of the above where present in the connection, NiFi will use the login identity provider (if configured). Login identity providers include either ldap or kerberos. With both of these options, NiFi will present users with a login screen.2. Authorization:Authorization is the mechanism that controls what features and components authenticated users are granted access. The default authorizer NiFi will use is the internal file based authorizer. There is an option to configure NiFi to use Ranger as the authorizer instead.The intent of this article is not to discuss how to setup NiFi to use any of the Authentication or Authorizer options. This article covers

Centralized Cache Management in HDFS is a mechanism that explicitly caches specific files or directories in memory for improved performance. This is useful for relatively small files that are accessed repeatedly. For example, reference/lookup tables or fact tables that are used in many joins. Once enabled, HDFS will automatically cache selected files, and periodically check for changes and recache the files.While HDFS and the underlying file system do some caching of files when memory is available, explicit caching using Centralized Cache Management prevents the data from being evicted from memory when processes consume all of the physical memory. As a corollary of this, if you ARE working on a lightly loaded system where there is free memory, you may not see any performance improvement from this method, as the data was already in disk cache. So, your performance testing needs to stress the system.Let’s look at some key terms and concepts:Cache PoolsA cache pool is an administrative en

Part 1: https://community.hortonworks.com/articles/82964/getting-started-with-apache-ambari-workflow-design.htmlPart 2: https://community.hortonworks.com/articles/82967/apache-ambari-workflow-designer-view-for-apache-oo.htmlPart 3: https://community.hortonworks.com/articles/82988/apache-ambari-workflow-designer-view-for-apache-oo-1.htmlPart 4: https://community.hortonworks.com/articles/83051/apache-ambari-workflow-designer-view-for-apache-oo-2.htmlPart 5: https://community.hortonworks.com/articles/83361/apache-ambari-workflow-manager-view-for-apache-ooz.htmlPart 6: https://community.hortonworks.com/articles/83787/apache-ambari-workflow-manager-view-for-apache-ooz-1.htmlPart 7: https://community.hortonworks.com/articles/84071/apache-ambari-workflow-manager-view-for-apache-ooz-2.htmlPart 8: https://community.hortonworks.com/articles/84394/apache-ambari-workflow-manager-view-for-apache-ooz-3.htmlPart 10: https://community.hortonworks.com/articles/85354/apache-ambari-workflow-manager-view-

In this article we will talk about one of the best feature of Ambari 2.4 in which we will be doing the ambari "setup-security" in non interactive mode to enable the HTTPs in ambari server. I assume that you have alredy created the ambari-server keys and certificate using open ssl. So we will see how can we do the HTTPs setup using a single line command:ambari-server setup-security --security-option=setup-https --api-ssl=true --api-ssl-port=8443 --import-cert-path=/etc/ambari-server/certs/sandbox.hortonworks.com.crt --import-key-path=/etc/ambari-server/certs/sandbox.hortonworks.com.key --pem-password=hadoopOutput of the above command can be as following, Notice that it will run in non interactive mode.Using python /usr/bin/pythonSecurity setup options...Do you want to configure HTTPS [y/n] (y)? SSL port [8443] ? Please enter password for Private Key: Importing and saving Certificate...done.Ambari server URL changed. To make use of the Tez View in Ambari please update the property tez.t

IntroductionGeospatial data is generated in huge volumes with the rise of the Internet of Things. IoT sensor networks are pushing the geospatial data rates even higher. There has been an explosion of sensor networks on the ground, mobile devices carried by people or mounted on vehicles, drones flying overhead, tethered aerostats (such as Google’s Project Loon), atmosats at high altitude, and microsats in orbit.OpportunityGeospatial analytics can provide us with the tools and methods we need to make sense of all that data and put it to use in solving problems we face at all scales.ChallengesGeospatial work requires atypical data types (e.g., points, shapefiles, map projections), potentially many layers of detail to process and visualize, and specialized algorithms—not your typical ETL (extract, transform, load) or reporting work. Apache Spark Role in Geospatial DevelopmentWhile Spark might seem to be influencing the evolution of accessory tools, it’s also becoming a default in the geosp

SYMPTOM:Oozie sqoop action fails with below error while inserting data into Hive.20217 [Thread-30] INFO org.apache.sqoop.hive.HiveImport - Sorry ! hive-shell is disabled use 'Beeline' or 'Hive View' instead. Please contact cluster administrators for further information 20218 [main] ERROR org.apache.sqoop.tool.ImportTool - Encountered IOException running import job: java.io.IOException: Hive exited with status 1 at org.apache.sqoop.hive.HiveImport.executeExternalHiveScript(HiveImport.java:389) at org.apache.sqoop.hive.HiveImport.executeScript(HiveImport.java:342) at org.apache.sqoop.hive.HiveImport.importTable(HiveImport.java:246) at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:524) at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:615) at org.apache.sqoop.tool.JobTool.execJob(JobTool.java:243) at org.apache.sqoop.tool.JobTool.run(JobTool.java:298) at org.apache.sqoop.Sqoop.run(Sqoop.java:147) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at org.apac

SYMPTOM When running Hive queries from Resource Manager log the following error is displayed:2017-02-07 15:08:32,140 ERROR impl.MetricsSinkAdapter (MetricsSinkAdapter.java:publishMetricsFromQueue(148)) - Got sink exception, retry in 4600ms org.apache.hadoop.metrics2.MetricsException: Failed to putMetrics at org.apache.hadoop.metrics2.sink.timeline.HadoopTimelineMetricsSink.putMetrics(HadoopTimelineMetricsSink.java:216) at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.consume(MetricsSinkAdapter.java:186) at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.consume(MetricsSinkAdapter.java:43) at org.apache.hadoop.metrics2.impl.SinkQueue.consumeAll(SinkQueue.java:87) at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.publishMetricsFromQueue(MetricsSinkAdapter.java:134) at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter$1.run(MetricsSinkAdapter.java:88) Caused by: java.net.UnknownHostException: http at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketIm

IntroductionUsing NiFi, data can be exposed in such a way that a receiver can pull from it by adding an Output Port to the root process group. For Spark, we will use this same mechanism - we will use the Site-to-Site protocol to pull data from NiFi's Output Ports. In this tutorial we learn to capture NiFi app log from the Sandbox and parse it using Java regex and ingest it to Phoenix via Spark or Directly using NiFi PutSql Processor.Prerequisites1) Assuming you already have latest version of NiFi-1.x/HDF-2.x downloaded as zip file (HDF and HDP cannot be managed by Ambari on same nodes as of now) on to your HW Sandbox Version 2.5, else execute below after ssh connectivity to sandbox is established:# mkdir /opt/HDF-2.1.1# cd /opt/HDF-2.1.1 # wget http://public-repo-1.hortonworks.com/HDF/2.1.1.0/nifi-1.1.0.2.1.1.0-2-bin.tar.gz # tar -xvf nifi-1.1.0.2.1.1.0-2-bin.tar.gz 2) Spark, Zeppelin, YARN and HDFS are Installed on your VM and started. 3) Hbase is Installed with phoeix Query Server.

Article Goals Setup Hive ODBC on Ubuntu 14.x.x with/without Kerberos Setup Hive ODBC on CentOS 6.x with/without Kerberos Enabling Debug logging useful for debugging connectivity issues Notes/Requirements Article assumes that you have a setup of Ubuntu/CentOS server Access to Ubuntu/CentOS repository Have libsasl2-modules-gssapi-mit installedHave "gcc" installed, required for compilation when building from sourceHave "make" installed, required for compilation when building from source Verify if you have a working Ubuntu repository by typing "sudo apt-get update" and see if this returns successfully Within CentOS as a non superuser you can run "sudo yum repolist" wherein the "base" and "epel" repositories should get listed with thousands as value in "status" field Here are the examples of working repository on either systemsThis node should be able to talk to the HiveServer2 via hostname/IP addressEnsure that firewall is disabled (ufw status for Ubuntu) & (service iptables/ip

Part 1: https://community.hortonworks.com/articles/82964/getting-started-with-apache-ambari-workflow-design.htmlPart 2: https://community.hortonworks.com/articles/82967/apache-ambari-workflow-designer-view-for-apache-oo.htmlPart 3: https://community.hortonworks.com/articles/82988/apache-ambari-workflow-designer-view-for-apache-oo-1.htmlPart 4: https://community.hortonworks.com/articles/83051/apache-ambari-workflow-designer-view-for-apache-oo-2.htmlPart 5: https://community.hortonworks.com/articles/83361/apache-ambari-workflow-manager-view-for-apache-ooz.htmlPart 6: https://community.hortonworks.com/articles/83787/apache-ambari-workflow-manager-view-for-apache-ooz-1.htmlPart 7: https://community.hortonworks.com/articles/84071/apache-ambari-workflow-manager-view-for-apache-ooz-2.htmlPart 9: https://community.hortonworks.com/articles/85091/apache-ambari-workflow-manager-view-for-apache-ooz-4.htmlPart 10: https://community.hortonworks.com/articles/85354/apache-ambari-workflow-manager-view-

Imagine we have a use case of exchanging data with an external party via AWS S3 buckets: we want to push these messages into our internal Kafka cluster, after enriching each message with additional metadata, in an event-driven fashion. In AWS, this is supported by associating notifications with an S3 bucket.These destinations can make use of a few different destinations, namely, SQS, SNS, and Lambda. We'll focus on the SQS approach, and will make use of NiFi's GetSQS processor.To configure this in AWS, navigate to the S3 bucket and then to the Properties tab, and scroll down to Advanced settings > Events. You'll need to create an SQS queue for this purpose.With this configured, a new SQS message will appear any time an object is created within our S3 bucket.We need to configure some IAM policies in order for our NiFi data flow to be authorized to read from the S3 bucket and to read from the SQS queue. We will authenticate from NiFi using the Access Key and Secret Key associated with

- We can see that ambari has it's image files and web contents present inside the "/usr/lib/ambari-server/web/" directory. This directory contains all the static stuff that are needed by the UI..- Suppose we want to make changes to ambari UI logo. That can be accessed from the URL:http://localhost:8080/img/logo.pnghttp://localhost:8080/img/logo-white.png- In this example we will try changing the "logo-white.png". So in order to do that we will need to get our own logo likehttp://test.example.com/jboss/wp-content/uploads/2015/09/MM-Banner-logo.pngNow we want to use the above image as ambari "logo-white.png". So in order to do that in need to do the following:# mkdir /tmp/images# cd /tmp/images# wget http://test.example.com/jboss/wp-content/uploads/2015/09/MM-Banner-logo.png# mv MM-Banner-logo.png logo-white.png# gzip logo-white.png- We have converted out images in compressed format. We can see the file as following, which we will need to move inside the "/usr/lib/ambari-server/web/img"

Reference:a) Githubb) AMBARI-132051) Install SNMP on a sandbox or test environmentyum install net-snmp net-snmp-utils net-snmp-libs –y2) Modify the script /etc/snmp/snmptrapd.conf to include "disableAuthorization yes"# Example configuration file for snmptrapd## No traps are handled by default, you must edit this file!## authCommunity log,execute,net public# traphandle SNMPv2-MIB::coldStart /usr/bin/bin/my_great_script colddisableAuthorization yes3) Copy APACHE-AMBARI-MIB.txt file from the above github or jira referencea) cp APACHE-AMBARI-MIB.txt /usr/share/snmp/mibs b) chmod 777 /usr/share/snmp/mibs/APACHE-AMBARI-MIB.txt 4) Copy snmp_mib_script.sh script from the above github or jira reference and set appropriate permission to execute the scriptcp snmp_mib_script.sh /tmp/snmp_mib_script.sh5) Run the below commandnohup snmptrapd -m ALL -A -n -Lf /tmp/traps.log &6) Invoke a test trap to ensure that the snmptrapd is logging appropriately to '/tmp/traps.log'snmptrap -v 2c -c publi

SYMPTOM:same hive insert query is failing with following different exception intermittently.grep 'Failed to move' hive.log ERROR [main]: metadata.Hive (Hive.java:copyFiles(2652)) - Failed to move: java.util.ConcurrentModificationException ERROR [main]: metadata.Hive (Hive.java:copyFiles(2652)) - Failed to move: java.util.ConcurrentModificationException ERROR [main]: metadata.Hive (Hive.java:copyFiles(2652)) - Failed to move: java.util.ConcurrentModificationException ERROR [main]: metadata.Hive (Hive.java:copyFiles(2652)) - Failed to move: java.util.NoSuchElementException ERROR [main]: metadata.Hive (Hive.java:copyFiles(2652)) - Failed to move: java.util.NoSuchElementException ERROR [main]: metadata.Hive (Hive.java:copyFiles(2652)) - Failed to move: java.lang.ArrayIndexOutOfBoundsException: -3 ROOT CAUSE:earlier it was observed that if query execution time is small but the Move Task which copies the part file to destination directory is actually taking too long to complete if the destin

bug (HIVE-15355) is being worked on by Hive engineering team at HoortonworksYou can use the following workaround :1- use "SORT BY 0" at the end of the query which will force to use one reducer , please use this only if you have a small query.2- try to use set hive.mv.files.thread=0; before running the query.If you have any question regarding the above, please let me know.

To begin log into ambari and from the Views section select workflow manager. Now select create new workflow. Give your workflow a name and then click on the line that connects the start and end nodes to add a step.Select the email step to add it to your flow and then select it and click the settings(gear) icon. Add your custom settings for all required fields and click save. Now we have a workflow capable of sending an email. That was easy and no XML needed to be modified(big reason many people have never used Oozie) Click Submit and provide a path (that doesn't already exist) for the workflow to be saved. Now go to the Dashboard and find your submitted workflow. You can click run from the dashboard to run the flow or you can select "run on submit" in the step before saving and submitting the flow.

The document lists the steps to be performed to enable users on windows workstation to access the HDP cluster hosted on a different realm. Windows Workstation (realm: WIN.EXAMPLE.COM) HDP (realm: HDP.EXAMPLE.COM) Accessing WebUI components for Namenode, YARN, MapReduce, Oozie etc. Install and Setup MIT KerberosInstall FirefoxEnable MIT Kerberos on FirefoxGet Kerberos Ticket using MIT Kerberos UtilityOpen NN, RM UIs using FirefoxInstall and Setup MIT Kerberos For 64-bit machines (includes both 32 and 64 bit libraries)http://web.mit.edu/kerberos/dist/kfw/4.0/kfw-4.0.1-amd64.msi For 32-bit machines (includes only 32 bit libraries) http://web.mit.edu/kerberos/dist/kfw/4.0/kfw-4.0.1-i386.msi Default location of configuration file in Windows machine is "C:\Program Files\MIT\Kerberos" directory. Note: This is a hidden folder. Copy the krb5.conf file (from the HDP KDC) to above mentioned location and rename krb5.conf to krb5.ini. Configure the following environment properties for MIT K

The new year brings new innovation and collaborative efforts. Various teams from the Apache community have been working hard for the last eighteen months to bring the EZ button to Apache Hadoop technology and Data Lake. In the coming months, we will publish a series of blogs introducing our Data Lake 3.0 architecture and highlighting our innovations within Apache Hadoop core and its related technologies. The WhatYou probably heard of the Deep Learning powered cucumber sorter from a Japanese farmer Makoto Koike! In their cucumber farm, Makoto’s mother spends up to eight hours per day classifying cucumbers into different classes. Makoto is a trained embedded systems designer but not a trained “Machine Learning” engineer. He leveraged TensorFlow, a deep learning framework, with minor configurations to automate his mom’s complex art of cucumber sorting so that they can focus more on cucumber farming instead.This simple, yet powerful example mirrors the trip we have embarked

ISSUE: We did see that Knox gateway servers failing to return Load balancer url after submitting a WebHDFS commands. It could be seen that the logs in the server as below,2017-02-15 20:15:51,050 DEBUG hadoop.gateway (UrlRewriteProcessor.java:rewrite(157)) - Rewrote URL: http://<Gateway-server-hostname>:50075/webhdfs/v1/user/<username>/test2?op=CREATE&user.name=<username>&namenoderpcaddress=<NN-server-hostname>:8020&overwrite=false, direction: OUT via explicit rule: WEBHDFS/webhdfs/outbound/namenode/headers/location to URL: https://<Gateway-server-hostname>:8443/gateway/production/webhdfs/data/v1/webhdfs/v1/user/<username>/test2?_=AAAACAAAABAAAACwXrr5dePzjWo4CD7w6g—lwAqK25Z-yUGo9MJf3qOHlOPn-oZzMWN3qF17Me78ia7H00bVqhPCLVCZNEbeoRY9Sct1cEkfqtmuqyWnj5LI68GDrc7iKr9loQheBkXuceCE4nf-9zXLqE-m8CVtdQQyxMSQnxcZMaAIPesoLoWDWOVXAgGLzWeVMs—uafWTPORGv5KRqok61gSU_KtCt4_9Igcoa1RrpnFEDtusyUHD9osMP612VJdu4ggzJJaVtLdg_btQhxw20CAUSE: It is seen that co

Part 1: https://community.hortonworks.com/articles/82964/getting-started-with-apache-ambari-workflow-design.htmlPart 2: https://community.hortonworks.com/articles/82967/apache-ambari-workflow-designer-view-for-apache-oo.htmlPart 3: https://community.hortonworks.com/articles/82988/apache-ambari-workflow-designer-view-for-apache-oo-1.htmlPart 4: https://community.hortonworks.com/articles/83051/apache-ambari-workflow-designer-view-for-apache-oo-2.htmlPart 5: https://community.hortonworks.com/articles/83361/apache-ambari-workflow-manager-view-for-apache-ooz.htmlPart 7: https://community.hortonworks.com/articles/84071/apache-ambari-workflow-manager-view-for-apache-ooz-2.htmlPart 8: https://community.hortonworks.com/articles/84394/apache-ambari-workflow-manager-view-for-apache-ooz-3.htmlPart 9: https://community.hortonworks.com/articles/85091/apache-ambari-workflow-manager-view-for-apache-ooz-4.htmlPart 10: https://community.hortonworks.com/articles/85354/apache-ambari-workflow-manager-view-

Build flow in HDF/Nifi to push tweets to HDPIn this tutorial, we will learn how to use HDF to create a simple event processing flow by:Install HDF/Nifi on sandbox using the Ambari serviceSetup Solr/Banana/Hive tableImport/Instantiate a prebuilt Nifi templateVerify tweets got pushed to HDFS, Hive using Ambari viewsVisualize tweets in Solr using Banana dashboardExplore provenance features of NifiChange log9/30: Automation script to deploy HDP clusters (on any cloud) with this demo already setup, is available here9/15: Updated: Demo Ambari service for Nifi updated to support HDP 2.5 sandbox and Nifi 1.0. Steps to manually install demo artifacts remains unchanged (but below Nifi screenshots need to be updated)ReferencesFor a primer on HDF, you can refer to the materials here to get a basic backgroundThanks to @bbende@hortonworks.com for his earlier blog post that helped make this tutorial possible Pre-RequisitesThe lab is designed for the HDP Sandbox VM. To run on Azure sandbox, Azure spec

Open NLP Example Apache NiFi ProcessorI wanted to be able to add NLP processing to my dataflow without calling to Apache Spark jobs or other disconnected ways. A custom processor let's me write fast Java 8 microservices to process functionality in my stream in a concise way. All the source code for this processor is available with the Apache license in github. So I wrote one.See the attached generated HTML documentation for the processor.If you would like to use this processor.git clone https://github.com/tspannhw/nifi-nlp-processormvn packagecp nifi-nlp-nar/target/nifi-nlp-nar-1.0.nar /usr/hdf/current/nifi/lib/You can also download a prebuilt NAR from github.Then restart NIFI via Ambari and you can start using it.This has been tested for HDF 2.x NiFi.Add the NLP Processor.Then set the properties, you need to set sentence that you want parsed. You can use expression language to grab a field from an attribute like I am doing to grab the Tweet.Send it a sentence, say from Twitter a

HBase supports several different compression algorithms which can be enabled on a ColumnFamily. Data block encoding attempts to limit duplication of information in keys, taking advantage of some of the fundamental designs and patterns of HBase, such as sorted row keys and the schema of a given table. Compressors reduce the size of large, opaque byte arrays in cells, and can significantly reduce the storage space needed to store uncompressed data. Compressors and data block encoding can be used together on the same ColumnFamily. Changes Take Effect Upon Compaction. If you change compression or encoding for a ColumnFamily, the changes take effect during compaction. Some codecs take advantage of capabilities built into Java, such as GZip compression. Others rely on native libraries. Native libraries may be available as part of Hadoop, such as LZ4. In this case, HBase only needs access to the appropriate shared library. Other codecs, such as Google Snappy, need to