Community Articles

IntroductionHive is one of the most common used databases on Hadoop, users of Hive are doubling per year due to the amazing enhancements and the addition of Tez and Spark that enabled Hive to by pass the MR era to a an in-memory execution that changed how people are using Hive.in this blog post, I will show you how to connect squirrel Sql Client to Hive, the concept is similar to any other clients out there as long as you are using the open-source libraries that matches the ones here you should be fine.PrerequisiteDownload Hortonworks Sandbox with HDP 2.2.4, Squirrel SQL ClientStep 1Follow the Squirrel documentation and run it on your Mac or PC.Step 2Follow the Hortonworks HDP Installation on VritualBox, VMware or Hyper-V and start up the virtual Instance.Step 3once you are HDP is up and running, connect it it using SSH as it shows on the console, once you are connected you need to download some JAR files in order to establish the connection.Step 4if you are using MacOS, simply while

Controlling the environment of an application is vital for it's functionality and stability. Especially in a distributed environment it is important for developers to have control over the version of dependencies. In such an scenario it's a critical task to ensure possible conflicting requirements of multiple applications are not disturbing each other.That is why frameworks like YARN ensure that each application is executed in a self-contained environment - typically in a Linux Container or Docker Container - that is controlled by the developer. In this post we show what this means for Python environments being used by Spark.YARN Application DeploymentAs mentioned earlier does YARN execute each application in a self-contained environment on each host. This ensures the execution in a controlled environment managed by individual developers. The way this works in a nutshell is that the dependency of an application are distributed to each node typically via HDFS.This figure s

Hey Everybody, Today I've got a continuation video about Attribute Based Access Control (ABAC) on CDP. ABAC allows you to Tag tables and columns and then set access policies for groups or users that will either grant or deny access to tagged data. At Cloudera Now Jonathan Hsieh showed us how to implement Column Masking with ABAC. So today I decided I would show you how to do the same thing. After that, I'll show you how security policies travel with the data; all because of SDX.   If you haven't checked out the first video go watch that first here: https://community.cloudera.com/t5/Community-Articles/How-to-use-ABAC-in-CDP/ta-p/288056     More information: Watch Cloudera Now https://www.cloudera.com/about/events/cloudera-now-cdp.html Documentation on CDP https://docs.cloudera.com/ More information on SDX https://www.cloudera.com/products/sdx.html

What is Cloudera Data Warehouse?Cloudera Data Warehouse is an auto-scaling, highly concurrent and cost effective analytics service that ingests high scale data anywhere, from structured, unstructured and edge sources. It supports hybrid and multi-cloud infrastructure models by seamlessly moving workloads between on-premises and any cloud for reports, dashboards, ad-hoc and advanced analytics, including AI, with consistent security and governance. Cloudera Data Warehouse offers zero query wait times, reduced IT costs and agile delivery.  See more information here Key Concepts:In the Cloudera Data Warehouse service, your data is stored in an object store in a data lake that resides in your specific cloud environment. The service is composed of:Database Catalogs: a metadata service associated to a CDP Data Lake which provides the data context for your defined tables and databases within the CDP Enterprise Data Cloud. Virtual Warehouses: compute resources running H

The purpose of this repo is to provide quick examples and utilities to work on a Spark and Hive integration on HDP 3.1.4 Prerequisites: HiveServer2Interactive (LLAP) must be installed, up and running Bash and Python interpreter must be available Ideally, for connections using HTTP transport protocol, in the Ambari -> Hive -> Configs, hive.server2.thrift.http.cookie.auth.enabled should be set to true Set hadoop.proxyuser.hive.hosts=* in the Ambari -> HDFS -> Configs -> Custom core-site section (core-site.xml, or mention at least the LLAP Hosts, HS2 Hosts and Spark client hosts separated by commas). Once the LLAP service is up and running, the next step for this setup requires the following properties to be configured in the Spark client: spark.datasource.hive.warehouse.load.staging.dir=spark.datasource.hive.warehouse.metastoreUri=spark.hadoop.hive.llap.daemon.service.hosts=spark.jars=spark.security.credentials.hiveserver2.enabled=spark.sql.hive.hiveserve

Implementing Streaming Use Case From REST to Hive with Apache NiFi and Apache KafkaPart 1With Apache Kafka 2.0, Apache NiFi 1.8 and many new features and abilities coming out. It's time to put them to the test.So to plan out what we are going to do, I have a high level architecture diagram. We are going to ingest a number of sources including REST feeds, Social Feeds, Messages, Images, Documents and Relational Data.We will ingest with NiFi, filter and process and segment it into Kafka topics. Kafka data will be in Apache Avro format with schemas specified in Hortonworks Schema Registry. Kafka Streams, Spark and NiFi will do additional event processing along with machine learning and deep learning. it will be stored in Druid for real-time analytics and summaries. Hive, HDFS and S3 will store for permanent storage. We will do dashboards with Superset and Spark SQL + Zeppelin. We will integrate machine learning with Spark ML, TensorFlow and Apache MXNet.We will also push back

Cloudera on Public Cloud makes being an administrator for a big data platform even easier, thanks to SDX. Watch me spend a day at a temp position for Aperture Cybertronics as their Data Admin. I'll quickly deploy clusters, grant access to users, and change performance parameters for the Aperture Cybertronics' staff.     More information: More information on Data Warehouse https://www.cloudera.com/products/data-warehouse.html Documentation on CDP https://docs.cloudera.com/

Below are some examples of how you would achieve this: Case1: Restrict to users in a single group – In this example, only the users who are members of “scientist” group are only allowed to login to ranger admin. User search filter parameter would look something like this: (&(sAMAccountName={0})(memberof=cn=scientist,ou=groups,dc=hwqe,dc=hortonworks,dc=com)) Case2: Restrict to users in multiple groups – In this example, only the users who are members of either “scientist” group OR “analyst” group are allowed to login to ranger admin. User search filter parameter would look something like this: (&(sAMAccountName={0})(|((memberof=cn=scientist,ou=groups,dc=hwqe,dc=hortonworks,dc=com)(memberof=cn=analyst,ou=groups,dc=hwqe,dc=hortonworks,dc=com)))) Case3: Restrict to given list of users – In this example, only the users whose cn (or common name) starts with “sam r” are allowed to login to ranger admin. User search filter parameter would look something like this: (&(sAMAccount

Issue:Nodemanager log:FATAL nodemanager.NodeManager (NodeManager.java:initAndStartNodeManager(540)) - Error starting NodeManager java.lang.UnsatisfiedLinkError: Could not load library. Reasons: [no leveldbjni64-1.6.0.2.4.0.0-169 in java.library.path, no leveldbjni-1.6.0.2.4.0.0-169 in java.library.path, no leveldbjni in java.library.path, No such file or directory] Resolution:a) If you are using Ambari, by default the library files reported as missing in the above error should be present under /var/lib/ambari-agent/tmp/hadoop_java_io_tmpdirYou can find this location by running "ps -ef | grep -i nodemanager" on any of the hosts in the cluster where nodemanager runs successfully and look for this property "-Djava.io.tmpdir"If you have a different location other than the default then make sure the library files are present in that location or copy from another hosts.b) You can also check to remove noexec from /tmp as below and then start node manager via ambari cat /etc/fstabmount -o remo

Hi Again, Today's video is all about Cloudera Data Warehouse (CDW). CDW is one of the many experiences you can use on the Cloudera. Cloudera Data warehouse packages up the projects you may already know and use such as Impala and Hive into a service. This Service runs on Kubernetes giving it the ability to pause, resume, scale up, or down quickly and automatically. The Data Warehouse service allows you to quickly deploy new use cases with the right data at the right time to tackle hard problems while meeting your SLA’s. It does all of this with significant cost savings and little effort on your part.   In this first video of the Data Warehouse collection we'll cover the different pieces of the service and layout of the UI. We'll show you the Data Analytics studio (DAS) as well to get you started querying your data warehouse.     More information: More information on Data Warehouse https://www.cloudera.com/products/data-warehouse.html Documentation on CDP https://docs

HDFS Balancer is a tool for balancing the data across the storage devices of a HDFS cluster. The Balancer was originally designed to run slowly so that the balancing activities do not affect the normal cluster activities and the running jobs. We have received feedback from the HDFS community that it is also desirable if Balancer can be configured to run faster. The use cases are listed below. Free up the spaces from some nearly full datanodes.Move data to some newly added datanodes in order to utilize the new machines.Run Balancer when the cluster load is low or in a maintenance window, instead of running it as a background daemon.We have changed Balancer for addressing these new use cases. After the changes, Balancer is able to run 100x faster while it still can be configured to run slowly as before. In one of our tests, we were able to bump the performance from a few gigabytes per minute to a terabyte per minute.In addition, we added two new features -- source datanodes and block

This community article assumes that we already have CDH 6.x and Kerberos enabled, in case we have to install Kerberos, please use the document below:   https://docs.cloudera.com/documentation/enterprise/6/6.3/topics/cm_sg_intro_kerb.html   1. Install a database In this case, we are using MySQL: https://docs.cloudera.com/csp/2.0.1/deployment/topics/csp-installing_mysql.html 2. Configure the database for schema registry and SMM https://docs.cloudera.com/csp/2.0.1/deployment/topics/csp-configuring-schema-registry-metadata-stores-in-mysql.html 3. Download Schema Registry and SMM parcels SMM https://www.cloudera.com/downloads/cdf/csm.html CSR https://www.cloudera.com/downloads/cdf/csp.html 4. Install the Parcels Install the services in this order:   1. Schema Registry 2. SRM (if no SRM installation, avoid this step) 3. SMM   https://docs.cloudera.com/csp/2.0.1/deployment/topics/csp-get-parcel-csd.html 5. Distribute and activate the parcels. In Schema registry point “S

PROBLEMAfter restarting the Zeppelin service from UI I have the following situation: Zeppelin service runs fine – “green” in AmbariGoing to http://<ZEPPELIN_HOST>:9995 - I am gettingChecking out zeppelin-zeppelin-<ZEPPELIN_HOST>.log file I can seeINFO [2017-02-04 07:47:09,081] ({main}ZeppelinServer.java[main]:114) - Starting zeppelin server INFO [2017-02-0407:47:09,084] ({main} Server.java[doStart]:327) - jetty-9.2.15.v20160210 WARN [2017-02-0407:47:09,129] ({main} WebAppContext.java[doStart]:514) - Failed startup ofcontexto.e.j.w.WebAppContext@3b938003{/,null,null}{/usr/hdp/current/zeppelin-server/lib/zeppelin-web-0.6.0.2.5.0.0-1245.war}java.lang.IllegalStateException: Failed to delete temp dir/usr/hdp/2.5.0.0-1245/zeppelin/webapps atorg.eclipse.jetty.webapp.WebInfConfiguration.configureTempDirectory(WebInfConfiguration.java:372) atorg.eclipse.jetty.webapp.WebInfConfiguration.resolveTempDirectory(WebInfConfiguration.java:260) atorg.eclipse.jetty.webapp.WebInfConfigurati

When an object gets created in Hive metastore, we expect to see the entry in Atlas almost instantaneously. Often times, might be because of disruption in availability of some of the core components on which Atlas is dependent on or the methodology that was used sometimes to create metastore objects(ex: Directly use metastore API to create Hive objects) where Atlas hook doesn't get invoked, we some times see missing tables in Atlas.This article aims at identifying the missing table entries in Atlas. Once the missing table entries are identified, we can post them to Atlas. The easiest method of doing that will be explained in a later article(Will update the reference link in this article when I am done writing the second article).Step#1: Identify the Hive tables from the metastore.There are multiple ways of gathering this information. The easiest way is to write an SQL over "hive" database to get all the information. In my example, the metastore is based on MySQL. From command line on H

Introduction to Application Timeline Server All the metrics of applications, either current or historic, can be retrieved from Yarn through Application Timeline Server. This will include pieces of information like the number of map tasks, reduce tasks, counters, etc. Application developers can publish the specific information to the Timeline server via TimelineClient in the Application Master and/or the application’s containers. The information is then queryable via REST APIs. The above said are the artifacts of the present application.   Also, generic information of completed applications like queue-name, user information a list of application-attempts that ran for an application, information about each application-attempt can be stored in the Application Timeline Server.   Let us delve into a live example of an Application Server. We are using HDP 3.1 cluster in this example.     Note: The following is for Hadoop Version: The following is a demons

SynopsisThis tutorial will demonstrate using Spark for data processing operations on a large set of data consisting of pipe delimited text files. Delimited text files are a common format seen in Data Warehousing: Random lookup for a single recordGrouping data with aggregation and sorting the output3 Different techniques will be used to solve the above 2 problems and then compare how they perform against each other: Using RDD’sUsing DataFramesUsing SparkSQLThe purpose of this tutorial is to provide you with code snippets for the above 3 techniques and to demonstrate how RDD’s outperform DataFrames and SparkSQL for certain types of data processing. See below at the end of this article for all code. Also, these tests are demonstrating the native functionality within Spark for RDDs, DataFrames, and SparkSQL without calling additional modules/readers for file format conversions or other optimizations. For joining datasets, DataFrames and SparkSQL are much more intuitive to use, especial

Some customers who upgrade from CDSW 1.5.x to 1.6.x - 1.7.x will have deprecated CDSW jobs which use base image v7 which run containers as root.  Base Image v8 now have enhanced security which disallow containers to be ran as root users, but rather as the cdsw user.    It is generally advised to re create these jobs again using base image v8 but in certain environments this is not a feasible option.  The following instructions will assist you in changing the CDSW base image for your jobs.  In this case v7 → v8.   To connect to the cdsw database, gain root on the master node and run kubectl exec -ti `kubectl get pods | grep Running | grep '^db' | awk '{print $1}'` -- psql -U sense Confirm the engine id select * from engine_images; Let's first explore which projects are using which engines. There are three tables involved: the "projects" table, the "engine_images" table, and the one that joins the two together, the "projects_engine_images" table. This

One Way Trust - MIT KDC to Active DirectoryMany security environments have strict policies on allowing administrative access to Active Directory. Some performance issues can also require that Hadoop cluster principals for Kerberos are not created directly in AD. To aid in these situations, it may be preferable to use a local MIT KDC in the Hadoop cluster to manage service principals while using a one-way trust to allow AD users to utilze the Hadoop environment. This tutorial describes the steps necessary to create such a trust.The following assumptions are made for this tutorial:An existing HDP clusterCluster has Kerberos enabled with an MIT KDCThe MIT KDC realm name is HDP.HORTONWORKS.COMThe MIT KDC server is named kdc-server.hdp.hortonworks.comThe AD domain/realm is AD.HORTONWORKS.COMStep 1: Configure the Trust in Active DirectoryCreate a KDC definition in Active Directory On the AD server, run a command window with Administrator privileges and create a definition for the KDC of the

This document seeks to explain the configuration steps required to access NiFi UI using Knox SSO and along with Knox proxy.   This eliminates the need to access NiFi UI using the regular port 9091 on the NiFi node     Configure Knox SSO for NiFi:     Create the CA signed Knox certificate:   Ensure that Knox certificate is signed with the same CA that NiFi certificates are signed. By default, knox creates a self-signed certificate which needs to be replaced by CA signed cert.   Go to knox server and do the following: Copy the CA signed certificate from the knox host as gateway.jks in the knox keystore path. $ cp /etc/security/certs/keystore.jks /var/lib/knox/data-3.1.0.0-78/security/keystores/gateway.jks $ chown knox:knox /var/lib/knox/data-3.1.0.0-78/security/keystores/gateway.jks   Knox certificate’s keystore password should match the knox master secret (let us say, it is Hadoop*123)   Now, let us change the new knox certificate’s password

As the release of Spark 2.0 finally came, the machine learning library of Spark has been changed from the mllib to ml. One of the biggest change in the new ml library is the introduction of so-called machine learning pipeline. It provides a high level abstraction of the machine learning flow and greatly simplified the creation of machine learning process. In this tutorial, we will walk through the steps on how to create a machine learning pipeline and also explain what is under the hood in the pipeline. (Note: the code in this article is based off the technical preview ml library from Spark 1.6 and with minor change, they should run on Spark 2.0 as well)In this tutorial, we will demonstrate the process to create a pipeline in Spark to predict airline flight delay. The dataset we use contain the airline flights information from 2007 and 2008. We use 2007 data as the training dataset and 2008 data as the testing dataset. We will predict flight delay using 2 classification models, logis

Problem Description (The Crash) : During an HDFS rolling restart, the Standby NameNode (SBNN) failed to load the FsImage, causing the SBNN to crash and interrupting the process of rolling restart (that is actually good news). At this stage, the SBNN is down (if it is shown as “started” in CM, you can manually stop it via the CM Web UI), but the Active NameNode (ANN) is still active and operates the HDFS service properly.  This means,   The service is still up and the client can issue Read / Write requests over HDFS.  Do NOT by any means try to restart the ANN.  On the SBNN logs, the following stack trace can be observed:      2020-02-10 13:51:56,845 ERROR org.apache.hadoop.hdfs.server.namenode.FSImage: Failed to load image from FSImageFile(file=/data/nn2/dfs/nn/current/fsimage_0000000007432739660, cpktTxId=0000000007432739660)java.lang.NullPointerException at org.apache.hadoop.hdfs.server.namenode.INodeDirectory.addChild(INod

- We will need to make sure that we have the phoenix client packages installed in the host where we want to run the sqlline utility. If it is not installed then we can install it as following: yum install phoenix -y- Now get the value of "zookeeper.znode.parent" from the Ambari Metrics Collector configurations. In ambari we can get the value from : "Ambari Metrics" -> "config" -> "Advanced ams-hbase-site", Then search for "zookeeper.znode.parent" normally the value will be "/ams-hbase-unsecure" in the AMS configs.- Now we will need to use the "sqlline" utility to connect to it. Here "c6403.ambari.apache.org" is the AMS hostname running in embedded mode and the embedded zookeeper port is 61181/usr/hdp/2.5.0.0-1245/phoenix/bin/sqlline.py c6403.ambari.apache.org:61181:/ams-hbase-unsecure.NOTE: For Kerberized Cluster use the value of "zookeeper.znode.parent" may be "/ams-hbase-secure" so we can connect to it as following:/usr/hdp/2.5.0.0-1245/phoenix/bin/sqlline.py c6403.ambari.ap

With the release of Apache NiFi 1.10 (http://nifi.apache.org/download.html), a new feature is available whereby you can configure a Parquet Record reader to read incoming Parquet formatted files.    Apache NiFi Record processing allows you to read and deal with a set of data as a single unit. By supplying a schema that matches the incoming data, you can perform record processing as if the data is a "table", and easily convert between formats (CSV, JSON, Avro, Parquet, XML). Before record processing was added to Apache NiFi, you would have had to split the file line by line, perform the transformation and then merge everything back together into a single output. This is extremely inefficient. With Record processing, we can do everything to a large unit of data as a single step which improves the speed by many factors.    Apache Parquet is a columnar storage format (https://parquet.apache.org/documentation/latest/), and the format includes the schema for the dat

I have just started working on updated Apache Tika and Apache OpenNLP processors for Apache 1.5 and while testing found an interesting workflow I would like to share.I am using a few of my processors in this flow: https://github.com/tspannhw/nifi-attributecleaner-processor <- Just updated thishttps://github.com/tspannhw/nifi-nlp-processorhttps://github.com/tspannhw/nifi-extracttext-processor <- this needs a major version refreshhttps://github.com/tspannhw/nifi-corenlp-processorHere is the flow that I was working on.Step 1 - Load Some PDFsStep 2 - Use the built-in Apache Tika Processor to extract metadata from the filesStep 3 - Pull Out the Text using my Apache Tika processorStep 4 - Split this into individual linesStep 5 - Extract out the text of the line into an attribute ((^.*$)) into a sentenceStep 6 - Run NLP to analyze for names and locations on that sentenceStep 7 - Run Stanford CoreNLP sentiment analysis on the sentenceStep 8 - I run my attribute cleaner to turn those att

  Introduction Image classification is a powerful tool that can be used to gather insight that is not contained in textual information like (titles, comments, tags, etc...). The objects in the image posted along with a tweet for example may have nothing to do with what was written in the post. In this exercise we will look at how a real time stream of tweets on any topic can be broken down and analyzed in order to classify the main object/subject of a photo by leveraging a deep learning model that has been deployed as a Rest Endpoint via CDSW.  Requirements NiFi CDSW Twitter Developer Tokens - developer.twitter.com A topic of interest Setup To get started we need to get a model deployed in CDSW that will serve as our endpoint for predictions of our incoming images.    In CDSW Create a project and then add the python code we will use to score any image URL:       import jsonimport csvimport keras#import pandas as pandasfrom keras.applications