Community Articles

Introduction Continuing my previous article on creating a CDP AWS environment, and a CDP data lake, this tutorial teaches you how to automate the creation of a simple data engineering data hub cluster. You'll notice that once a data lake is setup, launching data hub clusters is very easy!   The cluster generated has the following properties: Template: CDP 1.1 - Data Engineering: Apache Spark, Apache Hive, Apache Oozie Nodes: 1 Master m5.2xlarge 3 Workers m5.2xlarge   Here is the TL;DR: go to my github and run the scripts as instructed.   Automation scripts Step 1: Create Data Hub Cluster  cdp_create_dh_de.sh <prefix>   Step 2: Verify periodically until cluster status is AVAILABLE cdp_describe_dh_de.sh <prefix>   That's it!

IntroductionContinuing my previous article on creating a CDP AWS environment, this tutorial teaches you how to automate the creation of a datalake, including:Setting up proper roles and IDBroker mappings associated with your user and environmentCreating a datalakeSyncing users to FreeIPA Here is the TL;DR: go to my github and run the scripts as instructed. Automation scriptsStep 1: Create IAM and launch Data LakeCreate roles and mapping in your existing environment:cdp_create_group_iam.sh <base_dir> <prefix> <region> Create datalake:cdp_create_datalake.sh <base_dir> <prefix> Step 2: Verify periodically until datalake status is RUNNINGcdp_describe_dl.sh <prefix>  Step 3: Sync free IPA userscdp_sync_users.sh <base_dir> <prefix>  ConclusionThis was a short and sweet tutorial, more fun to come playing with data lake clusters and experiences!

Introduction Cloudera for public cloud has an amazing admin UI that drives you through a thorough wizards for setting up environment, data lakes, data hub clusters and experiences. Details of AWS pre-requisites can be found in CDP official documentation, but why use the easy button when you can leverage AWS and CDP cli and do it the hard way?My thoughts exactly.  Here is the TL;DR: go to my github and run the scripts as instructed. AWS minimal requirements for CDP environmentEven through cli, a lot of the configuration of a environment can be automated, including:Network (VPCs, routing, gateways, etc.)Security GroupsFree IPA instance(s)However, you will need to set this minimal set of elements to get an environment ready:In AWSPublic/Private keys (not automated/demonstrated here)S3 BucketAWS RolesAWS policiesIn CDPCredential (not automated/demonstrated here)To better understand how roles, policies and bucket interact, you can refer to this diagram:In this

A few years ago I switched to using VS Code as my main code / text editor. I find it meets all my personal code development needs. With the release of the new BYOE functionality in CDSW 1.6 and CML, you can now use VS Code to remotely edit (and debug) Python, R and probably Scala code too. Plus you can also run and edit Jupyter Notebooks, all inside VS Code. This is a quick how-to to get it working. Getting ConnectedTo start, you need to set up the Remote Editing feature for your CDSW/CML cluster. You must download the CLI client and add an SSH public key.  The next step is to authenticate and connect to the CDSW server using the CLI client from your local machine.One you are connected, you should see something like this:$ cdswctl ssh-endpoint -p ml-at-scale -m 4 -c 2Forwarding local port 7847 to port 2222 on session bhsb7k4eqmonap62 in project jfletcher/ml-at-scale.You can SSH to the session using    ssh -p 7847 cdsw@localhostNow you need to add an entry

Most of online article already specified the code and submission command for Spark connecting Kafka in kerberos environment. Here we talk other issues you probably encountered when doing debugging, which are of the kind of easily ignored. I experienced and be tortured, but no pain no gain. Here records them and hope do help to you.The environment is HDP 2.6.1.0 with FreeIPA kerberized, Spark 1.6.3 and Kafka 0.10.1 The test code I borrowed from below link, but added lines enabling it running in kerberized.https://github.com/eBay/Spark/blob/master/examples/src/main/java/org/apache/spark/examples/streaming/JavaDirectKafkaWordCount.java package spark.example;import java.util.HashMap;import java.util.HashSet;import java.util.Arrays;import java.util.regex.Pattern;import scala.Tuple2;import com.google.common.collect.Lists;import kafka.serializer.StringDecoder;import org.apache.spark.SparkConf;import org.apache.spark.api.java.function.*;import org.apache.spark.streaming.api.java.*;import org.a

At times customers want to change the default docker bridge network.  Perhaps their monitoring software utilizes that range for example.  The following illustrates the default IP range for docker0   # ip a show docker05: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group defaultlink/ether 02:42:be:4d:f3:e7 brd ff:ff:ff:ff:ff:ffinet 172.17.0.1/16 scope global docker0valid_lft forever preferred_lft forever   CDSW does not use the docker0 bridge within CDSW and this interface is down. To change it however a customer may create a daemon.json file in order to create a custom docker0 range.  Documentation on that is enclosed below.   - https://docs.docker.com/v17.09/engine/userguide/networking/default_network/custom-docker0/ /etc/docker/daemon.json{"bip": "192.168.1.5/24","fixed-cidr": "192.168.1.5/25"}   Thereafter you would stop docker on all nodes and ensure the Docker daemon is up and ifconfig shows the right val

Many customers have a high number of CDSW Models they wish to deploy in their environments.  Some customers have a large number of model requests coming in which would exceed the default 30 second timeout limit of these models. According the CDSW Documentation, Model Replicas are described as "The engines that serve incoming requests to the model."  Models are single threaded and can only process one request at a time   Replicas are utilized for models to ensure some level of load-balancing, fault tolerance, and serving multiple requests.  There is a maximum deployment of 9 replicas per model.   This UI limit within the model can be circumvented by scaling the model manually through Kubernetes commands.   NOTE:  Please perform these at your own risk.   One can attempt the following to scale up their model deployment.   Find model deployments. - kubectl get deployments -all-namespaces   Scale select deployment. - kubectl scale deploymen

Introduction In this article we’re going to look at Logistic regression which is a technique widely used in industry to solve binary classification problems. Unlike the linear regression model, which is used to make predictions on the numerical target variable, logistic regression has the ability to deal with discrete valued outcomes. For example, a numerical target variable might involve predicting house prices, whereas a discrete valued outcome might involve predicting who would win in the next presidential elections. Logistic regression is most commonly used in solving binary classification problems, such as tumor detection, credit card fraud detection, email spam detection, and so on.  How the logistic regression model works The logistic regression model makes use of the sigmoid function (also known as the logistic function) to measure the relationship between input variables and the output variable by estimating probability scores of the outcome. Unlike li

Step by step guide - Shell action in oozie workflow via HueStep1: Create a sample shell script and upload it to hdfs [root@sandbox shell]# cat ~/sample.sh#!/bin/bashecho "`date` hi" > /tmp/output hadoop fs -put sample.sh /user/hue/oozie/workspaces/ [root@sandbox shell]# hadoop fs -ls /user/hue/oozie/workspaces/-rw-r--r-- 3 root hdfs 44 2016-02-15 10:26 /user/hue/oozie/workspaces/sample.shStep 2: Login to Hue Web UI and select Oozie editor/dashboardStep3: Goto "Workflows" tab and click on "Create" buttonStep 4: Fill in the required details and click on save buttonStep 5: Drag shell action between start and end phaseStep 6: Fill in the required details about shell action and click on "Done" buttonStep 7: Submit your workflowYou will see that your job is in progressOutput 1Output 2

Introduction   In this article we’re going to look at linear regression, which is a technique for estimating linear relationships between various features and a continuous target variable. Linear regression provides the ability to measure a correlation between your input data and a response variable. We’ll illustrate this concept with a simple example. Suppose you have collected data for employees’ years of experience and their corresponding salaries. To understand the distribution of this data we can generate a plot as shown in Fig 1. You can see that there’s a linear relationship between years of experience (x) and salary (y). This correlation allows us to build a linear regression model that outputs a salary given years of experience.     Fig1 : Points plotted using training examples (years of experience, salary)   Let’s look at the internals of a linear regression algorithm. The  algorithm will fit a line in the form of y = β0 + β1x to your data. Here we

Overview As you may or may not know, Nifi-Registry 0.2.0 (included in HDF 3.2) allows the storage of versioned flows in a git repository. @Andrew Lim wrote a great article on how to configure this storage for nifi-registry as you can see here: https://community.hortonworks.com/articles/210286/storing-apache-nifi-versioned-flows-in-a-git-repos.html I thought I'd give a few pointers on how to do it in HDF 3.2, following these four simple steps: Step 1: Setup your git repository Step 2: Clone repository on HDF server Step 3: Configure nifi-registry via Ambari Step 4: Restart nifi-registry Step 1: Setup your git repository This step is essentially exactly what was highlighted by Andrew. Create a new repository Create a token Go to to GitHub’s “Developer settings” and create a new “Personal access token”, as follows: Step 2: Clone repository on HDF server Connect to the server running nifi-registry on your hdf cluster. Make sure that git is in

This article shows how to setup and secure a SolrCloud cluster with Kerberos and Ranger. Furthermore it outlines some important configurations that are necessary in order to use the combination Solr + HDFS + Kerberos. Tested on HDP 2.3.4, Ambari 2.1.2, Ranger 0.5, Solr 5.2.1; MIT Kerberos Pre-Requisites & Service Allocation You should have a running HDP cluster, including Kerberos, Ranger and HDFS. For this article I am going to use a 6 node (3 master + 3 worker) cluster with the following service allocation. Depending on the size and use case of your Solr environment, you can either install Solr on separate nodes (larger workloads and collections) or install them on the same nodes as the Datanodes. For this installation I have decided to install Solr on the 3 Datanodes. Note: The picture above is only showing the main services and components, there are additional clients and services installed (Yarn, MR, Hive, ...). Installing the S

Problem Statement : when using a local repository, installation fails due to empty baseurls being written to the Ambari repository (even though local repository baseurl values were provided). The installation fails with below error :stderr: Traceback (most recent call last): File "/var/lib/ambari-agent/cache/stack-hooks/before-INSTALL/scripts/hook.py", line 37, in <module> BeforeInstallHook().execute() File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 352, in execute method(env) File "/var/lib/ambari-agent/cache/stack-hooks/before-INSTALL/scripts/hook.py", line 33, in hook install_packages() File "/var/lib/ambari-agent/cache/stack-hooks/before-INSTALL/scripts/shared_initialization.py", line 37, in install_packages retry_count=params.agent_stack_retry_count) File "/usr/lib/ambari-agent/lib/resource_management/core/base.py", line 166, in __init__ self.env.run() File "/usr/lib/ambari-agent/lib/resource_management/core/environm

Edge Data Processing with Jetson Nano Part 3 - AI Integration Source:   https://github.com/tspannhw/minifi-jetson-nano Part 1:  https://www.datainmotion.dev/2019/07/edge-data-processing-with-jetson-nano.html Part 2:  https://www.datainmotion.dev/2019/07/edge-processing-with-jetson-nano-part-2.html   Cloudera Edge Management         Top Level NiFi Flow Receiving MiNiFi Agent Messages       Overview of our Apache NiFi Flow For Processing                 We format a new flow file to send to CDSW in JSON to the CDSW Job Environment       We Run Apache MXNet 1.3.1 (Java) SSD Against the Web Camera Image   Extract The Values From the FlowFile to Send to the Spark Job     Our JSON Results From the Logs           Log data has successfully arrived, consistent JSON rows are grabbed as they are written to the

Customers at times have users that utilized CDSW in the past but are no longer in the organization.  Unfortunately there are limitations within CDSW to 1) Delete those users, and 2) Delete projects associated with those users.  As of CDSW 1.6 administrators can only "disable" users.  Deleting projects associated with the users is a bit convoluted but possible.  This requires access to the underlying PSQL database within CDSW & Kubernetes. To accomplish the deletion of projects associated with a user (in this example let's say we are removing projects from /var/lib/cdsw/current/projects/projects/0/) you will have to create two queries to execute this.   1) Create query to find which users are banned/disabled within CDSW. # kubectl exec -it $(kubectl get pods -l role=db -o jsonpath='{.items[*].metadata.name}') -- psql -P pager=off -A -t -U sense -c "SELECT username FROM users where banned='t'" >> /tmp/list_usersdisabled.tx2) Cr

Ranger audit logs are stored in HDFS for long term storage, and can grow to a significant size after while. It might be useful to turn those logs into Hive tables using ORC as the backing store for compression and query speed. This will allow you to connect any external tool via ODBC to Hive and view your audit logs as SQL tables.  Each plugin for Ranger that you have enabled will log to a different sub directory within /ranger/audit on HDFS:  [hdfs@ip-10-0-1-75 ~]$ hdfs dfs -ls /ranger/audit/Found 2 itemsdrwx------ - hdfs hdfs 0 2019-10-10 14:19 /ranger/audit/hdfsdrwxr-xr-x - hive hive 0 2019-10-10 14:12 /ranger/audit/hiveServer2   And then within each of these, you will have a sub-directory for audit logs for every day: [hdfs@ip-10-0-1-75 ~]$ hdfs dfs -ls /ranger/audit/hdfsFound 16 itemsdrwx------ - hdfs hdfs 0 2019-09-25 09:55 /ranger/audit/hdfs/20190925drwxr-xr-x - hdfs hdfs 0 2019-09-26 00:00 /rang

IntroductionTransparent Data Encryption (TDE) encrypts HDFS data on disk ("at rest"). It works thorough an interaction of multiple Hadoop components and security keys. Although the general idea may be easy to understand, major misconceptions can occur and the detailed mechanics and many acronyms can get confusing. Having an accurate and detailed understanding can be valuable, especially when working with security stakeholders who often serve as strict gatekeepers in allowing technology in the enterprise.Transparent Data Encryption: What it is and is notTDE encrypts HDFS data on disk in way that is transparent to the user, meaning the user accesses the HDFS data encrypted on disk identical to accessing non-encrypted data. No knowledge or implementation changes are needed on the client side and the user sees the data in its unencrypted form. TDE works at the file path or encryption zone level: all files written to designated paths (zones) are encrypted on disk.The goal of TDE is to prev

This video describes how to configure HiveServer 2 HA and how Load Balancer works with HiveServer 2. Open the video on YouTube here

This article will explain how to setup a static edge node as an entry point to a Hadoop cluster deployed using Cloudbreak. This setup it's not supported by default, because it's not possible to join external vm's in a cluster deployed with Cloudbreak, but in the next article you will find the steps in order to do it.   Cloudbreak is a manager/operator tool to deploy clusters that provides some extra features like scaling, start/stop resources etc in a simple UI. It's a good tool that is growing and sometimes could be a bit rigid and for some core changes on the cluster you will need to deploy a new cluster, for example to change the type of VM or change the HDP version etc. For this reason in our case we decided to setup a static IaaS virtual machine as edge node, that give us a single and immutable entry point to the cluster, this will never change even we will deploy a new cluster. With this and Knox server we have the ability to deploy a new cluster and then make the switch wi

This video is about Hive Session, the timeout values and their effect on the Hive Services load. The video covers the following topics: Understanding Hive session and operations. Timeout parameters for Hive session and operations. Effect of idle sessions. Tuning the timeout values. Open the video on YouTube here

Using NiFi REST API for unsecured cluster is straight-forward like below:[root@<nifi-host> ~]# curl -v -X GET http://<nifi-host>:<port>/nifi-api/flow/current-user* About to connect() to <nifi-host> port <port> (#0)* Trying <IP address>...* Connected to <nifi-host> (<IP address>) port <port> (#0)> GET /nifi-api/flow/current-user HTTP/1.1> User-Agent: curl/7.29.0> Host: <nifi-host>:<port>> Accept: */*>< HTTP/1.1 200 OK< Date: Fri, 30 Jun 2017 22:15:09 GMT< X-Frame-Options: SAMEORIGIN< X-Frame-Options: SAMEORIGIN< X-Frame-Options: SAMEORIGIN< Cache-Control: private, no-cache, no-store, no-transform< Server: Jetty(9.4.3.v20170317)< Vary: Accept-Encoding, User-Agent< Date: Fri, 30 Jun 2017 22:15:09 GMT< Date: Fri, 30 Jun 2017 22:15:09 GMT< Content-Type: application/json< Content-Length: 439<* Connection #0 to host <nifi-host> left intact{"identity":"anonymous","ano

Source Code:  https://github.com/tspannhw/flume-to-nifi   Consume / Publish Kafka And Store to Files, HDFS, Hive 3.1, Kudu  Consume Kafka Flow    Merge Records And Store As AVRO or ORC Consume Kafka, Update Records via Machine Learning Models In CDSW And Store to Kudu  Source:  Apache Kafka Topics You enter a few parameters and start ingesting data with or without schemas.   Apache Flume had no Schema support.   Flume did not support transactions.   Sink:   Files   Storing to files in files systems, object stores, SFTP or elsewhere could not be easier.  Choose S3, Local File System, SFTP, HDFS or wherever. Sink:   Apache Kudu / Apache Impala  Storing to Kudu/Impala (or Parquet for that manner could not be easier with Apache NiFi).  Sink:   HDFS for Apache ORC Files When completes, the Conv

Oracle SQL Developer is one of the most common SQL client tool that is used by Developers, Data Analyst, Data Architects etc for interacting with Oracle and other relational systems. So extending the functionality of SQL developer to connect to hive is very useful for Oracle users. I found this original article on oracle’s website and made some additions based upon the issues that I ran into. Here is the original linkOracle SQL Developer support for HiveHere are the steps that i followedStep1) For the latest version of SQL developer you would need JDK 1.8 so you would need to install that on your mac and also change the JAVA_HOME path so that it points to JDK 1.8.Download JDK 1.8Step2) Download the latest version of Oracle SQL developer for mac, from oracle and unzip itOracle SQL Developer DownloadMove the SQL Developer file to your application, so that it is available for you. Now when you try to open up Oracle SQL developer on mac, it may not open. For me it showed up in the tray, bl

One of the first cases we get to see with Hbase is loading it up with Data, most of the time we will have some sort of data in some format like CSV availalble and we would like to load it in Hbase, lets take a quick look on how does the procedure looks like:lets examine our example data by looking at the simple structure that I have got for an industrial sensor id, temp:in,temp:out,vibration,pressure:in,pressure:out 5842, 50, 30, 4, 240, 340First of all make sure Hbase is started on your Sandbox as followingCreating the HBase Table Login as Root to the HDP Sandbox and and switch to the Hbase User root> su - hbase Go to the Hbase Shell by typinghbase> hbase shell Create the example table by typinghbase(main):001:0> create 'sensor','temp','vibration','pressure' lets make sure the table was created and examine the structure by typinghbase(main):001:0> list now, exit the shell by typing 'exit' and lets load some dataLoading the Data lets put the hbase.cs

ENVIRONMENT and SETUPI did test the below solution withHDP-2.6.0.3-8Ambari 2.5.0.3To configure this environment, follow up the steps defined for HDP 2.5 (Zeppelin 0.6) - https://community.hortonworks.com/articles/81069/how-to-enable-user-impersonation-for-sh-interprete.htmlAs the Zeppelin UI -> Interpreter window has changed, follow this up.I. Default config looks like below: II. Do the following change:- edit the interpreter - change The interpreter will be instantiated from Globally to Per User and shared to isolated - then you will be able to see and select User Impersonate check-box - also removezeppelin.shell.auth.typezeppelin.shell.keytab.locationzeppelin.shell.principal III. Changed config looks like below:After Saving the changes and running below as admin%shwhoamiWe get