Resources

Note: This article was written as of HDF 2.1.2 release which is based off Apache NiFi 1.1.0.NiFi has several processors that can be used to retrieve FlowFiles from a SFTP server. It is important to understand the different capabilities each provides so you know when you should be using one vs another.Lets start with the oldest of the available processors: GetSFTPThe GetSFTP processor is the original processor introduced for retrieving files from a remote SFTP server. The processor has the following configurable properties:Things to know about this processor:1. Disadvantage: This processor does not retain any state. This means it does keep track of which files it has previously retrieved. So if property "Delete Original" is set to "false", this processor will continue to retrieve the same file over and over again.2. This processor is not cluster friendly, meaning in a NiFi cluster it should be set to run on "primary node only" so that every node in the cluster is not competing to pull t

Avro is a popular file format within the Big Data and streaming space. Avro has 3 important characteristics that make it a great fit for both Big Data and streaming applications. Avro files are self describing. Meaning the Avro files can be opened and the schema definition viewed as standard JSON or inspected programmatically by numerous applications. This makes your application code much less brittle as the schema information can be obtained from the incoming Avro file itself rather than manually defining them in your code.Avro can handle a wide range of data type natively. This includes things like complex types, Maps, Arrays, and even raw bytes.Avro supports schema evolution which can come in very handy in streaming systems where the data flowing through the system can change without notice. Now that all of the pros of Avro have been called out there is a problem. In reality we rarely encounter Avro files while ingesting/streaming data. Why is this? Mostly because storing data in th

The Extreme Gradient Boosting or Xgboost has received a lot of attention recently and has been the star of the show across many data science workflows and competitions on Kaggle. It is a decision-tree-based model that creates hundreds of “weak learners,” i.e. trees that do not overfit the data. The final predictions or estimations are made using this group of trees, thereby reducing overfit and increasing generalization on unseen data.   For many machine learning use cases, especially in enterprises with exponential amounts of data, Xgboost is a great candidate for distributed ML model building, as it has been proven to scale well linearly along with the number of parallel instances. To achieve this, we can use DASK, an open-source parallel computing framework, inside of CML. While you can use other frameworks like Spark, we chose DASK because it's written natively in Python (Spark is written in Java) and doesn't have the overhead of running JVMs and context switch

The creation of large database schemas can be a very complicated task.  In this article, I am going to share how I used NiFi to fully automate a monstrous task.  For my project, I needed to create very large Avro Schemas, and corresponding Hive Tables for five or more Data Sources, each having from 400-500+ different CSV columns.   Doing this manually would have been a nightmare just to manage the initial schema creations.  Managing schema changes an even bigger task over time.   My answer was the Schema Generator API using NiFi and Schema Registry.     Please reference the following NiFi Template: Schema Generator API Demo Template  Schema Generator API The above NiFi template provides the following capabilities: A NiFi API is capable of the following: Accepting POST for a new table, given the tables data columns and data types is in CSV Format  Creating Schema Registry Entity (POST: create) Creating Schema Registry Avro Schema (POS

I recently ran into an issue with Zeppelin failing to restart all of a sudden. Upon further inspection, the log file never received any additional data as I tailed it. After looking in the log file for Ambari, I found that it was failing because of a missing interpreter.json file.    019-08-29 17:57:35,701 - get_user_call_output returned (0, u'{"RemoteException":{"exception":"FileNotFoundException","javaClassName":"java.io.FileNotFoundException","message":"File does not exist: /user/zeppelin/conf/interpreter.json"}}404', u'')2019-08-29 17:57:35,703 - Creating new file /user/zeppelin/conf/interpreter.json in DFS2019-08-29 17:57:35,704 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -sS -L -w '"'"'%{http_code}'"'"' -X PUT --data-binary @/etc/zeppelin/conf/interpreter.json -H '"'"'Content-Type: application/octet-stream'"'"' --negotiate -u : '"'"'<a href="<a href="http://removedhostname.info:50070/webhdfs/v1/user/zeppelin/conf/interpreter.json?op=CREATE&overwr

    Application deployment has been significantly proliferated by Kubernetes. However, true universal log capture with multi endpoint (downstream) support is lacking. Apache NiFi Stateless provides a possibility to bridge the gap between rapid application deployment and InfoSecs desire to continue to capture and monitor behaviors.   What is NiFi Stateless?  NiFi-Fn is a library for running NiFi flows as stateless functions. It provides delivery guarantees similar to NiFi, without the need for an on-disk repository, by waiting to confirm receipt of incoming data until it has been written to the destination (source NIFI-5922). Try it out Prerequisites K8s (local or cluster).  In this demonstration, Azure Kubernetes Service is used. Some familiarity with K8s & NiFi Assets Used NiFi on K8s https://github.com/sunileman/AKS-YAMLS/blob/master/apache-nifi.yaml Any instance of NiFi will do here.  It does not need to run on K8s. NiFi Regist

Spark job fails with below error when byte code for any particular method grows beyond 64KBspark.sql.codegen.wholeStage is enabled by default for internal optimization in Spark2 which can cause these kind of issues in some corner cases.Below is the detailed stack trace for your reference:org.codehaus.janino.JaninoRuntimeException: Code of method "processNext()V" of class "org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator" grows beyond 64 KBat org.codehaus.janino.CodeContext.makeSpace(CodeContext.java:949)at org.codehaus.janino.CodeContext.write(CodeContext.java:857)at org.codehaus.janino.UnitCompiler.writeShort(UnitCompiler.java:11072)at org.codehaus.janino.UnitCompiler.load(UnitCompiler.java:10744)at org.codehaus.janino.UnitCompiler.load(UnitCompiler.java:10729)at org.codehaus.janino.UnitCompiler.compileGet2(UnitCompiler.java:3824)at org.codehaus.janino.UnitCompiler.access$9100(UnitCompiler.java:206)at org.codehaus.janino.UnitCompiler$12.visitLocalVariableAcce

I am running a free mongodb instance (DBaaS) on at mlab.com.From NiFi, I can read from and write to MongoDB very easily. It is a great way to pull data out of a large collection of MongoDB databases; in some startups or enterprises a lot of little MEAN apps have been written and have small silos of data locked in MongoDB datasets. These can be streamed into a data lake very easily with NiFi. Once stored in HDFS, the data can be accessed via SparkSQL, Hive, Zeppelin and other tools very easily.An example of a Twitter tweet as a JSON document in a MongoDB collection in a MongoDB database stored in an online NoSQL store.The nifi flow for storing to MongoDB is trivial.A simple flow to read MongoDB JSON records and land them as JSON in HDFS.Here is an example of another source to store to HDFS or MongoDB for example. We use a GetHTTP processor to access an SSL protected resource. There are a few options for storing something to MongoDB. You need to format the mongodb URI correctly

Overview This document provides instructions and background information on how to change the Service accounts in an HDP cluster that is managed by Ambari. Service accounts in Hortonworks Data Platform The Hortonworks Data Platform uses services accounts to provide a common user to perform cluster operations and hold ownership of files and folders for a given service. All major services are assigned services account. The Ambari web interface provides a list of service accounts used by a specific cluster. The table below gives a list of common service accounts. Hadoop Service User Group HDFS hdfs hadoop YARN yarn hadoop MapReduce mapred hadoop, mapred Hive hive hadoop HCatalog/WebHCatalog hcat hadoop HBase hbase hadoop Falcon falcon hadoop Sqoop sqoop hadoop ZooKeeper zookeeper hadoop Oozie oozie hadoop Knox Gateway knox hadoop In the course of a standard Ambari installation the service accounts are created in the local Linux systems and assigned the appropr

This is a short video tutorial to configure cross-realm trust between two secure (kerberized) clusters with different realm names. Cluster 1 (c1232) has the realm name SUPPORTLAB.CLOUDERA.COM and Cluster 2 (c4232) has the realm name COELAB.CLOUDERA.COM. This video explains the steps to set up a cross-realm trust in order to perform distcp operation. Open the video on YouTube here

Hi Everyone, Cloudera AI is just one of the many experiences you can use on the Cloudera. Cloudera Machine Learning allows teams to immediately deploy machine learning workspaces that auto-scale to fit their needs and auto-suspend to save costs by using Kubernetes. All of this is packaged into a portable experience that can be accessed by multiple team members easily to provide a consistent experience across an organization.In today's video, I'll walk you through the different high-level features in CML on CDP public cloud.     More information: More information on Cloudera AI https://www.cloudera.com/products/machine-learning.html Documentation on CDP https://docs.cloudera.com/

Objective With the release of NiFi 1.8.0, flowfiles that remain on a disconnected node can be rebalanced to other active nodes in the cluster via offloading. Note: This tutorial assumes you have already setup a running NiFi cluster. The examples included are for a 2 node NiFi cluster.Environment This tutorial was tested using the following environment and components: Mac OS X 10.11.6 Apache NiFi 1.8.0Offload Nodes TutorialQueue FlowFiles First, generate some queued flowfiles: In the Global Menu at the top left, select "Cluster" to see the Cluster Management dialog: The queued flowfiles are distributed between both nodes in the cluster:Disconnect Node Select Disconnect for one of the nodes: The node is disconnected:Offload Node Select Offload () on the disconnected node. This will stop and terminate all processors and rebalance flowfiles to the other connected nodes in the cluster: Note: Offload would also stop transmitting on all remote process groups if they were in the fl

Hi Everyone, In this video I'll show you how to get started with Cloudera Data Warehouse in CDP public cloud. I'll walk you through activating an environment for use with the Data Warehouse experience, creating a Virtual Warehouse, and then loading in some data. After loading data in, I'll show you how to connect your Virtual Warehouse to Tableau. Although we're talking about Tableau specifically here, the same procedure can be replicated with other BI tools.     More information: More information on Data Warehouse https://www.cloudera.com/products/data-warehouse.html Documentation on CDP https://docs.cloudera.com/

Summary Below is an entire summary of different verticals to be considered while securing an Enterprise HDP cluster: FreeIPA / Active Directory for Authentication LDAP - User/Group level Authentication for WebUI & JDBC connections Kerberos - Service level Authentication for Services & JDBC Note: FreeIPA has been recommended here as it is a single solution for LDAP Directory Server, MIT Kerberos, NTP, DNS, Dogtag certificate system, SSSD, and others.Alternatively, Microsoft’s AD (Active Directory) can be used as well Atlas for Data Lineage & Data Governance ( Data Classification, Data Masking - PII, & Data Security) Ranger for Authorization & Auditing Resource-Based security policies on HDP Services TAG Based security policies for data tagged using ATLAS KNOX - SSO(Single Sign-On) and Perimeter Security Encryption (performance impact needs to be considered) Ranger KMS - for Data on Rest Encryption TLS/SSL - for Wire Data Encryption Data Steward Studio - C

I needed to build a quick SQL table from a JSON. There's some online tools, but I'd rather Java this process.It works okay enough, now I am wondering if this would make a good Apache NiFi processorpackage com.dataflowdeveloper.processors.process;import java.util.Iterator;import java.util.Map;import com.fasterxml.jackson.core.JsonFactory;import com.fasterxml.jackson.databind.JsonNode;import com.fasterxml.jackson.databind.ObjectMapper;public class JsonToDDL { /** * * @param tableName * @param json * @return String DDL SQL */ public String parse(String tableName, String json) { JsonFactory factory = new JsonFactory(); StringBuilder sql = new StringBuilder(256); sql.append("CREATE TABLE ").append(tableName).append(" ( "); ObjectMapper mapper = new ObjectMapper(factory); JsonNode rootNode = null; try { rootNode = mapper.readTree(json); } catch (Exception e) { e.printStackTrace(); } Iterator<Map.Entry<String, JsonNode>> fieldsIterator = rootNode.fields(); whi

ObjectiveThis tutorial will walk you through the process of using the PyHive Python module from Dropbox to query HiveServer2. You can read more about PyHive here: PyHivePrerequisites You should already have Python 2.7 installed.You should already have a version of the Hortonworks Sandbox 2.5 setup.ScopeThis tutorial was tested using the following environment and components: Mac OS X 10.12.3Anaconda 4.3.1 (Python 2.7.13)Hortonworks HDP Sandbox 2.5PyHive 0.1.5StepsInstall PyHive and DependanciesBefore we can query Hive using Python, we have to install the PyHive module and associated dependancies. Because I'm using Anaconda, I chose to use the conda command to install PyHive. Because the PyHive module is provided by a third party, Blaze, you must specify -c blaze with the command line.You can read more about Blaze PyHive for Anaconda here: Blaze PyHiveWe need to instal PyHive using the following command: $ conda install -c blaze pyhiveYou will be doing this installation on your local com

This article walks you through the setup of a default 7 node CDP OpDB Data Hub cluster for running Operational workloads in the Public Cloud.   In this article, I have used our internal Sandbox environment which was on CDP 7.0.2 release, in which the users are already set up and authenticated via Okta SSO. In this article, I am using AWS for Public Cloud. You can use Microsoft Azure as well. Support for Google Cloud Platform is currently targeted for later this year.    Navigate to Okta, or your SSO provider home page, if enabled. Search for CDP Public Cloud tile:Or, click directly on the CDP Public Cloud tile Click Data Hub Clusters in the portal: Click Create Data Hub:    Select the following options to create OpDB DataHub Cluster on AWS in the Sandbox environment. Feel free to enter your own Cluster Name under General Settings as shown below:   Also, review the Advanced Options. You need not update anything under it, for this exercise. Note the S

Problem Description:Ambari-infra-solr is running fine but using a "ps" command shows a password like below. According to security policy, this is consider as security breach.The issue occurred because the value of property infra_solr_trust_store_password and infra_solr_key_store_password showing cleartext passwords in java Options.$ ps -ef | grep -i 'ambari-infra'1008 25938 1 21 07:25 ?00:00:11 /usr/jdk64/jdk1.8.0_112/bin/java -server -Xms1024m -Xmx2048m -XX:NewRatio=3 -XX:SurvivorRatio=4 -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=8 -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:ConcGCThreads=4 -XX:ParallelGCThreads=4 -XX:+CMSScavengeBeforeRemark -XX:PretenureSizeThreshold=64m -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=50 -XX:CMSMaxAbortablePrecleanTime=6000 -XX:+CMSParallelRemarkEnabled -XX:+ParallelRefProcEnabled -verbose:gc -XX:+PrintHeapAtGC -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+PrintTenuringDistribution -XX:+PrintG

When using Ambari Metrics in Distributed Mode,after HDP-3/Ambari-2.7 upgrade, HBase metrics are not emitted due to an issue which will likely be fixed after Ambari-2.7.4. We will see similar kind of messages in AMS Collector Log.Error : 2019-06-10 02:42:59,215 INFO timeline timeline.HadoopTimelineMetricsSink: No live collector to send metrics to. Metrics to be sent will be discarded. This message will be skipped for the next 20Debug Error shows this :2019-06-14 20:35:29,538 DEBUG main timeline.HadoopTimelineMetricsSink: Trying to find live collector host from : exp5.lab.com,exp4.lab.com 2019-06-14 20:35:29,538 DEBUG main timeline.HadoopTimelineMetricsSink: Requesting live collector nodes : http://exp5.lab.com,exp4.lab.com:6188/ws/v1/timeline/metrics/livenodes 2019-06-14 20:35:29,557 DEBUG main timeline.HadoopTimelineMetricsSink: Unable to connect to collector, http://exp5.lab.com,exp4.lab.com:6188/ws/v1/timeline/metrics/livenodes 2019-06-14 20:35:29,557 DEBUG main timeline.HadoopT

Note: The ODBC connector is supported for Windows version of Tableau only. Download Hortonworks driver from the link belowhttps://hortonworks.com/downloads/#addons Chose Hortonworks ODBC Driver for Apache Hive (v2.1.10) from Hortonworks Data Platform Add-Ons section (Tested Windows 64 bit)Download and run/install the driver.Create a Data Source Name in ODBC Administrator: Open Data Sources (ODBC) from Control Panel/Administrative Tools.Highlight "Sample Hortonworks Hive DSN" from System DSN tab and click "Configure" buttonEnter the following info Host(s): <HOST_NAME>Port: <PORT>Database: DefaultMechanism: select "User name password" from dropdownUser name: <USER_NAME>Password: <PASSWORD>Delegation UID:Thrift Transport: select "HTTP" from dropdownClick on HTTP Options HTTP Path: gateway/default/hiveClick on SSL Options Check "Enable SSL" checkboxCheck "Allow Common name host name mismatch" checkbox - This option specifies whether a CA-issued SSL certificate

Apache Airflow is a popular environment for scheduling and monitoring workflows. Running workflows is useful in machine learning for automation of data acquisition or the building and monitoring of machine learning models.   This tutorial will take you through the process of setting up Airflow as an Application in CML. This can be very useful in building out testing and prototype pipelines. It also provides a mechanism for calling chains of Models or Jobs.   Create a new Python project in CML using a blank template:New Project Open up a Workspace and install Airflow, this can be scripted using the install instructions here. The following shell script and python scripts can be used to automate this process. The installation needs to be performed only once: install.py:import osos.system("./install.sh")​install.sh:#!/bin/bash -xexport AIRFLOW_HOME=~/airflow# install from pypi using pippip3 install apache-airflow# initialize the databaseairflow initdb​ This will install the Air

Running custom applications in CDSW/CML With the recent addition of Analytical Applications, data scientists and data engineers can now deploy their own custom applications and share them with other users.   Whilst simple applications may have all necessary code already baked into one script file, more complicated applications may require custom launchers, run flags that are set on execution and parameters that may be instance specific. In order to run these applications, we can leverage the Python subprocess module to run the commands that we would normally have manually entered into the terminal. Getting Started For this demonstration, I will show how to run Tensorboard and Hiplot, where both allow for the visualization of parameters from multiple runs of deep learning models.   Both applications rely on a custom command to trigger them as standalone applications: tensorboard for Tensorboard and hiplot for Hiplot. Requirements CDSW 1.7.1 or later versions Python 3 en

IntroductionHive is one of the most common used databases on Hadoop, users of Hive are doubling per year due to the amazing enhancements and the addition of Tez and Spark that enabled Hive to by pass the MR era to a an in-memory execution that changed how people are using Hive.in this blog post, I will show you how to connect squirrel Sql Client to Hive, the concept is similar to any other clients out there as long as you are using the open-source libraries that matches the ones here you should be fine.PrerequisiteDownload Hortonworks Sandbox with HDP 2.2.4, Squirrel SQL ClientStep 1Follow the Squirrel documentation and run it on your Mac or PC.Step 2Follow the Hortonworks HDP Installation on VritualBox, VMware or Hyper-V and start up the virtual Instance.Step 3once you are HDP is up and running, connect it it using SSH as it shows on the console, once you are connected you need to download some JAR files in order to establish the connection.Step 4if you are using MacOS, simply while

Controlling the environment of an application is vital for it's functionality and stability. Especially in a distributed environment it is important for developers to have control over the version of dependencies. In such an scenario it's a critical task to ensure possible conflicting requirements of multiple applications are not disturbing each other.That is why frameworks like YARN ensure that each application is executed in a self-contained environment - typically in a Linux Container or Docker Container - that is controlled by the developer. In this post we show what this means for Python environments being used by Spark.YARN Application DeploymentAs mentioned earlier does YARN execute each application in a self-contained environment on each host. This ensures the execution in a controlled environment managed by individual developers. The way this works in a nutshell is that the dependency of an application are distributed to each node typically via HDFS.This figure s

Hey Everybody, Today I've got a continuation video about Attribute Based Access Control (ABAC) on CDP. ABAC allows you to Tag tables and columns and then set access policies for groups or users that will either grant or deny access to tagged data. At Cloudera Now Jonathan Hsieh showed us how to implement Column Masking with ABAC. So today I decided I would show you how to do the same thing. After that, I'll show you how security policies travel with the data; all because of SDX.   If you haven't checked out the first video go watch that first here: https://community.cloudera.com/t5/Community-Articles/How-to-use-ABAC-in-CDP/ta-p/288056     More information: Watch Cloudera Now https://www.cloudera.com/about/events/cloudera-now-cdp.html Documentation on CDP https://docs.cloudera.com/ More information on SDX https://www.cloudera.com/products/sdx.html