Community Articles

IntroductionRecently a customer asked me how to change destination of a connection which still contains data, but the destination is stopped. Using NiFi REST Api we can change the flow, here in this article I am trying to capture steps to update destination of a connection using REST API. The requirement around this was to push incoming data to different flows on a timely manner.Prerequisites1) To test this, Make sure HDF-2.x version of NiFi is up an running2) Minimum 3 processors are on the canvas with connection like below:3) Note the destination's uuid [In my case 'PutNext' processor's uuid]4) Note the uuid of the connection that has to be made to the 'PutNext' processor'GET'ing the details of connection1) Execute the below command on your terminal with uuid of the connection:curl -i -X GET http://localhost:8080/nifi-api/connections/dcbee9dd-0159-1000-45a7-8306c28f2786 2) Now Copy the result of the GET curl command and update the below section with uuid of the 'PutNext' processor."d

There are a number of technology trends today that organizations are thinking about to take their business applications to the next level. Cloud adoption and machine learning are two of these. However, one of the biggest changes we’ve observed at our customers is the drive to build and scale out streaming based applications.    It’s a given that many organizations have legacy analytics systems that are built on traditional batch ingestion technology. Depending on the use case, that might be fine; but it’s clear that there are many advantages to considering the benefits of a real-time approach to data ingestion, processing, monitoring, and data analysis.   Whilst we’re not suggesting that you completely ditch your batch processing systems (well not just yet!), here are some considerations regarding your existing applications that may be designed in this way: By the time you process the data and perform any analysis - is the data already out of date? What is the opportuni

Sometimes when we create our clusters, we use a small amount of Kafka brokers. After some time, there may be a requirement for adding more brokers, usually because of load, performance or high availability. In this article, we provide the considerations and steps to keep in mind after adding new brokers to HDP/HDF clusters. If there is one Kafka broker in a cluster and plans are to add more brokers for high availability, it's important to mention that Kafka has an internal topic called __consumer_offsets. This topic is created by Kafka internally to store the consumer's committed offsets. At this point, if there is only one broker, Kafka will, by default, create this topic using one single replica and 50 partitions. As a result, it's highly recommended to change the number of replicas for this topic when adding more brokers to the cluster. These numbers are handled by the following properties:   offsets.topic.num.partitions: The number of partitions for the offset comm

To access the (remote) HDFS of a CDP Data Hub cluster with Spark from within a CML session, you need to provide the hdfs-site.xml and core-site.xml of that specific cluster by copying them into the /etc/hadoop/conf/ folder of your CML session.   Below is some sample code that shows how to write data from your CML session to the remote HDFS and read it from there. This will automatically pull the required config files from a defined Data Hub cluster (REMOTE_HDFS_MASTER) and place it in the correct location of your CML session.    # This example shows how to write a file to the (remote) HDFS of a CDP Data Hub cluster and read it from there.from __future__ import print_functionimport sys, re, subprocessfrom pyspark.sql import SparkSession#Set the user name which will be used to login to the HDFS Master.USER_NAME = subprocess.getoutput("klist | sed -n 's/.*principal://p' | cut -d@ -f1")#FQDN of the HDFS MasterREMOTE_HDFS_MASTER = '<REPLACE ME with FQDN of remote HDFS Ma

Hey Everyone, Column Masking and Row Filtering are powerful tools in CDP that admins can use to implement security rules without creating dozens of views that must be maintained over time. With Column Masking, admins can block or limit the extent of data specific users or groups are allowed to see. If instead an admin needs to limit the rows a user can see, (for example US users can only see data from the USA) Row Filtering can be used. In this video I walk through an example where we limit the amount of data call center employees are allowed to see. I'll show you how we set up these policies in Ranger and what effect it has on the user.     More information: Documentation on CDP https://docs.cloudera.com/ More information on SDX https://www.cloudera.com/products/sdx.html

JDK/Python Changes Causing Ambari Server/Agent Registration IssuesRecent JDK and Python updates have introduced behavior changes that can affect the Ambari Server to Ambari Agent registration process. The Ambari Server and Ambari Agent use TLS to register with each other securely. Recent changes to JDK and Python have been made to increase security by eliminating the use of insecure cipher suites and protocols. These changes ensure that more secure protocols and cipher suites are used by the Ambari Server when setting up its TLS sockets, and as a result, require the Ambari Agent’s Python client to be configured to use later versions of the TLS protocol to communicate with the Ambari Server.You may be affected by these changes if you’ve recently upgraded your Python or JDK version and are noticing that your Ambari Agents are not heartbeating back to the Ambari Server and you see entries that resemble this in your Ambari Agent logs:WARNING 2018-04-24 16:35:10,989 NetUtil.py:124 - Server

Hey everyone, We have a new video for the SDX collection and this time we're talking about Attribute Based Access Control (ABAC). Cloudera Allows you to use ABAC on data access across the entire platform. ABAC allows you to Tag tables and columns and then set access policies based on those tags to allow specific groups or users to access that tagged data. In this video I'll show you how to setup ABAC in CDP using Atlas and Ranger. I'll walk through a couple examples to show you how ABAC can be safer and easier than implementing resource based policies.     More information: Documentation on CDP https://docs.cloudera.com/ More information on SDX https://www.cloudera.com/products/sdx.html

When we face high CPU issues, the next step is to identify the application and the thread causing the same. This article explains how to identify the thread/s causing high CPU issues.   In order to identify high CPU issues, the first thing we have to do is to identify the PID. This can be done by using a simple "top" command, so if you see a PID constantly using more than 70-100% CPU, then run the following command:   top -H -b -n1 -p <PID>   The command above will show us all the threads associated with that specific PID. The following is an example:   [kafka@c489-node2 conf]$ top -H -b -n1 -p 25910top - 14:21:54 up 84 days, 12:32, 1 user, load average: 6.79, 6.62, 6.69Threads: 145 total, 0 running, 145 sleeping, 0 stopped, 0 zombie%Cpu(s): 18.3 us, 4.0 sy, 0.0 ni, 77.6 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 stKiB Mem : 20971520 total, 16874368 free, 1725732 used, 2371420 buff/cacheKiB Swap: 5242872 total, 5242872 free, 0 used. 17874368 av

  In this article we’ll take an end to end look at the execution of a hive query.  We will focus on the query execution in the context of both Hive on Tez, and Hive on Map Reduce.    We will assume that our data includes customer data, so we will want to apply granular access control down to the column level, to secure PCI, and PII data.  This means we will be using Hive in conjunction with Apache Ranger (the standard open source Apache project for securing data in Hive).  Lets begin: 1) Client issues the SQL Query The query will be issued from a client such as a JDBC client, or Beeline.  JDBC clients might include any of the standard BI tools such as Tableau.  Beeline is the supported CLI for Hive 2 and 3, and it will also use a JDBC connection to connect to the remote Hive Server.   The JDBC URL for these connections can be easily retrieved in CDP-Data Warehouse dashboard, as shown in the dashboard below:&nbsp

Exploring Apache NiFi 1.10:   Stateless Engine and ParametersApache NiFi is now available in 1.10!https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12344993You can now use JDK 8 or JDK 11!   I am running in JDK 11, seems a bit faster.A huge feature is the addition of Parameters!   And you can use these to pass parameters to Apache NiFi Stateless!A few lesser Processors have been moved from the main download, see here for migration hints:https://cwiki.apache.org/confluence/display/NIFI/Migration+GuidanceRelease Notes:   https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.10.0Example Source Code:   https://github.com/tspannhw/stateless-examplesMore New Features:ParquetReader/Writer (See:  https://www.datainmotion.dev/2019/10/migrating-apache-flume-flows-to-apache_7.html)Prometheus Reporting Task.   Expect more Prometheus stuff coming.Experimental Encr

Hey everyone, Cloudera Manager's look has been updated with the arrival of the Cloudera. Although CDP is largely configured and controlled through the Control Plane, there are still some options available to you in Cloudera Manager when working with an Environment or in a Data Hub cluster. This quick tour of the different views and menus will hopefully help you align yourself to the new layout.     More information: More information on Data Hub https://www.cloudera.com/products/data-hub.html Documentation on Cloudera Data Platform https://community.cloudera.com/t5/Community-Articles/Cloudera-Data-Platform-documentation/ta-p/270222

Over the last fews weeks as customers have started to ramp up their usage of CDP cloud assets (like DataHub, Experiences, etc), I have observed many of the ways they are leveraging on-prem engineering assets (code) in the cloud.  The concept of write-once-deploy-anywhere is fundamental to a well designed data strategy.  It's NOT a sales pitch.  It's a reality for enterprises who have invested in a Modern Data Architecture.  However, unlike on-prem where storage and services are tightly coupled, CDP flips that concept on its head.  We can now launch services independently and choose only the capabilities we need for the task at hand.  For example, streaming use cases typically require NiFi, Kafka, and Spark Streaming.  Each of those services would be separate DataHub clusters and scale independently.  This article focuses on using PySpark to read from a secured Kafka instance.  To be clear, this is one way (not the only way) of using PySpark

Target Audience: It is recommended for those who have some operational experience in Zeppelin. Intention of the article is to give a fair idea on key factors which plays significant role in zeppelin's stability and performance. Disclaimer: This article is based on my personal experience and knowledge. Don't take it as a standard guidelines, understand the concept and modify it for your environmental best practices and use case. A common question everyone has, How many users can use a single zeppelin instance? It may vary from 10 to 50 users, we can't provide a number as an answer since zeppelin is a Multi-purpose Notebook and used for variety of use case with different interpreters and interpreter modes. Below are the factors that needs to be considered while deciding how many users can share/use a single instance in an environment. Often, tweaking these factors can give lot of performance boost, try doing the below checks before considering to scaling out for more instances. For scali

In this article I am going to explain how to use the Schema Registry API to Create and Delete a Schema Entity.     I am working on a Use Case for creating schemas with over 500 columns on the fly within a NiFi Data Flow.  In order to complete this task I needed to figure out how to create, and update the schemas from outside of the Schema Registry UI.   While testing I also needed to delete the test schemas since there is no way to delete them from within the UI.   First, to get the artifacts I need (API urls, and post formats), I use the Schema Registry UI in Chrome with my Developer Tools open to capture what happens when the UI buttons are clicked.  When I add a test schema, I notice 2 POST calls.   Creates the Schema Entity (without an actual Schema) Posts the Schema to the newly created Entity Using the information in Developer Tools, I then create 2 Postman api calls to duplicate, debug, and validate they will work externally.   ** Note:

Introduction to Hive and LLAP  Hadoop is the backbone of big data applications. The hallmark of Hadoop is its efficiency to process large volumes of data on a cluster of commodity servers. Apache Hive is the open-source data warehouse system built on top of Hadoop for querying and analyzing large datasets stored in Hadoop files. One of the greatest challenges that Hive faced was its perceived slow performance. Hive on TEZ was a leap in the right direction for performance. It performed better than its predecessor MR execution engine. LLAP, known as Long Live and Process, essentially is the next-generation execution engine with a better performance and response time. LLAP is very similar to Hive, the main difference being the worker tasks runs inside LLAP Daemons and that too inside tiny computational units called Executors. However, in Hive, these tasks are run by containers. The LLAP daemons are persistent and execute fragments of Hive Queries.   What are the core components

 What to ask before making Data Flow When taking on a new responsibility for designing and maintaining data flows. What are the main question one should ask to ensure a good outcome?  Here I list the key questions for important topics, as well as an illustration of what typically goes wrong if the questions are not asked. The most important points if you are under pressure to deliver LocationThe questions: Where is the data, where should it go (and where can I process it). And of course: Do I have the required accessThe Nightmare: Data is spread across multiple systems, one of these may not be identified. After you finally figure out which tables you need you try to start and don’t have access. When you finally get the data you either don’t have a compliant place to put it, or you are missing a tool. Finally you have the data but it is unclear how to get it written to the target. In the end a 3 day job takes 6 weeks.ContextThe questions: What is the data,

Bulk loading is the process of preparing and loading HFiles (HBase’s own file format) directly into the RegionServers, thus bypassing the write path. This obviates many issues, such as:MemStores getting fullWALs getting biggerCompaction and flush queues getting swollenGarbage Collector getting out of control because of inserts range in the megabytesLatency shoots up while importing dataHBase already has a replication functionality called Cluster Replication (see official documentation here). Why isn’t that enough if you want to replicate the bulk loaded data as well?The “standard” replication uses a source-push methodology. When the active cluster (source) receives an edit to a column family with replication enabled, that edit is propagated to all destination clusters using the WAL for that column family on the RegionServer managing the relevant region.However, in the case of bulk load, only the event (the fact that there is bulk load happening) is captured in the WAL, with refere

Update 10.12.2016: Added filter to rewrite proxyUser as authenticated user Update 25.01.2017: Improved service.xml and rewrite.xml1 Configure a new service for Livy Server in Knox 1.1 Create a service definition $ sudo mkdir -p /usr/hdp/current/knox-server/data/services/livy/0.1.0/ $ sudo chown -R knox:knox /usr/hdp/current/knox-server/data/services/livy Create a file /usr/hdp/current/knox-server/data/services/livy/0.1.0/service.xml with <service role="LIVYSERVER" name="livy" version="0.1.0"> <routes> <route path="/livy/v1/sessions"> <rewrite apply="LIVYSERVER/livy/addusername/inbound" to="request.body"/> </route> <route path="/livy/v1/**?**"/> <route path="/livy/v1"/> <route path="/livy/v1/"/> </routes></service> Note that the name "livy" attribute and the path .../services/livy/... need to be the same.The route /livy/v1/sessions is a special treatment for the POST request to create a Livy sessio

In today's short video we'll talk about how CDP makes onboarding enterprise users easier. If your Identity Provider (that provides SSO services) can use SAML, then CDP can import users along with their group associations. CDP will also make sure to sync any policy modifications across your clusters and the different experiences.   Let us know by adding a comment below what topics you care about and keep checking back as we will be posting new videos about CDP and more at least once a week.   More information: Documentation on onboarding users https://docs.cloudera.com/management-console/cloud/user-management/topics/mc-onboarding-users.html What is Cloudera's Shared Data Experience (SDX)? https://community.cloudera.com/t5/Community-Articles/What-is-Cloudera-s-Shared-Data-Experience-SDX/t...  

Heterogeneous Storage in HDFSHadoop version 2.6.0 introduced a new feature heterogeneous storage. Heterogeneous storage can be different according to each play their respective advantages of the storage medium to read and write characteristics. This is very suitable for cold storage of data. Data for the cold means storage with large capacity and where high read and write performance is not required, such as the most common disk for thermal data, the SSD can be used to store this way. On the other hand when we required efficient read performance, even in rate appear able to do ten times or a hundred times the ordinary disk read and write speed, or even data directly stored memory, lazy loaded hdfs. HDFS heterogeneous storage characteristics are when we do not need to build two separate clusters to store cold thermal class II data within a cluster can be done, so this feature is still very large practical significance. Here I introduce heterogeneous storage type, and if the flexible con

ObjectiveThis tutorial is the final article of a three part series. We will look at the flow components in greater detail to highlight best practices and potential trouble spots when working with CDC in NiFi. Part 1 Part 2EnvironmentThis tutorial was tested using the following environment and components:Mac OS X 10.11.6 MySQL 5.7.13 Apache NiFi 1.3.0 CDC Flow Configuration Details, Best Practices & TroubleshootingPrerequisitesIt's helpful to have completed the first two articles in this series to have a point of reference for the items discussed below. But, this article will hopefully still be useful for those who are interested in some of the nuances of the CaptureChangeMysql, EnforceOrder and PutDatabaseRecord processors.State in CaptureChangeMysqlThe CaptureChangeMySQL processor has a "Distributed Map Cache Client" property which is set to a DistributedMapCacheClientService controller service to store State information and keep it updated. The DistributedMapCacheClientService co

In terms of Azure HDInsight environment, here are few things to be aware of in terms of infrastructure: You have option to install HDInsight on Windows or HDInight on Linux (only on ubuntu 12 LTS). Apache Ambari only comes only with Linux based install. Type of machines used for Linux-based install were limited to D3, D4 & D12. Not sure if this is because of my Azure account limitations. HDInsight version is 3.2.1 which comes with HDP 2.2 certain components. Separate cluster required for Hadoop, Hbase , Storm. And Spark is available as Technical preview.Uses Blob storage as default for HDFS. Not sure if there is a option to add VHD or SSD. HDInsight 3.2 does not contain Falcon, Flume, Accumulo, Ambari Metrics, Atlas, Kafka, Knox, Ranger, Ranger KMS & Slider. Also it has a bit older version of hadoop components.Attached is a file that has comparison of HDInsight 3.2.1 components to that of HDP 2.3.2.hdinsight-and-hdp-component-comparison.zipUpdate by @Ancil McBarnettHDInsight

ObjectiveThis tutorial walks you through the process of using the CaptureChangeMySQL, EnforceOrder and PutDatabaseRecord processors in Apache NiFi to replicate a database using MySQL event logs.This article is the first of a three part series. We will setup the demo environment and components, create a multi-row MySQL table (which generates binary log data) and import a NiFi CDC flow template. The second part of the series will walk-through the CDC flow showing how the MySQL bin logs can be used to create a copy of the table and how the flow keeps the copy in sync with any row-level changes to the source table. The last article in the series will delve into the finer details of the flow configuration, suggest best practices and highlight potential trouble spots.EnvironmentThis tutorial was tested using the following environment and components:Mac OS X 10.11.6 MySQL 5.7.13 Apache NiFi 1.3.0Note: CaptureChangeMySQL, EnforceOrder and PutDatabaseRecord processors were introduced in Apache

A temporary table is a convenient way for an application to automatically manage intermediate data generated during a complex query. Rather than manually deleting tables needed only as temporary data in a complex query, Hive automatically deletes all temporary tables at the end of the Hive session in which they are created.The data in these tables is stored in the user's scratch directory rather than in the Hive warehouse directory. The scratch directory effectively acts as the user' data sandbox, located by default in /tmp/hive-<username>.Hive users create temporary tables using the TEMPORARY keywordCREATE TEMPORARY TABLE tmp1(c1 string);CREATE TEMPORARY TABLE tmp2 AS..CREATE TEMPORARY TABLE tmp3 LIKE..Multiple Hive users can create multiple Hive temporary tables with the same name because each table resides in a separate session.Temporary tables support most table options, but not all. The following features are not supported:Partition columnsIndexesA temporary table with the

Hey everyone,  In our last video we created a Data Hub cluster in CDP. But what if you don't want to go with the default settings? In this video we'll cover what advanced options are available to you when creating a new Data Hub cluster. For example, you can change what instance types certain host groups run on. Or, specify if scripts should run while the cluster is being created.   Let us know by adding a comment below what topics you care about and keep checking back as we will be posting new videos about CDP and more at least once a week.     More information: Cloudera Data Hub - What You Should Know https://community.cloudera.com/t5/Community-Articles/Cloudera-Data-Hub-What-You-Should-Know/ta-p/284608 Cloudera Data Hub – How to Create a Cluster https://community.cloudera.com/t5/Community-Articles/Cloudera-Data-Hub-How-to-Create-a-Cluster/ta-p/285285 Documentation on CDP https://community.cloudera.com/t5/Community-Articles/Cloudera-Data-Platform-documentation/