Resources

The all new Cloudera Data Engineering ExperienceThe all new Cloudera Data Engineering Experience I recently had the opportunity to work with Cloudera Data Engineering to stream data from Kafka.  It's quite interesting how I was able to deploy code without much worry about how to configure the back end components.   DemonstrationThis demo will pull from the Twitter API using NiFi, write to payload to a Kafka topic named "twitter".  Spark Streaming on Cloudera Data Engineering Experience CDE will pull from the twitter topic, extract the text field from the payload (which is the tweet itself) and write back to another Kafka topic named "tweet" The following is an example of a twitter payload.  The objective is to extract only the text field:What is Cloudera Data Engineering?Cloudera Data Engineering (CDE) is a serverless service for Cloudera Data Platform that allows you to submit Spark jobs to an auto-scaling cluster. CDE enables you to spend

LLAP Service sometimes may fail to startup after HDP upgrade with the following error:    Error::Caused by: org.apache.hadoop.security.KerberosAuthException: failure to login: for principal: hive/prakash.cloudera.com@CLOUDERA.COM from keytab /grid/0/yarn/local/usercache/hive/appcache/application_**********/container_***********_01_000001/keytabs/llap0.keytab javax.security.auth.login.LoginException: Pre-authentication information was invalid (24)   To resolve this issue, run the following command and start LLAP:  yarn app -destroy llap0 If the issue persists, then do the following   Check the hive.service.keytab file is available in all the hosts, run kinit and test the connectivity. If the file is missing, recreate the keytabs on the missing hosts either through Ambari or manually (see this article for manual keytab creation). Verify the file in HDFS and copy it to the local. Perform kinit with the copied keytab. hdfs:///user/hive/.yarn/keytabs/hive/hive.ser

Accessing CDP Endpoints using Postman or cURL When automating, testing, or validating APIs within the Cloudera Data Platform within our AWS, Azure, or GCP infrastructures, most developers utilize either curl or postman.  In this article, we will highlight the proper API endpoints, proper credentialing through Knox, and the setting of workload passwords.     Details Needed to Access Endpoints There are several details to access specific application endpoints within CDP: The Endpoint URL for the Specific Application Your specific username Your Workload Password The Endpoint URL for the Specific Application Each application within a CDP environment will generate a unique URL for both GUI and API development.  As an example for DataHub Clusters, you can view the API endpoints by clicking on the Cluster details, scrolling to the bottom, and selecting "Endpoints."    Within the Endpoints section, you'll find the appropriate API endpoints for each of the

Hey Everyone! In this video we're going to go over some more advanced features of the Cloudera Data Engineering Experience. Using some publicly accessible Paycheck Protection Data, you'll see how to automatically setup Spark jobs to deploy by using the CDE CLI, making development and deployment times much quicker and painless. We'll also take the development cycle through to the end and get some visualization of the finished reports using the aforementioned PPP data.     More information: A tutorial on using the Cloudera Data Engineering CLI Using CLI-API to Automate Access to Cloudera Data Engineering (Tutorial) A related Blog post: Using Cloudera Data Engineering to Analyze the Paycheck Protection Program Data A related Meetup event: Creating an Automated Spark Development Environment with Analysis and Logging Documentation on Cloudera Data Engineering Cloudera Docs / Data Engineering

This video covers Livy's Feature, Operational Flow and Basic Demo.   Open the video on YouTube here

Inspired by @sunile_manjee 's article How to use K9s to fetch metrics and logs for Cloudera Data Warehouse Experience , I decided to create the same tutorial for Cloudera Data Engineering. The process is very similar, as you can see below. Pre-Requisites CDE environment setup K9s installed on your machine (instructions here) AWS user ARN AWS configure (CLI) pointing to your AWS env (instructions here) Step 1: Add your AWS user to the CDE environment In your AWS account console, go to IAM > Users > Search for your user name > copy your ARN: In the CDE main page, go to environment details: Then, go to access and add your ARN: Step 2: Download the kubeconfig In the CDE main page, click on the 3 dots, then Download Kube Config: Step 3: Run K9s You can now run k9s, by running:k9s --kubeconfig ~/Downloads/kubeconfig​Note: the path to your downloaded kubeconfig is subject to change of course.    

Hi, In this article, let us take a look at how to delete a schema from the Hortonworks Schema Registry. Let me start with a word of caution that the approach is not recommended for Production systems and use these steps at your own risk. Also, would like to thank Brian Goerlitz for his ideas towards this post. Currently it is not possible to delete a schema from the UI. So the steps below shows how to delete the schema from its backend datastore. I am using MySQL as my backend datastore for the schema registry and the queries will be related to MySQL. You should change them according to your database type. Step 1Verify that the two tables schema_version_info and schema_field_info have CASCADE ON UPDATE and CASCADE ON DELETE enabled. This can be done by the below queries on information_schema databaseselect UPDATE_RULE,DELETE_RULE,REFERENCED_TABLE_NAME from REFERENTIAL_CONSTRAINTS where table_name='schema_version_info';andselect UPDATE_RULE,DELETE_RULE,REFERENCED_TABLE_NAME from REFER

As we know many services like Atlas for lineage, Ranger for audit logs, log search and so on uses Ambari Infra (Solr) for indexing data. So moving Ambari Infra in production and keeping it stable and up is really important. This are the key points I came up with to make this happenHardware – Try to have minimum of 3 Ambari infra nodes with atleast 1-2TB disk for Solr data storage, but mainly depends on how many components ( like Ranger, Atlas , Log search.. ) and amount of data will feed into Solr of indexing. A major driving factor for Solr performance is RAM. Solr requires sufficient memory for two separate things: One is the Java heap, the other is free memory for the OS disk cache. Let's say that you have a Solr index size of 8GB. If your OS, Solr's Java heap, and all other running programs require 4GB of memory, then an ideal memory size for that server is at least 12GB. So How much memory do I need for Ambari Infra?? This is one of those questions that has no generic answe

In this article, I am going to explain how you can work with the Schema Registry directly in your NiFi Data Flow.  In my previous article Using the Schema Registry API I talk about the work required to expose the API methods needed to Create a Schema Registry Entity and update that Entity with an Avro Schema.   In this article, I am going to take it one step further and complete both operations directly in my NiFi Data Flow. I will also include the Use Case I am working on. This flow accepts a CSV Parameter List which contains the columns and data types found within the fields object of the Avro Schema, processes the contents of the file, builds the required Avro Schema and saves it to the Schema Registry Entity I created.   At the time of writing this article I have not handled data types, so in these examples, the schema data types are all strings.  In future updates to my flow, I will be mapping different types to appropriate data types for hive queries.  

XML:[hdfs@sandbox ~]$ hdfs dfs -mkdir -p /test/sequences [hdfs@sandbox ~]$ [hdfs@sandbox ~]$ vi sample.xml <sequences period=”5″> <moment game-clock=”300.00″ time=”1433732215737″ game-event-id=”” shot-clock=”24.00″ locations=”-1,-1,96.95182,1.98648,5.75987;9,173004,45.54661,17.35545,0;9,338365,24.04722,25.67399,0;9,457611,46.95292,27.93478,0;9,468895,48.59834,33.96586,0;9,552336,33.73381,24.05929,0;5,214152,59.26872,24.12006,0;5,253997,45.71551,17.41071,0;5,457186,48.59834,33.96586,0;5,531447,78.09629,34.24688,0;5,552806,47.79678,22.8155,0″/> <moment game-clock=”300.00″ time=”1433732215794″ game-event-id=”” shot-clock=”24.00″ locations=”-1,-1,97.79683,.89407,3.67626;9,173004,45.62283,17.34854,0;9,338365,24.04248,25.6784,0;9,457611,46.84978,27.8463,0;9,468895,48.52017,33.89189,0;9,552336,33.74064,24.03493,0;5,214152,59.27282,24.07895,0;5,253997,45.87101,17.38532,0;5,457186,48.52017,33.89189,0;5,531447,78.06394,34.2155,0;5,552806,47.8269,22.81393,0

Today we are using below command to sync all the AD users to get access to Ambari Views/Operation server.ambari-server sync-ldap --groups groups.txtWe can't schedule cron tab because we need to pass admin credentials at the time of sync. It's hard to run daily as the users get added to the group and deleted which are managed in AD.By using below curl command we can automate the Ambari LDAP sync. We can add this command to bash script and schedule it using confab. Whenever new users get added/deleted to the LDAP group, it will get automatically synced the same with Ambari Views. We need to run below command as Ambari user.This can avoid running ‘ambari-server sync-ldap’command. Here even no need to run any unset proxy commands.curl -k -u admin:<password> -H'X-Requested-By: ambari' -X POST -d '[{"Event": {"specs":[{"principal_type": "users", "sync_type":"specific", "names":"sredd66,sreddyuw"},{"principal_type":"groups","sync_type":"specific","names": "hadoopadmin,hadoopusers"}]}}]'

  Image Courtesy: k9s I recently ran into a scenario where I needed to gather Hive logs on the new Data Warehouse Experience on AWS.   The "old" way of fetching logs was to SSH into the nodes. Data Warehouse Experience is now deployed on K8s, so SSHing is off the table. Therefore a tool like K9s is key. This is a raw article to quickly demonstrate how to use K9s to fetch Data Warehouse Experience logs which are deployed on AWS K8s Prerequisites Data Warehouse Experience K9s installed on your machine  AWS ARN (instructions provided below) AWS configure (CLI) pointing to your AWS env.  Simply type AWS configure via CLI and point to the correct AWS subscription AWS ARN Your AWS ARN is required to successfully connect K9s to CDW(DW-X) On AWS, go to IAM > Users > Search for your user name: Click on your username to fetch the ARN: Kubeconfig Connecting to DW-X using K9s requires kubeconfig. DW-X makes this available under DW-X-> Environments > Y

Moving data from your local machine to the cloud has never been easier using NiFi site to site protocol and CDP Datahub. In this article, I will focus on how to set up a site to site communication between your local machine and CDP Cloud, without using the default Knox CDP Proxy.   This configuration assumes that you already have a local instance of NiFi (or MiNiFi) and a CDP Datahub Cluster running NiFi. If you want to learn how to use CDP Public Cloud, please visit our overview page and documentation.   This setup will be executed in 4 steps: Step 1: Open CDP to your local IP Step 2: Download and configure stores on your local machine Step 3: Configure a simple site-to-site flow Step 4: Authorize this flow in Ranger Step 1: Open CDP to your local IP Go to your CDP Management Console, and find your datahub (here pvn-nifi). At the bottom of the datahub page, click on Hardware and locate one of the instances running NiFi: Click on the instances and you will be redirec

Goal:This article provides script to extract the DDLs for all tables and partition in a given hive database. This scripts comes handy when migrating/creating Hive Tables from one cluster to another. You can modify and loop this script by passing all the databases via command line. #!/bin/bashhiveDBName=testdbname;showcreate="show create table "showpartitions="show partitions "terminate=";"tables=`hive -e "use $hiveDBName;show tables;"`tab_list=`echo "${tables}"`rm -f ${hiveDBName}_all_table_partition_DDL.txtfor list in $tab_listdo showcreatetable=${showcreatetable}${showcreate}${list}${terminate} listpartitions=`hive -e "use $hiveDBName; ${showpartitions}${list}"` for tablepart in $listpartitions do partname=`echo ${tablepart/=/=\"}` echo $partname echo "ALTER TABLE $list ADD PARTITION ($partname\");" >> ${hiveDBName}_all_table_partition_DDL.txt donedoneecho " ====== Create Tables ======= : " $showcreatetable## Remove the filerm -f ${hiveDBName}_extract_a

Problem Statement When a using CDP with default DHCP Option Set (Use AmazonProvided DNS & AmazonProvidedNameServer) attached to the VPC and need to resolve some on-prem server controlled by an internal DNS server, it does not work by default as those internal servers are not resolvable by Route53 AWS service. Is there a way to resolve the internal on-prem/cloud servers from CDP hosts on AWS?  Context The diagram below shows the typical flow for any DNS resolution request.  This document addresses the last part in the diagram when Route53 tries to resolve a domain name that is owned by a private DNS server. Requirement Minimum 2 subnets, each one in a separate AZ for Route53 Resolver to work. Assumptions Before going to the ‘how to’ part, let's look at the assumptions to make this solution work in your environment: If you use an on-premise DNS server, then you must have connectivity to this server from your VPC This document only depicts Outbound resolution reques

Hey Everyone! In this video, we go over the Cloudera Data Engineering Experience, a new way for data engineers to easily manage spark jobs in a production environment. We'll go over a few of the key features as well as a quick demo on how to launch your first simple python ETL spark job. You'll see how to schedule as well as analyze a job once the run is complete.     More information: Link to the related Tutorial: Getting Started with Data Engineering (Tutorial) See the Release Webinar Cloudera Data Engineering Experience Release Webinar Documentation on CDP https://docs.cloudera.com/

Nifi Flow for writing to S3, WASB and Google Storage.Run the flow, watch as the twitter messages are captured and then aggregated before putting them in storage. Azure Storage:Now you can go to your Azure Portal and look in the container and you should see aggregated messages organized by year/month/day.Google Storage:Open Google Cloud Platform and go to your Storage service. Google Storage will now contain. the aggregated messages organized by year/month/day.AWS Storage:S3 bucket in your AWS account will now have aggregated Twitter messages organized by year/month/day.Now let’s see what’s happening here. I will only focus on the three main important processors as the others make up the simple flow. The entire flow template is available as an xml file and you can download: nificloudstorage.xmlPutAzureStorage Processor:Azure:Create a Storage Account Get the Storage Account name and Key as shown in this screenshot. This is needed in the PutAzureObject Processor.PutS3Object ProcessorFrom

Scenarios: There are few scenarios where you may need to migrate ranger components from one host to another host in Ambari. Server that is hosting Ranger master components is faulty & down or need to be replaced for newer hardware. Reduce the administrative implementation of enabling Ranger HA with load balancer and still have some recovery mechanism to get ranger back online as quickly as possible should you encounter above sitaution. Design of Ranger has HA builtin where the policies are cached in every plugin such as namenode will have a copy of policies cached on disk in the event ranger admin is not available. If ranger admin/usersync is down administrators will only lose access to add/modify/delete policies and users. In many of the hadoop clusters, users may not need to have high availability access to Ranger UI. It may be ok to survive for few hours without ranger ui and usersync components as existing policies and users should continue to work. Method: Stop

This video covers the Default Zeppelin UI option and basic navigation demo. References: Cloudera Product document page: https://docs.cloudera.com/runtime/7.1...  Cloudera tutorials: https://www.cloudera.com/tutorials/ge...  Visit Apache Zeppelin Website: http://zeppelin.apache.org  Cloudera Community: https://community.cloudera.com/

This video covers zeppelin’s backend operations and overview on impersonation concepts in Zeppelin. References: Cloudera Product document page: https://docs.cloudera.com/runtime/7.1...  Cloudera tutorials: https://www.cloudera.com/tutorials/ge...  Visit Apache Zeppelin Website: http://zeppelin.apache.org  Cloudera Community: https://community.cloudera.com/

This video covers High level overview of Zeppelin’s architecture and Operational Flow. References: Cloudera Product document page: https://docs.cloudera.com/runtime/7.1... Cloudera tutorials: https://www.cloudera.com/tutorials/ge... Visit Apache Zeppelin Website: http://zeppelin.apache.org Cloudera Community: https://community.cloudera.com/

The Cloudera Data Platform (CDP) comes with many places to store your data, and it can be challenging to know which one to use. Though there is no formal decision tree, I hereby share the key considerations from my personal perspective. They can be visualized like this:  Explanation of each patha. Have large bulky files, that do not need to be queried > File and object storageThe exact kind of storage to be used will mostly be defined by your environment, in a classical cluster HDFS is available. In the public cloud, each provider object store will be leveraged, and on-premises Ozone will serve as the object-store.b. Have a table, either from large bulky files, or a set of messages > Hive for scale or Kudu for interactionIf you want to work with a table, and need to store it as such, it is clear you want to store your data as a table. Even if this may force you to think about how to implement the ingest in a sensible way. Kudu is great for fast insights, where hive tabl

Disclaimer: This article is based on my personal experience and knowledge. Don't take it as a standard guidelines, understand the concept and modify it for your environmental best practices and use case. Ambari unofficially supports a curl command which will help deleting the old stacks and packages from each of the host . the script is described in https://issues.apache.org/jira/browse/AMBARI-18435 and also in the article : https://community.hortonworks.com/articles/202904/how-to-remove-all-previous-version-hdp-directories.html But this above script doesn't work in ambari-2.7.3 version. Root cause : the script will fail with below exception : Traceback (most recent call last): File "/var/lib/ambari-agent/cache/custom_actions/scripts/remove_previous_stacks.py", line 119, in <module> RemovePreviousStacks().execute() File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 352, in execute method(env) File "/var/lib/ambari-agent/cache/custom

In this tutorial we will walkthrough the steps to enable YARN Node labels so that we can run LLAP on a specific set of nodes in the cluster. (Stack Version HDP 2.6.3)In the first step we will check if we have any NodeLabels created on the cluster and the command is as follows[$] yarn cluster --list-node-labels 18/02/05 10:32:29 INFO client.RMProxy: Connecting to ResourceManager at stocks.hdp/192.168.1.200:8050 18/02/05 10:32:31 INFO client.AHSProxy: Connecting to Application History server at stocks.hdp/192.168.1.200:10200 Node Labels: This shows that there are no node labels configured on the cluster. We now try to add a new node label, we will use this node label to assign to LLAP queue, so that LLAP runs on that specific node(s). For now we will keep the node label exclusive, so that only users in LLAP queue can access these nodes [$] yarn rmadmin -addToClusterNodeLabels "interactive(exclusive=true)” 18/02/05 10:33:25 INFO client.RMProxy: Connecting to ResourceManager at stocks.hdp/

I am capturing the steps to install supplementary spark version on your HDP version. Installing the version not shipped by ambari is unsupported and not recommended, however they are some times customer need it for testing purposes.Please find the steps belowHere are the steps :-1. Create the spark user on all nodes. Add it to the hdfs group.    useradd -G hdfs spark 2. Create conf and log directorymkdir -p /etc/spark2.4/confmkdir -p /var/log/spark2.4chmod 755 /var/log/spark2.4/chown spark:hadoop /var/log/spark2.4mkdir -p /var/run/spark2.4chown spark:hadoop /var/run/spark2.4chmod 775 /var/run/spark2.4mkdir /var/lib/spark2/chown spark:spark /var/lib/spark2/3. Create a directory in /usr/hdp and cd to the dirmkdir -p /usr/hdp/<your hdp version>/spark2.4 (root user)cd /usr/hdp/<your hdp version>/spark2.44. Download the tar file from location http://apache.claz.org/spark/spark-2.4.0/ wget http://apache.claz.org/spark/spark-2.4.0/spark-2.4.0-bin-hadoop2.7.tgz5. Extract