Community Articles

Introduction Cloudera Data Warehouse architecture leverage compute/storage separation, this is different from the standard Hadoop architecture.     Figure 1: Cloudera Modern Data Warehouse Architecture   The objective of this post is to show how to import the wide functions and code material that we have on the open-source community inside Cloudera Data Warehouse using the object storage architecture. For this, we'll use ESRI Spatial Framework as an example. Prerequisites We'll use github to download the ESRI project, Java and maven to build the necessary JAR files. Step 1: Download the files from ESRI Github repository Download the necessary files from ESRI Spatial Framework Github repository, this can be done using the following command: $ git clone https://github.com/Esri/spatial-framework-for-hadoop.git Figure 2: Cloning ESRI project This will create a dir called "spatial-framework-for-hadoop", enter in this directory to build the project, and generate the JAR file

- Please follow steps mentioned in below link in order to get the list of provenance event id's :Nifi- how to get provenance event id in nifi?- In order to get more information about a specific event : For Nifi running in Standalone mode :# curl -i -X GET http://<hostname>:9090/nifi-api/provenance-events/id For Nifi running in clustered mode:# curl -i -X GET http://<hostname>:9090/nifi-api/provenance-events/id?clusterNodeId=<NODE UUID> There may be multiple events with the same event id(one on each node), so you need to specify from which node you want to return that specific event.

    Recently I was engaged in a use case where CDE processing was required to be triggered once data landed on s3. The s3 trigger in AWS would be via a Lambda function. As the files/data land in s3, an AWS Lambda function would be triggered to then call CDE to process the data/files. Lambda functions at trigger time include the names and locations of the files the trigger was executed upon. The file locations/names would be passed onto the CDE engine to pick up and process accordingly. Prerequisites to run this demo AWS account s3 Bucket Some knowledge of Lambda CDP and CDE Artifacts AWS Lambda function code https://github.com/sunileman/spark-kafka-streaming/blob/master/src/main/awslambda/triggerCDE.py CDE Spark Job, main class com.cloudera.examples.SimpleCDERun  Code for class com.cloudera.examples.SimpleCDERun https://github.com/sunileman/spark-kafka-streaming Prebuilt jar https://sunileman.s3.amazonaws.com/CDE/spark-kafka-streaming_2.11-1.0.jar

In a browser put in: http://<AMBARI-HOST>:8080/api/v1/clusters/<CLUSTER-NAME>/requests/ Find the highest reqest number in the output. Now, we can go to each request by pasting in the number of the request, an example would be: http://<AMBARI-HOST>:8080/api/v1/clusters/<CLUSTER-NAME>/requests/110 The output should look like: { "href" : "http://mon-orlan:8080/api/v1/clusters/test/requests/110", "Requests" : { "aborted_task_count" : 0, "cluster_name" : "test", "completed_task_count" : 3, "create_time" : 1444340395849, "end_time" : 1444340449936, "exclusive" : false, "failed_task_count" : 0, "id" : 110, "inputs" : "{}", "operation_level" : "SERVICE", "progress_percent" : 100.0, "queued_task_count" : 0, "request_context" : "Restart all components with Stale Configs for HIVE", ..... If that is not your hung operation, go back one by one from the highest request number. What we are looking for is the request context matches your hung Operation, that is the one we want

There are several options available to achieve this use case. The easiest and the best approach would be HBase snapshots method to transfer the data.    Note: All actions need to be performed as the HBase user only, to ensure correct permissions. On source cluster:#hbase shell> snapshot ‘Test’_Table’, ‘Test_Table_SS’#hbase org.apache.hadoop.hbase.snapshot.SnapshotInfo -snapshot ‘Test_Table_SS’ -files -stats #hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot 'Test_Table_SS' -copy-to hdfs://<Destination_NN_hostname>:8020/hbase -mappers 16 -bandwidth 200 On destination cluster:#hbase org.apache.hadoop.hbase.snapshot.SnapshotInfo -snapshot ‘Test_Table_SS’ -files -stats#hbase shell> clone_snapshot ‘Test_Table_SS’, ‘Test_Table’#hbase shell> major_compact 'Test_Table' Once done, you can choose to delete the snapshots on both ms05 and as05:#hbase shell> delete_snapshot 'Test_Table_SS'​ But If you plan to use CopyTable, this will not work without

The Cloudera Operational Database (COD) experience is a managed dbPaaS solution. It can auto-scale based on the workload utilization of the cluster and will be adding the ability to auto-tune (better performance within the existing infrastructure footprint) and auto-heal (resolve operational problems automatically) later this year.    COD is backed by the proven, scalable technology in Apache HBase and Apache Phoenix.   There is an excellent blog post showing the basics of setting up a COD database and connecting to it using Python.   COD offers options to connect using both HBase APIs and ANSI SQL through Apache Phoenix. This article shows how to connect to a COD database through Phoenix using Golang.    All of the code for this article can be found in this Github repository.   Getting started with COD is incredibly easy - no clusters to deploy and secure, simply log in and click “Create Database”: Select your CDP Environment and enter a

Cloudera Data Warehouse (CDW) brings one of the most efficient compute isolation and rapid scale up and down of data warehousing workloads, leveraging the latest container and caching technologies.   One of the great features of this architecture is the ability to only bring compute on-demand, as illustrated by the figure below:   This default setup is the most cost-effective setup as only a few shared services nodes (small nodes running services like UIs, Viz, Zookeeper, etc.) are long lasting. Each Virtual Warehouse has a set of nodes that are only running if compute is needed (i.e. a new query on a non-cached dataset).  The caveat to this approach is that on a completely cold warehouse,  the warm up type from zero to compute is 1 minute or 2.   An alternative to this default architecture is to leverage compute-reserved nodes, that will be shared between virtual warehouses, as depicted below:  With this architecture, a pool of reserved nodes can

It may often come to pass that you need to utilize data that doesn't exist in your Environment's Data Lake. This article will cover the 3 different scenarios where you need to access data that is outside of your Data Lake: A bucket in the same account as your CDP Environment A public bucket A bucker in a different account (than your CDP Environment) Since the steps for variations 1 and 2 are the same, I'll cover them together. Since IDBroker maps CDP users to explicit IAM roles. You need to perform the following to access even Public Buckets (unless you intend on interacting with them via URL). Public Bucket / Bucket in the Same AccountFirst, identify the public bucket in question. For purposes of this article, I will use the "commoncrawl" public AWS bucket (https://registry.opendata.aws/commoncrawl/) You'll need the bucket ARN (in this case "arn:aws:s3:::commoncrawl"), the ability to create IAM policies and roles, as well as the EnvironmentAdmin role in CDP. Next, we'll create a po

The all new Cloudera Data Engineering ExperienceThe all new Cloudera Data Engineering Experience I recently had the opportunity to work with Cloudera Data Engineering to stream data from Kafka.  It's quite interesting how I was able to deploy code without much worry about how to configure the back end components.   DemonstrationThis demo will pull from the Twitter API using NiFi, write to payload to a Kafka topic named "twitter".  Spark Streaming on Cloudera Data Engineering Experience CDE will pull from the twitter topic, extract the text field from the payload (which is the tweet itself) and write back to another Kafka topic named "tweet" The following is an example of a twitter payload.  The objective is to extract only the text field:What is Cloudera Data Engineering?Cloudera Data Engineering (CDE) is a serverless service for Cloudera Data Platform that allows you to submit Spark jobs to an auto-scaling cluster. CDE enables you to spend

LLAP Service sometimes may fail to startup after HDP upgrade with the following error:    Error::Caused by: org.apache.hadoop.security.KerberosAuthException: failure to login: for principal: hive/prakash.cloudera.com@CLOUDERA.COM from keytab /grid/0/yarn/local/usercache/hive/appcache/application_**********/container_***********_01_000001/keytabs/llap0.keytab javax.security.auth.login.LoginException: Pre-authentication information was invalid (24)   To resolve this issue, run the following command and start LLAP:  yarn app -destroy llap0 If the issue persists, then do the following   Check the hive.service.keytab file is available in all the hosts, run kinit and test the connectivity. If the file is missing, recreate the keytabs on the missing hosts either through Ambari or manually (see this article for manual keytab creation). Verify the file in HDFS and copy it to the local. Perform kinit with the copied keytab. hdfs:///user/hive/.yarn/keytabs/hive/hive.ser

Accessing CDP Endpoints using Postman or cURL When automating, testing, or validating APIs within the Cloudera Data Platform within our AWS, Azure, or GCP infrastructures, most developers utilize either curl or postman.  In this article, we will highlight the proper API endpoints, proper credentialing through Knox, and the setting of workload passwords.     Details Needed to Access Endpoints There are several details to access specific application endpoints within CDP: The Endpoint URL for the Specific Application Your specific username Your Workload Password The Endpoint URL for the Specific Application Each application within a CDP environment will generate a unique URL for both GUI and API development.  As an example for DataHub Clusters, you can view the API endpoints by clicking on the Cluster details, scrolling to the bottom, and selecting "Endpoints."    Within the Endpoints section, you'll find the appropriate API endpoints for each of the

Hey Everyone! In this video we're going to go over some more advanced features of the Cloudera Data Engineering Experience. Using some publicly accessible Paycheck Protection Data, you'll see how to automatically setup Spark jobs to deploy by using the CDE CLI, making development and deployment times much quicker and painless. We'll also take the development cycle through to the end and get some visualization of the finished reports using the aforementioned PPP data.     More information: A tutorial on using the Cloudera Data Engineering CLI Using CLI-API to Automate Access to Cloudera Data Engineering (Tutorial) A related Blog post: Using Cloudera Data Engineering to Analyze the Paycheck Protection Program Data A related Meetup event: Creating an Automated Spark Development Environment with Analysis and Logging Documentation on Cloudera Data Engineering Cloudera Docs / Data Engineering

This video covers Livy's Feature, Operational Flow and Basic Demo.   Open the video on YouTube here

Inspired by @sunile_manjee 's article How to use K9s to fetch metrics and logs for Cloudera Data Warehouse Experience , I decided to create the same tutorial for Cloudera Data Engineering. The process is very similar, as you can see below. Pre-Requisites CDE environment setup K9s installed on your machine (instructions here) AWS user ARN AWS configure (CLI) pointing to your AWS env (instructions here) Step 1: Add your AWS user to the CDE environment In your AWS account console, go to IAM > Users > Search for your user name > copy your ARN: In the CDE main page, go to environment details: Then, go to access and add your ARN: Step 2: Download the kubeconfig In the CDE main page, click on the 3 dots, then Download Kube Config: Step 3: Run K9s You can now run k9s, by running:k9s --kubeconfig ~/Downloads/kubeconfig​Note: the path to your downloaded kubeconfig is subject to change of course.    

Hi, In this article, let us take a look at how to delete a schema from the Hortonworks Schema Registry. Let me start with a word of caution that the approach is not recommended for Production systems and use these steps at your own risk. Also, would like to thank Brian Goerlitz for his ideas towards this post. Currently it is not possible to delete a schema from the UI. So the steps below shows how to delete the schema from its backend datastore. I am using MySQL as my backend datastore for the schema registry and the queries will be related to MySQL. You should change them according to your database type. Step 1Verify that the two tables schema_version_info and schema_field_info have CASCADE ON UPDATE and CASCADE ON DELETE enabled. This can be done by the below queries on information_schema databaseselect UPDATE_RULE,DELETE_RULE,REFERENCED_TABLE_NAME from REFERENTIAL_CONSTRAINTS where table_name='schema_version_info';andselect UPDATE_RULE,DELETE_RULE,REFERENCED_TABLE_NAME from REFER

As we know many services like Atlas for lineage, Ranger for audit logs, log search and so on uses Ambari Infra (Solr) for indexing data. So moving Ambari Infra in production and keeping it stable and up is really important. This are the key points I came up with to make this happenHardware – Try to have minimum of 3 Ambari infra nodes with atleast 1-2TB disk for Solr data storage, but mainly depends on how many components ( like Ranger, Atlas , Log search.. ) and amount of data will feed into Solr of indexing. A major driving factor for Solr performance is RAM. Solr requires sufficient memory for two separate things: One is the Java heap, the other is free memory for the OS disk cache. Let's say that you have a Solr index size of 8GB. If your OS, Solr's Java heap, and all other running programs require 4GB of memory, then an ideal memory size for that server is at least 12GB. So How much memory do I need for Ambari Infra?? This is one of those questions that has no generic answe

In this article, I am going to explain how you can work with the Schema Registry directly in your NiFi Data Flow.  In my previous article Using the Schema Registry API I talk about the work required to expose the API methods needed to Create a Schema Registry Entity and update that Entity with an Avro Schema.   In this article, I am going to take it one step further and complete both operations directly in my NiFi Data Flow. I will also include the Use Case I am working on. This flow accepts a CSV Parameter List which contains the columns and data types found within the fields object of the Avro Schema, processes the contents of the file, builds the required Avro Schema and saves it to the Schema Registry Entity I created.   At the time of writing this article I have not handled data types, so in these examples, the schema data types are all strings.  In future updates to my flow, I will be mapping different types to appropriate data types for hive queries.  

XML:[hdfs@sandbox ~]$ hdfs dfs -mkdir -p /test/sequences [hdfs@sandbox ~]$ [hdfs@sandbox ~]$ vi sample.xml <sequences period=”5″> <moment game-clock=”300.00″ time=”1433732215737″ game-event-id=”” shot-clock=”24.00″ locations=”-1,-1,96.95182,1.98648,5.75987;9,173004,45.54661,17.35545,0;9,338365,24.04722,25.67399,0;9,457611,46.95292,27.93478,0;9,468895,48.59834,33.96586,0;9,552336,33.73381,24.05929,0;5,214152,59.26872,24.12006,0;5,253997,45.71551,17.41071,0;5,457186,48.59834,33.96586,0;5,531447,78.09629,34.24688,0;5,552806,47.79678,22.8155,0″/> <moment game-clock=”300.00″ time=”1433732215794″ game-event-id=”” shot-clock=”24.00″ locations=”-1,-1,97.79683,.89407,3.67626;9,173004,45.62283,17.34854,0;9,338365,24.04248,25.6784,0;9,457611,46.84978,27.8463,0;9,468895,48.52017,33.89189,0;9,552336,33.74064,24.03493,0;5,214152,59.27282,24.07895,0;5,253997,45.87101,17.38532,0;5,457186,48.52017,33.89189,0;5,531447,78.06394,34.2155,0;5,552806,47.8269,22.81393,0

Today we are using below command to sync all the AD users to get access to Ambari Views/Operation server.ambari-server sync-ldap --groups groups.txtWe can't schedule cron tab because we need to pass admin credentials at the time of sync. It's hard to run daily as the users get added to the group and deleted which are managed in AD.By using below curl command we can automate the Ambari LDAP sync. We can add this command to bash script and schedule it using confab. Whenever new users get added/deleted to the LDAP group, it will get automatically synced the same with Ambari Views. We need to run below command as Ambari user.This can avoid running ‘ambari-server sync-ldap’command. Here even no need to run any unset proxy commands.curl -k -u admin:<password> -H'X-Requested-By: ambari' -X POST -d '[{"Event": {"specs":[{"principal_type": "users", "sync_type":"specific", "names":"sredd66,sreddyuw"},{"principal_type":"groups","sync_type":"specific","names": "hadoopadmin,hadoopusers"}]}}]'

  Image Courtesy: k9s I recently ran into a scenario where I needed to gather Hive logs on the new Data Warehouse Experience on AWS.   The "old" way of fetching logs was to SSH into the nodes. Data Warehouse Experience is now deployed on K8s, so SSHing is off the table. Therefore a tool like K9s is key. This is a raw article to quickly demonstrate how to use K9s to fetch Data Warehouse Experience logs which are deployed on AWS K8s Prerequisites Data Warehouse Experience K9s installed on your machine  AWS ARN (instructions provided below) AWS configure (CLI) pointing to your AWS env.  Simply type AWS configure via CLI and point to the correct AWS subscription AWS ARN Your AWS ARN is required to successfully connect K9s to CDW(DW-X) On AWS, go to IAM > Users > Search for your user name: Click on your username to fetch the ARN: Kubeconfig Connecting to DW-X using K9s requires kubeconfig. DW-X makes this available under DW-X-> Environments > Y

Moving data from your local machine to the cloud has never been easier using NiFi site to site protocol and CDP Datahub. In this article, I will focus on how to set up a site to site communication between your local machine and CDP Cloud, without using the default Knox CDP Proxy.   This configuration assumes that you already have a local instance of NiFi (or MiNiFi) and a CDP Datahub Cluster running NiFi. If you want to learn how to use CDP Public Cloud, please visit our overview page and documentation.   This setup will be executed in 4 steps: Step 1: Open CDP to your local IP Step 2: Download and configure stores on your local machine Step 3: Configure a simple site-to-site flow Step 4: Authorize this flow in Ranger Step 1: Open CDP to your local IP Go to your CDP Management Console, and find your datahub (here pvn-nifi). At the bottom of the datahub page, click on Hardware and locate one of the instances running NiFi: Click on the instances and you will be redirec

Goal:This article provides script to extract the DDLs for all tables and partition in a given hive database. This scripts comes handy when migrating/creating Hive Tables from one cluster to another. You can modify and loop this script by passing all the databases via command line. #!/bin/bashhiveDBName=testdbname;showcreate="show create table "showpartitions="show partitions "terminate=";"tables=`hive -e "use $hiveDBName;show tables;"`tab_list=`echo "${tables}"`rm -f ${hiveDBName}_all_table_partition_DDL.txtfor list in $tab_listdo showcreatetable=${showcreatetable}${showcreate}${list}${terminate} listpartitions=`hive -e "use $hiveDBName; ${showpartitions}${list}"` for tablepart in $listpartitions do partname=`echo ${tablepart/=/=\"}` echo $partname echo "ALTER TABLE $list ADD PARTITION ($partname\");" >> ${hiveDBName}_all_table_partition_DDL.txt donedoneecho " ====== Create Tables ======= : " $showcreatetable## Remove the filerm -f ${hiveDBName}_extract_a

Problem Statement When a using CDP with default DHCP Option Set (Use AmazonProvided DNS & AmazonProvidedNameServer) attached to the VPC and need to resolve some on-prem server controlled by an internal DNS server, it does not work by default as those internal servers are not resolvable by Route53 AWS service. Is there a way to resolve the internal on-prem/cloud servers from CDP hosts on AWS?  Context The diagram below shows the typical flow for any DNS resolution request.  This document addresses the last part in the diagram when Route53 tries to resolve a domain name that is owned by a private DNS server. Requirement Minimum 2 subnets, each one in a separate AZ for Route53 Resolver to work. Assumptions Before going to the ‘how to’ part, let's look at the assumptions to make this solution work in your environment: If you use an on-premise DNS server, then you must have connectivity to this server from your VPC This document only depicts Outbound resolution reques

Hey Everyone! In this video, we go over the Cloudera Data Engineering Experience, a new way for data engineers to easily manage spark jobs in a production environment. We'll go over a few of the key features as well as a quick demo on how to launch your first simple python ETL spark job. You'll see how to schedule as well as analyze a job once the run is complete.     More information: Link to the related Tutorial: Getting Started with Data Engineering (Tutorial) See the Release Webinar Cloudera Data Engineering Experience Release Webinar Documentation on CDP https://docs.cloudera.com/

Nifi Flow for writing to S3, WASB and Google Storage.Run the flow, watch as the twitter messages are captured and then aggregated before putting them in storage. Azure Storage:Now you can go to your Azure Portal and look in the container and you should see aggregated messages organized by year/month/day.Google Storage:Open Google Cloud Platform and go to your Storage service. Google Storage will now contain. the aggregated messages organized by year/month/day.AWS Storage:S3 bucket in your AWS account will now have aggregated Twitter messages organized by year/month/day.Now let’s see what’s happening here. I will only focus on the three main important processors as the others make up the simple flow. The entire flow template is available as an xml file and you can download: nificloudstorage.xmlPutAzureStorage Processor:Azure:Create a Storage Account Get the Storage Account name and Key as shown in this screenshot. This is needed in the PutAzureObject Processor.PutS3Object ProcessorFrom