Resources

What is CRN?CRN is an acronym for Customer Resource Number and this is a Cloudera-specific identifier for an environment. The CRN uniquely identifies the environment as well as the CDP control plane that is used to create it.Where can I find the CRN?It is located prominently on the Details page of each CDP environment. Steps to find the CRN for an environment:Click Environments from the CDP Management Console in the left panel next to Dashboard Type in the name of the environment in the search bar. In the screenshot here, the search key is "orrep-aw-env"Click on the "Name" of the environment. This should show the details of the environmentClick Data LakeNow, click Copy to Clipboard icon next to "crn: ..." in the header This should copy the CRN to the clipboard. Now you can paste it as and where needed.

In this article, we will learn how to integrate Zeppelin JDBC (Phoenix) interpreter example.   1. Configuring the JDBC (Phoenix) interpreter:Login to Zeppelin UI -> Click on the user name (in my case, admin) at the right-hand corner. It will display a menu > click on Interpreter.   Click on + Create at the right-hand side of the screen.     It will display a popup menu. Enter Interpreter Name as jdbc and select Interpreter Group as jdbc. Then, it will populate Properties in table format.      Click on + button and add the Phoenix-related properties according to your cluster, and click on the Save button. phoenix.driver org.apache.phoenix.jdbc.PhoenixDriver phoenix.url jdbc:phoenix:localhost:2181:/hbase phoenix.user   phoenix.password        2. Creating the Notebook:   Click Notebook dropdown menu in the top left-hand corner and select Create new note and enter Note Name as Phoeni

This article describes various "recipes" on how to accomplish certain tasks with the NiFi processor ExecuteScript, with examples given in Groovy, Jython, Javascript (Nashorn), and JRuby. This is Part 3 in the series, I will be discussing advanced features such as dynamic properties, modules, state management, and accessing/using Controller Services. Part 1 - Introduction to the NiFi API and FlowFiles Getting a flow file from an incoming queue Creating new flow files Working with flow file attributes Transferring a flow file Logging Part 2 - FlowFile I/O and Error Handling Reading from a flow file Writing to a flow file Reading and writing to/from a flow file Error Handling Part 3 - Advanced Features Using Dynamic Properties Adding Modules State Management Accessing Controller Services   Advanced Features The first two articles in the series covered the basics of flow file operations, such as reading/writing attributes and content, as well as retrieving and transferring flow

Components required: jconsole (UI required) jmxterm (for Linux environment - CLI only) Kafka client (java producer/consumer) exposing JMX  Kafka Brokers exposing jmx Steps to get the available metrics (mbeans) available in a Kafka consumer (Environment with a UI available): Add the following JVM properties to your java consumer:-Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.port=9090 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false After starting, your consumer uses jconsole to connect to the host:port specified in Step 1. After we add the <producer-hostname>:<JMXPort>, we should be able to see the following in the jconsole UI. Here, we have to click on the mbeans tab and navigate through the available metrics, in this example, we want to see the "records-consumer-rate" which in the program has the following definition "The average number of records consumed per second".In this case, the average

Problem: Trying to use sudo to run commands(as root) in a cluster gives error " Stderr: Pseudo-terminal will not be allocated because stdin is not a terminal. sudo: sorry, you must have a tty to run sudo" This is a common situation where you are login as some other user in a terminal and then sudo to root to perform the task. And you can only ssh from the user you logged in and not from root. Solution: Note: You have to install pssh if it is not already installed. in your cluster.Keep all host names in a file say all_hosts This kind of execution will fail:pssh -i -h all_hosts "sudo whoami;hostname -f" Will fail to execute and give error: Stderr: sudo: sorry, you must have a tty to run sudo OR pssh -i -h all_hosts -x "-t" "sudo whoami;hostname -f" Will fail to execute and give error:Stderr: Pseudo-terminal will not be allocated because stdin is not a terminal. sudo: sorry, you must have a tty to run sudo The correct command should be run like this:pssh -i -h all_hosts -x "-t -t" "sudo

This article will show you how to interact with Atlas APIs in CDP-public to create tags and associate tags with entities (in preparation for use with Ranger's tag based policies)   In Cloudera CDP-public offering, Apache Atlas is a part of SDX DataLake cluster that is created when you create your first Environment: Introduction to Data Lakes Pre-requisites A. First, you will need to find the Atlas endpoint using the Cloudera CDP management console: Accessing Data Lake services Sample Atlas endpoint: https://pse-722-cdp-xxxxx.cloudera.site/pse-722-cdp-dl/cdp-proxy-api/atlas/api/atlas/   B. Next, you will need to set your user's workload password Setting the workload password Now you can use the following sample bash code to interact with Atlas APIs from a CentOS instance outside CDP:   From Atlas endpoint, you can extract the first 2 params below. You will also need to set your username and password: export datalake_name='pse-722-cdp-dl'export lake_ip='pse-722-cdp-xxxxx.

This article applies to HDP 2.5.x and below. For HDP 2.6, please check new article. Zeppelin can be configured to leverage an organization's Active Directory infrastructure for user authentication. By doing this, the existing Active Directory users can login to Zeppelin UI using their Active Directory credentials. This article discuss how to configure this kind of setup. Environment Setup: HDP 2.5 cluster / Sandbox - I'm using HDP 2.5 Sandbox on VirtualBox. Get one from here ! Ambari 2.4+ - I'm using Ambari 2.4.0.0 which comes with HDP 2.5 Sandbox 'Zeppelin Notebook' Service installed in Ambari - With HDP 2.5 Sandbox, it will be Zeppelin version 0.6.0 - If you don't have Zeppelin installed, it can be installed via 'Add Service' option in Ambari Active Directory - I'm using Active Directory 2012 R2 version - Make sure that you have 'working' Active Directory details handy like URI, bind DN/password, search base etc. Configuration Steps: 1. From Ambari Dashboard, navigate to Zeppelin Not

My previous article demonstrated how to use the CDE-managed Airflow instance to read files from CML. To close the loop, here is a short demonstration on how to write files to CML using CDE's managed Airflow instance.   Artifacts Airflow dag Demo  

I wanted to quickly share that we’ve released FIVE new Applied ML Prototypes (AMPs) in Cloudera AI and Cloudera AI Workbench! These AMPs solve a wide range of problems for data scientists and help jumpstart ML / AI projects, enabling them to deliver greater value faster across their organization. Here’s an overview of what was released: Summarization — This project demonstrates four automatic summarization models, including extractive and abstractive techniques. Why you should care: Summarization enables users to quickly extract important information from larger bodies of text. This is useful for any industry looking to accelerate research, competitive analysis, or businesses that need to process and understand large amounts of text information that would be time-prohibitive for a human to do. AutoML with TPot — This project enables automated machine learning on a sample of credit card fraud data. Why you should care: AutoML has the potential to accelerate many repetitive pro

Description NiFI performance comparison between Python and groovy script. This analysis shows the performance impact of programming scripts in NiFi. And also, it has an impact on setting concurrent tasks. Here, generate processor has input, and it is coming out as a attribute. Groovy and Python script read the attributes and replace them with the actual value for target_file_cfg. Below are the sample codes. Objective Translate Python script to groovy script and analyze the performance and set the concurrent task accordingly. Use Case Following is the input where the target file name will replace with the actual value.   input-target_file_cfg — /movement/%C/%D/MOVE%V/%F output-target-filename — /movement/JP/None/MOVE055/xxxxxx.xxxx.x Input target_file_cfg — /movement/%C/%D/MOVE%V/%F country-JP dns-country-code- JP dns-site-number- 055 source-file-country-code-WM source-filename-xxxxxx.xxxx.x target_file_cfg — /movement/%C/%D/MOVE%V/%F Output target-filename — /movement/JP/None/MOV

Introduction of Apache Kafka 3.0 Thanks to a collaborative effort between Cloudera and other parties, Apache Kafka continues to improve rapidly, and now, this has resulted in the release of Apache Kafka 3.0! The top two questions now are: What does this mean for the setup that I have today? Why are we excited about Apache Kafka 3.0? Any challenges we should be aware of? Let us address both of these. What does this mean for the setup that I have today? First of all, there is no need to be concerned about your current footprint. Every time Cloudera announces a release, it also announces an end of support date. The availability of a new version does not detract from this.  Furthermore, the Kafka 2 branch is very much still alive, and we are still planning to release improvements to this for a good period of time. Why are we excited about Apache Kafka 3.0? Any challenges we should be aware of? In short, we see Kafka 3.0 as a good foundation. There are many areas with small improvem

Migrating HDP clusters to CDP has been a journey many customers are going through these days. Migrating Ranger Audits + Atlas collections in infra-solr to CDP has been a challenging task. We hope the steps below will make simplify your journey. Preparation: Sample API calls to get the current status of collections in infra-solr. This is an important step to visualise how big these collections are and as a result, get an idea of how long the migration will take. Note: In the commands below, -k is used if https is used rather than http change http for https if the connection is secure check the port is correct depending on the version of infra-solr being used or if you have customized the port Infra-Solr API queries Gather the list of collections in infra-solr   ### To list collectionscurl --negotiate -u: -k 'http://solr_host:solr_port/solr/admin/collections?action=LIST'     Get the total number of records in your infra-solr collection. Example, ranger_audits

At Cloudera, we believe that data can make what is impossible today, possible tomorrow. There are many good uses of data. With data, we can monitor our business, the overall business, or specific business units. We can segment based on the customer verticals or whether they run in the public or private cloud. We can understand customers better, see usage patterns and main consumption drivers. We can find customer pain points, see where they get stuck, and understand how different bugs affect them. With data, we can discover new market opportunities, and review where we stand compared to the global market. We can track feature adoption, see how new features are picked up and what usage/consumption they generate. With data, we can set better goals, know where we are and where we want to go. And in the end, we can make better decisions.  At Cloudera, we practice what we preach. As the Cloudera gains popularity and more and more customers make it a critical piece of their in

1. We need to know on which database Hive metastore is hosted like my SQL, Oracle? 2. What is the version of jdbc driver used for a database connection?   if Oracle then, we need to know which version it has by this command unzip -p /usr/share/oracle-connector-java.jar | head > jar_version.txt Right now we are using ojdbc8.jar.   Solution: Using the following command, move the existing oracle-connector-java.jar from /usr/share/java/ to /tmp/mv /usr/share/java/oracle-connector-java.jar /tmp/ Using the following command, copy the existing ojdbc8.jar as oracle-connector-java.jarcp /usr/share/java/ojdbc8.jar oracle-connector-java.jar Using the following command, change the permissions to 775chmod 775 oracle-connector-java.jar Restart the Hive service Disclaimer: This article is contributed by an external user. The steps may not be verified by Cloudera and may not be applicable for all use cases and may be very specific to a particular distribution. Please follow with

If any user/admins are not able to view the container logs. And the applications are running with Spark service. Looks like ACLs are not configured. Add this in your clusters:    Go to Cloudera Manager -> Spark -> Configuration -> Spark Client Advanced Configuration Snippet (Safety Valve) for spark-conf/spark-defaults.confspark.acls.enable= truespark.admin.acls=username1,username2spark.admin.acls.groups=group1,group2 Set spark.ui.view.acls=username1,username2spark.ui.view.acls.groups=group1,group2 Set spark.modify.acls=user1,user2spark.modify.acls.groups=group1,group2  If you pass * as value it will give access to all the users. And both user[admin.acls] and groups[admin.acls.groups] accept both username and groups as comma-separated values.   Both user and group lists use a comma-separated list of entries. The wildcard character “*” allows access to anyone, and the single space “ “ allows access to no one.   Regards, Naresh.B   Disclaimer

 Short demo on how to use CDE's new Airflow UI to design pipelines.  Additionally leveraging Airflow triggers to orchestrate a typical job pipeline forking logic (triggers).  A "Typical job pipeline" often encapsulates logic regarding success and failure paths. Artifacts Demo Videohttps://youtu.be/G-53Uf4qrFwGit Repohttps://github.com/sunileman/cde-airflow-branch-operator  

Often a timestamp is found within a phoenix table to support various business taxonomy requirements. Generally I have seen a pattern simply adding a timestamp datatype field. Many may not realize part of the hbase data model rowkey design is the timestamp. When a cell is written in hbase by default it is versioned using timestamp when the cell was created. This is out of the box.Apache Phoenix provides a way of mapping HBase’s native row timestamp to a Phoenix column. This leverages various optimizations which HBase provides for time ranges on the store files as well as various query optimization capabilities built within Phoenix. - https://phoenix.apache.org/rowtimestamp.htmlBased on your use case, set this "column" to a value of your liking. Take advantage of the built in data model design without adding yet another timestamp field. Commonly I find many end up creating secondary index on these additional timestamp fields due to time always being a variable at query time.Lets ta

ObjectiveThis tutorial walks you through the process of installing the Docker version of the HDP 2.5 Hortonworks Sandbox on a Mac. This tutorial is part one of a two part series. The second article can be found here:HCC ArticlePrerequisitesYou should already have installed Docker for Mac. (Read more here Docker for Mac)You should already have downloaded the Docker version of the Hortonworks Sandbox (Read more here Hortonworks Sandbox)ScopeThis tutorial was tested using the following environment and components:Mac OS X 10.11.6HDP 2.5 on Hortonworks Sandbox (Docker Version)Docker for Mac 1.12.1NOTE: You should adjust your Docker configuration to provide at least 8GB of RAM. I personally find things are better with 10-12GB of RAM. You can follow this article for more information: https://hortonworks.com/tutorial/sandbox-deployment-and-install-guide/section/3/#for-macSteps1. Ensure the Docker daemon is running. You can verify by typing: $ docker imagesYou should see something similar

   Apache Airflow and Spark running on K8s provide unreasonable scalability.  Cloudera Machine Learning also runs on K8s and often the ML users leverage Apache Airflow to orchestrate their pipelines. Apache Airflow within Cloudera Data Engineering now has the capability to orchestrate CML jobs via the new Airflow CML Operator.  This time instead of writing up a detailed article, I recorded a live demo. Assets used in the demoCML Operatorhttps://github.com/curtishoward/cml_operator/Airflow Daghttps://github.com/sunileman/cde-airflow-branch-operatorDemohttps://www.youtube.com/watch?v=n9o_QimnggoEnjoy!

Article Short Description: This Tutorial describes how to add partition field to the content of the flowfile, create dynamic partitions based on flowfile content and store data into HDFS directory using NiFi. PartitionRecord Processor Params: Record Reader : Specifies the Controller Service to use for reading incoming data Record Writer: Specifies the Controller Service to use for writing out the records. Dynamic Properties: We need to add at least one property name and value(recordpath) that will be evaluated on each record of the flowfile. Each record will be grouped with the other like records in the group. **Note** No attribute will be added if the value returned for the RecordPath is null or is not a scalar value (i.e., the value is an Array, Map, or Record). Flow:   Flow Overview: 1. GenerateFlowfile //to generate csv data2. UpdateAttribute //add schema name attribute to the flowfile3. Update Record //add partition field to the flowfile content4. Partition Record //cre

Introduction This article is the final part in the series Scaling the Namenode (See  part 1, part 2, part 3 and part 4) In part 4 we discussed about monitoring Namenode Logs for Excessive Skews. In this part, we will look at a few optimizations around logging, access check, and block reports. Audience This article is for Hadoop administrators who are familiar with HDFS and its components. Audit log specific operations only when debug is enabled. By default, the following property is set to blank so none of the Namenode operations are restricted from making an entry into Audit log. Operations like getfileinfo results in fetching the metadata associated with a file, and in a large/read-heavy cluster, it can generate too much audit log. So, it is recommended to audit log getfileinfo only when audit log debug is enabled.   Change in hdfs-site.xml     <property> <name>dfs.namenode.audit.log.debug.cmdlist<\nam

A question about CDSW was asked by a customer. Question: We connect to external databases in CDSW. We found that some of the client IPs of the database are not in our LAN CIDR, and these IPs do not belong to the Node IP of CDSW. Why is this happening?   Response: CDSW uses Kubernetes as its infrastructure. Kubernetes maintains the IP CIDR of Pod and Service internally.When the Kubernetes workload communicates with the outside, it needs to pass the Service abstraction layer. The Service layer mainly implements packet forwarding between Node and Pod through NAT. Service has different modes. There are different ways to deal with the source IP NAT in different modes. Service may retain the source (Node) IP. This nature may be related to your question.For details, please refer to the official Kubernetes document: Using Source IP.   In addition, if you want to test what the source IP is when accessing an external database from the Pod in the CDSW Kubernetes cluster, you can refer

Many technologists ask "What is the easiest way to import data into Hive?" It's true, you can write a Sqoop job (making sure YARN containers are tuned properly in regards to # of mappers, the size of the mappers, and the sort allocated memory). And you'll deal with a two step process of moving the RDBMS tables into plain text and then creating an ORC table with an INSERT INTO statement.... I don't call this easy. What if there were a tool where you could simply drag-and-drop processes (processors) on a canvas and connect them together creating a workflow? There is a tool for this called NiFi! Before we go any further, I'll assume you know and understand NiFi - what is a processor, scheduling a processor, what is a connection (relationship), what are different relationship types, etc:. Small-Medium RDBMS tablesThis article will cover small to medium sized RDBMS tables. This means I can run a single select * from myrdbmstable;without returning millions of rows (where we're taxin

Accessing Kudu tables from Spark on the Cloudera Data Platform (CDP) is a common integration pattern for real-time analytic workloads that require fast inserts and updates while at the same time enabling efficient columnar scans across a single storage layer.   This article describes Kudu integration for Spark jobs running in the Cloudera Data Engineering Service by using the Kudu storage defined as part of a CDP Datahub cluster. Kudu Cluster We'll first configure the Kudu cluster. This can be done by creating a Datahub cluster of type Real-time Data Mart: Once the cluster has been provisioned successfully, you may need to add your IP address to the firewall rules for the cluster hosts in order to access some of the cluster's interfaces.  For example, with Amazon Web Services (AWS), a link to the EC2 instance configuration page is provided on the Datahub cluster page (on the EC2 instance page, locate the Security configuration and edit Security Group > Inbound Rule

Introduction In this article, I'll show how to stream data into CDP Public Cloud using Cloudera Dataflow/Streaming Datahub and query the data using Cloudera Data Warehouse. Pre-Requisites For this exercise you'll need Cloudera Data Platform with: Cloudera Data Warehouse; Datahub Flow Management; Datahub Streams Messaging; This exercise and flow are based on the sensor data/pipeline in edge2ai-workshop, but it is in a modified version that I'll share in another repository. 1. Create the Streaming Table in Cloudera Data Warehouse For this exercise we need two virtual warehouses: A Hive virtual warehouse: Used only to perform the compaction process (Streaming data ingest). For this example, I've named it latam-hive; A Unified Analytics Virtual Warehouse: Used to query data and visualization. Unified Analytics is a very exciting new feature for CDW customers at no extra fee!. You can learn more here. For this example, I've named it latam-vw; Figure 1: Virtual Warehouses used on