Community Articles

With the need of keeping metrics centralized into a single spot, people have sought a way to configure grafana for observability by attempting to collect the wide range of metrics available in a Datahub / Datalake in order to keep their dashboards arranged in a centralized hub. This tutorial will walk you through step-by-step in how to configure Grafana to query the metrics available in the Cloudera Manager of a datahub cluster. First, we need to ensure that the machine where Grafana is running has direct connection with the CM Server. In other words, it must be able to resolve the CM FQDN and establish a straight up connection with the service, bypassing knox completely. Unlike the Grafana utilized by other experiences, such as DWX, CDW and so forth of which use prometheus for integration, this grafana will have to integrate with CM Server for authentication and during the creation of a datasource when querying the metrics therein. This integration i

After Jun 1, 2021, the following changes are introduced to S3Guard in CDW service on CDP Public Cloud[1]:S3 Guard disabled for custom Database Catalogs and default Database Catalogs on AWS environmentsDuring new AWS environment activation in CDW:If you have not specified a DynamoDB table name during environment registration in Management Console, the S3 Guard feature is disabled for all Database Catalogs and Virtual Warehouses.If you have specified a DynamoDB table name during environment registration in Management Console, the S3 Guard feature is enabled for default Database Catalogs and Virtual Warehouses. However, it is disabled for non-default (custom) Database Catalogs.For all environments that were activated before the June 1, 2021 release, S3 Guard is enabled. [1] https://docs.cloudera.com/data-warehouse/cloud/release-notes/topics/dw-whats-new-24.html

Overview:This handbook focuses on Cloudera’s new adapter dbt-spark-cde that enables users to use dbt on top of Apache Spark by leveraging the use of CDE. Cloudera Data Engineering (CDE) is a serverless service for Cloudera Data Platform that allows you to submit Spark jobs to an auto-scaling cluster.Links to a dbt example project that can be used to bootstrap a user’s dbt journey can be found below.Version Notes:The adapter has been tested on the following version:python: 3.9Cloudera Data Engineering release (1.15-h1)dbt-core: 1.3.0Installing dbt and requirementsPrerequisites: Make sure python3, git, and pip are available on your system.Cloning the repo:Run the following command in your terminal to clone the example project repository:git clone https://github.com/cloudera/dbt-spark-cde-example/Installing the packages:Run the following commands in your terminal to install the required packages/dependencies using pip/pip3:pip install dbt-corepip install dbt-spark-cdeCDE and SparkFor

COD - CDE Spark-HBase tutorial In this article related to CDP Public Cloud, we will walk through steps required to be followed to read/ write to COD (Cloudera Operational Database) from Spark on CDE (Cloudera Data Engineering) using spark-hbase connector.If you are looking to leverage Phoenix instead, please refer to this community article. Assumption COD is already provisioned and database is created. Refer to this link for the same.For this example, we will assume "amallegni-cod" as the database name. CDE is already provisioned and the virtual cluster is already created. Refer to this link for the same. COD Download client configuration Spark in CDE to be able to talk to COD, would require the hbase-site.xml config of the COD cluster. Do the following steps to retrieve the same: Go to the COD control plane UI and click on "amallegni-cod" database. Under the Connect tab of COD database, look for HBase Client Configuration

With the release of HDP 2.6 comes the Hive ACID Merge feature. This is long sought after feature for people who want to batch update table content in ACID manner. This tutorial will walk you through step-by-step how to use this new feature with some really simple dataset. The example is done on HDP 2.6.0.3 installation. First, ACID in Hive need to be turned on in Ambari UIThen we will create 2 tables, one as the target of merge and one as the source of merge. Please note that the target table must be bucketed, set as transaction enabled and stored in orc format.CREATE DATABASE merge_data;CREATE TABLE merge_data.transactions( ID int, TranValue string, last_update_user string)PARTITIONED BY (tran_date string)CLUSTERED BY (ID) into 5 buckets STORED AS ORC TBLPROPERTIES ('transactional'='true');CREATE TABLE merge_data.merge_source( ID int, TranValue string, tran_date string)STORED AS ORC;Then we will populate the target and source table with some data.INSERT INTO merge_data.transactions PA

IntroductionMigration PathsHDP 2.6 to CDPExportImportHDP 3.X to CDPBackupRestoreUseful Links & ScriptsIntroduction Apache Atlas provides open metadata management and governance capabilities for organisations to build a catalog of their data assets, classify and govern these assets and provide collaboration capabilities around these data assets for data scientists, analysts and the data governance team. This article focuses on backup and restore of Atlas data during HDP to CDP migration. Migration Paths HDP 2.6 to CDP ExportNote: Following commands need to be executed on HDP ClusterDownload the migration script to download on to Atlas server host.  ## Download the scriptwget https://archive.cloudera.com/am2cm/hdp2/atlas-migration-exporter-0.8.0.2.6.6.0-332.tar.gz## Untartar zxvf atlas-migration-exporter-0.8.0.2.6.6.0-332.tar.gz##copy the contentsmkdir /usr/hdp/2.6.5.0-292/atlas/tools/migration-exporter/cp atlas-migration-exporter-0.8.0.2.

OverviewThis article focuses on Cloudera’s new adapter dbt-spark-livy that enables users to use dbt on top of Apache Spark by connecting to it using Apache Livy. Livy is a REST web service for managing long-running Spark Contexts and submitting Spark jobs. Instead of running the Spark Contexts in the Server itself, Livy manages Contexts running on the cluster managed by a Resource Manager like YARN.Cloudera Data Platform provides Spark with Livy in the Datahub form factor. Links to a dbt example project that can be used to bootstrap a user’s dbt journey can be found below.Version Notes:The adapter has been tested on the following version:python: 3.9.13Cloudera Manager CM VERSION 7.6.2 RUNTIME VERSION 7.2.15CDP Datahub with the template - 7.2.14 - Data Engineering: Apache Spark2/Spark3dbt-core: 1.3.0Installing dbt and requirementsPrerequisites: Make sure python3, git, and pip are available on your system.Cloning the repo:Run the following command in your terminal to clone the examp

     There are a numerous ways of doing event driven architectures in Cloud with Cloudera Data Platform(CDP) but, lately I was working on a use case which required Spark inferences and transformations on streaming log data in Azure cloud environment on a near realtime basis. The use case is to kickoff Spark transformations as soon as the files get landed in the Azure storage container.                                  The preferred way of doing Spark transformations for Data Engineering use cases in CDP is by leveraging Cloudera Data Engineering(CDE) which runs Spark on Kubernetes. CDE provides much better isolation, job orchestration with help of Airflow, Spark version independence, scales faster, efficiently utilizes the cloud native resources and the best part is that it does dependency management like a rockstar. Just makes your life easy and simple.      &nb

This has been a recurring question in the Community forum. I replied to those a few times so I've decided to post the answer here as an article so that's easier to find and reference. The question is: if you have a ZIP file that contains other ZIP files (which may in turn contain other ZIP files, etc.), how can I extract all the ZIP files, recursively, using NiFi? One way to do that is to implement the following data flow:  The RouteOnAttribute processor checks the file name and if it's a ZIP file it sends it for unpacking. Otherwise sends the non-zip file downstream.  You can find the flow definition attached below. HTH,André

With the May 2022 release of Cloudera Machine Learning, administrators now have the ability to specify a unique subdomain for the CML Workspace.  This enables a new CML workspace to leverage the endpoint of a previously deleted CML Workspace. This ensures that deployed models and applications deployed in the new Workspace will have the same endpoint as the same models and applications deployed in the old Workspace.  The text box to enter the Subdomain is available once you toggle the Advanced Options switch during  Workspace creation. For more information, see CML Static Subdomain in Provisioning ML Workspaces.

We are pleased to announce the release of the dbt-impala adapter v1.1.0. This release is compatible and tested with dbt-core v1.1.0.https://pypi.org/project/dbt-impala/1.1.0/https://github.com/cloudera/dbt-impala/releases/tag/v1.1.0 Full dbt-core v1.1.0 release notes can be found here, including a number of bug-fixes and improvements and support for Python 3.10 under the hood. Adapter Enhancements and New FeaturesAdopted new dbt methods provided in dbt 1.1.0Fix for an issue with get_catalog that caused empty catalog to be generated (#36)Ensure lower case for pattern passed to "show tables in" query (#37)Add timeout for snowplow endpoint to handle air gapped environments (#38)Bump version number to 1.1.0 to align with dbt-core versioningPython 3.10 compatibility as a result of migrating to dbt-core=1.1.0Getting started with dbt-impalaRead the Getting started with dbt-impala & Cloudera Data Warehouse guideWatch the video Using the dbt-impala example

The video below will take you through using the dbt-impala-example repo. This is an example dbt project, containing a few seeds, tests and models to help you bootstrap your own dbt project. If you haven't gotten started with dbt & Impala on CDP yet, take a look at the previous post Getting started with dbt-impala & Cloudera Data Warehouse. 

Kerberos cross realm trust for distcp This article is to demonstrate how to setup cross realm trust for distcp between two secure HDP clusters with their own Kerberos realms(KDC’s). Prerequisites Both HDP clusters must be running JDK 1.7 or higher. JDK 1.6 has some known issues Lets assume first HDP DEV cluster realm : HDPDEV.DEV.COM Lets assume second HDP QA cluster realm : HDPQA.QA.COM Step 1 : To set up cross realm trust between HDPDEV.DEV.COM and HDPQA.QA.COM, for example a client of realm HDPDEV.DEV.COM to access a service in realm HDPDQA.QA.COM, both realms must share a key for a principal name krbtgt/ HDPDQA.QA.COM@ HDPDEV.DEV.COM and both keys must have the same key version number associated with them. Cross realm trust is unidirectional by default. So for clients in HDPQA.QA.COM also to have access services in HDPDEV.DEV.com, both realms must share a key for principal krbtgt/ HDPDDEV.DEV.COM@ HDPQA.QA.COM. Add both krbtgt principals on both clusters #HDP DEV Cluster kadmin.l

The ListS3 and FetchS3 processors in Apache NiFi are commonly used to retrieve objects from Amazon S3 buckets, but they can be easily configured to retrieve objects from IBM Cloud buckets. Assume, I have an IBM Cloud bucket that contains three CSV files:  First, get the following from your IBM Cloud bucket configuration :Bucket NamePrivate Endpoint Then, from the Service Credentials of your Cloud Object Storage, get:Access Key IDSecret Access Key Note: If you don't have Service Credentials for the storage instance, create a new one with HMAC set to "true" (https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-uhc-hmac-credentials-main) Create or confirm that your IBM Cloud user has the necessary Bucket Access Policy to view and download objects (https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-iam-bucket-permissions) :  With this setup information confirmed, add to and connect ListS3 and FetchS3 proc

We all love Hue. However, it isn't always the perfect tool for doing a lot of querying, so you've probably looked toward some 3rd party tools for database connectivity. There are many such tools, but DBeaver has a very capable free version and is among the most popular. Connecting it to a Hive virtual warehouse isn't difficult, but there are a few gotchas that can make it frustrating. So let's conquer those obstacles. Here is a video walkthrough of the process. Provision a Hive Virtual Warehouse Log into a CDP instance Navigate to Data Warehouse Enable your Data Warehouse Environment & Database Catalog Create a New Virtual Warehouse Give your virtual warehouse a unique name Select Hive Select your database catalog from the dropdownSSO may be enabled or disabled, the choice is yours! Set Availability Zone and User Groups per your requirements or leave as-is Pick the size (aka decide how much money you want to spend)The remaining options are going to depend on your needs, but the

What is Apache Knox?In summary, Apache Knox was designed to provide access to the Big Data environment through a reverse proxy gateway, enabling perimeter protection when combined with a firewall. The Cloudera Data Platform supports Apache Knox and makes it simpler to install and administer by integrating it with the other components of the platform. Please, check the Cloudera Security Overview for additional information about how to increase the security in CDP. The figure below shows a high-level architecture of Apache Knox:   When using the Apache Knox Gateway we benefit from a number of advantages, such as:Single sign-on and enterprise authenticationPerimeter securityCentral access managementGranular access control to cluster servicesProxied JDBC connections and streamingExtensible APIEtc. Please, check the Apache Knox official site for more information. However, as all connections go through Knox, it becomes a critical piece of access to the envi

Introduction The AM2CM is an offline tool that converts the Ambari blueprint to Cloudera Manager Deployment template.Import the converted template to Cloudera Manager, start the services through the Cloudera Manager UI, and validate the cluster. The latest version of the tool can be downloaded here:Software download matrix for 3.1.5 to CDP 7.1.xNote: This article is written on am2cm version 2.0.4.0-4. Latest version might have some new features which are not covered in this article. Internals of workingFiles comes with AM2CM script, configs files, and lib directories.  am2cm-2.0.4.0-4 % ls -1am2cm-2.0.0.2.0.4.0-4.jaram2cm.shambari7_blueprint.jsoncm_migration.logconfconfigs_summary.logkerberos_summary.jsonlibrestore_collections.py  AM2CM script uses blueprint from Ambari cluster as input, and based on user input HDP version, makes a decision regarding the config mapping file need to be used. This mapping files consists of mapping rule from Ambari world to C

I just tried out the new NiFi 0.7.0 version's Slack.SourceI used Twitter, since it has some fun data and gets you a nice big stream. Sometimes with Twitter Feeds will be limited and Twitter will give you the 420 Enhance Your Calm Message. https://httpstatusdogs.com/420-enhance-your-calm Usually you can just wait 5-20 minutes and you will be serving again. Sometimes you might need to use a different of your apps, reset the tokens in your app or create a new app (https://apps.twitter.com/). ProcessingUse the Pull Key Attributes, to Find Only Tweets (remove null) Sink to SlackFor the PutToSlack ProcessorSet the Webhook URL to the URL generated by the incoming webhook page in slack.com.Set the Webhook Text to ${twitter.msg}, this will send your Twitter message to slack.Set the Channel to #general, or a channel of your choosing.I created a slack board for receiving my messages, https://nifi-se.slack.com/messages/general/.You can easily create your own (or using your existing Slack b

In our organization we have noticed that the native Power BI connector for Impala that is provided by Microsoft is not able to communicate with a Cloudera CDP cluster that is "behind" an Apache Knox reverse proxy (using Apache Knox means that among other things the transport mode 'HTTP' is needed for ODBC connections). While the different ways to connect via ODBC to Python, R, or other common end-user tools are extensively documented here on this site, we have so far not found any documentation of a way to set up Power BI with Direct Query capabilities in a cluster with Impala behind Apache Knox. Using a custom connector enables Direct Query with Impala via Apache Knox In order to achieve connectivity with Impala with Direct Query capabilities in Power BI (computations are sent to the cluster via ODBC), we created a fork of the Impala connector that Microsoft provides for Power BI and inserted the changes to the connection string that are needed in the setup I sketched above. Since the

For a recent project I was tasked with configuring DBeaver to connect to Phoenix running in an instance of the CDP Datahub. CDP provides a means of creating a Datahub for running an Operational Database (HBase) and using JDBC via Phoenix to query it.Let's start a Datahub from an Operational Database template.Provision an Operational DatabaseLog into a CDP InstanceSelect Data Hub ClustersSelect Create Data HubChoose the Cloud environmentChoose the template 7.2.2 - Operational Database with SQLProvide a unique name for the clusterClick Provision Cluster This will start a Datahub cluster running HBase and Phoenix as well as all of the security dependencies provided by SDX. For example, Knox - this will be important when connecting to our instance. Once the cluster has started we need to collect some configuration details. This is best done on the Datahub information page and in the Cloudera Manager Console. We will use the Phoenix thin driver, and this requires a JDBC string of

Cyber/Log data ingestion is super simple using CDP NiFi; however, it alone does not complete the full picture InfoSec & Threat hunters are interested in. Often these uber pros are looking for breadcrumbs/clues to shorten their "hunting" exercise. CDP DataViz immediately provides rich visualizations on high fidelity cyber/log data to shorten mean time to vulnerability detection. Additionally, I've observed these teams often need the ability to run ad hoc analysis, enrichment, and/or correlation on stream using an intuitive interface;  not to mention, without writing code. Ah yes, SQL on Stream.  The entire demo looks less than a few hours to build; proving in cyber, we don't always have to engineer (i.e. Grok) our way through difficult data challenges.   Demo:  

The CDP Public Cloud Release Summary for the month of April is now available. It summarizes all major features introduced in CDP Public Cloud Management Console, Data Hub, and data services. See CDP Public Cloud Release Summary - April 2022.

When I first joined Hortonworks I wrote an article on integrating Apache Hive and Spring Boot since I came from Pivotal and was a huge fan of Spring Boot. It's been almost 1.5 years and Spring Boot is now in 2.0 and Apache Hive is now in LLAP with ACID transactional tables. So it's time for a remix!It is crazy easy to start your project, just head to https://start.spring.io/ and build a bomb POM and let's being. You will need to modify the built POM.XML (Maven) to include some Hadoop and Hive references. No big deal.Clone it then own it:git clone https://github.com/tspannhw/hivereader.gitYou can build the code via build.sh.mvn install Then run it via run.sh (change HDP version to match yours).java -Xms512m -Xmx2048m -Dhdp.version=2.6.4 -Djava.net.preferIPv4Stack=true -jar target/hivereader-0.0.1-SNAPSHOT.jarYou must have the Java JDK, Git and Maven installed on your machine. If you don't why are you running Java programs. Obviously this can be inside a docker container or vi

This article explains how to use the Snowflake Connector for Spark in Cloudera Machine Learning. Save your Snowflake password for your account. Go to Account Settings > Environment Variables and create a new entry with Name as "SNOW_PASSWORD" and Value as your <password>. Create a new session in your CML project. Use any of the editors, and the Python 3.7 kernel. You also need to enable Spark for this session and select the Spark 2.4.7 version. Download required dependencies.To initiate the connection, you need to have a Snowflake Connector for Spark and a Snowflake JDBC Driver that's compatible with CML's Spark version. You can get these from the Maven Central Repository: spark-connector and jdbc-driver. Place them in a ./jars folder in your CML Project.cdsw@ysezv3fm94edq4pb:~$ ls -l ./jars/total 28512-rw-r--r-- 1 cdsw cdsw 28167364 Mar 17 21:03 snowflake-jdbc-3.13.16.jar-rw-r--r-- 1 cdsw cdsw 1027154 Jan 27 01:18 spark-snowflake_2.

This article explains how to connect to Snowflake from Cloudera Machine Learning.   Save your Snowflake password for your account.Go to Account Settings > Environment Variables and create a new entry with Name: "SNOW_PASSWORD" and Value: <your password> Create a new session in your CML Project. You can use any of the editors, and a Python kernel: Install required Python packagespip install pandas snowflake-connector-python snowflake-connector-python[pandas] Initiate the Snowflake connection. You need to set your Snowflake Account ID and your username for the connection. Your Snowflake password is retrieved from the environment variable that you configured in the Account Settings. I'm using the default Snowflake warehouse and a sample database/schema.import os import pandas as pdimport snowflake.connectorconn = snowflake.connector.connect( account='<Account ID>', user='<Username>', password=os.environ['SNOW_PASSWORD'], warehouse='COMPU