Resources

With the May 2022 release of Cloudera Machine Learning, administrators now have the ability to specify a unique subdomain for the CML Workspace.  This enables a new CML workspace to leverage the endpoint of a previously deleted CML Workspace. This ensures that deployed models and applications deployed in the new Workspace will have the same endpoint as the same models and applications deployed in the old Workspace.  The text box to enter the Subdomain is available once you toggle the Advanced Options switch during  Workspace creation. For more information, see CML Static Subdomain in Provisioning ML Workspaces.

We are pleased to announce the release of the dbt-impala adapter v1.1.0. This release is compatible and tested with dbt-core v1.1.0.https://pypi.org/project/dbt-impala/1.1.0/https://github.com/cloudera/dbt-impala/releases/tag/v1.1.0 Full dbt-core v1.1.0 release notes can be found here, including a number of bug-fixes and improvements and support for Python 3.10 under the hood. Adapter Enhancements and New FeaturesAdopted new dbt methods provided in dbt 1.1.0Fix for an issue with get_catalog that caused empty catalog to be generated (#36)Ensure lower case for pattern passed to "show tables in" query (#37)Add timeout for snowplow endpoint to handle air gapped environments (#38)Bump version number to 1.1.0 to align with dbt-core versioningPython 3.10 compatibility as a result of migrating to dbt-core=1.1.0Getting started with dbt-impalaRead the Getting started with dbt-impala & Cloudera Data Warehouse guideWatch the video Using the dbt-impala example

The video below will take you through using the dbt-impala-example repo. This is an example dbt project, containing a few seeds, tests and models to help you bootstrap your own dbt project. If you haven't gotten started with dbt & Impala on CDP yet, take a look at the previous post Getting started with dbt-impala & Cloudera Data Warehouse. 

Kerberos cross realm trust for distcp This article is to demonstrate how to setup cross realm trust for distcp between two secure HDP clusters with their own Kerberos realms(KDC’s). Prerequisites Both HDP clusters must be running JDK 1.7 or higher. JDK 1.6 has some known issues Lets assume first HDP DEV cluster realm : HDPDEV.DEV.COM Lets assume second HDP QA cluster realm : HDPQA.QA.COM Step 1 : To set up cross realm trust between HDPDEV.DEV.COM and HDPQA.QA.COM, for example a client of realm HDPDEV.DEV.COM to access a service in realm HDPDQA.QA.COM, both realms must share a key for a principal name krbtgt/ HDPDQA.QA.COM@ HDPDEV.DEV.COM and both keys must have the same key version number associated with them. Cross realm trust is unidirectional by default. So for clients in HDPQA.QA.COM also to have access services in HDPDEV.DEV.com, both realms must share a key for principal krbtgt/ HDPDDEV.DEV.COM@ HDPQA.QA.COM. Add both krbtgt principals on both clusters #HDP DEV Cluster kadmin.l

The ListS3 and FetchS3 processors in Apache NiFi are commonly used to retrieve objects from Amazon S3 buckets, but they can be easily configured to retrieve objects from IBM Cloud buckets. Assume, I have an IBM Cloud bucket that contains three CSV files:  First, get the following from your IBM Cloud bucket configuration :Bucket NamePrivate Endpoint Then, from the Service Credentials of your Cloud Object Storage, get:Access Key IDSecret Access Key Note: If you don't have Service Credentials for the storage instance, create a new one with HMAC set to "true" (https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-uhc-hmac-credentials-main) Create or confirm that your IBM Cloud user has the necessary Bucket Access Policy to view and download objects (https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-iam-bucket-permissions) :  With this setup information confirmed, add to and connect ListS3 and FetchS3 proc

We all love Hue. However, it isn't always the perfect tool for doing a lot of querying, so you've probably looked toward some 3rd party tools for database connectivity. There are many such tools, but DBeaver has a very capable free version and is among the most popular. Connecting it to a Hive virtual warehouse isn't difficult, but there are a few gotchas that can make it frustrating. So let's conquer those obstacles. Here is a video walkthrough of the process. Provision a Hive Virtual Warehouse Log into a CDP instance Navigate to Data Warehouse Enable your Data Warehouse Environment & Database Catalog Create a New Virtual Warehouse Give your virtual warehouse a unique name Select Hive Select your database catalog from the dropdownSSO may be enabled or disabled, the choice is yours! Set Availability Zone and User Groups per your requirements or leave as-is Pick the size (aka decide how much money you want to spend)The remaining options are going to depend on your needs, but the

What is Apache Knox?In summary, Apache Knox was designed to provide access to the Big Data environment through a reverse proxy gateway, enabling perimeter protection when combined with a firewall. The Cloudera Data Platform supports Apache Knox and makes it simpler to install and administer by integrating it with the other components of the platform. Please, check the Cloudera Security Overview for additional information about how to increase the security in CDP. The figure below shows a high-level architecture of Apache Knox:   When using the Apache Knox Gateway we benefit from a number of advantages, such as:Single sign-on and enterprise authenticationPerimeter securityCentral access managementGranular access control to cluster servicesProxied JDBC connections and streamingExtensible APIEtc. Please, check the Apache Knox official site for more information. However, as all connections go through Knox, it becomes a critical piece of access to the envi

Introduction The AM2CM is an offline tool that converts the Ambari blueprint to Cloudera Manager Deployment template.Import the converted template to Cloudera Manager, start the services through the Cloudera Manager UI, and validate the cluster. The latest version of the tool can be downloaded here:Software download matrix for 3.1.5 to CDP 7.1.xNote: This article is written on am2cm version 2.0.4.0-4. Latest version might have some new features which are not covered in this article. Internals of workingFiles comes with AM2CM script, configs files, and lib directories.  am2cm-2.0.4.0-4 % ls -1am2cm-2.0.0.2.0.4.0-4.jaram2cm.shambari7_blueprint.jsoncm_migration.logconfconfigs_summary.logkerberos_summary.jsonlibrestore_collections.py  AM2CM script uses blueprint from Ambari cluster as input, and based on user input HDP version, makes a decision regarding the config mapping file need to be used. This mapping files consists of mapping rule from Ambari world to C

I just tried out the new NiFi 0.7.0 version's Slack.SourceI used Twitter, since it has some fun data and gets you a nice big stream. Sometimes with Twitter Feeds will be limited and Twitter will give you the 420 Enhance Your Calm Message. https://httpstatusdogs.com/420-enhance-your-calm Usually you can just wait 5-20 minutes and you will be serving again. Sometimes you might need to use a different of your apps, reset the tokens in your app or create a new app (https://apps.twitter.com/). ProcessingUse the Pull Key Attributes, to Find Only Tweets (remove null) Sink to SlackFor the PutToSlack ProcessorSet the Webhook URL to the URL generated by the incoming webhook page in slack.com.Set the Webhook Text to ${twitter.msg}, this will send your Twitter message to slack.Set the Channel to #general, or a channel of your choosing.I created a slack board for receiving my messages, https://nifi-se.slack.com/messages/general/.You can easily create your own (or using your existing Slack b

In our organization we have noticed that the native Power BI connector for Impala that is provided by Microsoft is not able to communicate with a Cloudera CDP cluster that is "behind" an Apache Knox reverse proxy (using Apache Knox means that among other things the transport mode 'HTTP' is needed for ODBC connections). While the different ways to connect via ODBC to Python, R, or other common end-user tools are extensively documented here on this site, we have so far not found any documentation of a way to set up Power BI with Direct Query capabilities in a cluster with Impala behind Apache Knox. Using a custom connector enables Direct Query with Impala via Apache Knox In order to achieve connectivity with Impala with Direct Query capabilities in Power BI (computations are sent to the cluster via ODBC), we created a fork of the Impala connector that Microsoft provides for Power BI and inserted the changes to the connection string that are needed in the setup I sketched above. Since the

For a recent project I was tasked with configuring DBeaver to connect to Phoenix running in an instance of the CDP Datahub. CDP provides a means of creating a Datahub for running an Operational Database (HBase) and using JDBC via Phoenix to query it.Let's start a Datahub from an Operational Database template.Provision an Operational DatabaseLog into a CDP InstanceSelect Data Hub ClustersSelect Create Data HubChoose the Cloud environmentChoose the template 7.2.2 - Operational Database with SQLProvide a unique name for the clusterClick Provision Cluster This will start a Datahub cluster running HBase and Phoenix as well as all of the security dependencies provided by SDX. For example, Knox - this will be important when connecting to our instance. Once the cluster has started we need to collect some configuration details. This is best done on the Datahub information page and in the Cloudera Manager Console. We will use the Phoenix thin driver, and this requires a JDBC string of

Cyber/Log data ingestion is super simple using CDP NiFi; however, it alone does not complete the full picture InfoSec & Threat hunters are interested in. Often these uber pros are looking for breadcrumbs/clues to shorten their "hunting" exercise. CDP DataViz immediately provides rich visualizations on high fidelity cyber/log data to shorten mean time to vulnerability detection. Additionally, I've observed these teams often need the ability to run ad hoc analysis, enrichment, and/or correlation on stream using an intuitive interface;  not to mention, without writing code. Ah yes, SQL on Stream.  The entire demo looks less than a few hours to build; proving in cyber, we don't always have to engineer (i.e. Grok) our way through difficult data challenges.   Demo:  

The CDP Public Cloud Release Summary for the month of April is now available. It summarizes all major features introduced in CDP Public Cloud Management Console, Data Hub, and data services. See CDP Public Cloud Release Summary - April 2022.

When I first joined Hortonworks I wrote an article on integrating Apache Hive and Spring Boot since I came from Pivotal and was a huge fan of Spring Boot. It's been almost 1.5 years and Spring Boot is now in 2.0 and Apache Hive is now in LLAP with ACID transactional tables. So it's time for a remix!It is crazy easy to start your project, just head to https://start.spring.io/ and build a bomb POM and let's being. You will need to modify the built POM.XML (Maven) to include some Hadoop and Hive references. No big deal.Clone it then own it:git clone https://github.com/tspannhw/hivereader.gitYou can build the code via build.sh.mvn install Then run it via run.sh (change HDP version to match yours).java -Xms512m -Xmx2048m -Dhdp.version=2.6.4 -Djava.net.preferIPv4Stack=true -jar target/hivereader-0.0.1-SNAPSHOT.jarYou must have the Java JDK, Git and Maven installed on your machine. If you don't why are you running Java programs. Obviously this can be inside a docker container or vi

This article explains how to use the Snowflake Connector for Spark in Cloudera Machine Learning. Save your Snowflake password for your account. Go to Account Settings > Environment Variables and create a new entry with Name as "SNOW_PASSWORD" and Value as your <password>. Create a new session in your CML project. Use any of the editors, and the Python 3.7 kernel. You also need to enable Spark for this session and select the Spark 2.4.7 version. Download required dependencies.To initiate the connection, you need to have a Snowflake Connector for Spark and a Snowflake JDBC Driver that's compatible with CML's Spark version. You can get these from the Maven Central Repository: spark-connector and jdbc-driver. Place them in a ./jars folder in your CML Project.cdsw@ysezv3fm94edq4pb:~$ ls -l ./jars/total 28512-rw-r--r-- 1 cdsw cdsw 28167364 Mar 17 21:03 snowflake-jdbc-3.13.16.jar-rw-r--r-- 1 cdsw cdsw 1027154 Jan 27 01:18 spark-snowflake_2.

This article explains how to connect to Snowflake from Cloudera Machine Learning.   Save your Snowflake password for your account.Go to Account Settings > Environment Variables and create a new entry with Name: "SNOW_PASSWORD" and Value: <your password> Create a new session in your CML Project. You can use any of the editors, and a Python kernel: Install required Python packagespip install pandas snowflake-connector-python snowflake-connector-python[pandas] Initiate the Snowflake connection. You need to set your Snowflake Account ID and your username for the connection. Your Snowflake password is retrieved from the environment variable that you configured in the Account Settings. I'm using the default Snowflake warehouse and a sample database/schema.import os import pandas as pdimport snowflake.connectorconn = snowflake.connector.connect( account='<Account ID>', user='<Username>', password=os.environ['SNOW_PASSWORD'], warehouse='COMPU

We frequently get asked questions by customers about how to run, deploy and/or debug Cloudera’s Operational Database, which is based on Apache HBase & Phoenix. This post shares the most frequently asked questions and how they were addressed.   Can I run HBase on HDP 7.1.4 for my production environment?Answer: HDP 7.1.4 is only an intermediate step in the upgrade from HDP2 or HDP 3 to CDP. Once HDP 7.1.4 is deployed, you need to use the AM2CM migration tool to complete the upgrade to CDP. HDP 7 is not supported for production environments. HBase is not expected to be stable in HDP 7 but will be stable after the completion of the upgrade to CDP. How can I repair my HBase metadata?Answer: Use the hbck2 meta repair feature. For more information about this feature, see this github link. Alternatively, if there is no important data there, you can clean up the whole HDFS HBase root directory and also remove the HBase ‘znode’ from ZooKeeper using the deleteall/hbase comman

In this post, we will learn how to create a Kafka topic and produce and consume messages from a Kafka topic. After testing the basic producer and consumer example, we will test it with Spark using spark-examples.jar file.   Creating a Kafka topic:   # kafka bootstrap serverKAFKA_BROKERS="localhost:9092"# kafka topic nameTOPIC_NAME="word_count_topic"# group nameGROUP_NAME="spark-kafka-group"# creating a topic/opt/cloudera/parcels/CDH/lib/kafka/bin/kafka-topics.sh --create --topic ${TOPIC_NAME} --bootstrap-server ${KAFKA_BROKERS}# describing a topic/opt/cloudera/parcels/CDH/lib/kafka/bin/kafka-topics.sh --describe --topic ${TOPIC_NAME} --bootstrap-server ${KAFKA_BROKERS}     Producing messages to Kafka topic:   # producing kafka messages/opt/cloudera/parcels/CDH/lib/kafka/bin/kafka-console-producer.sh --topic ${TOPIC_NAME} --broker-list ${KAFKA_BROKERS}     Consuming messages from Kafka topic:   # consuming kafka messages/opt/cloudera/parcel

Intro Online shopping is on the rise as more of us stay at home and let our credit cards do the walking. Keeping pace with that trend is an unfortunate increase in credit card fraud. It’s no surprise, really. According to Forbes, online fraud has been a growing problem for the past few years. And now, as consumers and businesses adapt to the worldwide pandemic and make more credit card transactions in the card-not-present (CNP) space, the resulting uptick in online shopping and e-commerce has opened up an even bigger playground for fraudsters to try out new tricks. Fraud detection has been a major issue for financial services and institutions. But artificial intelligence has an enormous potential to reduce financial fraud. Artificial intelligence applications have a great potential to detect and prevent fraud. Therefore, we will start a series of articles talking about that and how we can use Cloudera mechanisms to implement a whole Credit Card Fraud detection solution. But first, le

If you have erasure coded some directory and perform some operations on the directory you might have observed WARN messages like belowWARN erasurecode.ErasureCodeNative: Loading ISA-L failed: Failed to load libisal.so.2 (libisal.so.2: cannot open shared object file: No such file or directory)WARN erasurecode.ErasureCodeNative: ISA-L support is not available in your platform... using builtin-java codec where applicableThis WARN messages are due to ISA Library not being present on the node. Below are the steps to enable the library1) Clone the isa-l github repository.# git clone https://github.com/01org/isa-l.git2) Go to the cloned directory# cd isa-l3) Install yasm if you do not have it already# yum install -y yasm ---> centOS# apt-get install yasm ----> ubuntu4) Build the library # make -f Makefile.unx5) Copy the library files to lib directory# cp bin/libisal.so bin/libisal.so.2 /lib646) Verify that isa-l library is enabled properly# hadoop checknativeExpected output18/10/12 10

This video contains a step by step process that shows how to connect to Hive running on a secure cluster while using a JDBC Uber driver from MS Windows. Open the video on YouTube here   Prerequisites: Validate username belong to the same @DOMAIN /Realm as the one setup on cluster nodes. Install  DbVisualizer. Download Hive Uber driver for same version as HDP. Kerberos Java  Config Get Kerberos /etc/krb5.conf from cluster, scp this file from any cluster node to c:\windows. Rename krb5.conf to c:\windows\krb5.ini. Edit krb5.ini and add property: (see video)  property udp_preference_limit = 1   DbVisualizer Setup Add Hive Uber driver .jar to DBVisualizer as driver for Hive (see video). Add startup Java command line options to DBVisualizer for kerberos , under Tools (see video): -Dsun.security.krb5.debug=true-Djavax.security.auth.useSubjectCredsOnly=false-Djava.security.krb5.conf=c:\windows\krb5.in

in August 2021, we released CDP Private Cloud Base 7.1.7 LTS.This article will outline what that means.For each software release, we provide a version, General Availability (GA) date, and End of Service (EoS) date. These are all published and described here. CDP Private Cloud Base 7.1.7 LTS has a GA date of August 2021 and an EoS date of August 2025, providing a 4-year support window. We then offer two paths,Service Pack TrainThe Service Pack train is for customers who value a stable platform, service a lot of users or use-cases, and want a low risk, low costs in maintenance and operations. Rather than taking the new feature release, customers can instead take regular Cumulative Hotfix (CHF)*. These hotfixes are released pro-actively to address break-fix issues and CVEs. At certain pre-planned release date, the hotfixes will be rolled up into a Service Pack (e.g. SP1)**. * Cumulative Hotfix means each hotfix includes those that came before up to the initial release

Steps to connect from Legacy Cluster to CDP Public Cloud Hive:   Build a connection string as follows using Hive Server Endpoint from DataHub by just adding the sslTrustStore and trustStorePassword along with workload username and password for PAM authentication.jdbc:hive2://master0.repro.cloudera/default;transportMode=http;httpPath=new/cdp-proxy-api/hive;ssl=true;sslTrustStore=gateway-client-trust.jks;trustStorePassword=*****;user=chella; Get the JKS file downloaded from CDP Knox (Token Integration) and copy it to HDP/CDH host. Get the latest JDBC jar "hive-jdbc-3.1.0-SNAPSHOT-standalone.jar" from CDW.CDP Control Plane > Management Console > Data Warehouse > click the 3 dots from any of Hive VW and "Download JDBC JAR" Copy the jar "hive-jdbc-3.1.0-SNAPSHOT-standalone.jar"  to HDP/CDH host. Get into beeline from HDP/CDH host Add the jar using the following commandBeeline version 3.1.0.3.1.5.26-1 by Apache Hive 0: jdbc:hive2://c1544-node2> !addlocaldriverjar /roo

Perform the following steps to access the Hive MetaStore database :- SSH to the Data lake CM Node as 'cloudbreak' user.Run 'sudo -i' to gain root access.source activate_salt_envexport PGPASSWORD=$(salt-call pillar.get postgres:hive:password 2>/dev/null| tail -n 1 | awk '{print $1}')HIVE_DB=$(salt-call pillar.get postgres:hive:database 2>/dev/null| tail -n 1 | awk '{print $1}')HIVE_DB_USER=$(salt-call pillar.get postgres:hive:user 2>/dev/null| tail -n 1 | awk '{print $1}')psql -U ${HIVE_DB_USER} -d ${HIVE_DB}e.g. [root@xxxx-master0 ~]# source activate_salt_env(salt_3001.8) [root@xxxx-master0 ~]# export PGPASSWORD=$(salt-call pillar.get postgres:hive:password 2>/dev/null| tail -n 1 | awk '{print $1}')(salt_3001.8) [root@xxxx-master0 ~]# HIVE_DB=$(salt-call pillar.get postgres:hive:database 2>/dev/null| tail -n 1 | awk '{print $1}')(salt_3001.8) [root@xxxx-master0 ~]# HIVE_DB_USER=$(salt-call pillar.get postgres:hive:user 2>/dev/null| tail -n 1 | awk '{print $1

In NiFi the data being passed between operators is referred to as a FlowFile and can be accessed via various scripting languages in the ExecuteScript operator.In order to access the data in the FlowFile you need to understand a few requirements first.In this example we will access the json data being passed into the ExecuteScript operator via a getTwitter operator.This could be any data ingestion operator.Using the ExecuteScript (Note: Execute script will use Jython libraries which are good but limited. If you have a python script which uses other libs and produces an output you can use Execute Process instead which will execute the python script on the machine using the full python lib and the output will become your flow file. )  operator set the script type to python and in the "script body" section enter the following code:import jsonimport java.iofrom org.apache.commons.io import IOUtilsfrom java.nio.charset import StandardCharsetsfrom org.apache.nifi.processor.io import Stre