Community Articles

Introduction Cloudera Data Platform Base doesn't have one Quickstart/Sandbox VM like the ones for CDH/HDP releases that helped a lot of people (including me), to learn more about the open-source components and also see the improvements from the community in CDP Runtime.  The objective of this tutorial is to enable and create a VM from scratch via some automation (Shell Script and Cloudera Template) that can help whoever wants to use and/or learn Cloudera CDP in a Sandbox/Quickstart like environment in your machine. Pre-Requisites This exercise is performed on a Mac OS but you can install Vagrant/Virtualbox on Windows/Linux machines (https://www.vagrantup.com/docs/installation).The versions below were tested at the moment of writing this blog and may change in the future. The machine needs to have at least:80 GB of free disk space;12 GB free RAM;8 free VCPU;Good internet connection;Install Virtualbox and Vagrant These are the software that we'll use to

For this article let’s take the supposition that we have a MDM Software handling the enterprise organization.When we add/change an item in a MDM, it goes throw a workflow (validation, dedup, etc…).For synchronizing in real-time to Ranger “Users and Policies Management” the last action on the workflow must be publishing to a broker the information, that can be a JMS based broker(AMQ, RabbitMQ etc ..) or Kafka.Putting the message into a broker gives you the safety of never losing the event.We use HDF to consume, built the payload and call Ranger API.List of hidden Ranger useful API's.To invoke the API'scurl -u admin:admin -v -i -s -X POST -H "Accept: application/json" -H "Content-Type: application/json" http://<IP>:6080/service/xusers/secure/usersUpdate a userhttp://<IP>:6080/service/xusers/secure/groupsUpdate a grouphttp://<IP>:6080/service/xusers/groups?name=haGet particular group details

IntroductionSpark doesn't supply a mechanism to have data pushed to it - instead, it wants to pull data from other sources. In NiFi, this data can be exposed in such a way that a receiver can pull from it by adding an Output Port to the root process group. For Spark, we will use this same mechanism - we will use the Site-to-Site protocol to pull data from NiFi's Output Ports.Prerequisite1) Assuming you already have latest version of NiFi-0.4.1/HDF-1.1.1 downloaded on your HW Sandbox, else execute below after ssh connectivity to sandbox is established:# cd /opt/# wget http://public-repo-1.hortonworks.com/HDF/1.1.1.0/nifi-1.1.1.0-12-bin.tar.gz# tar -xvf nifi-1.1.1.0-12-bin.tar.gz2) Download Compatible version [in our case 0.4.1] of "nifi-spark-receiver" and "nifi-site-to-site-client" to Sandbox in a specific location:# mkdir /opt/spark-receiver# cd /opt/spark-receiver# wget http://central.maven.org/maven2/org/apache/nifi/nifi-site-to-site-client/0.4.1/nifi-site-to-site-client-0.4.1.jar

After upgrading from CDH 6.1.1 to CDP 7.1.7, we saw that spark was handling parquet files with older timestamps in non-UTC cluster incorrectly. Symptoms: In older saved parquet files via CDH, when we were reading these files in CDP, it was changing the timestamps for older dates and adding below property in when checking via parquet tools hadoop jar /opt/cloudera/parcels/CDH/jars/parquet-tools-1.10.99.7.1.7.0-551.jar meta  extra: writer.date.proleptic = falseextra: writer.time.zone = Australia/Sydney For us, 1985-02-01 onwards, the date was not changed and appeared normal for all outputs. Another observation was when this parquet data was read by pyspark it was normal but it did not return correct values when running spark-submit. Explanation:  There are two types of parquet readers and writers in CDP. One is Hive based and other are Spark native. Hive-supplied parquet readers and writers

In case customer wants to install CDV on top of their CDSW environment but they have an air gapped cluster and they are not able to connect to internet to install CDV, we can use below steps to install it. 1. If customer has some Dev environment with access to internet, they can enable Cloudera data visualisation there using below guide. https://docs.cloudera.com/data-visualization/7/start-cdsw/topics/viz-add-engine-cdsw-runtime.htmlIn case there is no internet access at all for Dev, Prod cluster of CDSW, Cloudera can be contacted and they can share the latest image for your current version. 2. If you have Dev. CDSW environment with CDV running, please identify the name of image from Engine Image section in CDV and check the running images by running below command in CDSW node.$ docker images 3. We need to save the full image (not the runtime image) to a docker file so that it can be copied over to the destination CDSW cluster.sudo docker save -o <cdv_image_name&

Table of ContentsUse CaseDesignImplementationPrerequisitesStep #1 - Setup NiFi FlowStep #2 - Setup PySpark JobStep #3 - Identify impacted members in Hue editorStep #4 - View Hive tables in AtlasUse CaseAs a healthcare provider / public health official, I want to respond equitably to the COVID-19 pandemic as quickly as possible, and serve all the communities that are adversely impacted in the state of California.I want to use health equity data reported by California Department of Public Health (CDPH) to identify impacted members and accelerate the launch of outreach programs.DesignCollect - Ingest data from https://data.chhs.ca.gov/dataset/covid-19-equity-metrics using NiFi.Enrich - Transform the dataset using Spark and load Hive tables.Report - Gather insights using Hue Editor.  ImplementationPrerequisitesAn existing CDP Private Cloud environment with following services - Hadoop Distributed File System (HDFS), Atlas, NiFi, Spark, Hive on Te

Table of ContentsUse CaseDesignImplementationPrerequisitesStep #1 - Setup NiFi FlowStep #2 - Setup Cloudera DataFlow (CDF)Step #3 - Setup Cloudera Data Engineering (CDE)Step #4 - Setup Cloudera Data Warehouse (CDW)Step #5 - Setup Cloudera Cloudera Data Visualization (Data VIZ) DashboardStep #6 - Identify impacted members in Hue editorStep #7 - View Hive tables in Cloudera Data CatalogStep #8 - Setup Cloudera AIUse CaseAs a healthcare provider / public health official, I want to respond equitably to the COVID-19 pandemic as quickly as possible, and serve all the communities that are adversely impacted in the state of California.I want to use health equity data reported by California Department of Public Health (CDPH) to identify impacted members and accelerate the launch of outreach programs.DesignCollect - Ingest data from https://data.chhs.ca.gov/dataset/covid-19-equity-metrics using NiFi.Enrich - Transform the dataset using Spark and load Hive tables.Rep

If you made it this far, you probably already know why you're here.  Connecting Dbeaver to an Impala virtual warehouse isn't difficult, but there are a few gotchas that can make it frustrating. So let's conquer those obstacles. Provision an Impala Virtual WarehouseLog into a CDP instanceNavigate to Data WarehouseEnable your Data Warehouse Environment & Database CatalogCreate a New Virtual WarehouseGive your virtual warehouse a unique nameSelect ImpalaSelect your database catalog from the dropdownSSO may be enabled or disabled, the choice is yours!Set Availability Zone and User Groups per your requirements or leave as-isPick the size (aka decide how much money you want to spend)The remaining options are going to depend on your needs, but the defaults are fine for our purposes.Click Create.  Expect approximately 5 minutes to create your virtual warehouse.  Download the JDBC DriverFrom your virtual warehouse tile, click on the kebab icon in

Table of ContentsOverviewDesignImplementationPrerequisitesStep #1 - Cloudera Data FlowStep #2 - Cloudera Data Warehouse (CDW)Step #3 - Cloudera Data VisualizationOverviewIn this article, let's build a real-time data visualization to analyze Twitter feeds using Cloudera Data Platform.Design  Explanation:NiFi Flow invokes Twitter API v2 (every 15 seconds), and stages all tweets in an AWS S3 bucket.Hive External Tables points to the staging location (i.e. AWS S3 bucket).Cloudera Data Visualization uses Hive External Tables as its data source, to create visuals. All visuals are refreshed every 20 seconds.ImplementationPrerequisitesA Cloudera Public Cloud environment on Amazon Web Services (AWS). If you don't have an existing environment, follow instructions here to set one up - CDP/AWS Quick Start Guide.An app in Twitter's Developer Portal. This is needed to call Twitter API v2. If this is your first time using Twitter API v2, follow these instructions - Step-by-step guide t

OverviewCloudera has implemented dbt adapters for Hive, Spark (CDE and Livy), and Impala. In addition to providing the adapters, Cloudera is offering a turn-key solution to be able to manage the end-to-end software development life cycle (SDLC) of dbt models. This solution is available in all clouds as well as on-prem deployments of CDP. It is useful for customers who would not like to use dbt Cloud for security reasons or the lack of availability of adapters in dbt cloud. We have identified the following requirements for any solution that supports the end-to-end SDLC of data transformation pipelines using dbt. Have multiple environmentsDevStage/TestProdHave a dev setup where different users can do the following in an isolated way:Make changes to modelsTest changesSee logs of testsUpdate docs in the models and see docsHave a CI/CD pipeline to push committed changes in the git repo to stage/prod environmentsSee logs in stage/prod of the dbt runsSee dbt docs in stage/prodO

NiFi Identity Conversion In a secure NiFi environment, the identity of a user can be determined in a number of ways depending on the authentication configuration. Machines also have an identity that needs to be determined upon authentication. Determining the identity of an entity is important to ensure proper authorization and access to resources.Machine Identity The identity of the node in a NiFi cluster is determined by the SSL certificate that is used for secure communication with other nodes in the cluster. This certificate can be generated by the internal Certificate Authority provided with HDF, or by an external CA. Once SSL is enabled on the cluster using the certificates, they will be stored (by default) in the /etc/nifi/conf/keystore.jks keystore. To get the node's identity as specified in the certificate, first get the keystore password from the nifi.properties file, then run the keytool command: cat /etc/nifi/conf/nifi.properties | grep keystorePasswdnifi.security.keystor

This week’s release includes:Adapters & docker images:dbt-spark-livy - 1.3.1 (Dec 5th, 2022)dbt-spark-cde - 1.3.0 (Dec 5th, 2022)dbt-hive - 1.3.1 (Dec 5th, 2022)dbt-impala - 1.3.1 (Dec 5th, 2022)For CML/CDSW deploymentpublic.ecr.aws/d7w2o6p0/dbt-cml:1.2.0 (with Jupyter Interface):NameVersiondbt-core1.2.0dbt-impala1.2.0dbt-hive1.2.0dbt-spark-cde1.2.0dbt-spark-livy1.2.0CML Base imageml-runtime-jupyterlab-python3.9-standard:2022.04.1-b6.py scripts (Utility)n/apublic.ecr.aws/d7w2o6p0/dbt-cdsw:1.2.0 (with workbench editor): NameVersiondbt-core1.2.0dbt-impala1.2.0dbt-hive1.2.0dbt-spark-cde1.2.0dbt-spark-livy1.2.0CML Base imageml-runtime-workbench-python3.9-standard:2022.04.1-b6.py scripts (Utility)n/a Note: Both CML and CDSW docker image works with CDSW, though later one only support workbench editor while setting up jobs.Supported infrastructure:All adapters we have released support CDP Public Cloud LDAP with KnoxOur Impala and Hive adapters support CDP Private Cloud with Kerb

Imagine a scenario where you're using CDP to host your Kafka environment, but you also are using Snowflake and need to get data from here to there with as little effort as possible.   It turns out that CDP plays very nicely with 3rd party vendors such as Snowflake.   Let's find out how through Streams Messaging Manager & KConnect.   Prerequisites:Streams Messaging data hubSnowflake account (if you don't have one, you can set up a free trial which will work fine for this)30 minutes of free time Installing the Snowflake Sink ConnectorThe default setup for a Streams Messaging data hub in CDP includes many Kafka Connect sources & sinks, but obviously every connector can't be included.  To use a 3rd party connector you need to download the jar(s) to each broker node in your Streams Messaging cluster.  Where exactly to put the jar files depends on what sort of jar it is.  If it is completely contained in one "fat" jar, then

The Kafka console producer & consumer client utilities are very simple to use in an unsecured environment, such as with many legacy CDH clusters or a standalone install.  However CDP-Public Cloud spins up secured by default, which means that we must get our feet wet with Kerberos if we want to use the Kafka client tools.  Kerberos isn't the easiest topic, but thankfully we won't need to get too deep into it. This isn't meant to be a Kerberos primer; we'll cover just enough to achieve the minimal goal of producing and consuming basic messages. Pre-requisitesA CDP environmentA Streams Messaging data hubroot access to the data hub hosts Building a Kafka ConfigurationThe Kafka client utilities require a number of ssl related properties in order to talk to Kafka. We'll put the properties into a configuration file named kafka-ssl.config for ease of use, which will look something like this boilerplate:    security.protocol = SASL_SSLsasl.m

Introduction   Stock market is always a thing when it comes to data analysis and examples, so how can we easily create an ingestion pipeline that allows quick insights with Cloudera Data Platform?   How about using the new Iceberg table format available in CDP to get new features with everything integrated and secured? Introducing Apache Iceberg in Cloudera Data Platform   In this example, I'll show how to use Cloudera Public Cloud to get data from Rest API and (with only clicks/parameters input) start quickly querying your data, dashboard, and advanced analytics.   In this example we will: Load intraday stock data into the object storage periodically (D-1 data) with Cloudera Dataflow; Process with Spark the data to make available in Cloudera Data Engineering and schedule with Airflow)This process will check if the table is new and then MERGE INTO (new) stock ICEBERG (new) table; Code is here. Analyze and query data with Cloudera Data Warehouse/Cloudera Data Visua

OverviewCloudera has implemented dbt adapters for Hive, Spark (CDE and Livy), and Impala. In addition to providing the adapters, Cloudera is offering a turn-key solution to be able to manage the end-to-end software development life cycle (SDLC) of dbt models. This solution is available in all clouds as well as on-prem deployments of CDP. It is useful for customers who would not like to use dbt Cloud for security reasons or the lack of availability of adapters in dbt cloud.We have identified the following requirements for any solution that supports the end-to-end SDLC of data transformation pipelines using dbt. Have multiple environmentsDevStage/TestProdHave a dev setup where different users can do the following in an isolated way:Make changes to modelsTest changesSee logs of testsUpdate docs in the models and see docsHave a CI/CD pipeline to push committed changes in the git repo to stage/prod environmentsSee logs in stage/prod of the dbt runsSee dbt docs in stage/prodOrchestratio

Cloudera maintains a list of current Authorized Sub-processors and Affiliates on Cloudera.com. When a change occurs, Cloudera will update the Authorized Sub-processors and Affiliates page and place a comment on this article to notify anyone who has subscribed to updates.For those who wish to receive notifications of these updates, you will need to join the community and subscribe to this article.   Becoming A Member   Joining the community is quite simple. Those who have an existing account with Cloudera will click the sign-in button and enter their account credentials. The first time logging into the community, new members will also choose a community nickname. This nickname will show alongside any posts by the member.     Those without an existing Cloudera account will click the register now button to begin the account creation process.     Subscribing To Updates   Members must subscribe to this article to receiv

Cloudera Data Engineering is Cloudera's new Spark as a Service offering on Public Cloud. It features kubernetes auto-scaling of Spark workers for efficient cost optimization, a simple UI interface for job management, and an integrated Airflow Scheduler for managing your production-grade workflows. In this article, we will not focus on service deployment. For that, refer to Enabling Cloudera Data Engineering. What we will cover is how to adjust your applications to fit CDE's paradigm, deploy them and schedule them by working through some example applications in Python and Scala. Developing Apps for Cloudera Data Engineering In CDE, an application is called a job. This takes the form of a main jar file or Python script. Depending on which language you wish to use, the development setup will be slightly different. We will go through those at a high level later. For creating and managing jobs, you can interact with CDE through the UI or the CLI. We will mainly use the UI here. For instr

Table of ContentsOverviewDesignPrerequisitesImplementationConclusionOverviewOver the last few years, Cloudera has evolved to become the Hybrid Data Company. Inarguably, the future of the data ecosystem is a hybrid one that enables common security and governance across on-premise and multi-cloud environments. In this article you will learn how Cloudera enables key hybrid capabilities like application portability, and data replication in order to quickly move workloads and data to the cloud. Let's explore how Cloudera excels at following hybrid use cases, through data pipeline replication and  data pipeline migration exercises to the public cloud.Develop Once and Run AnywhereDe-risk Cloud MigrationDesignThe following diagram shows a data pipeline in private & public form factors -PrerequisitesA data pipeline in a private cloud environment. Please follow the instructions provided in cdp-pvc-data-pipeline to set one up.Cloudera on Amazon Web Services

Oftentimes data hubs will need additional steps performed in order to support your specific workload or use case.  For me this usually is something like adding a jar to enable database connectivity on a Data Flow or Streams Messaging hub.  Recently I was working with Kafka Connect on a Streams Messaging data hub for MySQL change data capture.  This required uploading a particular jar file to each node in the cluster, placing it in a specific folder, renaming it, and changing the permissions.  The tasks themselves weren't difficult, but by the time I had done it on the 3rd broker node I was ready for a better solution.  By the end of this article we will have that better way implemented. Enter CDP Recipes.  Recipes are essentially bash or python scripts that execute while the cluster is spinning up.  They are analogous to "user data" when you spin up an EC2 instance, which can run commands to make initializing an instance easier so it's

OpenLDAP is an open-source implementation of the Lightweight Directory Access Protocol and is used for central management of accounts (users, hosts, and services) and can be used in concert with a KDC to provide authentication within the Hadoop ecosystem. Fundamentally, LDAP functions like a database in many ways and can be used to store any information.We will assume that you have a fresh CentOS 7 host available that will host OpenLDAP. Make sure you have network connectivity between any clients and this server and that DNS resolution is working. Let's ssh to the host, and install, as root, the packages we need with yum:yum -y install openldap compat-openldap openldap-clients openldap-servers openldap-servers-sql openldap-develWe'll also start the LDAP daemon (called slapd) and enable it to auto-start on system boot:systemctl start slapd.servicesystemctl enable slapd.serviceNext, run the slappasswd command to create an LDAP root password. Please take note of this root password, the en

HBase Region NormalizerRegion Normalizer a new feature available in HBase-1.2, can be used to perform normalization of inefficiently created HBase table regions. It basically carries out a set of pre-calculated merge/split actions to resize certain regions that are either too large or too small compared to the average region size for a given table. Region Normalizer when invoked computes a normalization plan for all of the tables in HBase. System tables (such as hbase:meta, hbase:namespace, Phoenix system tables etc) and user tables with normalization disabled are ignored while computing this normalization plan. For normalization enabled tables, normalization plan is carried out in parallel across multiple tables.Normalizer can be enabled or disabled globally for the entire cluster using ‘normalizer_switch’ command from HBase shell. Normalization can also be controlled per table basis, which is disabled by default when a table is created. Normalization for a table can be enabled or d

Command to delete a particular solr collection:-Example:-curl "http://<SOLR-HOSTNAME>:8886/solr/audit_logs/update?commit=true" -H "Content-Type: text/xml" --data-binary "<delete><query>evtTime:[* TO NOW-7DAYS]</query></delete>"In this example,audit_logs - is the collection name.TO NOW-7DAYS - Will delete the data older than 7 days.

Hi All,Below are the steps to setup load balancer on top of multiple Knox gateways. I will assume that you have KnoxSSO already set and services are going through it.   Install knox gateway on another node Ambari/CM -> Hosts -> Select any host(where you want to install knox) and add knox gateway.Installation of HAproxy and Rsyslog:   yum install -y haproxy rsyslog    Step 3: Update the Rsyslog for checking the haproxy logs /etc/rsyslog.conf:   ## Uncomment below line$ModLoad imudp$UDPServerRun 514##Add following directive in the same file.# HAProxy Logginglocal2.* /var/log/haproxy.log    Create SSL certificate for haproxy:   openssl genrsa -out /etc/haproxy/haproxy.key 2048openssl req -new -key /etc/haproxy/haproxy.key -out /etc/haproxy/haproxy.csr -subj "/C=US/ST=North Carolina/L=Raleigh/O=HWX/OU=Support/CN=$(hostname -f)"openssl x509 -req -days 365 -in /etc/haproxy/haproxy

Introduction Apache Ranger delivers a comprehensive approach to security for a Hadoop cluster. It provides a centralised platform to define, administer and manage security policies consistently across Hadoop components.More details about Ranger can be found here https://www.cloudera.com/products/open-source/apache-hadoop/apache-ranger.html  Ranger API policy documentation https://ranger.apache.org/apidocs/index.html  This article focuses on Export and Import of Ranger policies using API calls during HDP to CDP migration. Export List of Services configured in Ranger  ### Command to get list of services curl -s -u admin:pass -H "Accept: application/json" -H "Content-Type: application/json" -X GET "http://<hostname>:<ranger-port>/service/public/v2/api/service" > services.json Export of Policies  ### Export all policies To export all policiescurl -X GET --header "text/json" -H "Content-Type: text/json