Resources

NiFi Identity Conversion In a secure NiFi environment, the identity of a user can be determined in a number of ways depending on the authentication configuration. Machines also have an identity that needs to be determined upon authentication. Determining the identity of an entity is important to ensure proper authorization and access to resources.Machine Identity The identity of the node in a NiFi cluster is determined by the SSL certificate that is used for secure communication with other nodes in the cluster. This certificate can be generated by the internal Certificate Authority provided with HDF, or by an external CA. Once SSL is enabled on the cluster using the certificates, they will be stored (by default) in the /etc/nifi/conf/keystore.jks keystore. To get the node's identity as specified in the certificate, first get the keystore password from the nifi.properties file, then run the keytool command: cat /etc/nifi/conf/nifi.properties | grep keystorePasswdnifi.security.keystor

This week’s release includes:Adapters & docker images:dbt-spark-livy - 1.3.1 (Dec 5th, 2022)dbt-spark-cde - 1.3.0 (Dec 5th, 2022)dbt-hive - 1.3.1 (Dec 5th, 2022)dbt-impala - 1.3.1 (Dec 5th, 2022)For CML/CDSW deploymentpublic.ecr.aws/d7w2o6p0/dbt-cml:1.2.0 (with Jupyter Interface):NameVersiondbt-core1.2.0dbt-impala1.2.0dbt-hive1.2.0dbt-spark-cde1.2.0dbt-spark-livy1.2.0CML Base imageml-runtime-jupyterlab-python3.9-standard:2022.04.1-b6.py scripts (Utility)n/apublic.ecr.aws/d7w2o6p0/dbt-cdsw:1.2.0 (with workbench editor): NameVersiondbt-core1.2.0dbt-impala1.2.0dbt-hive1.2.0dbt-spark-cde1.2.0dbt-spark-livy1.2.0CML Base imageml-runtime-workbench-python3.9-standard:2022.04.1-b6.py scripts (Utility)n/a Note: Both CML and CDSW docker image works with CDSW, though later one only support workbench editor while setting up jobs.Supported infrastructure:All adapters we have released support CDP Public Cloud LDAP with KnoxOur Impala and Hive adapters support CDP Private Cloud with Kerb

Imagine a scenario where you're using CDP to host your Kafka environment, but you also are using Snowflake and need to get data from here to there with as little effort as possible.   It turns out that CDP plays very nicely with 3rd party vendors such as Snowflake.   Let's find out how through Streams Messaging Manager & KConnect.   Prerequisites:Streams Messaging data hubSnowflake account (if you don't have one, you can set up a free trial which will work fine for this)30 minutes of free time Installing the Snowflake Sink ConnectorThe default setup for a Streams Messaging data hub in CDP includes many Kafka Connect sources & sinks, but obviously every connector can't be included.  To use a 3rd party connector you need to download the jar(s) to each broker node in your Streams Messaging cluster.  Where exactly to put the jar files depends on what sort of jar it is.  If it is completely contained in one "fat" jar, then

The Kafka console producer & consumer client utilities are very simple to use in an unsecured environment, such as with many legacy CDH clusters or a standalone install.  However CDP-Public Cloud spins up secured by default, which means that we must get our feet wet with Kerberos if we want to use the Kafka client tools.  Kerberos isn't the easiest topic, but thankfully we won't need to get too deep into it. This isn't meant to be a Kerberos primer; we'll cover just enough to achieve the minimal goal of producing and consuming basic messages. Pre-requisitesA CDP environmentA Streams Messaging data hubroot access to the data hub hosts Building a Kafka ConfigurationThe Kafka client utilities require a number of ssl related properties in order to talk to Kafka. We'll put the properties into a configuration file named kafka-ssl.config for ease of use, which will look something like this boilerplate:    security.protocol = SASL_SSLsasl.m

Introduction   Stock market is always a thing when it comes to data analysis and examples, so how can we easily create an ingestion pipeline that allows quick insights with Cloudera Data Platform?   How about using the new Iceberg table format available in CDP to get new features with everything integrated and secured? Introducing Apache Iceberg in Cloudera Data Platform   In this example, I'll show how to use Cloudera Public Cloud to get data from Rest API and (with only clicks/parameters input) start quickly querying your data, dashboard, and advanced analytics.   In this example we will: Load intraday stock data into the object storage periodically (D-1 data) with Cloudera Dataflow; Process with Spark the data to make available in Cloudera Data Engineering and schedule with Airflow)This process will check if the table is new and then MERGE INTO (new) stock ICEBERG (new) table; Code is here. Analyze and query data with Cloudera Data Warehouse/Cloudera Data Visua

OverviewCloudera has implemented dbt adapters for Hive, Spark (CDE and Livy), and Impala. In addition to providing the adapters, Cloudera is offering a turn-key solution to be able to manage the end-to-end software development life cycle (SDLC) of dbt models. This solution is available in all clouds as well as on-prem deployments of CDP. It is useful for customers who would not like to use dbt Cloud for security reasons or the lack of availability of adapters in dbt cloud.We have identified the following requirements for any solution that supports the end-to-end SDLC of data transformation pipelines using dbt. Have multiple environmentsDevStage/TestProdHave a dev setup where different users can do the following in an isolated way:Make changes to modelsTest changesSee logs of testsUpdate docs in the models and see docsHave a CI/CD pipeline to push committed changes in the git repo to stage/prod environmentsSee logs in stage/prod of the dbt runsSee dbt docs in stage/prodOrchestratio

Cloudera maintains a list of current Authorized Sub-processors and Affiliates on Cloudera.com. When a change occurs, Cloudera will update the Authorized Sub-processors and Affiliates page and place a comment on this article to notify anyone who has subscribed to updates.For those who wish to receive notifications of these updates, you will need to join the community and subscribe to this article.   Becoming A Member   Joining the community is quite simple. Those who have an existing account with Cloudera will click the sign-in button and enter their account credentials. The first time logging into the community, new members will also choose a community nickname. This nickname will show alongside any posts by the member.     Those without an existing Cloudera account will click the register now button to begin the account creation process.     Subscribing To Updates   Members must subscribe to this article to receiv

Cloudera Data Engineering is Cloudera's new Spark as a Service offering on Public Cloud. It features kubernetes auto-scaling of Spark workers for efficient cost optimization, a simple UI interface for job management, and an integrated Airflow Scheduler for managing your production-grade workflows. In this article, we will not focus on service deployment. For that, refer to Enabling Cloudera Data Engineering. What we will cover is how to adjust your applications to fit CDE's paradigm, deploy them and schedule them by working through some example applications in Python and Scala. Developing Apps for Cloudera Data Engineering In CDE, an application is called a job. This takes the form of a main jar file or Python script. Depending on which language you wish to use, the development setup will be slightly different. We will go through those at a high level later. For creating and managing jobs, you can interact with CDE through the UI or the CLI. We will mainly use the UI here. For instr

Table of ContentsOverviewDesignPrerequisitesImplementationConclusionOverviewOver the last few years, Cloudera has evolved to become the Hybrid Data Company. Inarguably, the future of the data ecosystem is a hybrid one that enables common security and governance across on-premise and multi-cloud environments. In this article you will learn how Cloudera enables key hybrid capabilities like application portability, and data replication in order to quickly move workloads and data to the cloud. Let's explore how Cloudera excels at following hybrid use cases, through data pipeline replication and  data pipeline migration exercises to the public cloud.Develop Once and Run AnywhereDe-risk Cloud MigrationDesignThe following diagram shows a data pipeline in private & public form factors -PrerequisitesA data pipeline in a private cloud environment. Please follow the instructions provided in cdp-pvc-data-pipeline to set one up.Cloudera on Amazon Web Services

Oftentimes data hubs will need additional steps performed in order to support your specific workload or use case.  For me this usually is something like adding a jar to enable database connectivity on a Data Flow or Streams Messaging hub.  Recently I was working with Kafka Connect on a Streams Messaging data hub for MySQL change data capture.  This required uploading a particular jar file to each node in the cluster, placing it in a specific folder, renaming it, and changing the permissions.  The tasks themselves weren't difficult, but by the time I had done it on the 3rd broker node I was ready for a better solution.  By the end of this article we will have that better way implemented. Enter CDP Recipes.  Recipes are essentially bash or python scripts that execute while the cluster is spinning up.  They are analogous to "user data" when you spin up an EC2 instance, which can run commands to make initializing an instance easier so it's

OpenLDAP is an open-source implementation of the Lightweight Directory Access Protocol and is used for central management of accounts (users, hosts, and services) and can be used in concert with a KDC to provide authentication within the Hadoop ecosystem. Fundamentally, LDAP functions like a database in many ways and can be used to store any information.We will assume that you have a fresh CentOS 7 host available that will host OpenLDAP. Make sure you have network connectivity between any clients and this server and that DNS resolution is working. Let's ssh to the host, and install, as root, the packages we need with yum:yum -y install openldap compat-openldap openldap-clients openldap-servers openldap-servers-sql openldap-develWe'll also start the LDAP daemon (called slapd) and enable it to auto-start on system boot:systemctl start slapd.servicesystemctl enable slapd.serviceNext, run the slappasswd command to create an LDAP root password. Please take note of this root password, the en

HBase Region NormalizerRegion Normalizer a new feature available in HBase-1.2, can be used to perform normalization of inefficiently created HBase table regions. It basically carries out a set of pre-calculated merge/split actions to resize certain regions that are either too large or too small compared to the average region size for a given table. Region Normalizer when invoked computes a normalization plan for all of the tables in HBase. System tables (such as hbase:meta, hbase:namespace, Phoenix system tables etc) and user tables with normalization disabled are ignored while computing this normalization plan. For normalization enabled tables, normalization plan is carried out in parallel across multiple tables.Normalizer can be enabled or disabled globally for the entire cluster using ‘normalizer_switch’ command from HBase shell. Normalization can also be controlled per table basis, which is disabled by default when a table is created. Normalization for a table can be enabled or d

Command to delete a particular solr collection:-Example:-curl "http://<SOLR-HOSTNAME>:8886/solr/audit_logs/update?commit=true" -H "Content-Type: text/xml" --data-binary "<delete><query>evtTime:[* TO NOW-7DAYS]</query></delete>"In this example,audit_logs - is the collection name.TO NOW-7DAYS - Will delete the data older than 7 days.

Hi All,Below are the steps to setup load balancer on top of multiple Knox gateways. I will assume that you have KnoxSSO already set and services are going through it.   Install knox gateway on another node Ambari/CM -> Hosts -> Select any host(where you want to install knox) and add knox gateway.Installation of HAproxy and Rsyslog:   yum install -y haproxy rsyslog    Step 3: Update the Rsyslog for checking the haproxy logs /etc/rsyslog.conf:   ## Uncomment below line$ModLoad imudp$UDPServerRun 514##Add following directive in the same file.# HAProxy Logginglocal2.* /var/log/haproxy.log    Create SSL certificate for haproxy:   openssl genrsa -out /etc/haproxy/haproxy.key 2048openssl req -new -key /etc/haproxy/haproxy.key -out /etc/haproxy/haproxy.csr -subj "/C=US/ST=North Carolina/L=Raleigh/O=HWX/OU=Support/CN=$(hostname -f)"openssl x509 -req -days 365 -in /etc/haproxy/haproxy

Introduction Apache Ranger delivers a comprehensive approach to security for a Hadoop cluster. It provides a centralised platform to define, administer and manage security policies consistently across Hadoop components.More details about Ranger can be found here https://www.cloudera.com/products/open-source/apache-hadoop/apache-ranger.html  Ranger API policy documentation https://ranger.apache.org/apidocs/index.html  This article focuses on Export and Import of Ranger policies using API calls during HDP to CDP migration. Export List of Services configured in Ranger  ### Command to get list of services curl -s -u admin:pass -H "Accept: application/json" -H "Content-Type: application/json" -X GET "http://<hostname>:<ranger-port>/service/public/v2/api/service" > services.json Export of Policies  ### Export all policies To export all policiescurl -X GET --header "text/json" -H "Content-Type: text/json

With the need of keeping metrics centralized into a single spot, people have sought a way to configure grafana for observability by attempting to collect the wide range of metrics available in a Datahub / Datalake in order to keep their dashboards arranged in a centralized hub. This tutorial will walk you through step-by-step in how to configure Grafana to query the metrics available in the Cloudera Manager of a datahub cluster. First, we need to ensure that the machine where Grafana is running has direct connection with the CM Server. In other words, it must be able to resolve the CM FQDN and establish a straight up connection with the service, bypassing knox completely. Unlike the Grafana utilized by other experiences, such as DWX, CDW and so forth of which use prometheus for integration, this grafana will have to integrate with CM Server for authentication and during the creation of a datasource when querying the metrics therein. This integration i

After Jun 1, 2021, the following changes are introduced to S3Guard in CDW service on CDP Public Cloud[1]:S3 Guard disabled for custom Database Catalogs and default Database Catalogs on AWS environmentsDuring new AWS environment activation in CDW:If you have not specified a DynamoDB table name during environment registration in Management Console, the S3 Guard feature is disabled for all Database Catalogs and Virtual Warehouses.If you have specified a DynamoDB table name during environment registration in Management Console, the S3 Guard feature is enabled for default Database Catalogs and Virtual Warehouses. However, it is disabled for non-default (custom) Database Catalogs.For all environments that were activated before the June 1, 2021 release, S3 Guard is enabled. [1] https://docs.cloudera.com/data-warehouse/cloud/release-notes/topics/dw-whats-new-24.html

Overview:This handbook focuses on Cloudera’s new adapter dbt-spark-cde that enables users to use dbt on top of Apache Spark by leveraging the use of CDE. Cloudera Data Engineering (CDE) is a serverless service for Cloudera Data Platform that allows you to submit Spark jobs to an auto-scaling cluster.Links to a dbt example project that can be used to bootstrap a user’s dbt journey can be found below.Version Notes:The adapter has been tested on the following version:python: 3.9Cloudera Data Engineering release (1.15-h1)dbt-core: 1.3.0Installing dbt and requirementsPrerequisites: Make sure python3, git, and pip are available on your system.Cloning the repo:Run the following command in your terminal to clone the example project repository:git clone https://github.com/cloudera/dbt-spark-cde-example/Installing the packages:Run the following commands in your terminal to install the required packages/dependencies using pip/pip3:pip install dbt-corepip install dbt-spark-cdeCDE and SparkFor

I want to change my company name from (xxx) to (xxx AA) as there is no option to edit from my account setting.

COD - CDE Spark-HBase tutorial In this article related to CDP Public Cloud, we will walk through steps required to be followed to read/ write to COD (Cloudera Operational Database) from Spark on CDE (Cloudera Data Engineering) using spark-hbase connector.If you are looking to leverage Phoenix instead, please refer to this community article. Assumption COD is already provisioned and database is created. Refer to this link for the same.For this example, we will assume "amallegni-cod" as the database name. CDE is already provisioned and the virtual cluster is already created. Refer to this link for the same. COD Download client configuration Spark in CDE to be able to talk to COD, would require the hbase-site.xml config of the COD cluster. Do the following steps to retrieve the same: Go to the COD control plane UI and click on "amallegni-cod" database. Under the Connect tab of COD database, look for HBase Client Configuration

With the release of HDP 2.6 comes the Hive ACID Merge feature. This is long sought after feature for people who want to batch update table content in ACID manner. This tutorial will walk you through step-by-step how to use this new feature with some really simple dataset. The example is done on HDP 2.6.0.3 installation. First, ACID in Hive need to be turned on in Ambari UIThen we will create 2 tables, one as the target of merge and one as the source of merge. Please note that the target table must be bucketed, set as transaction enabled and stored in orc format.CREATE DATABASE merge_data;CREATE TABLE merge_data.transactions( ID int, TranValue string, last_update_user string)PARTITIONED BY (tran_date string)CLUSTERED BY (ID) into 5 buckets STORED AS ORC TBLPROPERTIES ('transactional'='true');CREATE TABLE merge_data.merge_source( ID int, TranValue string, tran_date string)STORED AS ORC;Then we will populate the target and source table with some data.INSERT INTO merge_data.transactions PA

IntroductionMigration PathsHDP 2.6 to CDPExportImportHDP 3.X to CDPBackupRestoreUseful Links & ScriptsIntroduction Apache Atlas provides open metadata management and governance capabilities for organisations to build a catalog of their data assets, classify and govern these assets and provide collaboration capabilities around these data assets for data scientists, analysts and the data governance team. This article focuses on backup and restore of Atlas data during HDP to CDP migration. Migration Paths HDP 2.6 to CDP ExportNote: Following commands need to be executed on HDP ClusterDownload the migration script to download on to Atlas server host.  ## Download the scriptwget https://archive.cloudera.com/am2cm/hdp2/atlas-migration-exporter-0.8.0.2.6.6.0-332.tar.gz## Untartar zxvf atlas-migration-exporter-0.8.0.2.6.6.0-332.tar.gz##copy the contentsmkdir /usr/hdp/2.6.5.0-292/atlas/tools/migration-exporter/cp atlas-migration-exporter-0.8.0.2.

OverviewThis article focuses on Cloudera’s new adapter dbt-spark-livy that enables users to use dbt on top of Apache Spark by connecting to it using Apache Livy. Livy is a REST web service for managing long-running Spark Contexts and submitting Spark jobs. Instead of running the Spark Contexts in the Server itself, Livy manages Contexts running on the cluster managed by a Resource Manager like YARN.Cloudera Data Platform provides Spark with Livy in the Datahub form factor. Links to a dbt example project that can be used to bootstrap a user’s dbt journey can be found below.Version Notes:The adapter has been tested on the following version:python: 3.9.13Cloudera Manager CM VERSION 7.6.2 RUNTIME VERSION 7.2.15CDP Datahub with the template - 7.2.14 - Data Engineering: Apache Spark2/Spark3dbt-core: 1.3.0Installing dbt and requirementsPrerequisites: Make sure python3, git, and pip are available on your system.Cloning the repo:Run the following command in your terminal to clone the examp

     There are a numerous ways of doing event driven architectures in Cloud with Cloudera Data Platform(CDP) but, lately I was working on a use case which required Spark inferences and transformations on streaming log data in Azure cloud environment on a near realtime basis. The use case is to kickoff Spark transformations as soon as the files get landed in the Azure storage container.                                  The preferred way of doing Spark transformations for Data Engineering use cases in CDP is by leveraging Cloudera Data Engineering(CDE) which runs Spark on Kubernetes. CDE provides much better isolation, job orchestration with help of Airflow, Spark version independence, scales faster, efficiently utilizes the cloud native resources and the best part is that it does dependency management like a rockstar. Just makes your life easy and simple.      &nb

This has been a recurring question in the Community forum. I replied to those a few times so I've decided to post the answer here as an article so that's easier to find and reference. The question is: if you have a ZIP file that contains other ZIP files (which may in turn contain other ZIP files, etc.), how can I extract all the ZIP files, recursively, using NiFi? One way to do that is to implement the following data flow:  The RouteOnAttribute processor checks the file name and if it's a ZIP file it sends it for unpacking. Otherwise sends the non-zip file downstream.  You can find the flow definition attached below. HTH,André