Resources

OverviewIn this article I will show 3 different ways of connecting to Cloudera Data Lake using CML and R kernel. The first two approaches use Spark as the primary compute engine, while the last one uses JDBC connection directly. Connect directly to Data Lake through Hive Warehouse Connector (HWC)Connect to Data Lake via Cloudera Data Warehouse (CDW) through HWC Connect to Data Lake via Cloudera Data Warehouse (CDW) through JDBC driver (Note: this method is not recommended for reads larger than 1GB) Prerequisites1. Cloudera Public Cloud Environment2. Data Lake3. Cloudera Machine Learning4. Cloudera Data Warehouse (Hive/Impala)5. Workload Username and Password6. Permission both on environment and datasets7. JDBC Driver   Step-by-Step Preparing CML1. Prepare your workload username and password2. Create project in CML with below configurationInitial Setup: GitGit URL: https://github.com/servaed/cml-r-connection-template.gitRuntime Setup: BasicKernel: R (3

 This article lists issues that may arise while using dbt adapters and their resolution steps.  Issue: Error recreating a dbt model in Impala which was originally created using dbt HiveIssue: Permission issue for table accessIssue: Spark2 read managed tables issue via dbtIssue: SSL certificate failure or Invalid certificate errorIssue: Permission issue while creating tableIssue: Latency while executing dbt commands with dbt-spark-cde adapterIssue: Missing yarn.env file in current working directory Issue: Command '['tar', '-zcf', '/tmp/dbt-workspace.tar.gz', 'xxx']' returned non-zero exit status 2Issue: Troubleshooting Connection with Warehouse Please reach out to us at innovation-feedback@cloudera.com if you see any issues not listed here.Issue: Error recreating a dbt model in Impala which was originally created using dbt HiveDescription: Recreating a dbt model in Impala which was originally created using dbt Hive produces an error similar toMetaException: Cann

Introduction This is part 3 of a series of articles on Data Enrichment with NiFi: Part 1: Data flow enrichment with LookupRecord and SimpleKV Lookup Service is available here Part 2: Data flow enrichment with LookupAttribute and SimpleKV Lookup Service is available here Part 3: Data flow enrichment with LookupRecord and MongoDB Lookup Service is available here Enrichment is a common use case when working on data ingestion or flow management. Enrichment is getting data from external source (database, file, API, etc) to add more details, context or information to data being ingested. In Part 1 and 2 of this series, I showed how to use LookupRecord and LookupAttribute to enrich the content/metadata of a flow file with a Simple Key Value Lookup Service. Using this lookup service helped us implement an enrichment scenario without deploying any external system. This is perfect for scenarios where reference data is not too big and don't evolve too much. However, managing entries in the

This article focuses on demonstrating how to post external user/group mappings in to CM UI using CM API call.   Target state on Cloudera Manger LDAP GroupCM RoleCMGroup1ROLE_ADMINCMGroup2ROLE_CONFIGURATORCMGroup3ROLE_AUDITOR, ROLE_LIMITED Step 1: Get the current authRoles GET /authRoles curl -g -X GET -u admin:admin -H "Content-Type: application/json" "http://cmhost:7180/api/v43/authRoles" > cm_authroles.json Download the existing authroles from the CM UI and validate if the CM roles are mentioned in target state available in the file cm_authroles.json Step 2: Create the user mapping template example shown below  cm_user_mapping.json { "items": [ { "name": "CMGroup1", "type": "LDAP", "authRoles": [ { "name": "ROLE_ADMIN" } ] }, { "name": "CMGroup2", "type": "LDAP", "authRoles": [ { "name": "ROLE_CONFIGURATOR" } ] }, { "name": "CMGroup3", "type": "LDAP",

IntroductionSteps to Backup on HDP3Steps to Import on CDP Introduction This article focuses on backup and restore of Atlas data during HDP3 to CDP migration. Steps to Backup on HDP3Run the following commands on the Atlas server host of HDP3. Command to get the metrics From Atlas API or Atlas UI  curl -k -g -X GET -u admin:admin -H "Content-Type: application/json" -H"Cache-Control: no-cache" "https://atlas_host:21443/api/atlas/admin/metrics" > atlas_metrics.json Extract the entityActive from atlas_metrics.json and make it as a list sample is shown below # cat metics_types.listhive_db_ddlhive_tablehive_dbhbase_namespacehive_processhive_storagedeschdfs_pathhbase_tablehive_column_lineagehbase_column_familyhive_columnhive_process_executionhive_table_ddl​ Export API Script to export all entities and save it as zip file mkdir /tmp/atlas_backupcd /tmp/atlas_backupfor t in `cat metics_types.list`domkdir -p $tcurl -k -X POST -u admin:admi

Symptoms: In versions prior to CDH 6.3.1, Node Managers can enter unhealthy state with below error observed in NM logs 2022-10-20 15:31:32,487 ERROR logaggregation.AggregatedLogFormat (AggregatedLogFormat.java:logErrorMessage(299)) - Error aggregating log file. Log file : /hadoop/ssd01/yarn/log/application_1665989140069_135925/container_e93_1665989140069_135925_01_000002/history.txt.appattempt_1665989140069_135925_000001. /hadoop/ssd01/yarn/log/application_1665989140069_135925/container_e93_1665989140069_135925_01_000002/history.txt.appattempt_1665989140069_135925_000001 (Permission denied)2022-10-20 15:28:19,556 INFO nodemanager.ContainerExecutor (ContainerExecutor.java:logOutput(532)) - Exit code: 352022-10-20 15:28:19,556 INFO nodemanager.ContainerExecutor (ContainerExecutor.java:logOutput(532)) - Exception message: Launch container failed2022-10-20 15:28:19,556 INFO nodemanager.ContainerExecutor (ContainerExecutor.java:logOutput(532)) - Shell error output: Could not cr

The objective of this article is to clearly demonstrate how to use NiFi's REST API to create a template on a secured cluster. I plan to publish multiple articles on using NiFi rest api based on community feedback around the use cases in the field. No pie in the sky type of knowledge share. Adding complexity to the workflows as I publish new articles and how they look via rest api publishing. Use Case Setup: Create template of my process group UVindex via rest api. UVindex resides within a process group hierarchy Root->SunileManjeeProcessGroup->RestAPI Test->UNIndex UNindex is a simply process groups which polls Chiago uv index data every 5 minutes and stores on hdfs. CurrentWeatherData is in the same process group but for this use case I only want to create template for UVIndex process group. Steps Fetch Access Token Fetch Client ID Fetch Process Group attributes Create Snippet Create Template Step 1 - Fetch Access Token Access token is

The intent of this article is cover exactly what happens when a user clicks the "terminate" button on a processor component that has an actively running task.  Before we can discuss the "terminate" option, we need to understand a few basics about the NiFi application and a bit of history:1. NiFi is a java application and the execution of any component (processors, controller service, reporting tasks, funnel, input/output ports, etc) happens within that single Java Virtual Machine (JVM) process. NiFi does not create a child process fro the execution of each component.  2. Since NiFi operates within a single JVM, it is not possible to "kill" a thread for an individual component without killing the entire JVM.3. NiFi consists of well over 400 unique components and many of them are not executing native NiFi code.  Many use client libraries not managed or controlled by NiFi. Others can be configured to execute command external to NiFi (ExecuteStreamCommand, ExecuteP

Do you love Machine Learning? Do you think your code is so sublime that it should serve as a template for the rest of the ML community? Do you like winning prizes? If you answered yes to these questions, come join Cloudera’s Applied ML Prototype Hackathon where you will have free reign to create an ML project that may be integrated into Cloudera Machine Learning (CML) as an Applied ML Prototype. Cloudera’s Applied Machine Learning Prototypes, (AMPs) are fully built end-to-end data science solutions that allow data scientists to go from an idea to a fully working machine learning in a fraction of the time. Accessible with a single click from Cloudera Machine learning or via public github repositories, AMPs provide an end-to-end framework for building, deploying and monitoring business-ready ML applications instantly.   Winning entrants will receive a cash prize, and their projects will be reviewed by Cloudera Fast Forward Labs and added to the AMP

IntroductionWhen users interact with services in CDP Public, services generate audits to detail:Which resource the user is attempting to accessAction the user is attempting to perform (e.g. SELECT)Result of the policy evaluation (i.e. Allowed or Denied)The time the request was madeAdministrators can inspect these audits using the Ranger Web UI. The audits are also archived to object storage as JSON-per-line files which can be easily consumed by downstream applications for monitoring purposes.Audits include a wide range of data points that describe the nature of the request and the result of the policy evaluation (see the Apache wiki for a complete list).In our example, we use Flink to read these files and calculate the number of denied requests made by each user within a single session.Flink exampleThe complete project can be found on GitHub [link].Clone this repository to follow along as we break down the application logic.Data SourceThe Ranger audits are stored in the Data Lake Stora

Apache Ranger is a powerful tool to manage authorization policies across the entire Hadoop ecosystem. It enforces security for services like HDFS, Hive, Hbase, Kafka, etc.. but also for Object Storage like Apache Ozone, Amazon S3 and Azure ADSL. By default, Ranger’s policies are matched against the username or the groups the requesting user belongs to, but there are some situations where users information should be taken from other data sources, like for example Active Directory extended attributes or external RDBMS. Here is where Dynamic Policy Hooks come in handy, allowing Ranger’s administrator to extend the capabilities of the software, enriching the information passed to the authorization engine and making decisions applying a custom logic. Writing a Dynamic Policy Hook is as simple as extending two abstract classes provided by Ranger’s core: RangerAbstractContextEnricher and RangerAbstractConditionEvaluator. Use caseLet’s imagine a use case in which we have a group

In a CDP environment containing both Spark2 and Spark3, Jupyter notebook will use the default path provided in the builds and will refer to spark2 in this case. We tried adding a new build in Jupyter by providing below json format file where Spark3 was copied over to CDH directory, but it did not work cat /data1/python3.6.10/share/jupyter/kernels/pyspark3/kernel.json{ "argv": [ "/data1/python3.6.10/bin/python3.6", "-m", "ipykernel_launcher", "-f", "{connection_file}" ], "display_name": "PySpark3", "language": "python","env":{"JAVA_HOME":"/usr/java/latest","PYSPARK_PYTHON":"/data1/python3.6.10/bin/python3.6","SPARK_HOME":"/opt/cloudera/parcels/CDH/lib/spark3","HADOOP_CONF_DIR":"/opt/cloudera/parcels/CDH/lib/spark3/conf/yarn-conf","SPARK_CONF_DIR":"/opt/cloudera/parcels/CDH/lib/spark3/conf","PYTHONPATH":"/opt/cloudera/parcels/CDH/lib/spark3/python/lib/py4j-0.10.9.2-src.zip:opt/cloudera/parcels/CDH/lib/spark3/python/:","PATH":"$SPARK_HOME/bin:$JAVA_HOME/bin:$PATH" , "PYTHON_STA

To duplicate table and merge partitions from TWO PARTITIONS (date VAR /string VAR) to ONE PARTITION (STRING var):  1 STEP: create new table with just one partition variable:-------CREATE TABLE `caz_bb96_cdnbank.jg4_CONSOL_LOAN_LSE_ACCT_MTH_SNAPSHOT_20220727`CREATE TABLE ${hivevar:CON_DB}.CONSOL_LOAN_LSE_ACCT_MTH_SNAPSHOT2(`acct_num` string,`app_num` int,`host_nm` string,---.....`payment_extension` string,`final_ltv` string,`cbb_used_vehcl_val_rrough_amt` double, `eff_dt` date)PARTITIONED BY (`acct_tp` string) Populate having conditions for PARTITION var FIRST and then for the other PARTITION variable to avoid shifted variables E.g.:set hive.support.quoted.identifiers=none;set hive.exec.dynamic.partition.mode=nonstrict;INSERT OVERWRITE table `caz_bb96_cdnbank.jg4_CONSOL_LOAN_LSE_ACCT_MTH_SNAPSHOT_20220730` partition (acct_tp)selectacct_num,app_num--.....FROM edaf.CONSOL_LOAN_LSE_ACCT_MTH_SNAPSHOT cllWHERE eff_dt IN(SELECT MAX(eff_dt) FROM edaf.CONSOL_LOAN_LSE_ACCT_MTH_SNAPSHOT wh

Let's assume we have a Hive table with the name test and the table is stored under /tmp directory. In the test table, data is stored as below: hdfs dfs -ls -R /tmp/testdrwxr-xr-x - hive hive 0 2022-08-24 09:15 /tmp/test/dir1-rw-r--r-- 3 hive hive 685 2022-08-24 09:15 /tmp/test/dir1/000000_0drwxr-xr-x - hive hive 0 2022-08-24 09:15 /tmp/test/dir2-rw-r--r-- 3 hive hive 685 2022-08-24 09:15 /tmp/test/dir2/000000_0 Generally, the above kind of data will be generated while doing union all operations in Hive. By using spark, if we try to load the hive table data, we will get the following exception: scala> spark.sql("SELECT * FROM test").show()java.io.IOException: Not a file: hdfs://localhost:8020/tmp/test/dir1 at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:340) at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:204) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:273)

Let's understand the information_schema database:   Hive supports the ANSI-standard information_schema database, which we can query for information about tables, views, columns, and our Hive privileges. The information_schema data reveals the state of the system, similar to sys database data, but in a user-friendly, read-only way.   Example:   SELECT * FROM information_schema.tables WHERE is_insertable_into='YES' limit 2;...+--------------------+-------------------+-----------------|tables.table_catalog|tables.table_schema|tables.table_name+--------------------+-------------------+-----------------|default |default |students2|default |default |t3     Now we will try to access the following table under the information_schema database.     spark.sql("select * from information_schema.schemata").show()     We will get the following exception:     org.apache.spark.sql.AnalysisExceptio

Introduction Cloudera Data Platform Base doesn't have one Quickstart/Sandbox VM like the ones for CDH/HDP releases that helped a lot of people (including me), to learn more about the open-source components and also see the improvements from the community in CDP Runtime.  The objective of this tutorial is to enable and create a VM from scratch via some automation (Shell Script and Cloudera Template) that can help whoever wants to use and/or learn Cloudera CDP in a Sandbox/Quickstart like environment in your machine. Pre-Requisites This exercise is performed on a Mac OS but you can install Vagrant/Virtualbox on Windows/Linux machines (https://www.vagrantup.com/docs/installation).The versions below were tested at the moment of writing this blog and may change in the future. The machine needs to have at least:80 GB of free disk space;12 GB free RAM;8 free VCPU;Good internet connection;Install Virtualbox and Vagrant These are the software that we'll use to

For this article let’s take the supposition that we have a MDM Software handling the enterprise organization.When we add/change an item in a MDM, it goes throw a workflow (validation, dedup, etc…).For synchronizing in real-time to Ranger “Users and Policies Management” the last action on the workflow must be publishing to a broker the information, that can be a JMS based broker(AMQ, RabbitMQ etc ..) or Kafka.Putting the message into a broker gives you the safety of never losing the event.We use HDF to consume, built the payload and call Ranger API.List of hidden Ranger useful API's.To invoke the API'scurl -u admin:admin -v -i -s -X POST -H "Accept: application/json" -H "Content-Type: application/json" http://<IP>:6080/service/xusers/secure/usersUpdate a userhttp://<IP>:6080/service/xusers/secure/groupsUpdate a grouphttp://<IP>:6080/service/xusers/groups?name=haGet particular group details

IntroductionSpark doesn't supply a mechanism to have data pushed to it - instead, it wants to pull data from other sources. In NiFi, this data can be exposed in such a way that a receiver can pull from it by adding an Output Port to the root process group. For Spark, we will use this same mechanism - we will use the Site-to-Site protocol to pull data from NiFi's Output Ports.Prerequisite1) Assuming you already have latest version of NiFi-0.4.1/HDF-1.1.1 downloaded on your HW Sandbox, else execute below after ssh connectivity to sandbox is established:# cd /opt/# wget http://public-repo-1.hortonworks.com/HDF/1.1.1.0/nifi-1.1.1.0-12-bin.tar.gz# tar -xvf nifi-1.1.1.0-12-bin.tar.gz2) Download Compatible version [in our case 0.4.1] of "nifi-spark-receiver" and "nifi-site-to-site-client" to Sandbox in a specific location:# mkdir /opt/spark-receiver# cd /opt/spark-receiver# wget http://central.maven.org/maven2/org/apache/nifi/nifi-site-to-site-client/0.4.1/nifi-site-to-site-client-0.4.1.jar

After upgrading from CDH 6.1.1 to CDP 7.1.7, we saw that spark was handling parquet files with older timestamps in non-UTC cluster incorrectly. Symptoms: In older saved parquet files via CDH, when we were reading these files in CDP, it was changing the timestamps for older dates and adding below property in when checking via parquet tools hadoop jar /opt/cloudera/parcels/CDH/jars/parquet-tools-1.10.99.7.1.7.0-551.jar meta  extra: writer.date.proleptic = falseextra: writer.time.zone = Australia/Sydney For us, 1985-02-01 onwards, the date was not changed and appeared normal for all outputs. Another observation was when this parquet data was read by pyspark it was normal but it did not return correct values when running spark-submit. Explanation:  There are two types of parquet readers and writers in CDP. One is Hive based and other are Spark native. Hive-supplied parquet readers and writers

In case customer wants to install CDV on top of their CDSW environment but they have an air gapped cluster and they are not able to connect to internet to install CDV, we can use below steps to install it. 1. If customer has some Dev environment with access to internet, they can enable Cloudera data visualisation there using below guide. https://docs.cloudera.com/data-visualization/7/start-cdsw/topics/viz-add-engine-cdsw-runtime.htmlIn case there is no internet access at all for Dev, Prod cluster of CDSW, Cloudera can be contacted and they can share the latest image for your current version. 2. If you have Dev. CDSW environment with CDV running, please identify the name of image from Engine Image section in CDV and check the running images by running below command in CDSW node.$ docker images 3. We need to save the full image (not the runtime image) to a docker file so that it can be copied over to the destination CDSW cluster.sudo docker save -o <cdv_image_name&

Table of ContentsUse CaseDesignImplementationPrerequisitesStep #1 - Setup NiFi FlowStep #2 - Setup PySpark JobStep #3 - Identify impacted members in Hue editorStep #4 - View Hive tables in AtlasUse CaseAs a healthcare provider / public health official, I want to respond equitably to the COVID-19 pandemic as quickly as possible, and serve all the communities that are adversely impacted in the state of California.I want to use health equity data reported by California Department of Public Health (CDPH) to identify impacted members and accelerate the launch of outreach programs.DesignCollect - Ingest data from https://data.chhs.ca.gov/dataset/covid-19-equity-metrics using NiFi.Enrich - Transform the dataset using Spark and load Hive tables.Report - Gather insights using Hue Editor.  ImplementationPrerequisitesAn existing CDP Private Cloud environment with following services - Hadoop Distributed File System (HDFS), Atlas, NiFi, Spark, Hive on Te

Table of ContentsUse CaseDesignImplementationPrerequisitesStep #1 - Setup NiFi FlowStep #2 - Setup Cloudera DataFlow (CDF)Step #3 - Setup Cloudera Data Engineering (CDE)Step #4 - Setup Cloudera Data Warehouse (CDW)Step #5 - Setup Cloudera Cloudera Data Visualization (Data VIZ) DashboardStep #6 - Identify impacted members in Hue editorStep #7 - View Hive tables in Cloudera Data CatalogStep #8 - Setup Cloudera AIUse CaseAs a healthcare provider / public health official, I want to respond equitably to the COVID-19 pandemic as quickly as possible, and serve all the communities that are adversely impacted in the state of California.I want to use health equity data reported by California Department of Public Health (CDPH) to identify impacted members and accelerate the launch of outreach programs.DesignCollect - Ingest data from https://data.chhs.ca.gov/dataset/covid-19-equity-metrics using NiFi.Enrich - Transform the dataset using Spark and load Hive tables.Rep

If you made it this far, you probably already know why you're here.  Connecting Dbeaver to an Impala virtual warehouse isn't difficult, but there are a few gotchas that can make it frustrating. So let's conquer those obstacles. Provision an Impala Virtual WarehouseLog into a CDP instanceNavigate to Data WarehouseEnable your Data Warehouse Environment & Database CatalogCreate a New Virtual WarehouseGive your virtual warehouse a unique nameSelect ImpalaSelect your database catalog from the dropdownSSO may be enabled or disabled, the choice is yours!Set Availability Zone and User Groups per your requirements or leave as-isPick the size (aka decide how much money you want to spend)The remaining options are going to depend on your needs, but the defaults are fine for our purposes.Click Create.  Expect approximately 5 minutes to create your virtual warehouse.  Download the JDBC DriverFrom your virtual warehouse tile, click on the kebab icon in

Table of ContentsOverviewDesignImplementationPrerequisitesStep #1 - Cloudera Data FlowStep #2 - Cloudera Data Warehouse (CDW)Step #3 - Cloudera Data VisualizationOverviewIn this article, let's build a real-time data visualization to analyze Twitter feeds using Cloudera Data Platform.Design  Explanation:NiFi Flow invokes Twitter API v2 (every 15 seconds), and stages all tweets in an AWS S3 bucket.Hive External Tables points to the staging location (i.e. AWS S3 bucket).Cloudera Data Visualization uses Hive External Tables as its data source, to create visuals. All visuals are refreshed every 20 seconds.ImplementationPrerequisitesA Cloudera Public Cloud environment on Amazon Web Services (AWS). If you don't have an existing environment, follow instructions here to set one up - CDP/AWS Quick Start Guide.An app in Twitter's Developer Portal. This is needed to call Twitter API v2. If this is your first time using Twitter API v2, follow these instructions - Step-by-step guide t

OverviewCloudera has implemented dbt adapters for Hive, Spark (CDE and Livy), and Impala. In addition to providing the adapters, Cloudera is offering a turn-key solution to be able to manage the end-to-end software development life cycle (SDLC) of dbt models. This solution is available in all clouds as well as on-prem deployments of CDP. It is useful for customers who would not like to use dbt Cloud for security reasons or the lack of availability of adapters in dbt cloud. We have identified the following requirements for any solution that supports the end-to-end SDLC of data transformation pipelines using dbt. Have multiple environmentsDevStage/TestProdHave a dev setup where different users can do the following in an isolated way:Make changes to modelsTest changesSee logs of testsUpdate docs in the models and see docsHave a CI/CD pipeline to push committed changes in the git repo to stage/prod environmentsSee logs in stage/prod of the dbt runsSee dbt docs in stage/prodO