Community Articles

To install Apache NiFi on Macbook, do the following:  Download the Zip file from Apache nifi Downloads. Ensure that you download the latest version zip file Apache NiFi Binary 1.19.1 [OpenPGP] [SHA-256] [SHA-512].  Download it to MBP, and then unzip it: chmod +x bin/nifi.sh Run the following command for User Generate: bin/nifi.sh set-single-user-credentials zzeng <my long password> Run the following command to Start: bin/nifi.sh start bin/nifi.sh status netstat -an | grep LIST Check the  8443 port is LISTEN: tcp4 0 0 127.0.0.1.8443 *.* LISTEN Login from https://localhost:8443/nifi/

Why scale vertically?Once a DL or DH is provisioned with given parameters (instance types, root volume size etc), later there is currently no way in CDP / cloudbreak to change those parameters. However, with time the load may increase and it may not be possible to terminate and re-launch the DL or DH. In this case you might need to scale vertically via Azure console.On the other hand this scaling plays only partially with CDP features, so the long-term fix is rather upgrading to a larger duty size.Overview Both VM and root disk resize steps do require the VM to be in a stopped state. So you can stop the DH or DL and perform both steps.Scaling the virtual machine sizeBased on the azure doc for resizing VMs. After resizing the VM the new size will be in effect until you do a repair or upgrade. After that you will need to do the resize again, if you do not have HA setup.If, however, you have a HA setup, keep in mind to change both VMs: in that way, when the repair changes the pr

Datagen lets you generate data on all services provided by Cloudera (HDFS, Hive, HBase, Ozone, Kafka, Kudu, SolR, Local files), in any format (CSV, JSON, Avro, Parquet, ORC). Users can generate pre-defined data (such as customers or transactions) or create their own data model which supports a wide variety of types. Users can schedule data generation and provide a large set of configuration (such as replication factor, keys etc…).  Datagen is a basic web server exposing APIs that deploys natively as a service in CDP 7.1.7+. Why ?What ?InstallationBasic Data GenerationConfigurationAPICustom Data GenerationResources Why ? There could be many reasons to install and use Datagen, here is a list of some:Test your installationBenchmark your platformReproduce another environment and data (usually a production in a build)Test your code on sample dataTest your code on data at scale before going to productionMake a Proof Of Concept cluster with test dataTest one

OverviewIn this post we will discuss using dbt with the Cloudera Data Platform, and show you how to get started by connecting dbt to your Impala Data Warehouse. You’ll also find links to a dbt example project that you can use to bootstrap your dbt journey.Version Notes:The adapter has been tested on the following version:python: 3.9Cloudera Data Engineering release (1.15-h1)dbt-core: 1.3.0Cloudera Data WarehouseCloudera Data Warehouse (CDW) is a CDP Public Cloud service for self-service creation of independent data warehouses and data marts that autoscale up and down to meet your varying workload demands. The Data Warehouse service provides isolated compute instances for each data warehouse/mart, automatic optimization, and enables you to save costs while meeting SLAs. Both Apache Impala and Apache Hive are available through Cloudera Data Warehouse.What is dbt?dbt is quickly gaining popularity as a key component of the modern data stack; a tool that enables the creation of data pipelin

IntroductionUse of data in Excel spreadsheets is commonplace in the enterprise and often results in widespread yet highly manual and loosely defined dataflows. The manual and loose nature of Excel data flows are at odds with the automated way we expect data to move through the enterprise. As a result, these Excel data flows lack the accuracy, speed and transparency leveraged by automated flows. Shouldn’t Excel data flows be automated too?In this article I show how to use NiFi to automate the reading, writing and validation of Excel contents (using the full capability set humans use) and how to fit this into a larger workflow. I focus here on building an Excel client to operate on Excel data during your workflow. Designing your own Excel workflows depends on how you do things manually today and on your ability to innovate new ways to leverage Excel data in your larger data landscape and initiatives.TechnologiesThe key technology for reading and writing to Excel is Apache’s POI Java

 In this article, we will walk through the steps to integrate Data Engineering Experience & Operational DataBase Experience within a few clicks with an example of reading & writing data into Phoenix tables using PySpark script.  As you know Cloudera Operational Database supports both Traditional relational schema and Evolutionary schema with the capability to transact on petabytes of structured and unstructured data generated by businesses today further details are attached. Whereas Cloudera Data Engineering streamlines and operationalizes data pipelines securely at any scale to increase efficiency and accelerate time to value further details attached.Example: Customer 360 - Cloudera Operational Database will store customer profiling data for business applications and the data is modeled & prepared with Cloudera Data Engineering. Both capabilities come with High Availability, Scalability, Mission Critical Performance, and Fl

OverviewCloudera has implemented dbt adapters for Hive, Spark, and Impala. In addition to providing the adapters, Cloudera is also offering dbt capabilities to our on-prem customers (CDP PvC-Base). Our solution satisfies our customers' stringent security and privacy requirements while providing an easy-to-use turnkey solution for practitioners.In this document, we will show how users can run dbt on YARN without having to worry about anything else. We have packaged everything as a .tar.gz file.If you are new to dbt, refer to the article to learn more about dbt. PrerequisitesThe cluster should have Python 3.7+The user should have the ability to ssh into a gateway machine The user should have access to the git repo with the dbt models and has the ability to create PRs in that git repo with changes. If you are creating your own repo as an analyst, refer to this Getting started with dbt Core The user SSH key can be configured for access to the git repoCompatibility>= CentO

IntroductionCloudera recently announced the open-source dbt adapters for all the engines in Cloudera Data Platform (CDP)—, Apache Impala, and Apache Spark, with added support for Apache Livy and Cloudera Data Engineering.  In addition to providing the adapters, Cloudera is offering a turn-key solution to be able to manage the end-to-end software development life cycle (SDLC) of dbt models. This solution is available on all clouds, as well as on-prem deployments of CDP.  In this article, we show how our customer data teams can streamline their data transformation pipelines in the Cloudera Data Platform and deliver high-quality data that their business can trust. Our solution satisfies the stringent security and privacy requirements of our customers while providing an easy-to-use turnkey solution for practitioners.dbt’s end-to-end software development life cycleA key advantage of using dbt is that it provides a framework for analysts to easily follow software engineer

Installing and configuring Livy on CDH 6.x.x Livy is a preferred way to run Spark jobs on several Hadoop installations, but not on CDH. While preparing for a CDP migration, one of our use-cases switched to Apache Airflow to run jobs without requiring an edge node or "bastion node" and they wanted to begin using Airflow before the CDP migration, so they asked me to install Livy on a CDH edge node.   A search online for Livy on CDH returned little helpful information, but I did find information on how to download and install it at https://livy.apache.org/ Step 1: Determine which account will be used to run Livy Linux security will allow an application to access or execute any program or file the executing account can access unless you configure selinux or another access management software. Pick an account to run Livy from. Step 2: Set up a keytab You'll need a Kerberos principal. If you use Active Directory principals with your CDH deployment, then this account will be outsid

Hi All, When we use MySQL 8 for HUE DB we get the below error  Unexpected error. Unable to verify the database connectionIn Cloudera manager server logs :  2022-11-04 09:18:56,152 INFO scm-web-5325:com.cloudera.enterprise.JavaMelodyFacade: Exiting HTTP Operation: Method:POST, Path:/dbTestConn/checkConnectionResult, Status:2002022-11-04 09:18:56,575 INFO CommandPusher-1:com.cloudera.server.cmf.CommandPusherThread: Acquired lease lock on DbCommand:5472022-11-04 09:18:56,578 INFO CommandPusher-1:com.cloudera.cmf.service.AbstractOneOffHostCommand: Unsuccessful 'HueTestDatabaseConnection'2022-11-04 09:18:56,578 INFO CommandPusher-1:com.cloudera.cmf.service.AbstractDbConnectionTestCommand: Command exited with code: 12022-11-04 09:18:56,578 INFO CommandPusher-1:com.cloudera.cmf.service.AbstractDbConnectionTestCommand: self.ensure_connection()File "/opt/cloudera/parcels/CDH-7.1.6-1.cdh7.1.6.p0.10506313/lib/hue/build/env/lib/python2.7/site-packages/Django-1.11.29-py2.7.

OverviewIn this article I will show 3 different ways of connecting to Cloudera Data Lake using CML and R kernel. The first two approaches use Spark as the primary compute engine, while the last one uses JDBC connection directly. Connect directly to Data Lake through Hive Warehouse Connector (HWC)Connect to Data Lake via Cloudera Data Warehouse (CDW) through HWC Connect to Data Lake via Cloudera Data Warehouse (CDW) through JDBC driver (Note: this method is not recommended for reads larger than 1GB) Prerequisites1. Cloudera Public Cloud Environment2. Data Lake3. Cloudera Machine Learning4. Cloudera Data Warehouse (Hive/Impala)5. Workload Username and Password6. Permission both on environment and datasets7. JDBC Driver   Step-by-Step Preparing CML1. Prepare your workload username and password2. Create project in CML with below configurationInitial Setup: GitGit URL: https://github.com/servaed/cml-r-connection-template.gitRuntime Setup: BasicKernel: R (3

 This article lists issues that may arise while using dbt adapters and their resolution steps.  Issue: Error recreating a dbt model in Impala which was originally created using dbt HiveIssue: Permission issue for table accessIssue: Spark2 read managed tables issue via dbtIssue: SSL certificate failure or Invalid certificate errorIssue: Permission issue while creating tableIssue: Latency while executing dbt commands with dbt-spark-cde adapterIssue: Missing yarn.env file in current working directory Issue: Command '['tar', '-zcf', '/tmp/dbt-workspace.tar.gz', 'xxx']' returned non-zero exit status 2Issue: Troubleshooting Connection with Warehouse Please reach out to us at innovation-feedback@cloudera.com if you see any issues not listed here.Issue: Error recreating a dbt model in Impala which was originally created using dbt HiveDescription: Recreating a dbt model in Impala which was originally created using dbt Hive produces an error similar toMetaException: Cann

Introduction This is part 3 of a series of articles on Data Enrichment with NiFi: Part 1: Data flow enrichment with LookupRecord and SimpleKV Lookup Service is available here Part 2: Data flow enrichment with LookupAttribute and SimpleKV Lookup Service is available here Part 3: Data flow enrichment with LookupRecord and MongoDB Lookup Service is available here Enrichment is a common use case when working on data ingestion or flow management. Enrichment is getting data from external source (database, file, API, etc) to add more details, context or information to data being ingested. In Part 1 and 2 of this series, I showed how to use LookupRecord and LookupAttribute to enrich the content/metadata of a flow file with a Simple Key Value Lookup Service. Using this lookup service helped us implement an enrichment scenario without deploying any external system. This is perfect for scenarios where reference data is not too big and don't evolve too much. However, managing entries in the

This article focuses on demonstrating how to post external user/group mappings in to CM UI using CM API call.   Target state on Cloudera Manger LDAP GroupCM RoleCMGroup1ROLE_ADMINCMGroup2ROLE_CONFIGURATORCMGroup3ROLE_AUDITOR, ROLE_LIMITED Step 1: Get the current authRoles GET /authRoles curl -g -X GET -u admin:admin -H "Content-Type: application/json" "http://cmhost:7180/api/v43/authRoles" > cm_authroles.json Download the existing authroles from the CM UI and validate if the CM roles are mentioned in target state available in the file cm_authroles.json Step 2: Create the user mapping template example shown below  cm_user_mapping.json { "items": [ { "name": "CMGroup1", "type": "LDAP", "authRoles": [ { "name": "ROLE_ADMIN" } ] }, { "name": "CMGroup2", "type": "LDAP", "authRoles": [ { "name": "ROLE_CONFIGURATOR" } ] }, { "name": "CMGroup3", "type": "LDAP",

IntroductionSteps to Backup on HDP3Steps to Import on CDP Introduction This article focuses on backup and restore of Atlas data during HDP3 to CDP migration. Steps to Backup on HDP3Run the following commands on the Atlas server host of HDP3. Command to get the metrics From Atlas API or Atlas UI  curl -k -g -X GET -u admin:admin -H "Content-Type: application/json" -H"Cache-Control: no-cache" "https://atlas_host:21443/api/atlas/admin/metrics" > atlas_metrics.json Extract the entityActive from atlas_metrics.json and make it as a list sample is shown below # cat metics_types.listhive_db_ddlhive_tablehive_dbhbase_namespacehive_processhive_storagedeschdfs_pathhbase_tablehive_column_lineagehbase_column_familyhive_columnhive_process_executionhive_table_ddl​ Export API Script to export all entities and save it as zip file mkdir /tmp/atlas_backupcd /tmp/atlas_backupfor t in `cat metics_types.list`domkdir -p $tcurl -k -X POST -u admin:admi

Symptoms: In versions prior to CDH 6.3.1, Node Managers can enter unhealthy state with below error observed in NM logs 2022-10-20 15:31:32,487 ERROR logaggregation.AggregatedLogFormat (AggregatedLogFormat.java:logErrorMessage(299)) - Error aggregating log file. Log file : /hadoop/ssd01/yarn/log/application_1665989140069_135925/container_e93_1665989140069_135925_01_000002/history.txt.appattempt_1665989140069_135925_000001. /hadoop/ssd01/yarn/log/application_1665989140069_135925/container_e93_1665989140069_135925_01_000002/history.txt.appattempt_1665989140069_135925_000001 (Permission denied)2022-10-20 15:28:19,556 INFO nodemanager.ContainerExecutor (ContainerExecutor.java:logOutput(532)) - Exit code: 352022-10-20 15:28:19,556 INFO nodemanager.ContainerExecutor (ContainerExecutor.java:logOutput(532)) - Exception message: Launch container failed2022-10-20 15:28:19,556 INFO nodemanager.ContainerExecutor (ContainerExecutor.java:logOutput(532)) - Shell error output: Could not cr

The objective of this article is to clearly demonstrate how to use NiFi's REST API to create a template on a secured cluster. I plan to publish multiple articles on using NiFi rest api based on community feedback around the use cases in the field. No pie in the sky type of knowledge share. Adding complexity to the workflows as I publish new articles and how they look via rest api publishing. Use Case Setup: Create template of my process group UVindex via rest api. UVindex resides within a process group hierarchy Root->SunileManjeeProcessGroup->RestAPI Test->UNIndex UNindex is a simply process groups which polls Chiago uv index data every 5 minutes and stores on hdfs. CurrentWeatherData is in the same process group but for this use case I only want to create template for UVIndex process group. Steps Fetch Access Token Fetch Client ID Fetch Process Group attributes Create Snippet Create Template Step 1 - Fetch Access Token Access token is

The intent of this article is cover exactly what happens when a user clicks the "terminate" button on a processor component that has an actively running task.  Before we can discuss the "terminate" option, we need to understand a few basics about the NiFi application and a bit of history:1. NiFi is a java application and the execution of any component (processors, controller service, reporting tasks, funnel, input/output ports, etc) happens within that single Java Virtual Machine (JVM) process. NiFi does not create a child process fro the execution of each component.  2. Since NiFi operates within a single JVM, it is not possible to "kill" a thread for an individual component without killing the entire JVM.3. NiFi consists of well over 400 unique components and many of them are not executing native NiFi code.  Many use client libraries not managed or controlled by NiFi. Others can be configured to execute command external to NiFi (ExecuteStreamCommand, ExecuteP

Do you love Machine Learning? Do you think your code is so sublime that it should serve as a template for the rest of the ML community? Do you like winning prizes? If you answered yes to these questions, come join Cloudera’s Applied ML Prototype Hackathon where you will have free reign to create an ML project that may be integrated into Cloudera Machine Learning (CML) as an Applied ML Prototype. Cloudera’s Applied Machine Learning Prototypes, (AMPs) are fully built end-to-end data science solutions that allow data scientists to go from an idea to a fully working machine learning in a fraction of the time. Accessible with a single click from Cloudera Machine learning or via public github repositories, AMPs provide an end-to-end framework for building, deploying and monitoring business-ready ML applications instantly.   Winning entrants will receive a cash prize, and their projects will be reviewed by Cloudera Fast Forward Labs and added to the AMP

IntroductionWhen users interact with services in CDP Public, services generate audits to detail:Which resource the user is attempting to accessAction the user is attempting to perform (e.g. SELECT)Result of the policy evaluation (i.e. Allowed or Denied)The time the request was madeAdministrators can inspect these audits using the Ranger Web UI. The audits are also archived to object storage as JSON-per-line files which can be easily consumed by downstream applications for monitoring purposes.Audits include a wide range of data points that describe the nature of the request and the result of the policy evaluation (see the Apache wiki for a complete list).In our example, we use Flink to read these files and calculate the number of denied requests made by each user within a single session.Flink exampleThe complete project can be found on GitHub [link].Clone this repository to follow along as we break down the application logic.Data SourceThe Ranger audits are stored in the Data Lake Stora

Apache Ranger is a powerful tool to manage authorization policies across the entire Hadoop ecosystem. It enforces security for services like HDFS, Hive, Hbase, Kafka, etc.. but also for Object Storage like Apache Ozone, Amazon S3 and Azure ADSL. By default, Ranger’s policies are matched against the username or the groups the requesting user belongs to, but there are some situations where users information should be taken from other data sources, like for example Active Directory extended attributes or external RDBMS. Here is where Dynamic Policy Hooks come in handy, allowing Ranger’s administrator to extend the capabilities of the software, enriching the information passed to the authorization engine and making decisions applying a custom logic. Writing a Dynamic Policy Hook is as simple as extending two abstract classes provided by Ranger’s core: RangerAbstractContextEnricher and RangerAbstractConditionEvaluator. Use caseLet’s imagine a use case in which we have a group

In a CDP environment containing both Spark2 and Spark3, Jupyter notebook will use the default path provided in the builds and will refer to spark2 in this case. We tried adding a new build in Jupyter by providing below json format file where Spark3 was copied over to CDH directory, but it did not work cat /data1/python3.6.10/share/jupyter/kernels/pyspark3/kernel.json{ "argv": [ "/data1/python3.6.10/bin/python3.6", "-m", "ipykernel_launcher", "-f", "{connection_file}" ], "display_name": "PySpark3", "language": "python","env":{"JAVA_HOME":"/usr/java/latest","PYSPARK_PYTHON":"/data1/python3.6.10/bin/python3.6","SPARK_HOME":"/opt/cloudera/parcels/CDH/lib/spark3","HADOOP_CONF_DIR":"/opt/cloudera/parcels/CDH/lib/spark3/conf/yarn-conf","SPARK_CONF_DIR":"/opt/cloudera/parcels/CDH/lib/spark3/conf","PYTHONPATH":"/opt/cloudera/parcels/CDH/lib/spark3/python/lib/py4j-0.10.9.2-src.zip:opt/cloudera/parcels/CDH/lib/spark3/python/:","PATH":"$SPARK_HOME/bin:$JAVA_HOME/bin:$PATH" , "PYTHON_STA

To duplicate table and merge partitions from TWO PARTITIONS (date VAR /string VAR) to ONE PARTITION (STRING var):  1 STEP: create new table with just one partition variable:-------CREATE TABLE `caz_bb96_cdnbank.jg4_CONSOL_LOAN_LSE_ACCT_MTH_SNAPSHOT_20220727`CREATE TABLE ${hivevar:CON_DB}.CONSOL_LOAN_LSE_ACCT_MTH_SNAPSHOT2(`acct_num` string,`app_num` int,`host_nm` string,---.....`payment_extension` string,`final_ltv` string,`cbb_used_vehcl_val_rrough_amt` double, `eff_dt` date)PARTITIONED BY (`acct_tp` string) Populate having conditions for PARTITION var FIRST and then for the other PARTITION variable to avoid shifted variables E.g.:set hive.support.quoted.identifiers=none;set hive.exec.dynamic.partition.mode=nonstrict;INSERT OVERWRITE table `caz_bb96_cdnbank.jg4_CONSOL_LOAN_LSE_ACCT_MTH_SNAPSHOT_20220730` partition (acct_tp)selectacct_num,app_num--.....FROM edaf.CONSOL_LOAN_LSE_ACCT_MTH_SNAPSHOT cllWHERE eff_dt IN(SELECT MAX(eff_dt) FROM edaf.CONSOL_LOAN_LSE_ACCT_MTH_SNAPSHOT wh

Let's assume we have a Hive table with the name test and the table is stored under /tmp directory. In the test table, data is stored as below: hdfs dfs -ls -R /tmp/testdrwxr-xr-x - hive hive 0 2022-08-24 09:15 /tmp/test/dir1-rw-r--r-- 3 hive hive 685 2022-08-24 09:15 /tmp/test/dir1/000000_0drwxr-xr-x - hive hive 0 2022-08-24 09:15 /tmp/test/dir2-rw-r--r-- 3 hive hive 685 2022-08-24 09:15 /tmp/test/dir2/000000_0 Generally, the above kind of data will be generated while doing union all operations in Hive. By using spark, if we try to load the hive table data, we will get the following exception: scala> spark.sql("SELECT * FROM test").show()java.io.IOException: Not a file: hdfs://localhost:8020/tmp/test/dir1 at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:340) at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:204) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:273)

Let's understand the information_schema database:   Hive supports the ANSI-standard information_schema database, which we can query for information about tables, views, columns, and our Hive privileges. The information_schema data reveals the state of the system, similar to sys database data, but in a user-friendly, read-only way.   Example:   SELECT * FROM information_schema.tables WHERE is_insertable_into='YES' limit 2;...+--------------------+-------------------+-----------------|tables.table_catalog|tables.table_schema|tables.table_name+--------------------+-------------------+-----------------|default |default |students2|default |default |t3     Now we will try to access the following table under the information_schema database.     spark.sql("select * from information_schema.schemata").show()     We will get the following exception:     org.apache.spark.sql.AnalysisExceptio