Community Articles

 [FLaNK] Smart Weather Applications with Flink SQL    Sometimes you want to acquire, route, transform, live query, and analyze all the weather data in the United States while those reports happen. With FLaNK, it's a trivial process to do.   From Kafka to Kudu for Any Schema of Any Type of Data - No Code, Two Steps The Schema Registry has full Swagger-ized Runnable REST API Documentation. Integrate, DevOps, and Migration in a simple script.  Here's your schemas, upload, edit, and compare. Validating Data Against a Schema With Your Approved Level of Tolerance. You want extra fields allowed, you got it: Feed that data to beautiful visual applications running in Cloudera Machine Learning. You like drill-down maps, you got them: Query your data fast with Apache Hue against Apache Kudu tables through Apache Impala: Let's ingest all the US weather stations even though they are a zipped directory of a ton of XML files: Weather Ingest is Easy

 Recently I ran into a scenario requiring to connect my Spark Intellij IDE to Kafka DataHub.   I'm not going to claim the status of a pro at IDE secure setup.  Therefore for novices in the security realm alike, they may find this article useful This article will go through steps setting up an Spark Scala IDE (Intellij)  (with a supplied working code example) to connect securely to a Kafka DataHub over SASL_SSL protocol using PLAIN SASL mechanism.Artifactshttps://github.com/sunileman/spark-kafka-streamingScala Objecthttps://github.com/sunileman/spark-kafka-streaming/blob/master/src/main/scala/KafkaSecureStreamSimpleLocalExample.scalaThe scala object accepts 2 inputsTarget Kafka topicKafka broker(s)Prequequites Kafka DataHub InstancesPermission setup on Ranger to be able to read/write from KafkaIntellij (or similar) with the Scala plugin installedWorkload username and passwordTrustStoreAndre Sousa Dantas De Araujo did a great job explaining (very simply

Objective:- Atlas, by default comes with certain types for Hive, Storm, Falcon etc. However, there might be cases where you would like to capture some custom metadata in Atlas. This can be metadata related to ETL processes, enterprise-operations etc. The article explains how to create custom Atlas types and provides some insight on establishing lineage between said types. Use Case:- Consider a simple Use Case where Raw Textual data is analyzed via a ML process and the results are stored in HDFS. For instance, the raw source data is a dump of access logs on professors and research assistants referring research papers. The ML process would try to come up with recommendations on research papers for further reading for these end users. To capture metadata and lineage for this workflow, we would want to have three custom types in Atlas.a.) ResearchPaperAccessDataset: To capture the metadata for the input dataset. b.) ResearchPaperRecommendationResults: To capture the metadata for t

As explained in this Parent article, here is the first step in our hybrid cloud implementation: replicating bank branches and employee data from an on-prem cluster to CDP Public Cloud.PrerequisitesA CDP Base Cluster (with admin rights and user part of HDFS supergroup)A CDP Public Cloud environment (with admin rights)CDW virtual warehouse attached to Public Cloud environmentNote: you can find the datasets used for this exercise hereStep 1: Register CDP Base as classic clusterStart registrationIn your CDP Public Cloud management console, go to Classic Clusters > Add Cluster > CDH and enter your CDP Base cluster information:You will then see your cluster registration in progress:Install a secure communication tunnelClick on the Files button in Install files and follow the instructions:The following are some example instructions to do this on your CM node.Download from your management console the ssh_tunnel_setup_files.zip archive:Copy it to your CM node  $ s

A continuation article from my IaaS Hadoop performance testing. My previous performance test was on BigStep.ObjectiveTest 1 Terabyte of data using the Tera Suite (TeraGen, TeraSort, and TeraValidate) on similar hardware profiles using core baseline settings across multiple IaaS providing and Hadoop as a Service offerings. Here we will capture EMR performance statistics using EMRFS(s3), an object storage.AWS EMRThe natural next step is to test the Tera suite on AWS EMR which is Amazon's Hadoop as a Service offering. I used EMR with "EMRFS which is an implementation of HDFS which allows EMR clusters to store data on Amazon S3". Object storage with Hadoop has not traditionally performed well. I was very interested in testing the new EMRFS. EMRFS/S3 was chosen as the storage device for this test due to fact much of the s3's (with EMR) allure is around EMR's ability to store and process data directly off s3. Using EMR's local storage (not s3) mayincrease performance. HardwareInstan

I'm back with a new article series like I did previously with news author personality recognition, beast mode quotient, and AI to edge (though this one got recently replaced with identifying Magic: The Gathering cards). In this series, I will showcase how to harness the true power of Cloudera Hybrid cloud capabilities. Throughout the series you will learn how to use CDP Private Cloud Base, Replication Manager, CDP Public Cloud, Nifi, Kafka on data hub, Cloudera Data Warehouse, and Cloudera Viz.Reminder: CDP VisionCDP is designed to seamlessly enable you to deploy any data workloads (data collection, streaming, enrichment, engineering, serving, and AI/ML), on any infrastructure, with the latest engines while maintaining a coherent layer of security and governance (SDX). Case Study: Worldwide BankFor the purpose of this article, I will use an example of a fake bank (Worldwide Bank). Worldwide Bank is a large international bank that leverages a traditional big data architec

A great way to connect Power BI desktop to CDP Public Cloud is using the Cloudera Data Warehouse Experience with Hive LLAP.PrerequisitesCDP Public Cloud environment with Data Warehouse Experience using either Hive LLAP or ImpalaUser will need a CDP login and will need to set up their Workload PasswordFrom their CDP login, they will need the Workload User Name and Workload Password NoteBoth Impala and Hive LLAP Virtual Warehouses will work with the Hive LLAP driver that is included with Power BI.  The steps below will work for both.  It is important to include port 443 when entering the server string in Step 6.Steps to connect Power BI to CDW with Hive LLAPFrom within the DW experience in the CDP Management console, select the Virtual Warehouse you want to use and click the  and select Copy JDBC URL:  From that string, you will only need the bold portion such as the following:jdbc:hive2://hs2-azure-hive-1.env-5iskrk.dwx.cloudera.site/default;trans

Parsing Web Pages for Images with Apache NiFiThis could be used to build a web crawler that downloads images. I am downloading awesome images from Pixabay! URL: https://pixabay.com/en/photos/?image_type=&cat=&min_width=&min_height=&q=data+science&order=popularI wanted to be able to grab every image from a page, I have some web sites I want to backup my images from. So I added a processor that uses JSoup to do.Once you download the NAR from github and deploy to your /usr/hdf/current/nifi/lib directories and restart Apache NiFi you will have a new processor. It is ImageProcessor listed version 1.6.0.You can examine and test the Java source code if you wish.Here is an example flow of grabbing all the images from a pixabay URL then filtering out the empty images. Then we split into individual image URLs. We pull out that tag and then download those images. If they are not blank or small I route to TensorFlow to run some inception on it. I extract image

Note: This article is all thanks to @sprakash for diving into the Ranger source code and explaining to me how to do this!   When developing new plugins for Ranger, these plugins need to be able to download the corresponding Ranger policies. In a secure Ranger setup (which is the preferred way), you download them via the secure download API endpoint: /plugins/secure/policies/download/{serviceName}   This API requires authentication. In this article, I will highlight how to set up a read-only user to download a policy.   Step 1: Create a read-only user  As a Ranger administrator, go to Settings > Users/Groups/Roles:   Then, click on Add New User and create a user with the auditor role, as follows :   Step 2: Give download access to this user In the Ranger home screen, edit the service you want to download (here Hadoop SQL, or it's technical name cm_hive):   Add the recently created user to the policy.download.auth.users configurati

This article is inspired by another article Spark Structured Streaming example with CDE, which talks about how to use CDE to read/write from Kafka using Spark Streaming. The difference in this article is that this article talks about how to connect to "Kafka" of the CDP Streams Messaging data hub cluster.   The following are the differences in CDP Streams Messaging data hub cluster:  Kafka of CDP datahub is SASL_SSL enabled. CDE needs the CA certificate of Kafka to connect. Prerequisites Knowledge of CDP, Nifi, Kafka, and CDE CDE cluster provisioned from CDP Environment CDP Flow Management Datahub Cluster (NiFi)  CDP Streams Messaging Datahub Cluster (Kafka, Schema Registry) CDE CLI The assumption is that the CDP environment, Data lake, Flow Management, and Streaming Messaging datahub cluster are already created. Flow Management Datahub Cluster Once the above clusters are provisioned, setup NiFi to read tweets from twitter and publish it to Kafka. Use thi

NiFi is most effectively used as an "always-on" system, meaning that the data flows are often always operational (running). Doing batch processing is a more difficult task and usually requires some user intervention (such as stopping a source processor). For relational databases (RDBMS), a common use case is to migrate, replicate, or otherwise move the data from the source RDBMS to some target. If ExecuteSQL were used to get all data from a table (for example), then the processor will execute the query each time it is triggered to run, and will return whatever results correspond to the query. If the goal is to simply move all rows to some target, then ExecuteSQL could be started then immediately stopped, such that it would only execute once, and all results will be sent down the flow. However a more common use case is that the source database is being updated from some external process (user, webapp, ERP/CRM/etc. system). In order to get the new rows, the table needs to be queried agai

Issue:In a heavily utilized kafka cluster AMS will keep on crashing with error "ERROR org.apache.hadoop.hbase.client.AsyncProcess: Cannot get replica 0 location for {"totalColumns":5,"row":"kafka.server.FetcherLagMetrics."Solution:1. Run the following command to gather the amount of metrics being collected:curl http://<Ambari-metrics-collector-host>:6188/ws/v1/timeline/metrics/metadata2. From Ambari UI -> Kafka -> Configs -> Flitersearch for: "external.kafka.metrics.exclude.prefix"3. Add the following at the end:kafka.log.Log4. Restart Kafka.This will exclude additional metrics from getting captured and will increase the stability of the AMS.

Cloudera Data Platform Public Cloud recently introduced the ability to backup and restore datalake from a saved location. Specifically, the backup operation saves a full snapshot of data from all SDX services: Atlas: Audit events, saved in HBase tables Lineage data, saved as Janus graph data in HBase tables Edge, vertex, and full text indexes, saved in Solr collections Ranger: Audit logs, saved as a Solr collection Permissions and tags, saved in RDBMS tables HMS Metadata, saved in RDBMS tables   In this article, I will detail how to run backup and restore in CDP Public Cloud in AWS, via the CDP CLI. Pre-Requisites Stop operations that could affect backup Make sure that no HMS affecting operations are running (e.g. creating a table from CDW or a datahub) Go to your Datalake Cloudera Manager, and shut down: Atlas Ranger HMS Make sure you have the proper IAM permissions Datalake backup uses both the Ranger Audit Role and Datalake Admin Roles to write the backups (more det

While trying to perform an import/export data from/to MS Parallel Data Warehouse following error is observed Sqoop Command Output: sqoop export --connect "jdbc:sqlserver://<DB_URL>:<PORT>;database=<DB_NAME>; --driver com.microsoft.sqlserver.jdbc.SQLServerDriver --username <USERNAME> --password ***--table "<TB_NAME>" --input-fields-terminated-by ',' --export-dir <EXPORT_DIR> -m 1 /usr/hdp/2.5.3.0-37//sqoop/conf/sqoop-env.sh: line 23: HADOOP_CLASSPATH=${hcat -classpath}: bad substitution Warning: /usr/hdp/2.5.3.0-37/accumulo does not exist! Accumulo imports will fail. Please set $ACCUMULO_HOME to the root of your Accumulo installation. SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/usr/hdp/2.5.3.0-37/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/hdp/2.5.3.0-37/hive/lib/phoenix-client.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See htt

How to access your secured NiFi instance or cluster:So you have secured your NiFi instance (meaning you have configured it for https access) and now you are trying to access the https web UI. Once NiFi is secured, any entity interacting with the UI will need to successfully authenticate and then be authorized to access the particular NiFi resource(s). As of HDF 2.1.1 or Apache NiFi 1.1.0, NiFi supports authentication via user certificates (default - always enabled), Kerberos/Spnego, or username and password based authentication via LDAP, LDAPS, or kerberos. The intent of this article is not to cover the authentication process, but rather to cover the initial admin authorization process. We assume for this article that authentication is successful. How do you know? A quick look in the nifi-user.log will tell you if your users authentication was successful.Following successful authentication comes NiFi Authorization. NiFi authorization can be handled by NiFi's default built in file ba

Hey Everyone! In this video, we’ll go over how to use Cloudera AI to validate a complex predictive model. Using a publicly available NASA dataset that simulates how jet engines degrade over time, we’ll use machine learning concepts in a cloud environment to go from simulation data to a cost benefit analysis in just a few steps. We’ll also show how we can run experiments to track specific metrics from many different scenarios that our predictive model could possibly be implemented in.     More information:

 Link to Related Tutorial: https://www.cloudera.com/tutorials/cml-validate-jet-engine-predictive-models.html Link to Related Blog: https://blog.cloudera.com/using-cml-to-build-a-predictive-maintenance-model-for-jet-engines/ Link to Related Meetup: https://www.meetup.com/futureofdata-austin/events/273929240/ Cloudera User's Page www.cloudera.com/users

 


The CDP Upgrade Advisor is a utility available on MyCLoudera for Cloudera customers. This tool performs an evaluation of diagnostic data to determine the CDP readiness of your CDH or HDP cluster environment.   Running the upgrade advisor against the cluster in question is one of your first steps to adopting CDP, followed by an in-depth conversation with your Cloudera account team to review the specific results. This utility raises awareness of clusters that may present risks during an upgrade to CDP due to, for example, an unsupported of the operating system currently in use.   The upgrade advisor utility is focused on the environment and platform in use but doesn’t take into consideration use-cases, the actual cluster data, or workflows in use. Analysis of these critical areas occurs as part of your CDP Journey Workshop with your Cloudera account team and Professional Services.   To run the Upgrade Advisor: Click Upgrade Path once logged in to https://my.clo

In the CDP DC cluster, once you use the AMToCM migration script to migrate all configurations, you will not be able to fetch a yarn application logs from the command line when using the following command: $yarn logs -applicationId application_160120642WARNING: YARN_OPTS has been replaced by HADOOP_OPTS. Using value of YARN_OPTS.No class configured for IndexedFormatCan not find any log file matching the pattern: [ALL] for the application: application_1601206427676_0011Can not find the logs for the application: application_1601206427676_0011 with the appOwner: hive This issue occurs because the yarn.log-aggregation.file-formats gets updated with the IndexedFormat configuration during the AMtoCM migration.   To resolve this issue, change it to the ifile and Tfile.  Once you change the configuration as mentioned in the above image, save and restart the yarn service, you will able to fetch the yarn logs using the command line.    Reference Log aggregation propert

Spark is notoriously knobby when it comes to tuning applications and requesting resources. While this added freedom gives us an incredible amount of control and flexibility when running our applications, it also gives us an opportunity to make a lot of mistakes. This guide is for Spark 1.5 and earlier. Spark 1.6 introduces a new setting that allows for a less rigid mapping of the memory areas.There are a lot of knobs in spark to tune, knowing which ones do what will allow you to maximize your memory usage, and understand why you’re doing what you're doing.Most people think when requesting 10g for a worker with Spark, your application has 10g of memory for caching, but its actually quite more complex than that. A Spark executor has several memory areas allocated, and knowing what these areas are for will help understand the best allocation for resources.An executor has two regions, room for caching, and room for aggregations pre-shuffle. Depending on the workload, we can optimize t

By default, Cloudera Machine Learning (CML) ships Jupyter kernel as part of the base engine images. Data Scientists often prefer to use a specialized custom kernel in Jupyter that makes their work more efficient. In this community post, we will walk through how to customize a Docker container image with a sparkmagic Jupyter kernel and how to deploy it to a CML workspace.  Prerequisites: Admin privileges in a CML workspace Local Docker client with access to Docker Hub or internal Docker registry   Step 1. Choose a custom Jupyter kernel. Jupyter kernels are purpose-built add-ons to the basic Python notebook editor. For this tutorial, I chose sparkmagic as the kernel that provides convenient features for working with Spark, like keeping SQL syntax clean in a cell. Sparkmagic relies on Livy to communicate with the Spark cluster. As of this writing, Livy is not supported in CML when running Spark on Kubernetes. However, your classic Spark cluster (for example on Data Hub) will

The webinar for this Prototype is available on this Cloudera Events page.   This is the third Applied Machine Learning Prototypes. (Here are the links for the Customer Churn and Anomaly Detection prototypes). These are prototypes that will help you build a fully working machine learning example in CML. The Templates will include source data, and walk through various steps: Ingest data into a useful place in CDP (e.g. a Hive Table) Explore the data set Create a plan to build a model Train the model Deploy the model Build and deploy an application Once you have deployed the template and all the CML artifacts that go with it, you can unpick and work it backward to map the process to your own data in your own environment.  Our latest Prototype - Fraud Detection - is now available. To get up and running with it, do the following: Log in to your CML workspace and create a project using the following repo: https://github.com/fastforwardlabs/cml_sentiment_analysis​ Thi

Zookeeper does not allow listing or editing znodes if the current ACL doesn't have a set of permissions for the user or group. This is observed as a security authentication of znodes in all Cloudera Distros inherited from Apache Zookeeper. There are few references for the workaround, just compiling them together for Cloudera Managed clusters.   For the following error: Authentication is not valid There are two ways to address them: Disable any ACL validation in Zookeeper (Not recommended): Add the following config in CM > Zookeeper config > Search for 'Java Configuration Options for Zookeeper Server':-Dzookeeper.skipACL=yes Then Restart and refresh the stale configs. Add a Zookeeper super auth: Skip the part added in <SKIP> if you want to use ‘password' as the auth key. <SKIP>cd /opt/cloudera/parcels/CDH/lib/zookeeper/java -cp "./zookeeper.jar:lib/*" org.apache.zookeeper.server.auth.DigestAuthenticationProvider super:passwordUse the last line from the follo

In this video, we'll review how to access data in S3 from the command line of a Data Hub cluster host using IDBroker. Some components in CDP work out of the box with IDBroker. However, most command-line tools like the Hadoop file system commands require a couple of additional steps to access data in S3. We'll demonstrate retrieving a keytab file for a workload user and using it to kinit on the Data Hub cluster host, enabling data access via IDBroker.   Open the video on YouTube here   Many command-line tools in CDP Public Cloud Data Hub clusters require a Kerberos ticket granting ticket (TGT) for a workload user in order to obtain a short-term access token for S3 or ADLS Gen 2 via IDBroker.  This video demonstrates the following steps:  Granting a data access role to a workload user Retrieving a keytab file for the workload user Copying the keytab file to a host in the data hub cluster Using the keytab file to kinit Confirming the TGT using klist Accessing

Hey Everyone! In this video, we're going to use Cloudera Machine Learning to run through some housing data and create a simple linear regression model. We'll then take this model and deploy it in Cloudera Machine Learning in order to interact with it through a third-party React application. You'll see how quick and easy it is to create and deploy and interact with your own created model.   More information: Link to Related Tutorial: Build, Deploy and Access a Model in Cloudera Machine Learning Cloudera User's Page: www.cloudera.com/users Documentation on Cloudera AI Cloudera Docs / Cloudera Machine Learning

In Cloudera Machine Learning experience (or CDSW for the on-prem version), projects are backed with git.You might want to use GitHub on your projects, so here is a simple way to do that.   First things first: there are basically two ways of interacting with git/GitHub: HTTPS or SSH; We'll use the latter to make the authentication easy. You might also consider SSO or 2FA for enhancing security, here we'll focus on the basics. To make this authentication going on under the hood, copy our SSH key from CML to Github. Find your SSH key in the Settings of CML: Copy that key and add it in Github, under the SSH and GPG keys in your github.com settings: Add SSH key. Put cdsw in the Title and paste your ssh content in the Key: Let's start with creating a new project on github.com: The important thing here is the access mode we want to use: SSH In CML, start a new project with a template: Open a Terminal window in a new session: Convert the project to a git project:cdsw@qp7h1qllr