Resources

Parsing Web Pages for Images with Apache NiFiThis could be used to build a web crawler that downloads images. I am downloading awesome images from Pixabay! URL: https://pixabay.com/en/photos/?image_type=&cat=&min_width=&min_height=&q=data+science&order=popularI wanted to be able to grab every image from a page, I have some web sites I want to backup my images from. So I added a processor that uses JSoup to do.Once you download the NAR from github and deploy to your /usr/hdf/current/nifi/lib directories and restart Apache NiFi you will have a new processor. It is ImageProcessor listed version 1.6.0.You can examine and test the Java source code if you wish.Here is an example flow of grabbing all the images from a pixabay URL then filtering out the empty images. Then we split into individual image URLs. We pull out that tag and then download those images. If they are not blank or small I route to TensorFlow to run some inception on it. I extract image

Note: This article is all thanks to @sprakash for diving into the Ranger source code and explaining to me how to do this!   When developing new plugins for Ranger, these plugins need to be able to download the corresponding Ranger policies. In a secure Ranger setup (which is the preferred way), you download them via the secure download API endpoint: /plugins/secure/policies/download/{serviceName}   This API requires authentication. In this article, I will highlight how to set up a read-only user to download a policy.   Step 1: Create a read-only user  As a Ranger administrator, go to Settings > Users/Groups/Roles:   Then, click on Add New User and create a user with the auditor role, as follows :   Step 2: Give download access to this user In the Ranger home screen, edit the service you want to download (here Hadoop SQL, or it's technical name cm_hive):   Add the recently created user to the policy.download.auth.users configurati

This article is inspired by another article Spark Structured Streaming example with CDE, which talks about how to use CDE to read/write from Kafka using Spark Streaming. The difference in this article is that this article talks about how to connect to "Kafka" of the CDP Streams Messaging data hub cluster.   The following are the differences in CDP Streams Messaging data hub cluster:  Kafka of CDP datahub is SASL_SSL enabled. CDE needs the CA certificate of Kafka to connect. Prerequisites Knowledge of CDP, Nifi, Kafka, and CDE CDE cluster provisioned from CDP Environment CDP Flow Management Datahub Cluster (NiFi)  CDP Streams Messaging Datahub Cluster (Kafka, Schema Registry) CDE CLI The assumption is that the CDP environment, Data lake, Flow Management, and Streaming Messaging datahub cluster are already created. Flow Management Datahub Cluster Once the above clusters are provisioned, setup NiFi to read tweets from twitter and publish it to Kafka. Use thi

NiFi is most effectively used as an "always-on" system, meaning that the data flows are often always operational (running). Doing batch processing is a more difficult task and usually requires some user intervention (such as stopping a source processor). For relational databases (RDBMS), a common use case is to migrate, replicate, or otherwise move the data from the source RDBMS to some target. If ExecuteSQL were used to get all data from a table (for example), then the processor will execute the query each time it is triggered to run, and will return whatever results correspond to the query. If the goal is to simply move all rows to some target, then ExecuteSQL could be started then immediately stopped, such that it would only execute once, and all results will be sent down the flow. However a more common use case is that the source database is being updated from some external process (user, webapp, ERP/CRM/etc. system). In order to get the new rows, the table needs to be queried agai

Issue:In a heavily utilized kafka cluster AMS will keep on crashing with error "ERROR org.apache.hadoop.hbase.client.AsyncProcess: Cannot get replica 0 location for {"totalColumns":5,"row":"kafka.server.FetcherLagMetrics."Solution:1. Run the following command to gather the amount of metrics being collected:curl http://<Ambari-metrics-collector-host>:6188/ws/v1/timeline/metrics/metadata2. From Ambari UI -> Kafka -> Configs -> Flitersearch for: "external.kafka.metrics.exclude.prefix"3. Add the following at the end:kafka.log.Log4. Restart Kafka.This will exclude additional metrics from getting captured and will increase the stability of the AMS.

Cloudera Data Platform Public Cloud recently introduced the ability to backup and restore datalake from a saved location. Specifically, the backup operation saves a full snapshot of data from all SDX services: Atlas: Audit events, saved in HBase tables Lineage data, saved as Janus graph data in HBase tables Edge, vertex, and full text indexes, saved in Solr collections Ranger: Audit logs, saved as a Solr collection Permissions and tags, saved in RDBMS tables HMS Metadata, saved in RDBMS tables   In this article, I will detail how to run backup and restore in CDP Public Cloud in AWS, via the CDP CLI. Pre-Requisites Stop operations that could affect backup Make sure that no HMS affecting operations are running (e.g. creating a table from CDW or a datahub) Go to your Datalake Cloudera Manager, and shut down: Atlas Ranger HMS Make sure you have the proper IAM permissions Datalake backup uses both the Ranger Audit Role and Datalake Admin Roles to write the backups (more det

While trying to perform an import/export data from/to MS Parallel Data Warehouse following error is observed Sqoop Command Output: sqoop export --connect "jdbc:sqlserver://<DB_URL>:<PORT>;database=<DB_NAME>; --driver com.microsoft.sqlserver.jdbc.SQLServerDriver --username <USERNAME> --password ***--table "<TB_NAME>" --input-fields-terminated-by ',' --export-dir <EXPORT_DIR> -m 1 /usr/hdp/2.5.3.0-37//sqoop/conf/sqoop-env.sh: line 23: HADOOP_CLASSPATH=${hcat -classpath}: bad substitution Warning: /usr/hdp/2.5.3.0-37/accumulo does not exist! Accumulo imports will fail. Please set $ACCUMULO_HOME to the root of your Accumulo installation. SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/usr/hdp/2.5.3.0-37/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/hdp/2.5.3.0-37/hive/lib/phoenix-client.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See htt

How to access your secured NiFi instance or cluster:So you have secured your NiFi instance (meaning you have configured it for https access) and now you are trying to access the https web UI. Once NiFi is secured, any entity interacting with the UI will need to successfully authenticate and then be authorized to access the particular NiFi resource(s). As of HDF 2.1.1 or Apache NiFi 1.1.0, NiFi supports authentication via user certificates (default - always enabled), Kerberos/Spnego, or username and password based authentication via LDAP, LDAPS, or kerberos. The intent of this article is not to cover the authentication process, but rather to cover the initial admin authorization process. We assume for this article that authentication is successful. How do you know? A quick look in the nifi-user.log will tell you if your users authentication was successful.Following successful authentication comes NiFi Authorization. NiFi authorization can be handled by NiFi's default built in file ba

Hey Everyone! In this video, we’ll go over how to use Cloudera AI to validate a complex predictive model. Using a publicly available NASA dataset that simulates how jet engines degrade over time, we’ll use machine learning concepts in a cloud environment to go from simulation data to a cost benefit analysis in just a few steps. We’ll also show how we can run experiments to track specific metrics from many different scenarios that our predictive model could possibly be implemented in.     More information:

 Link to Related Tutorial: https://www.cloudera.com/tutorials/cml-validate-jet-engine-predictive-models.html Link to Related Blog: https://blog.cloudera.com/using-cml-to-build-a-predictive-maintenance-model-for-jet-engines/ Link to Related Meetup: https://www.meetup.com/futureofdata-austin/events/273929240/ Cloudera User's Page www.cloudera.com/users

 


The CDP Upgrade Advisor is a utility available on MyCLoudera for Cloudera customers. This tool performs an evaluation of diagnostic data to determine the CDP readiness of your CDH or HDP cluster environment.   Running the upgrade advisor against the cluster in question is one of your first steps to adopting CDP, followed by an in-depth conversation with your Cloudera account team to review the specific results. This utility raises awareness of clusters that may present risks during an upgrade to CDP due to, for example, an unsupported of the operating system currently in use.   The upgrade advisor utility is focused on the environment and platform in use but doesn’t take into consideration use-cases, the actual cluster data, or workflows in use. Analysis of these critical areas occurs as part of your CDP Journey Workshop with your Cloudera account team and Professional Services.   To run the Upgrade Advisor: Click Upgrade Path once logged in to https://my.clo

In the CDP DC cluster, once you use the AMToCM migration script to migrate all configurations, you will not be able to fetch a yarn application logs from the command line when using the following command: $yarn logs -applicationId application_160120642WARNING: YARN_OPTS has been replaced by HADOOP_OPTS. Using value of YARN_OPTS.No class configured for IndexedFormatCan not find any log file matching the pattern: [ALL] for the application: application_1601206427676_0011Can not find the logs for the application: application_1601206427676_0011 with the appOwner: hive This issue occurs because the yarn.log-aggregation.file-formats gets updated with the IndexedFormat configuration during the AMtoCM migration.   To resolve this issue, change it to the ifile and Tfile.  Once you change the configuration as mentioned in the above image, save and restart the yarn service, you will able to fetch the yarn logs using the command line.    Reference Log aggregation propert

Spark is notoriously knobby when it comes to tuning applications and requesting resources. While this added freedom gives us an incredible amount of control and flexibility when running our applications, it also gives us an opportunity to make a lot of mistakes. This guide is for Spark 1.5 and earlier. Spark 1.6 introduces a new setting that allows for a less rigid mapping of the memory areas.There are a lot of knobs in spark to tune, knowing which ones do what will allow you to maximize your memory usage, and understand why you’re doing what you're doing.Most people think when requesting 10g for a worker with Spark, your application has 10g of memory for caching, but its actually quite more complex than that. A Spark executor has several memory areas allocated, and knowing what these areas are for will help understand the best allocation for resources.An executor has two regions, room for caching, and room for aggregations pre-shuffle. Depending on the workload, we can optimize t

By default, Cloudera Machine Learning (CML) ships Jupyter kernel as part of the base engine images. Data Scientists often prefer to use a specialized custom kernel in Jupyter that makes their work more efficient. In this community post, we will walk through how to customize a Docker container image with a sparkmagic Jupyter kernel and how to deploy it to a CML workspace.  Prerequisites: Admin privileges in a CML workspace Local Docker client with access to Docker Hub or internal Docker registry   Step 1. Choose a custom Jupyter kernel. Jupyter kernels are purpose-built add-ons to the basic Python notebook editor. For this tutorial, I chose sparkmagic as the kernel that provides convenient features for working with Spark, like keeping SQL syntax clean in a cell. Sparkmagic relies on Livy to communicate with the Spark cluster. As of this writing, Livy is not supported in CML when running Spark on Kubernetes. However, your classic Spark cluster (for example on Data Hub) will

The webinar for this Prototype is available on this Cloudera Events page.   This is the third Applied Machine Learning Prototypes. (Here are the links for the Customer Churn and Anomaly Detection prototypes). These are prototypes that will help you build a fully working machine learning example in CML. The Templates will include source data, and walk through various steps: Ingest data into a useful place in CDP (e.g. a Hive Table) Explore the data set Create a plan to build a model Train the model Deploy the model Build and deploy an application Once you have deployed the template and all the CML artifacts that go with it, you can unpick and work it backward to map the process to your own data in your own environment.  Our latest Prototype - Fraud Detection - is now available. To get up and running with it, do the following: Log in to your CML workspace and create a project using the following repo: https://github.com/fastforwardlabs/cml_sentiment_analysis​ Thi

Hi,I have created a post and I got few replies but suddenly I have my posted marked as spam. I tried to create new post but still I could see all my questions are marking as spam  

Zookeeper does not allow listing or editing znodes if the current ACL doesn't have a set of permissions for the user or group. This is observed as a security authentication of znodes in all Cloudera Distros inherited from Apache Zookeeper. There are few references for the workaround, just compiling them together for Cloudera Managed clusters.   For the following error: Authentication is not valid There are two ways to address them: Disable any ACL validation in Zookeeper (Not recommended): Add the following config in CM > Zookeeper config > Search for 'Java Configuration Options for Zookeeper Server':-Dzookeeper.skipACL=yes Then Restart and refresh the stale configs. Add a Zookeeper super auth: Skip the part added in <SKIP> if you want to use ‘password' as the auth key. <SKIP>cd /opt/cloudera/parcels/CDH/lib/zookeeper/java -cp "./zookeeper.jar:lib/*" org.apache.zookeeper.server.auth.DigestAuthenticationProvider super:passwordUse the last line from the follo

In this video, we'll review how to access data in S3 from the command line of a Data Hub cluster host using IDBroker. Some components in CDP work out of the box with IDBroker. However, most command-line tools like the Hadoop file system commands require a couple of additional steps to access data in S3. We'll demonstrate retrieving a keytab file for a workload user and using it to kinit on the Data Hub cluster host, enabling data access via IDBroker.   Open the video on YouTube here   Many command-line tools in CDP Public Cloud Data Hub clusters require a Kerberos ticket granting ticket (TGT) for a workload user in order to obtain a short-term access token for S3 or ADLS Gen 2 via IDBroker.  This video demonstrates the following steps:  Granting a data access role to a workload user Retrieving a keytab file for the workload user Copying the keytab file to a host in the data hub cluster Using the keytab file to kinit Confirming the TGT using klist Accessing

Hey Everyone! In this video, we're going to use Cloudera Machine Learning to run through some housing data and create a simple linear regression model. We'll then take this model and deploy it in Cloudera Machine Learning in order to interact with it through a third-party React application. You'll see how quick and easy it is to create and deploy and interact with your own created model.   More information: Link to Related Tutorial: Build, Deploy and Access a Model in Cloudera Machine Learning Cloudera User's Page: www.cloudera.com/users Documentation on Cloudera AI Cloudera Docs / Cloudera Machine Learning

In Cloudera Machine Learning experience (or CDSW for the on-prem version), projects are backed with git.You might want to use GitHub on your projects, so here is a simple way to do that.   First things first: there are basically two ways of interacting with git/GitHub: HTTPS or SSH; We'll use the latter to make the authentication easy. You might also consider SSO or 2FA for enhancing security, here we'll focus on the basics. To make this authentication going on under the hood, copy our SSH key from CML to Github. Find your SSH key in the Settings of CML: Copy that key and add it in Github, under the SSH and GPG keys in your github.com settings: Add SSH key. Put cdsw in the Title and paste your ssh content in the Key: Let's start with creating a new project on github.com: The important thing here is the access mode we want to use: SSH In CML, start a new project with a template: Open a Terminal window in a new session: Convert the project to a git project:cdsw@qp7h1qllr

Introduction Cloudera Data Warehouse architecture leverage compute/storage separation, this is different from the standard Hadoop architecture.     Figure 1: Cloudera Modern Data Warehouse Architecture   The objective of this post is to show how to import the wide functions and code material that we have on the open-source community inside Cloudera Data Warehouse using the object storage architecture. For this, we'll use ESRI Spatial Framework as an example. Prerequisites We'll use github to download the ESRI project, Java and maven to build the necessary JAR files. Step 1: Download the files from ESRI Github repository Download the necessary files from ESRI Spatial Framework Github repository, this can be done using the following command: $ git clone https://github.com/Esri/spatial-framework-for-hadoop.git Figure 2: Cloning ESRI project This will create a dir called "spatial-framework-for-hadoop", enter in this directory to build the project, and generate the JAR file

- Please follow steps mentioned in below link in order to get the list of provenance event id's :Nifi- how to get provenance event id in nifi?- In order to get more information about a specific event : For Nifi running in Standalone mode :# curl -i -X GET http://<hostname>:9090/nifi-api/provenance-events/id For Nifi running in clustered mode:# curl -i -X GET http://<hostname>:9090/nifi-api/provenance-events/id?clusterNodeId=<NODE UUID> There may be multiple events with the same event id(one on each node), so you need to specify from which node you want to return that specific event.

    Recently I was engaged in a use case where CDE processing was required to be triggered once data landed on s3. The s3 trigger in AWS would be via a Lambda function. As the files/data land in s3, an AWS Lambda function would be triggered to then call CDE to process the data/files. Lambda functions at trigger time include the names and locations of the files the trigger was executed upon. The file locations/names would be passed onto the CDE engine to pick up and process accordingly. Prerequisites to run this demo AWS account s3 Bucket Some knowledge of Lambda CDP and CDE Artifacts AWS Lambda function code https://github.com/sunileman/spark-kafka-streaming/blob/master/src/main/awslambda/triggerCDE.py CDE Spark Job, main class com.cloudera.examples.SimpleCDERun  Code for class com.cloudera.examples.SimpleCDERun https://github.com/sunileman/spark-kafka-streaming Prebuilt jar https://sunileman.s3.amazonaws.com/CDE/spark-kafka-streaming_2.11-1.0.jar

In a browser put in: http://<AMBARI-HOST>:8080/api/v1/clusters/<CLUSTER-NAME>/requests/ Find the highest reqest number in the output. Now, we can go to each request by pasting in the number of the request, an example would be: http://<AMBARI-HOST>:8080/api/v1/clusters/<CLUSTER-NAME>/requests/110 The output should look like: { "href" : "http://mon-orlan:8080/api/v1/clusters/test/requests/110", "Requests" : { "aborted_task_count" : 0, "cluster_name" : "test", "completed_task_count" : 3, "create_time" : 1444340395849, "end_time" : 1444340449936, "exclusive" : false, "failed_task_count" : 0, "id" : 110, "inputs" : "{}", "operation_level" : "SERVICE", "progress_percent" : 100.0, "queued_task_count" : 0, "request_context" : "Restart all components with Stale Configs for HIVE", ..... If that is not your hung operation, go back one by one from the highest request number. What we are looking for is the request context matches your hung Operation, that is the one we want

There are several options available to achieve this use case. The easiest and the best approach would be HBase snapshots method to transfer the data.    Note: All actions need to be performed as the HBase user only, to ensure correct permissions. On source cluster:#hbase shell> snapshot ‘Test’_Table’, ‘Test_Table_SS’#hbase org.apache.hadoop.hbase.snapshot.SnapshotInfo -snapshot ‘Test_Table_SS’ -files -stats #hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot 'Test_Table_SS' -copy-to hdfs://<Destination_NN_hostname>:8020/hbase -mappers 16 -bandwidth 200 On destination cluster:#hbase org.apache.hadoop.hbase.snapshot.SnapshotInfo -snapshot ‘Test_Table_SS’ -files -stats#hbase shell> clone_snapshot ‘Test_Table_SS’, ‘Test_Table’#hbase shell> major_compact 'Test_Table' Once done, you can choose to delete the snapshots on both ms05 and as05:#hbase shell> delete_snapshot 'Test_Table_SS'​ But If you plan to use CopyTable, this will not work without

The Cloudera Operational Database (COD) experience is a managed dbPaaS solution. It can auto-scale based on the workload utilization of the cluster and will be adding the ability to auto-tune (better performance within the existing infrastructure footprint) and auto-heal (resolve operational problems automatically) later this year.    COD is backed by the proven, scalable technology in Apache HBase and Apache Phoenix.   There is an excellent blog post showing the basics of setting up a COD database and connecting to it using Python.   COD offers options to connect using both HBase APIs and ANSI SQL through Apache Phoenix. This article shows how to connect to a COD database through Phoenix using Golang.    All of the code for this article can be found in this Github repository.   Getting started with COD is incredibly easy - no clusters to deploy and secure, simply log in and click “Create Database”: Select your CDP Environment and enter a