Resources

The Hive Streaming API enables the near real-time data ingestion into Hive. This two part posting reviews some of the design decisions necessary to produce a health Hive Streaming ingest process from which you can in a near real-time execute queries on the ingested data.Implementing a real-time Hive Streaming exampleImplementing a Hive Streaming data feed requires we make tradeoffs between the load on the NameNode versus the business SLAs for low latency data queries. Hive Streaming is able to work in a near real-time basis through the creation of a new ‘delta’ file on a bucketed Hive table with every table commit. For example, a Hive Streaming client is committing 1 row every second (it can actually commit at much faster rates), then after 1 minute there would exist 60 new delta files added o HDFS. After a day, there would be 86,400 new delta files and 15.5 million after 6 months just on your streamed data. With this many new HDFS Hive delta files, eventually the Hadoop cluster would

Introduction When working with CDE in CDP Public Cloud, there may be a need to allocate fractions of CPU to our Spark Jobs, without losing parallelism. Following are some of the real scenarios: Spark application that reads from HBase and performs CPU-light processing, causing a significant I/O wait. Reducing the number of executors or cores per executor is not optimal because the parallel reads from HBase scale linearly, increase the job duration. Spark application that needs high parallelism for the data processing, but before writing the data to HDFS, the partitions are coalesced to a smaller number to avoid creating many small files. So the single job uses all the assigned cores during processing and less cores during the output, leaving some CPU idle. Steps To allocate fractions of CPUs to Spark in CDE, we need to set the 'spark.kubernetes.executor.request.cores' config. This could be set to 0.1, 500m, 1.5, 5, etc. More details on official Spar

A - Key components needed to make use of time series data provided by the OSIsoft Pi System For real time processing NiFi for reading the data from Pi and push to HDF cluster Kafka for queuing data samples for stream processingStorm / Spark Streaming For batch processing NiFi for reading the data from Pi and push to HDP cluster Hive - for data, storage, pre-processing and analytical processing HBase - for real time use cases and iterative processing of windows of dataB - Ways HDF can be used to access data from the Pi System and move it into the data lake - File based ingestion - This is the easiest setup and does not require any additional license from OSI Soft. Pi System admins typically know how to export the data files (by tags and/or by a window of time raw/interpolated). The file export process involves setting up, then automating, an extract script (typically this is a .bat script that runs the command line utility to extract

Introduction This post covers the steps required to build a custom runtime for Cloudera Data Engineering. The process pulls the base image from container.repository.cloudera.com and builds a custom image based on the Dockerfile provided and uploads the custom image to Amazon ECR using AWS CodeBuild. All the files mentioned in this post can be downloaded from here.     Steps Setup ECR & IAM role using AWS CloudFormation The CloudFormation template cloudformation-ecr-codebuild.yml creates the Amazon Elastic Container Repository and the IAM role required for AWS CodeBuild Update the files cloudformation-ecr-codebuild.yml, cloudformation-parameters.json, and cloudformation-tags.json as required Create the Cloudformation stack using the following command aws cloudformation create-stack \ --stack-name vkar-ecr \--template-body file://cloudformation-ecr-codebuild.yml \--parameters file://cloudformation-parameters.json \--t

There may be a need to use Anaconda to set up a Python virtual environment or an R environment to run SparklyR on the CDP Public Cloud Datahub cluster. CDP Public Cloud provides "Recipe" as extensions to install additional software on top of the base image. In this post, we will use Recipe with a sample script to accomplish this. Download this bootstrap/ recipe script for Anaconda install. wget https://raw.githubusercontent.com/karthikeyanvijay/cdp-publiccloud/main/datahub-recipes/setup-anaconda.sh Ensure the parameters in the first section of the script. Modify them as required. The USER_GROUP_ADMIN will be the owner of the directory where Anaconda will be installed. ACLs are added to the directory for USER_GROUP_1 and USER_GROUP_2 for sufficient access. If you just have one user group, remove all lines containing USER_GROUP_2 from the script. TAR_FILE_PATH is the packaged Python/ R tarball that can be submitted in the YARN jobs (this way we do not have to i

Introduction When working with CDP Public Cloud, there may be a need to access other AWS services from Apache Spark. One such scenario is to get a secret (like a database password) from the AWS Secrets Manager. One approach to this is to use AWS Access keys, but using long-term security credentials in a program may not be feasible or desirable. Within CDP,  you can obtain the temporary AWS credentials from ID Broker and then use the AWS Java SDK to access AWS secrets manager.  Steps Here are the steps to try this in a Spark shell first: Ensure that the IAM role mapped to the user has 'Read' access to the AWS Secrets Manager. To test out this program, you can launch a Spark shell using the following command:   spark-shell --master=yarn \ --conf "spark.jars.packages=com.amazonaws:aws-java-sdk:1.11.984,org.scalaj:scalaj-http_2.11:0.3.15" Run the commands in the file interactively to see the results.   // Change variables hereval id_broker_h

In case you would like to move JournalNode service to another host, here are the steps to do so: 1. Put HDFS in safemode su - hdfs -c 'hdfs dfsadmin -fs hdfs://<active node>:8020 -safemode enter' 2. Execute a save namespace of the Active NameNode su - hdfs -c 'hdfs dfsadmin -fs hdfs://<active node>:8020 -saveNamespace' 3. Stop all services via Ambari 4. Add journal node to the Ambari database through the API call: export AMBARI_USER=adminexport PASSWORD=adminexport AMBARI_HOST=localhostexport CLUSTER_NAME=clusterexport MOVE_FROM='old-host.abcd.com'export MOVE_TO='new-host.abcd.com'# Tell to ambari we want to install this component on new_hostcurl -u $AMBARI_USER:$AMBARI_PASSWORD -H "X-Requested-By:ambari" -i -X POST http://$AMBARI_HOST:8080/api/v1/clusters/$CLUSTER_NAME/hosts/$MOVE_TO/host_components/JOURNALNODEORcurl -v -u admin:admin -H "X-Requested-By:ambari" -i -X POST -d '{"host_components" : [{"HostRoles":{"component_name":"JOURNALNODE"}}] }' http://<ambari-server&

Introduction If the AWS IAM roles are not configured correctly, and if you are running into issues using these IAM roles, the best way to troubleshoot it is by using the AWS CLI. This post contains the steps and the required setup for this. Steps If the data lake admin role is not configured correctly, you will see the following error during the data lake deployment.   Error: Failed to create HDFS directory., Command [CreateRangerAuditDir], with id [1546333865] failed: Failed to create HDFS directory s3a://<bucketname>/data/ranger/audit/hdfs., Command [RangerPluginCreateAuditDir].   To get to the root cause of the issue, check the S3 permissions on the IAM role. If you want to further troubleshoot, then you can assume the role from the ID Broker's instance profile and use AWS CLI to reproduce and troubleshoot the issue. To do this, use the following troubleshooting steps: Log in to the ID broker host as a Cloudbreak user using the pem file provided during the environ

The kubectl tool provides direct administrative access to the Kubernetes cluster underlying a CDE service, which is useful for troubleshooting, among other things. This video will demonstrate how to set up kubectl access. To enable kubectl, we will need a couple of prerequisites. We will need the kubeconfig file from the CDE service. We will need to get and authorize the IAM user, and then we will need to make sure that everything is set up correctly, both for kubectl and some other tools like k9s.   There are a couple of prerequisites for this operation, and we are not going to cover their installation in this video. You will need to follow the process relevant to your platform of choice to get the AWS CLI installed and configured with an IAM user. You will also need to get kubectl installed.   Key steps: Download the kubeconfig file from the CDE service Retrieve the ARN for the chosen IAM user Grant administrative access for the IAM user Verify kubectl access is successf

Infrastructure Virtual Warehouse Impala will be running under Kubernetes Pods. The user can find the Kubernetes cluster running in the AWS account. The pods will be running in one of the K8s nodes. The JDBC URL can be found from the Virtual Warehouse.  Example:     jdbc:impala://<IMPALA_Coordinator_URL>:443/default;AuthMech=3;transportMode=http;httpPath=cliservice;ssl=1;UID=xxxx;PWD=PASSWORD       The server name would be <IMPALA_Coordinator_URL> and the port is 443. Access to Cloudera Manager, other cluster UIs, and endpoints (such as JDBC) are always through the secure Knox gateway on port 443. Users are automatically logged in with their CDP credentials.   If you get "Error while trying to connect" message from the visualization tool, check if port 443 is allowed for Impala coordinator and Knox gateway server from the server where the visualization tool is hosted.   Have come across some visualization tools like DOMO; its co

Short Description: This article covers steps on adding a Spark 2 Dashboard to Grafana in Ambari-Metrics, in order to monitor Spark applications for detailed resource usage statistics. ArticleMonitoring Spark 2 Performance via Grafana in Ambari-Metrics Configuration of Advanced spark2-metrics-properties Adding a new data source to Grafana Building a Spark 2 dashboard in GrafanaPre-Requisites You are running HDP 2.6.4 or higher, with Spark2 and Ambari-Metrics installed. A standalone Grafana installation may be used instead of Ambari-Metrics. A time-series database with support for the graphite protocol. In this article, I will be use Graphite. Alternatively, Prometheus with the graphite-exporter plugin can be used depending on personal preference. Steps In order for Spark components to forward metrics to our time-series database, we need to add a few items to our configuration in Ambari -> Spark2 -> Configs -> Advanced spark2-metrics-p

SYMPTOM:Problem was seen when the client was running Metastore HA and both configured with org.apache.hadoop.hive.ql.txn.compactor.Initiator. this is what we observed from the logs of both metastore service// metastore 1 ERROR compactor.Worker (Worker.java:run(181)) - Caught an exception in the main loop of compactor worker lnxhdpap02.smrcy.com-33, MetaException(message:Unable to connect to transaction database com.mysql.jdbc.exceptions.jdbc4.MySQLTransactionRollbackException: Deadlock found when trying to get lock; try restarting transaction // metastore 2 ERROR txn.CompactionTxnHandler (CompactionTxnHandler.java:findNextToCompact(194)) - Unable to select next element for compaction, Deadlock found when trying to get lock; try restarting transaction ROOT CAUSE:org.apache.hadoop.hive.ql.txn.compactor.Initiator is a thread that runs inside the metastore service to manage compactions of ACID tables. There should be exactly 1 instance of this thread (even with multiple Thrift services).WO

This article will describe the steps to access the remote CDP cluster HBase data from another CDP cluster using Spark in Kerberized environment. Assume we have two clusters say 'Cluster_A' and 'Cluster_B'. 'Cluster_A' is having the HBase data. 'Cluster_B' is having Spark. Now, we are trying to access the HBase data available in 'Cluster_A' using Spark from 'Cluster_B'. Prerequisites: Both clusters 'Cluster_A' and 'Cluster_B' with keytabs need to have the same REALM. Follow these steps:  Login to 'Cluster_A' edge node Obtain the Kerberos ticket using kinit:kinit -kt <key_tab_file> <principal_name>​ Login to the Hbase shell and create the 'person' table:ranga]# hbase shellhbase(main):001:0> listTABLE0 row(s)hbase(main):013:0> create 'person', 'p'Created table personTook 8.2307 seconds=> Hbase::Table - personhbase(main):014:0> put 'person',1,'p:id','1'Took 0.0173 secondshbase(main):015:0> put 'person',1,'p:name','Ranga Reddy'Took 0.0045 secondsh

Would you like to experiment with CDP Private Cloud Base (formerly known as CDP Data Center), OpenShift 4, and YuniKorn, but all you have is a Bare Metal server or laptop?   The answer to this question lies In this GitHub link (includes videos as well). This link includes the following how-tos:   Automate and speed up the installation of CDP Private Cloud Base (formerly known as CDP Data Center) in a dockerless container Automate the install of OpenShift 4.5+ Deploy CDP Private Cloud components onto an existing on-premises OpenShift 4.5+ / Kubernetes cluster Experiment with YuniKorn ( on OpenShift 4.5+ to unleash the power of resource scheduling for running Big Data & ML on Kubernetes(  All this on one Baremetal server!   DISCLAIMER: This is for lab and learning purposes only.  

In this video, we’ll go over how you can use both Cloudera Public Cloud to both Ingest data through Cloudera Data Engineering as well as explore it through Hue and Impala within Cloudera Data Warehouse. You'll see how easy it is to run queries that give you insight into your data and how you can use a built-in data visualization tool to create a dashboard to share your results. More information: Link to related tutorial: Data Exploration and Reporting with Cloudera Data Warehouse  Link to related meetup: Data Exploration and Dashboard Creation in a Cloud-Based Data Warehouse  Cloudera user's page www.cloudera.com/users

In this article, you'll learn how to: Build and run a Cloudera CDP Private Cloud Base (formerly known as CDP Data Center) trial environment in a dockerless container Automate the install of OpenShift 4.5+ with YuniKorn scheduler for BigData and ML on your laptop, virtual machine or Bare Metal server using CodeReady containers Want to experiment with Cloudera CDP Private Cloud Base (formerly known as CDP Data Center)? You can set up a trial environment in a container as shown in this Github link. The trial environment also includes a wizard that provides step-by-step guidance for installing CDP Private Cloud components onto an existing on-premises Kubernetes/OpenShift cluster.So, would you also like to automate the install of OpenShift 4.5+ with YuniKorn scheduler for BigData and ML on your laptop, virtual machine or Bare Metal server using CodeReady containers?   Find out how to do this at the above Github link.

NiFi for XML / RSS / REST Feed IngestI want to retrieve the status from various Cloud providers and services, including Cloudera, AWS, Azure, and Google. I have found many of the available status APIs will return XML/RSS. We love that format for Apache NiFi, so let's do that.Note: If you are doing development in a non-production environment, try the new NiFi 1.13.1. If you need to run your flows in production on-premise, private cloud, or in the public cloud, then use Cloudera Flow Management.I have separated the processing module "Status" from the input, so I can pass in the input anyway I want. When I move this to a K8 environment, this will become a parameter that I pass in. Stay tuned to Cloudera releases.The flow is pretty simple to process RSS status data. We call the status URL and in the next step easily convert RSS into JSON for easier processing. I split these records and grab just the fields I like.I can easily add additional fields from my metadata for unique

While running sandbox-hdp docker container on Ubuntu 20.04 LTS, I came across a problem and found a solution, which is pretty simple: Download .zip file with .sh file for build docker container from here. Unzip the file Change line in docker-deploy-hdp<version>.shfromdocker run --privileged --name $name -h $hostname --network=cda --network-alias=$hostname -d "$registry/$name:$version" to docker run --privileged -m=<at least 12g works for me> --name $name -h $hostname --network=cda --network-alias=$hostname -d "$registry/$name:$version"   Now, you can run a script, but probably you will need to re-run the second part after line #Deploy the proxy container. To make sure, run docker ps and check if sandbox-proxy container is running; if not, just run the second part of the script. Like I said, for me that works, hope it will work for you.

The following steps can be used to configure Intellij to run Spark jobs on Cloudera Data Engineering experience (CDE). This way the developers will be able to test their jobs on CDE without having to leave the IDE. The following steps are shown for an Intellij installed on a Windows PC. Complete setting up CDE CLI with Git Bash as per this article. In IntelliJ, go to File > Settings, modify the "shell path" to the following:C:\Program Files\Git\bin\bash.exe" --login -i Test this setting by launching a terminal from within the IntelliJ project and running CDE CLI commands. If you are unable to run CDE CLI commands (with environment variable CDE_CONFIG not configured), you will have the option to setup/override it in the next step. Add a Run/Debug configuration with a new Shell script with the following details. Ignore the error stating that the Shell script is not found. You have the option of overriding the CDE_CONFIG environment variable (in case you want to submit

Download the Cloudera Data Engineering CLI from the Cloudera Data Engineering > CDE Services > Virtual Clusters. Click the Cluster details page, and click the CLI TOOL button to download the CDE CLI. Move the CLI to the desired directory. Here, it is moved to D:\Users\Vijay.Karthikeyan\Documents\utils\cde. Add this directory to PATH, as shown as follows: Create the directory ~/AppData/Local/cloudera/cde to store the CDE configuration files. Make a note of your workload username from the profile page of the CDP control plane, and set workload password for the environment if not done already. Create a file containing the workload password Create the certificate file ca.pem holding the root certificates. Get the certificates here.  Concatenate them and save them as ~/AppData/Local/cloudera/cde/ca.pem. Get the virtual cluster endpoint (This will be entered as the value vcluster-endpoint in the config.yaml file) Create a file config.yaml with the fo

Objective This tutorial consists of two articles. The first walks you through a NiFI flow that utilizes the ValidateRecord processor and Record Reader/Writer controller services to: Convert a CVS file into JSON format Validate the data against a given schema Write the JSON data to either a 'valid' relationship or 'invalid' relationship The second article, modifies the flow to demonstrated the effects of enabling/disabling the "Strict Type Checking" property of the ValidateRecord processor. Note: The ValidateRecord processor was introduced in NiFi 1.4.0.Environment This tutorial was tested using the following environment and components: Mac OS X 10.11.6 Apache NiFi 1.4.0ValidateRecord FlowSupport Files Here is a template of the flow discussed in this tutorial: validaterecord.xml In the spirit of Halloween last week, let's ingest some data that ranks the most popular candies: candy-data-dirty.txt (Change the extension from .txt to .csv after downloading) Note: The CSV dat

New Features of Apache NiFi 1.13.0 Check it out:   https://twitter.com/pvillard31/status/1361569608327716867?s=27 Download today: Apache NiFi Downloads Release Notes: Release Notes (Apache NiFi) Migration:  Migration Guidance New Features ListenFTP UpdateHiveTable - Hive DDL changes - Hive Update Schema i.e. Data Drift i.e. Hive Schema Migration!!!! SampleRecord - different sampling approaches to records (Interval Sampling, Probabilistic Sampling, Reservoir Sampling) CDC updates Kudu updates AMQP and MQTT integration upgrades ConsumeMQTT - readers, and writers added HTTP access to NiFi by default is now configured to accept connections to 127.0.0.1/localhost only. If you want to allow broader access for some reason for HTTP, and you understand the security implications, you can still control that as always by changing the 'nifi.web.http.host' property in nifi.properties as always. That said, take the time to configure proper HTTPS. We offer

In this article, we will learn how to build Twitter NiFi template in a flow management datahub. Pre-Requisite  CDP environment is created and flow management datahub is installed. In that, the required users are added. Steps Login to NiFi by clicking on the NiFi log as shown below: Add GetTwitter process group and put a file from the process group as follows: Configure Twitter process group by right-clicking on GetTwitter process group, and then select Configure; add required parameters: Consumer Key, Consumer Secret, Access Token, Access Token Secret as shown as follows: Configure the PUT file process to store the Twitter JSON files in one of the NiFi hosts.To verify the data received, log in to one of the NiFi hosts, and check the directory mentioned in the above step. 

Running 'pyspark' applications in CML for model generation and prediction, with data residing in COD With the recent addition of Cloudera Operation Database Experience to the CDP Public Cloud, we want to explore how it can be leveraged in the real-life 'DataFlow' end-user scenario. This article talks about how to execute Spark/pyspark job in CML to run modeling task using the data residing in COD. We read the table present in COD and also write back the score table back to the COD once the prediction is done. Getting Started CDP Runtime (supporting COD) >=7.2.2 We assume that CDP environment, datalake, datahub (Data Engineering) have been provisioned. We further assume that experiences COD and CML have been provisioned for the CDP target environment. Note: Please refer to The world’s first enterprise data cloud, if you are just starting with CDP, and get to know how all the requirements can be in place with ease.  Some of the following steps are already documen

It's not a simple process, but one that can be easily done by anyone with a little database admin experience. You have to add a user by restarting cbd with new user and password in Profile. Then you go into the database and move the encrypted password from the new user to the old. This will change the password and leave your old user with access to the clusters and resource you've already built.OverviewEdit the Profile and change both the UAA_DEFAULT_USER_PW to the desired new password and UAA_DEFAULT_USER_EMAIL to a different address. cbd restart – This will add a 2nd user with the new password to the database docker exec -ti cbreak_commondb_1 bash – Starts a bash shell in the database container pg_dump -Fc -U postgres uaadb > uaadb.dump – Makes a backup of the user database psql -U postgres – Starts a postgres shell postgres=# \c uaadb; – Connects to the user database in the postgres shell uaadb=# select * from users; – Shows the two accounts and their encrypted passwordsupdate u