Community Articles

How to test/ create the Livy interactive sessions The following session is an example of how we can create a Livy session and print out the Spark version: Create a session with the following command:curl -X POST --data '{"kind": "spark"}' -H "Content-Type: application/json" http://172.25.41.3:8998/sessions​ Wait for the application to spawn, replace the session ID:curl -X POST --data '{"kind": "spark","code":"sc.version"}' -H "Content-Type: application/json" http://172.25.41.3:8998/sessions/25/statements Replace the session ID and get the result:curl -X GET --data '{"kind": "spark"}' http://172.25.41.3:8998/sessions/25/statements *Livy objects properties for interactive sessions   How to test the Batch Applications Using the Livy API Following is the SparkPi test job submitted through Livy API: To submit the SparkPi job using Livy, you should upload the required jar files to HDFS before running the job. This is the main difference between the Livy API and spark-submit.

In this article, we will learn how to integrate Spark with Hive JdbcStorageHandler in CDP.   JdbcStorageHandler By using the JdbcStorageHandler, we can connect Apache Hive to JDBC Data Sources (MySQL, PostgreSQL, Oracle, DB2, or Derby etc). Currently writing to a JDBC data source is not supported. To use JdbcStorageHandler, we need to create an external table using JdbcStorageHandler.   Example:   CREATE EXTERNAL TABLE employee( id BIGINT, name STRING, age INT, salary DOUBLE)STORED BY 'org.apache.hive.storage.jdbc.JdbcStorageHandler'TBLPROPERTIES ( "hive.sql.database.type" = "MYSQL", "hive.sql.jdbc.driver" = "com.mysql.jdbc.Driver", "hive.sql.jdbc.url" = "jdbc:mysql://localhost/test", "hive.sql.dbcp.username" = "username", "hive.sql.dbcp.password" = "password", "hive.sql.table" = "EMPLOYEE", "hive.sql.dbcp.maxActive" = "1");   Supported Data Types: The column data type for a Hive JdbcStorageHandler table can be: Numeric data ty

In Beeline, the command-line options such as sslTrustStore, trustStorePassword, showDbInPromp etc are case sensitive.   For example, below is a working connection string from a test bed:  beeline -u "jdbc:hive2://host-A-fqdn:21051/default;principal=impala/host-A-fqdn@COE.CLOUDERA.COM;ssl=true;sslTrustStore=/opt/cloudera/security/truststore.jks"  In the above example, the common mistakes are principal mentioned as Principal and sslTrustStore mentioned as ssltruststore.   Here, if the case sensitivity is not followed, Beeline silently ignores the command line options and drops them:  //Sample stringbeeline -u "jdbc:hive2://host-A-fqdn:21051/default;Principal=impala/host-A-fqdn@COE.CLOUDERA.COM;ssl=true;ssltruststore=/opt/cloudera/security/truststore.jks" If you use the above connection string, at first, you will encounter a Kerberos issue as the property "principal" will be dropped and the actual Kerberos authentication w

For non-SSL enabled NiFi below should work: curl --tlsv1.2 -i -H'Content-Type: application/json' -XPUT -d'{"id":"cdb54c9a-0158-1000-5566-c45ca9692f85","state":"RUNNING"}'localhost:8080/nifi-api/flow/process-groups/cdb54c9a-0158-1000-5566-c45ca9692f85 Start Process Group for SSL enabled NiFi : Generate access token: [root@nifi-ambari-01~]# curl --tlsv1.2 https://000.00.00.000:9091/nifi-api/access/token--data 'username=awadhwani&password=password' -k O/P: eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJjbj1BcnRpIFdhZGh3YW5pLG91PVBlb3BsZSxkYz1zbWUsZGM9aHd4IiwiaXNzIjoiTGRhcFByb3ZpZGVyIiwiYXVkIjoiTGRhcFByb3ZpZGVyIiwicHJlZmVycmVkX3VzZXJuYW1lIjoiQXJ0aSBXYWRod2FuaSIsImtpZCI6MywiZXhwIjoxNDg5MjMyNDQyLCJpYXQiOjE0ODkxODkyNDJ9.17iHL3XX7Bw6dXv5lCByimu_asQOaSwW11o2IQEFO0s[root@nifi-ambari-01~]# Start PG by passing the above access token [root@nifi-ambari-01~]# curl --tlsv1.2 -ik -H 'Content-Type: application/json' -H 'Authorization:BearereyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJjbj1BcnRpIFdhZGh3YW5pLG91PVBlb3BsZSxkY

Configuring Ranger Usersync with AD/LDAP is highly dependent on the customer environment. This requires understanding of the customer environment as well as the specific requirements for syncing users and groups.In order to get some background on User management in Hadoop, please refer to this Dataworks Summit Presentation. Corresponding slides are available at: SlidesIn this article we will take a common use case and see how Ranger Usersync can be configured in that scenario. Ranger Usersync supports various configuration options with AD/LDAP and details are available hereUse case: Sync all the users who are members of a specific group(s)This use case can be handled in multiple ways and is also based on the AD/LDAP attributes available in the server. In case of Active Directory, users contain information of the group(s) they belong to using the “memberof” attribute. Similarly groups contain information about the user(s) that belong to them using the “member” attribute. Whereas in defa

slapd-conf.tar.gzEnvironment:Hortonworks Sandbox “HDP_2.3.2_virtualbox" Ranger Version: 0.5.0.2.3 LDAP Version: openldap-2.4.40-7.el6_7.x86_64LDAP setup url : LDAP setup tutorial 
[Note: The ldap setup here does not have ssl implemented. Pls find working “slapd.conf”attached]Steps:1.Configure openldap server on sandbox as mentioned in the Ldap setup url above.2.Login to Ambari console using “admin” user.3.Click on “Ranger” service -> “Configs”4.Go to “LDAP Settings” and set the below properties –ranger.ldap.user.searchfilter=(uid={0})ranger.ldap.user.dnpattern=cn=Manager,dc=hortonworks,dc=comranger.ldap.url=ldap://127.0.0.1:389ranger.ldap.referral=ignore ranger.ldap.group.roleattribute=uid ranger.ldap.bind.password=*****ranger.ldap.bind.dn=cn=Manager,dc=hortonworks,dc=comranger.ldap.base.dn=dc=hortonworks,dc=comNote: ldap.bind.password=<Admin password of openldap>5.Go to “Advanced ranger-admin-site” and set below properties -ranger.ldap.group.searchfilter=(member=uid={0},ou=Us

Security/Governance/GDPR Demo on CDP-Private Cloud Base 7.x Summary This article explains how to quickly set up Cloudera Security/Governance/GDPR (Worldwide Bank) demo using Cloudera Data Platform - Private Cloud Base (formerly known as CDP-Data Center). It can be deployed either on AWS using AMI or on your own setup via provided script What's included Single node CDP 7.1.7 including: Cloudera Manager (60-day trial license included) - for managing the services Kerberos - for authentication (via local MIT KDC) Ranger - for authorization (via both resource/tag-based policies for access and masking) Atlas - for governance (classification/lineage/search) Zeppelin - for running/visualizing Hive queries Impala/Hive 3 - for Sql access and ACID capabilities Spark/HiveWarehouseConnector - for running secure SparkSQL queries Worldwide Bank artifacts Demo hive tables Demo tags/attributes and lineage in Atlas Demo Zeppelin notebooks to walk through a demo scenario Ranger policies across HDF

The Hive Streaming API enables the near real-time data ingestion into Hive. This two part posting reviews some of the design decisions necessary to produce a health Hive Streaming ingest process from which you can in a near real-time execute queries on the ingested data.Implementing a real-time Hive Streaming exampleImplementing a Hive Streaming data feed requires we make tradeoffs between the load on the NameNode versus the business SLAs for low latency data queries. Hive Streaming is able to work in a near real-time basis through the creation of a new ‘delta’ file on a bucketed Hive table with every table commit. For example, a Hive Streaming client is committing 1 row every second (it can actually commit at much faster rates), then after 1 minute there would exist 60 new delta files added o HDFS. After a day, there would be 86,400 new delta files and 15.5 million after 6 months just on your streamed data. With this many new HDFS Hive delta files, eventually the Hadoop cluster would

Introduction When working with CDE in CDP Public Cloud, there may be a need to allocate fractions of CPU to our Spark Jobs, without losing parallelism. Following are some of the real scenarios: Spark application that reads from HBase and performs CPU-light processing, causing a significant I/O wait. Reducing the number of executors or cores per executor is not optimal because the parallel reads from HBase scale linearly, increase the job duration. Spark application that needs high parallelism for the data processing, but before writing the data to HDFS, the partitions are coalesced to a smaller number to avoid creating many small files. So the single job uses all the assigned cores during processing and less cores during the output, leaving some CPU idle. Steps To allocate fractions of CPUs to Spark in CDE, we need to set the 'spark.kubernetes.executor.request.cores' config. This could be set to 0.1, 500m, 1.5, 5, etc. More details on official Spar

A - Key components needed to make use of time series data provided by the OSIsoft Pi System For real time processing NiFi for reading the data from Pi and push to HDF cluster Kafka for queuing data samples for stream processingStorm / Spark Streaming For batch processing NiFi for reading the data from Pi and push to HDP cluster Hive - for data, storage, pre-processing and analytical processing HBase - for real time use cases and iterative processing of windows of dataB - Ways HDF can be used to access data from the Pi System and move it into the data lake - File based ingestion - This is the easiest setup and does not require any additional license from OSI Soft. Pi System admins typically know how to export the data files (by tags and/or by a window of time raw/interpolated). The file export process involves setting up, then automating, an extract script (typically this is a .bat script that runs the command line utility to extract

Introduction This post covers the steps required to build a custom runtime for Cloudera Data Engineering. The process pulls the base image from container.repository.cloudera.com and builds a custom image based on the Dockerfile provided and uploads the custom image to Amazon ECR using AWS CodeBuild. All the files mentioned in this post can be downloaded from here.     Steps Setup ECR & IAM role using AWS CloudFormation The CloudFormation template cloudformation-ecr-codebuild.yml creates the Amazon Elastic Container Repository and the IAM role required for AWS CodeBuild Update the files cloudformation-ecr-codebuild.yml, cloudformation-parameters.json, and cloudformation-tags.json as required Create the Cloudformation stack using the following command aws cloudformation create-stack \ --stack-name vkar-ecr \--template-body file://cloudformation-ecr-codebuild.yml \--parameters file://cloudformation-parameters.json \--t

There may be a need to use Anaconda to set up a Python virtual environment or an R environment to run SparklyR on the CDP Public Cloud Datahub cluster. CDP Public Cloud provides "Recipe" as extensions to install additional software on top of the base image. In this post, we will use Recipe with a sample script to accomplish this. Download this bootstrap/ recipe script for Anaconda install. wget https://raw.githubusercontent.com/karthikeyanvijay/cdp-publiccloud/main/datahub-recipes/setup-anaconda.sh Ensure the parameters in the first section of the script. Modify them as required. The USER_GROUP_ADMIN will be the owner of the directory where Anaconda will be installed. ACLs are added to the directory for USER_GROUP_1 and USER_GROUP_2 for sufficient access. If you just have one user group, remove all lines containing USER_GROUP_2 from the script. TAR_FILE_PATH is the packaged Python/ R tarball that can be submitted in the YARN jobs (this way we do not have to i

Introduction When working with CDP Public Cloud, there may be a need to access other AWS services from Apache Spark. One such scenario is to get a secret (like a database password) from the AWS Secrets Manager. One approach to this is to use AWS Access keys, but using long-term security credentials in a program may not be feasible or desirable. Within CDP,  you can obtain the temporary AWS credentials from ID Broker and then use the AWS Java SDK to access AWS secrets manager.  Steps Here are the steps to try this in a Spark shell first: Ensure that the IAM role mapped to the user has 'Read' access to the AWS Secrets Manager. To test out this program, you can launch a Spark shell using the following command:   spark-shell --master=yarn \ --conf "spark.jars.packages=com.amazonaws:aws-java-sdk:1.11.984,org.scalaj:scalaj-http_2.11:0.3.15" Run the commands in the file interactively to see the results.   // Change variables hereval id_broker_h

In case you would like to move JournalNode service to another host, here are the steps to do so: 1. Put HDFS in safemode su - hdfs -c 'hdfs dfsadmin -fs hdfs://<active node>:8020 -safemode enter' 2. Execute a save namespace of the Active NameNode su - hdfs -c 'hdfs dfsadmin -fs hdfs://<active node>:8020 -saveNamespace' 3. Stop all services via Ambari 4. Add journal node to the Ambari database through the API call: export AMBARI_USER=adminexport PASSWORD=adminexport AMBARI_HOST=localhostexport CLUSTER_NAME=clusterexport MOVE_FROM='old-host.abcd.com'export MOVE_TO='new-host.abcd.com'# Tell to ambari we want to install this component on new_hostcurl -u $AMBARI_USER:$AMBARI_PASSWORD -H "X-Requested-By:ambari" -i -X POST http://$AMBARI_HOST:8080/api/v1/clusters/$CLUSTER_NAME/hosts/$MOVE_TO/host_components/JOURNALNODEORcurl -v -u admin:admin -H "X-Requested-By:ambari" -i -X POST -d '{"host_components" : [{"HostRoles":{"component_name":"JOURNALNODE"}}] }' http://<ambari-server&

Introduction If the AWS IAM roles are not configured correctly, and if you are running into issues using these IAM roles, the best way to troubleshoot it is by using the AWS CLI. This post contains the steps and the required setup for this. Steps If the data lake admin role is not configured correctly, you will see the following error during the data lake deployment.   Error: Failed to create HDFS directory., Command [CreateRangerAuditDir], with id [1546333865] failed: Failed to create HDFS directory s3a://<bucketname>/data/ranger/audit/hdfs., Command [RangerPluginCreateAuditDir].   To get to the root cause of the issue, check the S3 permissions on the IAM role. If you want to further troubleshoot, then you can assume the role from the ID Broker's instance profile and use AWS CLI to reproduce and troubleshoot the issue. To do this, use the following troubleshooting steps: Log in to the ID broker host as a Cloudbreak user using the pem file provided during the environ

The kubectl tool provides direct administrative access to the Kubernetes cluster underlying a CDE service, which is useful for troubleshooting, among other things. This video will demonstrate how to set up kubectl access. To enable kubectl, we will need a couple of prerequisites. We will need the kubeconfig file from the CDE service. We will need to get and authorize the IAM user, and then we will need to make sure that everything is set up correctly, both for kubectl and some other tools like k9s.   There are a couple of prerequisites for this operation, and we are not going to cover their installation in this video. You will need to follow the process relevant to your platform of choice to get the AWS CLI installed and configured with an IAM user. You will also need to get kubectl installed.   Key steps: Download the kubeconfig file from the CDE service Retrieve the ARN for the chosen IAM user Grant administrative access for the IAM user Verify kubectl access is successf

Infrastructure Virtual Warehouse Impala will be running under Kubernetes Pods. The user can find the Kubernetes cluster running in the AWS account. The pods will be running in one of the K8s nodes. The JDBC URL can be found from the Virtual Warehouse.  Example:     jdbc:impala://<IMPALA_Coordinator_URL>:443/default;AuthMech=3;transportMode=http;httpPath=cliservice;ssl=1;UID=xxxx;PWD=PASSWORD       The server name would be <IMPALA_Coordinator_URL> and the port is 443. Access to Cloudera Manager, other cluster UIs, and endpoints (such as JDBC) are always through the secure Knox gateway on port 443. Users are automatically logged in with their CDP credentials.   If you get "Error while trying to connect" message from the visualization tool, check if port 443 is allowed for Impala coordinator and Knox gateway server from the server where the visualization tool is hosted.   Have come across some visualization tools like DOMO; its co

Short Description: This article covers steps on adding a Spark 2 Dashboard to Grafana in Ambari-Metrics, in order to monitor Spark applications for detailed resource usage statistics. ArticleMonitoring Spark 2 Performance via Grafana in Ambari-Metrics Configuration of Advanced spark2-metrics-properties Adding a new data source to Grafana Building a Spark 2 dashboard in GrafanaPre-Requisites You are running HDP 2.6.4 or higher, with Spark2 and Ambari-Metrics installed. A standalone Grafana installation may be used instead of Ambari-Metrics. A time-series database with support for the graphite protocol. In this article, I will be use Graphite. Alternatively, Prometheus with the graphite-exporter plugin can be used depending on personal preference. Steps In order for Spark components to forward metrics to our time-series database, we need to add a few items to our configuration in Ambari -> Spark2 -> Configs -> Advanced spark2-metrics-p

SYMPTOM:Problem was seen when the client was running Metastore HA and both configured with org.apache.hadoop.hive.ql.txn.compactor.Initiator. this is what we observed from the logs of both metastore service// metastore 1 ERROR compactor.Worker (Worker.java:run(181)) - Caught an exception in the main loop of compactor worker lnxhdpap02.smrcy.com-33, MetaException(message:Unable to connect to transaction database com.mysql.jdbc.exceptions.jdbc4.MySQLTransactionRollbackException: Deadlock found when trying to get lock; try restarting transaction // metastore 2 ERROR txn.CompactionTxnHandler (CompactionTxnHandler.java:findNextToCompact(194)) - Unable to select next element for compaction, Deadlock found when trying to get lock; try restarting transaction ROOT CAUSE:org.apache.hadoop.hive.ql.txn.compactor.Initiator is a thread that runs inside the metastore service to manage compactions of ACID tables. There should be exactly 1 instance of this thread (even with multiple Thrift services).WO

This article will describe the steps to access the remote CDP cluster HBase data from another CDP cluster using Spark in Kerberized environment. Assume we have two clusters say 'Cluster_A' and 'Cluster_B'. 'Cluster_A' is having the HBase data. 'Cluster_B' is having Spark. Now, we are trying to access the HBase data available in 'Cluster_A' using Spark from 'Cluster_B'. Prerequisites: Both clusters 'Cluster_A' and 'Cluster_B' with keytabs need to have the same REALM. Follow these steps:  Login to 'Cluster_A' edge node Obtain the Kerberos ticket using kinit:kinit -kt <key_tab_file> <principal_name>​ Login to the Hbase shell and create the 'person' table:ranga]# hbase shellhbase(main):001:0> listTABLE0 row(s)hbase(main):013:0> create 'person', 'p'Created table personTook 8.2307 seconds=> Hbase::Table - personhbase(main):014:0> put 'person',1,'p:id','1'Took 0.0173 secondshbase(main):015:0> put 'person',1,'p:name','Ranga Reddy'Took 0.0045 secondsh

Would you like to experiment with CDP Private Cloud Base (formerly known as CDP Data Center), OpenShift 4, and YuniKorn, but all you have is a Bare Metal server or laptop?   The answer to this question lies In this GitHub link (includes videos as well). This link includes the following how-tos:   Automate and speed up the installation of CDP Private Cloud Base (formerly known as CDP Data Center) in a dockerless container Automate the install of OpenShift 4.5+ Deploy CDP Private Cloud components onto an existing on-premises OpenShift 4.5+ / Kubernetes cluster Experiment with YuniKorn ( on OpenShift 4.5+ to unleash the power of resource scheduling for running Big Data & ML on Kubernetes(  All this on one Baremetal server!   DISCLAIMER: This is for lab and learning purposes only.  

In this video, we’ll go over how you can use both Cloudera Public Cloud to both Ingest data through Cloudera Data Engineering as well as explore it through Hue and Impala within Cloudera Data Warehouse. You'll see how easy it is to run queries that give you insight into your data and how you can use a built-in data visualization tool to create a dashboard to share your results. More information: Link to related tutorial: Data Exploration and Reporting with Cloudera Data Warehouse  Link to related meetup: Data Exploration and Dashboard Creation in a Cloud-Based Data Warehouse  Cloudera user's page www.cloudera.com/users

In this article, you'll learn how to: Build and run a Cloudera CDP Private Cloud Base (formerly known as CDP Data Center) trial environment in a dockerless container Automate the install of OpenShift 4.5+ with YuniKorn scheduler for BigData and ML on your laptop, virtual machine or Bare Metal server using CodeReady containers Want to experiment with Cloudera CDP Private Cloud Base (formerly known as CDP Data Center)? You can set up a trial environment in a container as shown in this Github link. The trial environment also includes a wizard that provides step-by-step guidance for installing CDP Private Cloud components onto an existing on-premises Kubernetes/OpenShift cluster.So, would you also like to automate the install of OpenShift 4.5+ with YuniKorn scheduler for BigData and ML on your laptop, virtual machine or Bare Metal server using CodeReady containers?   Find out how to do this at the above Github link.

NiFi for XML / RSS / REST Feed IngestI want to retrieve the status from various Cloud providers and services, including Cloudera, AWS, Azure, and Google. I have found many of the available status APIs will return XML/RSS. We love that format for Apache NiFi, so let's do that.Note: If you are doing development in a non-production environment, try the new NiFi 1.13.1. If you need to run your flows in production on-premise, private cloud, or in the public cloud, then use Cloudera Flow Management.I have separated the processing module "Status" from the input, so I can pass in the input anyway I want. When I move this to a K8 environment, this will become a parameter that I pass in. Stay tuned to Cloudera releases.The flow is pretty simple to process RSS status data. We call the status URL and in the next step easily convert RSS into JSON for easier processing. I split these records and grab just the fields I like.I can easily add additional fields from my metadata for unique

While running sandbox-hdp docker container on Ubuntu 20.04 LTS, I came across a problem and found a solution, which is pretty simple: Download .zip file with .sh file for build docker container from here. Unzip the file Change line in docker-deploy-hdp<version>.shfromdocker run --privileged --name $name -h $hostname --network=cda --network-alias=$hostname -d "$registry/$name:$version" to docker run --privileged -m=<at least 12g works for me> --name $name -h $hostname --network=cda --network-alias=$hostname -d "$registry/$name:$version"   Now, you can run a script, but probably you will need to re-run the second part after line #Deploy the proxy container. To make sure, run docker ps and check if sandbox-proxy container is running; if not, just run the second part of the script. Like I said, for me that works, hope it will work for you.