Community Articles

ObjectiveThis tutorial walks you through the process of installing the Docker version of the HDP 2.5 Hortonworks Sandbox on a Mac. This tutorial is part one of a two part series. The second article can be found here:HCC ArticlePrerequisitesYou should already have installed Docker for Mac. (Read more here Docker for Mac)You should already have downloaded the Docker version of the Hortonworks Sandbox (Read more here Hortonworks Sandbox)ScopeThis tutorial was tested using the following environment and components:Mac OS X 10.11.6HDP 2.5 on Hortonworks Sandbox (Docker Version)Docker for Mac 1.12.1NOTE: You should adjust your Docker configuration to provide at least 8GB of RAM. I personally find things are better with 10-12GB of RAM. You can follow this article for more information: https://hortonworks.com/tutorial/sandbox-deployment-and-install-guide/section/3/#for-macSteps1. Ensure the Docker daemon is running. You can verify by typing: $ docker imagesYou should see something similar

   Apache Airflow and Spark running on K8s provide unreasonable scalability.  Cloudera Machine Learning also runs on K8s and often the ML users leverage Apache Airflow to orchestrate their pipelines. Apache Airflow within Cloudera Data Engineering now has the capability to orchestrate CML jobs via the new Airflow CML Operator.  This time instead of writing up a detailed article, I recorded a live demo. Assets used in the demoCML Operatorhttps://github.com/curtishoward/cml_operator/Airflow Daghttps://github.com/sunileman/cde-airflow-branch-operatorDemohttps://www.youtube.com/watch?v=n9o_QimnggoEnjoy!

Article Short Description: This Tutorial describes how to add partition field to the content of the flowfile, create dynamic partitions based on flowfile content and store data into HDFS directory using NiFi. PartitionRecord Processor Params: Record Reader : Specifies the Controller Service to use for reading incoming data Record Writer: Specifies the Controller Service to use for writing out the records. Dynamic Properties: We need to add at least one property name and value(recordpath) that will be evaluated on each record of the flowfile. Each record will be grouped with the other like records in the group. **Note** No attribute will be added if the value returned for the RecordPath is null or is not a scalar value (i.e., the value is an Array, Map, or Record). Flow:   Flow Overview: 1. GenerateFlowfile //to generate csv data2. UpdateAttribute //add schema name attribute to the flowfile3. Update Record //add partition field to the flowfile content4. Partition Record //cre

Introduction This article is the final part in the series Scaling the Namenode (See  part 1, part 2, part 3 and part 4) In part 4 we discussed about monitoring Namenode Logs for Excessive Skews. In this part, we will look at a few optimizations around logging, access check, and block reports. Audience This article is for Hadoop administrators who are familiar with HDFS and its components. Audit log specific operations only when debug is enabled. By default, the following property is set to blank so none of the Namenode operations are restricted from making an entry into Audit log. Operations like getfileinfo results in fetching the metadata associated with a file, and in a large/read-heavy cluster, it can generate too much audit log. So, it is recommended to audit log getfileinfo only when audit log debug is enabled.   Change in hdfs-site.xml     <property> <name>dfs.namenode.audit.log.debug.cmdlist<\nam

A question about CDSW was asked by a customer. Question: We connect to external databases in CDSW. We found that some of the client IPs of the database are not in our LAN CIDR, and these IPs do not belong to the Node IP of CDSW. Why is this happening?   Response: CDSW uses Kubernetes as its infrastructure. Kubernetes maintains the IP CIDR of Pod and Service internally.When the Kubernetes workload communicates with the outside, it needs to pass the Service abstraction layer. The Service layer mainly implements packet forwarding between Node and Pod through NAT. Service has different modes. There are different ways to deal with the source IP NAT in different modes. Service may retain the source (Node) IP. This nature may be related to your question.For details, please refer to the official Kubernetes document: Using Source IP.   In addition, if you want to test what the source IP is when accessing an external database from the Pod in the CDSW Kubernetes cluster, you can refer

Many technologists ask "What is the easiest way to import data into Hive?" It's true, you can write a Sqoop job (making sure YARN containers are tuned properly in regards to # of mappers, the size of the mappers, and the sort allocated memory). And you'll deal with a two step process of moving the RDBMS tables into plain text and then creating an ORC table with an INSERT INTO statement.... I don't call this easy. What if there were a tool where you could simply drag-and-drop processes (processors) on a canvas and connect them together creating a workflow? There is a tool for this called NiFi! Before we go any further, I'll assume you know and understand NiFi - what is a processor, scheduling a processor, what is a connection (relationship), what are different relationship types, etc:. Small-Medium RDBMS tablesThis article will cover small to medium sized RDBMS tables. This means I can run a single select * from myrdbmstable;without returning millions of rows (where we're taxin

Accessing Kudu tables from Spark on the Cloudera Data Platform (CDP) is a common integration pattern for real-time analytic workloads that require fast inserts and updates while at the same time enabling efficient columnar scans across a single storage layer.   This article describes Kudu integration for Spark jobs running in the Cloudera Data Engineering Service by using the Kudu storage defined as part of a CDP Datahub cluster. Kudu Cluster We'll first configure the Kudu cluster. This can be done by creating a Datahub cluster of type Real-time Data Mart: Once the cluster has been provisioned successfully, you may need to add your IP address to the firewall rules for the cluster hosts in order to access some of the cluster's interfaces.  For example, with Amazon Web Services (AWS), a link to the EC2 instance configuration page is provided on the Datahub cluster page (on the EC2 instance page, locate the Security configuration and edit Security Group > Inbound Rule

Introduction In this article, I'll show how to stream data into CDP Public Cloud using Cloudera Dataflow/Streaming Datahub and query the data using Cloudera Data Warehouse. Pre-Requisites For this exercise you'll need Cloudera Data Platform with: Cloudera Data Warehouse; Datahub Flow Management; Datahub Streams Messaging; This exercise and flow are based on the sensor data/pipeline in edge2ai-workshop, but it is in a modified version that I'll share in another repository. 1. Create the Streaming Table in Cloudera Data Warehouse For this exercise we need two virtual warehouses: A Hive virtual warehouse: Used only to perform the compaction process (Streaming data ingest). For this example, I've named it latam-hive; A Unified Analytics Virtual Warehouse: Used to query data and visualization. Unified Analytics is a very exciting new feature for CDW customers at no extra fee!. You can learn more here. For this example, I've named it latam-vw; Figure 1: Virtual Warehouses used on

Short Description:Describes ways to manually regenerate keytabs for services through Ambari REST API ArticleMake sure KDC credentials are added to Ambari credentials store. You can follow this Article to perform. Once KDC credentials are added. You can use the below Ambari's REST API to regenerate keytabs.curl -H "X-Requested-By:ambari" -u <Ambari_Admin_username>:<Amabri_Admin_password> -X PUT -d '{ "Clusters": { "security_type" : "KERBEROS" } }' http://<Ambari_HOST>:8080/api/v1/clusters/<Cluster_Name>/?regenerate_keytabs=allExample : curl -H "X-Requested-By:ambari" -u admin:admin -X PUT -d '{ "Clusters": { "security_type" : "KERBEROS" } }' http://172.26.108.142:8080/api/v1/clusters/vinod/?regenerate_keytabs=all&ignore_config_updates=trueOnce the Keytabs are regenerated it requires Service restart to use the newly generated keytabs.

Pre-requisites: 1. You need to have the correct passwords2. The PrivateKey of the CA is required to sign off all the agent certificates.3. Password for the pass.txt file will be present in clear-text format. And currently, there is no way to encrypt it. Steps: Back up and delete all the crt and csr files that start with hostname at /var/lib/ambari-server/keys.tar -cvzf keys_directory.tar.gz /var/lib/ambari-server/keys/* Empty /var/lib/ambari-server/keys/db/index.txt filemv /var/lib/ambari-server/keys/db/index.txt /tmp Delete any certificates under /var/lib/ambari-server/keys/db/newcerts/mv /var/lib/ambari-server/keys/db/newcerts/* /tmp Configure CA-signed certificate artifacts in Ambari Server by: Copy the CA-signed certificate, key files to /var/lib/ambari-server/keys/ e.g. rootca.crt, root_server.key Create PKCS keystore file from your certificate and key files.openssl pkcs12 -export -in rootca.crt -inkey /var/lib/ambari-server/keys/root_server.ke

Objective This article will share what options a user has when they have lost or forgot the password set for the nifi.sensitive.props.key in the nifi.properties file. What is the nifi.sensitive.props.key?---------------------------------------------- The nifi.sensitive.props.key is used to encode any NiFi component added to the NiFi's canvas that has a sensitive property configured with a value (passwords). These "enc{...}" strings are set in the flow.xml.gz that is generated by NIFi, which includes everything dataflow-related set through the NiFi UI. How the nifi.sensitive.props.key gets set has varied over the history of NiFi. In some releases, NiFi would automatically generate a random password when the user did not set the property when NiFi was first launched. The current release requires that a user set this property or NiFi will not start. The nifi.sensitive.props.key is coupled closely with the flow.xml.gz. The flow.xml.gz file can be migrated to other

Objective This tutorial walks you through how to install and setup a local Hortonworks Registry to interact with Apache NiFi.Environment This tutorial was tested using the following environment and components: Mac OS X 10.11.6 MySQL 5.7.13 Apache NiFi 1.3.0 Hortonworks Registry 0.2.1 Note: The record-oriented processors and controller services used in the demo flow of this tutorial were in introduced in NiFi 1.2.0. As such, the tutorial needs to be done running Version 1.2.0 or later. Currently, Hortonworks Registry 0.2.1 is the version compatible with NiFi 1.3.0.Environment ConfigurationHortonworks Registry Installation Download the 0.2.1 Registry release: hortonworks-registry-0.2.1.tar.gz Extract the tar: tar xzvf hortonworks-registry-0.2.1.tar.gzMySQL Database Setup Login to your MySQL instance and create the schema registry database and necessary users and privileges: unix> mysql -u root -punix> Enter password:<enter>mysql> create database schema_regis

With the new runtimes feature available for both CML and CDSW, it is now possible to make better use of the remote editing capabilities with VS Code. To use this feature you will need VS Code installed locally and a CML/ CDSW instance that supports runtimes, and has the remote editing enabled (it is enabled by default but it can be disabled in the Admin settings) and that you can reach the server from the remote point you're connecting from. If you have all of that enabled, you are good to go. Step 1 - Configure cdswctl CLI For the first step, you will need to have a copy of the cdswctl command line tool on your local machine. You can get the cli directly from the CML/CDSW instance by going to User Settings > Remote Editing. The installation process is documented here and you need to get a version that works for your local OS. I'm on a Mac and I have the cdswctl cli in my /usr/local/bin directory:   % which cdswctl/usr/local/bin/cdswctl   To connect to your CML/CDSW

To demonstrate what is happening here, see these steps and output.Create a database at a custom location: 0: jdbc:hive2://hostname.cloudera.co> create database grandtourlocation 'hdfs://hostname.cloudera.com:8020/bdr-test/grandtour.db';<TRUNCATED>INFO : Executing command(queryId=hive_20210818180811_d96dd7f8-2713-440f-8e9f-8eebd2954d05): create database grandtour location 'hdfs://hostname.cloudera.com:8020/bdr-test/grandtour.db'INFO : Starting task [Stage-0:DDL] in serial modeINFO : Completed executing command(queryId=hive_20210818180811_d96dd7f8-2713-440f-8e9f-8eebd2954d05); Time taken: 0.031 secondsINFO : OKNo rows affected (0.083 seconds) Create a table and insert a record: 0: jdbc:hive2://hostname.cloudera.co> use grandtour;<TRUNCATED>INFO : Completed executing command(queryId=hive_20210818180835_f365f042-e2a9-4f53-a9ed-317d21dcfc07); Time taken: 0.008 secondsINFO : OKNo rows affected (0.065 seconds)0: jdbc:hive2://hostname.cloudera.co> creat

In this example, I am importing encryption keys from HDP 3.1.5 cluster to an HDP 2.6.5 cluster.Create key "testkey" in Ranger KMS HDP 3.1.5 cluster with steps: List and Create Keys. In HDP 3.1.5, the current master key is:Encryption Key:Create an encryption zone with the "testkey": [hdfs@c241-node3 ~]$ hdfs crypto -createZone -keyName testkey -path/testEncryptionZoneAdded encryption zone /testEncryptionZone List to confirm the zone and keys: [hdfs@c241-node3 ~]$ hdfs crypto -listZones/testEncryptionZone testkey Export the keys:Log in to KMS hostexport java homecd /usr/hdp/current/ranger-kms./exportKeysToJCEKS.sh $filenameThe output will look as follows: [root@c241-node3 ranger-kms]# export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.292.b10-1.el7_9.x86_64/jre[root@c241-node3 ranger-kms]# ./exportKeysToJCEKS.sh /tmp/hdp315keys.keystoreEnter Password for the keystore FILE :Enter Password for the KEY(s) stored in the keystore:Keys from Ranger KMS Database

Introduction Thrift proxy is a modern micro-service framework when comparing to other existing frameworks such as SOAP/JSON-RPC/Rest proxy. The Thrift proxy API has a higher performance, is more scalable, and is multi-language supported. (C++, Java, Python, PHP, Ruby, Perl, C#, Objective-C, JavaScript, NodeJs, and other languages). The application can interact with HBase via Thrift proxy. This article will discuss how to use correct libraries and methods to interact with HBase via Thrift proxy.  Outline The basic concept of Thrift proxy and how the thrift language bindings are generated. How Python thrift functions align with the correct settings of HBase configurations from Cloudera Manager. Sample client codes in security disabled/ enabled HBase clusters. Some known bugs when using TSaslClientTransport with Kerberos enabled in some CDP versions. The basic concept of Thrift proxy and how the Thrift cross-language bindings are generated The Apache Thrift library provides cross-

In Hortworks HCP they use an ElasticSearch Mpack for version 5.x of Elasticsearch and Kibana:http://public-repo-1.hortonworks.com/HCP/centos7/1.x/updates/1.6.0.0/tars/metron/elasticsearch_mpack-1.6.0.0-7.tar.gzI have taken this and expanded it to include version 6.3.2 of ElasticSearch, Logstash, Kibana, FileBeat, and MetricBeat.My cluster is 6 nodes.ElasticSearch is installed on Nodes 4,5,6(4 Master & 5,6 Data Nodes).Logstash is Installed on Node 3.FileBeat & MetricBeat are installed on all 6 nodes.Kibana is installed on Node 4.The rest of the cluster is configured normally for a minimal Install.DownloadsOrginal HCP Mpackelasticsearch-mpack-1600-7.tar.gzMy MPack 6.3.0 - Version Change + Logstash + Beatselasticsearch-mpack-2500-9.tar.gzMy Mpack 6.3.2 - Version Change + Logstash (multi-node & jvm settings config) + sudo non root capabilitieselasticsearch-mpack-2600-9.tar.gzInstallation Steps1. Deliver Management Pack tar.gz to local filesystem on Ambari-Serverupload to hdfs v

CEM What is Cloudera Edge Management (CEM)?Refer to What is Cloudera Edge Management. There is no compatibility matrix between CEM and HDF because these are independent products and MiNiFi agents from CEM will be perfectly fine to send data into HDF/NiFi, if they need to. SMM From Install or Upgrade Ambari, HDF, and HDP document, we can see that the rge cluster must be managed by Ambari 2.7.x. It can be an HDF 3.3.x, 3.4.0 or 3.4.1.1 cluster, or an HDP 3.1 or 3.1.1 cluster. And in Set up DP Platform document, you can see that before installing Streams Messaging Manager, you must first install or upgrade to DataPlane Platform 1.2.x   My understanding is: SMM is based on Dataplane Platform, so if you installed the latest Dataplane Platform, there is no way you cannot install the latest SMM. So, I believe DPS (DataPlane Platform) 1.3.1 or 1.3.0 and SMM 2.0.0 are compatible.   Actually, if you look at Installing DataPlane document, it looks to me th

   My Kubernetes series  (Part1, Part2) was strictly focused on MiNiFi on K8S.  NiFi and MiNiFi may communicate over Site 2 Site; however, often the pattern is to leverage Kafka for a clean message handoff.  NiFi within these patterns is generally the central router and transformer of messages.  Think of it like "FedEx" for data.   Till now most have deployed NiFi on bare metal or VMs.  Natural evolution kicks in. Deploy NiFi on k8s and yes it's super simple.  In this article I will demonstrate how to deploy both a NiFi and ZooKeeper cluster (none being a single pod!) on Azure Kubernetes Service (AKS); however, all artifacts may be leveraged to deploy on virtually any kubernetes offering. PrerequisitesSome knowledge of Kubernetes and NiFiAKS / k8s Clusterkubectl cliNiFi image available in a registry (ie dockerhub) DeploymentAll the contents in the ymls below can be placed into single file for deployment.  F

Introduction Phoenix is a popular solution to provide OLTP and operational analytics on top of HBase for low latency. Hortonworks Data Platform (HDP), Cloudera are the most popular platforms for Phoenix to interact with HBase. Nowadays, many customers choose to migrate to Cloudera Data Platform to better manage their Hadoop clusters and implement the latest solutions in big data. This article discussed how to migrate Phoenix data/index tables to the newer version CDP Private Cloud Base. Environment Source cluster HDP 2.6.5 , HDP 3.1.5 Target cluster CDP PvC 7.1.5, CDP PvC 7.1.6, CDP PvC 7.1.7 Migration steps The SYSTEM table will be automatically created when Phoenix-sqlline initially starts. It will contain the metadata of Phoenix tables. In order to show Phoenix data/index tables in the target cluster, we need to migrate SYSTEM tables from the source cluster as well. Stop Phoenix service on the CDP clusterYou can stop the service on Cloudera Manager > Services > Phoenix

Introduction This article is part of a series (see part 1, part 2, part 3 and part 5). Here we look at how to avoid some common NameNode performance issues. Audience This article is for Apache Hadoop administrators who are familiar with HDFS and its components. If you are using Ambari or Cloudera Manager, you should know how to manage services and configurations. Avoid Disk Throughput Issues The performance of the HDFS NameNode depends on being able to flush its edit logs to disk relatively quickly. Any delay in writing edit logs will hold up valuable RPC handler threads and can have cascading effects on the performance of your entire Hadoop cluster. You should use a dedicated hard disk i.e. dedicated spindle for the edit logs to keep edits flowing to the disk smoothly. The location of the edit log is configured using the dfs.namenode.edits.dir setting in hdfs-site.xml. If dfs.namenode.edits.dir is not configured, it defaults to the same value as dfs.namenode.name.dir. Exampl

Hi All, I have faced some issues. We want to dedicate one of the Zeppelin node to a different agency. So, if they create any notebook, those are getting synced to every one after a restart because we are using the central storage system as HDFS. So, Zeppelin nodes are picking all notebooks and interpreter settings from the HDFS single path. To avoid this, we isolated the settings and configurations by changing Zeppelin to use the local storage, so each node will act independently. For the same, the following are helpful steps in case anyone is facing the same. Thanks.   Verify the procedures in a lower environment before executing the same in production.sudo -u <zeppelin-user> mkdir -p /var/lib/zeppelin/conf /var/lib/zeppelin/notebookkinit <zeppelin-user>hdfs dfs -get "/user/zeppelin/notebook/*" /var/lib/zeppelin/notebookhdfs dfs -get /user/zeppelin/conf/notebook-authorization.json /var/lib/zeppelin/conf​   In the Zeppelin configuration page, search for an

In this tutorial, we will learn how to create Apache Ozone volumes, buckets, and keys. After that, we will see how to create an Apache Hive table using Apache Ozone, and finally how we can insert/read the data from Apache Spark. Ozone Create the volume with the name vol1.# ozone sh volume create /vol121/08/25 06:23:27 INFO rpc.RpcClient: Creating Volume: vol1, with root as owner. Create the bucket with the name bucket1 under vol1.# ozone sh bucket create /vol1/bucket121/08/25 06:24:09 INFO rpc.RpcClient: Creating Bucket: vol1/bucket1, with Versioning false and Storage Type set to DISK and Encryption set to false     Hive Launch the beeline shell. Create the employee table in Hive. Note: Update the om.host.example.com value.   CREATE DATABASE IF NOT EXISTS ozone_db;USE ozone_db;CREATE EXTERNAL TABLE IF NOT EXISTS `employee`( `id` bigint, `name` string,

Summary of the article I have encountered issues of the following scenario several times: A customer removes a host from a cluster managed by Cloudera Manager and adds the host to another cluster managed by Cloudera Manager. At this time, problems often occur. These problems are often caused by the old cluster files remaining on the host, and these old files cause the new Cloudera Manager to be unable to control the host normally. So I verified what files will be generated on a host after adding it to a CDP cluster managed by Cloudera Manager.In other words, after removing a host from Cloudera Manager, what files do we need to delete manually? Introduction to the Test environment CDP Runtime version: CDP PvC Base 7.1.6 CM version: Cloudera Manager 7.3.1 Whether to enable Kerberos: Yes Whether to enable TLS: Yes Auto-TLS: Yes Auto-TLS Use Case: Use Case 1 - Using Cloudera Manager to generate an internal CA and corresponding certificates (Refer to Configuring TLS Encryption for Clo

A common request from CDE users is the ability to specify a timeout (or SLA) for their Spark job. This is easily configured by leveraging CDE's embedded Airflow sub-service, which provides a rich set of workflow management and scheduling features, along with Cloudera Data Platform (CDP-specific) operators such as CDEJobRunOperator and CDWOperator.   As a simple example, the steps below create a CDE Spark job, which is then triggered from a CDE Airflow Task, also making use of the execution_timeout Airflow Task parameter to specify an SLA after which the Spark job should be killed. We'll use the pi.py sample code as our CDE Spark job, and the following Airflow DAG definition (airflow_dag_with_task_timeout.py)   from datetime import datetime, timedeltafrom airflow import DAGfrom airflow.operators.dummy_operator import DummyOperatorfrom cloudera.cdp.airflow.operators.cde_operator import CDEJobRunOperatordefault_args = { 'owner': 'cdeuser1', 'retry_delay': timed