Community Articles

When a Hive replication policy executes in Replication Manager, the admin can monitor the details of the policy and its execution by checking the underlying command details on the Datalake cluster's CM. An admin can determine further details about the policy by checking “All Recent Commands”. The first step is the Export Metadata step that is performed on the source cluster. This setup in turn exports the data and metadata of the input Hive databases and tables specified in the policy. From the command details, the admin can find logs of the check metadata step. Next is the Import metadata step that updates the Hive database (and table) metadata for a given replication policy. Finally, the last step imports the Exported Sentry permissions into Ranger policies on the Data Lake cluster. If the above steps ran into an error, the admin can determine the specific error by checking the stderr logs of the specific step that ran into an error to troubleshoot the actual error.

A step-by-step guide that explains the process to register a CDH Cluster as a Classic Cluster is outlined below. This process currently outlines the CDH clusters that aren’t directly visible to the CDP control plane.    Admin needs to select the Classic Cluster tab in the CDP control plane and proceed further with the next steps: CDH Cluster is discovered by its internal IP address which is not visible to the outside world. The wizard provides step-by-step instructions on how to proceed further with the classic cluster registration process. Once the administrator installs the required packages, the admin can verify whether the service is installed and started properly by logging into the CM server host Run the status command to verify whether the service is running properly. Proceed further with the classic cluster registration process. Provide the Cloudera Manager login credentials here. Provide further details such as location, data center name (which is arbitrary) about t

1. Get the output from Ambari serverhttp://AMBARI-HOST:8080/api/v1/stacks/HDP/versions?fields=repository_versions/operating_systems/repositories/*,repository_versions/operating_systems/OperatingSystems/*&repository_versions/RepositoryVersions/repository_version=3.1.0.0-782. Copy the json of "operating_systems" outer block of step1 output and create repo.json file -it would looks like repo.json3. update repo.json file to reflect your changes/modifications.4. now run below API call to update.curl -insecure  -H "X-Requested-By:ambari" --user admin:admin --data @repo.json  -X PUT  http://AMBARI-HOST:8080/api/v1/stacks/HDP/versions/3.1/repository_versions/1Note: please make sure you include correct repo version id at the end. in this case it is 15. Now go and check in UI to see if those changes reflects.Note: This tried and tested in Ambari2.7.x

The Qualys tool reports vulnerabilities in ZooKeeper, even when the ZooKeeper security configuration is applied (HDP doc, CDP doc). There are two kinds of reports Qualys makes that are not addressable by Cloudera: The security guidance keeps several znodes in the affected services as world-readable, like so:/zookeeper/quota sasl:zookeeper:cdrwa,world:anyone:rAny world-readable znode will appear on the scanner, and will require an exception filed for it. This is the position of Qualys, as reported by our customers who use Qualys. The security guidance does not cover several services. The following components require no action according to our documentation: Calcite Knox MapReduce Spark Tez Zeppelin The Qualys tool will still report znodes owned by these services. Note: it is possible to harden the ACLs beyond the Best Practices recommendation in the documentation, and to harden the ACLs of services not covered in the Best Practices. However, Cloudera cannot provide what the

This article will cover the following:   The purpose of this article Problem Statement Troubleshooting Step-1: Use the kubectl describe pod command to view the details of the Pod of the Session. Step-2: Use kubectl describe node to view the status of all the nodes. Step-3: Verify the kubelet process argument. Step-4: Check the nodefs and imagefs recognized by kubelet. Summary The purpose of this article This article records the troubleshooting about CDSW unable to start the Session. The reason for the problem involved is that the remaining capacity of nodels and imagefs used by Kubelet has reached the threshold of evicting the node, resulting in the node being labeled as NoSchedule taint by Kubernetes. Problem Statement One of my customers reported that they found their CDSW failed to start sessions. The following is a screenshot of the failure to start a session on CDSW. Troubleshooting Use the Kubectl describe pod comman

This article will explain the following:   How to connect to CDSW database using kubectl exec? How to forward the database service of CDSW to the host? Change user's password if you cannot set up smtp service to change password First of all, we need to know that in CDSW/Kubernetes, we use SDN (Software Defined Network) to create a network that is only used by Pod/Service within the cluster. For example, the IP CIDR used by the CDSW nodes is 192.168.0.0/24. Then the Pod IP CIDR used by CDSW/Kubernetes will generally not overlap with 192.168.0.0/24 such as 100.66.0.0/16.    kubectl cluster-info dump | grep -m 1 cluster-cidr# "--cluster-cidr=100.66.0.0/16",   How to connect to CDSW database using kubectl exec? For how to connect to CDSW database, you can refer to this official documentation. CDSW_NAMESPACE="default" # All CDSW internal applications are running in "default" NameSpace.db_pod=$(kubectl get

Purpose of this article Our purpose is to replace the nodes of Kafka Broker. We have two nodes newly added to the CDP PvC Base cluster. We will migrate two Kafka Brokers that were originally in use to these two new nodes. This article uses Cloudera CDP official documentation as a guide. Summary Environment Steps of Broker Migration Step-1: Startup the new broker as a member of the old cluster Step-2: Stop both the new broker and the old broker that it is replacing Step-3: Change broker.id of the new broker to the broker.id of the old one both in Cloudera Manager and in data directory/meta.properties. Step-4: Startup the new broker. Environment Here are the currently in use Kafka Brokers: Broker Host IP 1546345005 c3669-node3.coelab.cloudera.com 172.25.36.133 1546345009 c3669-node4.coelab.cloudera.com 172.25.36.207 1546345001 c3669-node2.coelab.cloudera.com 172.25.36.206 Steps of Broker Migration Startup the new broker as a member of the old clus

Below are the high level requirements which are needed to connect to Secure Hbase cluster- hbase-client- Hbase config file- Kerberos config files and keytab for userPom file and Sample Code are given below:Java Class: (Change Paths for config files and kerberos related parameters):package com.hortonworks.hbase;import java.io.IOException;import org.apache.hadoop.conf.Configuration;import org.apache.hadoop.fs.Path;import org.apache.hadoop.hbase.HBaseConfiguration;import org.apache.hadoop.hbase.TableName;import org.apache.hadoop.hbase.client.Connection;import org.apache.hadoop.hbase.client.ConnectionFactory;import org.apache.hadoop.hbase.client.ResultScanner;import org.apache.hadoop.hbase.client.Scan;import org.apache.hadoop.hbase.client.Table;import org.apache.hadoop.security.UserGroupInformation;public class HBaseConnection { publicstatic void main(String ar[]) throws IOException { //System Properties (Change Path/Properties according to env) //copy krb5.conf from cluster System.set

In this article I'd like to focus on how to automatically remove Solr documents from a collection. I ran into a lot of questions and issues recently that all came back to TTL (time-to-live) of documents, e.g. nodes running out of space company policies requiring to remove old audit logs auto-purging etc. SOLR-5795 introduced a new UpdateProcesser called DocExpirationUpdateProcessorFactory, which allows us to add an expiration date to solr documents and making sure expired documents are removed automatically. How does it work? Every indexed document will get an expiration field, which contains the date and time at which the document will "expire". This field is calculated relative to the time it is indexed as well as by using the configuration provided by a field called _ttle_ (this is the default name, you can always rename it if you want). _ttl_ sets the lifetime of documents, e.g. +10DAYS, +2WEEKS,+4HOURS,... For example: Current Time is: 2016-10-26 20:14:00_ttl_ is define

  A number of customers want to run CDSW R sessions within the Jupyter Notebook.  This is unsuccessful within CDSW when a customer attempts to perform this and results in the following errors depending on how you execute Jupyter.   Execution haltedWARNING:root:kernel 8166ed20-6142-44d1-92b8-9a0ae11777a9 restartedError in ok_device(filename, ...) : X11 is not availableCalls: <Anonymous> ... evaluate -> dev.new -> do.call -> <Anonymous> -> ok_deviceExecution halted   By default the Jupyter R Kernel is not shipped with CDSW 1.5+.  In order to get this to work you will have to install the Jupyter R Kernel manually.  The instructions are below as follows:   - https://irkernel.github.io/installation/   Instructions on installing Jupyter R Kernel from source. 1/2) Installing from source ZMQ You'll need zmq development headers to compile pbdZMQ (Click your OS): LINUXOS XWINDOWS R packages Start R in the same te

Ingesting HL7 Data in NiFi Pre-requisites We will assume you have the following: Basic understanding of Linux CLI HDF cluster or sandbox with NiFi, Kafka, and HDFS installed and running Basic knowledge of connecting components in NiFi and configuring settings Overview Here's the finished product: This NiFi flow accomplishes the following: Read the data from a Kafka stream Format the data to comply with HL7 specifications Extract the HL7 attributes from the formatted content block Transform the attribute list into a JSON block Impose the data's natural hierarchy onto the JSON rather than leaving it as a flat-mapped structure Write the data to HDFS Here's an example of our input data: MSH|^~\&|XXXXXX||HealthOrg01||||ORU^R01|Q1111111111111111111|P|2.3|<cr>PID|||000000001||SMITH^JOHN||19700101|M||||||||||999999999999|123456789|<cr>PD1||||1234567890^LAST^FIRST^M^^^^^NPI|<cr>OBR|1|341856649^HNAM_ORDERID|000000000000000000|648088^Basic Metabolic Panel|||201501010001

Accessing AWS Cloudera Data Warehouse to query data on Azure Cloudera Data Warehouse Introduction Cloudera Data Platform enables in a single console to work with different public cloud providers. With this, you can have a true hybrid environment with only one admin console. Cloudera Data Warehouse is a public cloud service that allows fast analytics in your preferred cloud provider. In this article, I'll show how easy it is to connect between two Virtual Warehouses located in different cloud providers using Cloudera Data Warehouse. Scenario We're using two different cloud providers for Cloudera Data Warehouse: one in AWS with TPC-DS data and another in Azure with the same TPC-DS data. We'll use Hive ACID to update the customer table on Azure and merge it with the customer table in AWS. 1.Pre-Requisites 1.1 - Cloudera CDP Control Plane Access and Register two environments For this exercise, you will need access to the Cloudera Data Platform. More information can be accessed here. Also

Motivation:When Hadoop components (HDFS et. al.) are configured to connect to external sources like LDAP, the LDAP bind passwords need to be given in configuration file (core-site.xml) in clear text. For many enterprise environments, having password in clear text is not allowed and is often flagged as risk in Security Audits.This article teaches Hadoop administrators on how to secure these plaintext password in Hadoop configuration.Configuration Steps:1. Before starting, make sure that LDAP bind is working with plain text password. So, the HDFS should be configured with these settings in core-site.xml (the following values should be changed to match your LDAP/AD environment. These are working example values from my AD setup)hadoop.security.group.mapping=org.apache.hadoop.security.LdapGroupsMapping hadoop.security.group.mapping.ldap.base=ou=CorpUsers,dc=lab,dc=hortonworks,dc=nethadoop.security.group.mapping.ldap.bind.user=cn=ldap-reader,ou=ServiceUsers,dc=lab,dc=hortonworks,dc=neth

In order to convert JSON to XML, you can use ConvertRecord processor available as part of the NiFi tool (I worked on version: 1.11.4). Here are the steps to configure ConvertRecord processor to convert JSON input to XML output: Add a processor – ConvertRecord and right-click to “configure”: Click on the “Record Reader value” and select “Create New Service”. Select “JSONTreeReader” and click CREATE: Similarly, Click on the “Record Writer value” and select “Create New Service”. Select “XMLRecordSetWriter” and click CREATE: Final configuration of Convert Record should look like this: Click on “à to configure “JSONTreeReader” to configure details about the schema which should be read by the registry. The controller service form should open: Click on “configure icon” of “JSONTreeReader” to start configuring the JSON Reader and configure as per below screenshot: When you select “Create New Service” in the above screenshot, another form opens up. Fill it up based on below screenshot and cli

Following are the configurations for connecting Apache Ranger with LDAP/LDAPS. There's an important tool that will help identify some settings in your AD AD Explorer - Windows Sysinternals | Microsoft Docs. This configuration will sync LDAP users and link them with their LDAP groups every 12 hours, so later from Apache Ranger, you can give permission based on LDAP groups as well. For connecting using LDAPS, ensure you have the proper certificates added in the same server that contains the Ranger's UserSync service. Configuration Name Configuration Value Comment ranger.usersync.source.impl.class org.apache.ranger.ldapusersync.process.LdapUserGroupBuilder   ranger.usersync.sleeptimeinmillisbetweensynccycle 12 hour   ranger.usersync.ldap.url ldaps://myldapserver.example.com ldaps or ldap based on your LDAP security ranger.usersync.ldap.binddn myuser@example.com   ranger.usersync.ldap.ldapbindpassword mypassword   ranger.usersync.ldap.searchBase OU=

IntroductionFor a simple PySpark application, you can use `--py-files` to specify its dependencies. A large PySpark application will have many dependencies, possibly including transitive dependencies. Sometimes a large application needs a Python package that has C code to compile before installation. And, there are times when you might want to run different versions of python for different applications. For such scenarios with large PySpark applications, `--py-files` is inconvenient. Fortunately, in the Python world you can create a virtual environment as an isolated Python runtime environment. We recently enabled virtual environments for PySpark in distributed environments. This eases the transition from local environment to distributed environment with PySpark. In this article, I will talk about how to use virtual environment in PySpark. (This feature is currently only supported in yarn mode.)Prerequisites Hortonworks supports two approaches for setting up a virtual environment: vi

Note: Cloudera does not support Antivirus software of any kind. This article contains recommendations for excluding CDH components and directories from AV scans and monitoring. It is important to note that these recommendations do not apply to each service, and further, some services will have additional items to exclude which are unique to them. These details will be addressed in individual articles dedicated to the service in question. The three primary locations you will want to exclude from antivirus are: Data directories: These can be very large, and therefore, take a long time to scan; they can also be very write-heavy, and therefore suffer performance impacts or failures if the AV holds up writes. Log directories: These are write-heavy. Scratch directories: These are internal locations used by some services for writing temporary data, and can also cause performance impacts or failures if the AV holds up writes. In general, consider excluding the following directories and

Note: Cloudera does not support Antivirus software of any kind.This article contains recommendations for excluding CDP components and directories from AV scans and monitoring. It is important to note that these recommendations do not apply to each service, and further, some services will have additional items to exclude that are unique to them. These details will be addressed in individual articles dedicated to the service in question.The three primary locations you will want to exclude from Antivirus are:Data directories: These can be very large, and therefore, take a long time to scan; they can also be very write-heavy, and therefore suffer performance impacts or failures if the AV holds up writes.Log directories: These are write-heavy.Scratch directories: These are internal locations used by some services for writing temporary data, and can also cause performance impacts or failures if the AV holds up writes.In general, consider excluding the following directories and all of their s

How to read fsimage:We can use Offline Image Viewer tool to view the fsimage data in a human readable format. Sometimes, this becomes more essential to analyse the fsimage to understand the usage pattern, how many 0 bite files are created, what is the space consumption pattern and is the fsimage corrupt. Donwload the fsimage: hdfs dfsadmin –fetchImage /fsimage This will download the latest fsimage from Name node 16/12/27 05:40:43 INFO namenode.TransferFsImage: Opening connection to

The hive testbench consists of a data generator and a standard set of queries typically used for benchmarking hive performance. This article describes how to generate data and run a query in using beeline and Hive 2.0 with and without LLAP. It also shows how to use explain to see the difference in query plans.If you don't have a cluster already configured for LLAP, you can provision one in AWS using Hortonworks Cloud. See this article for instructions on how to provision a 2.5 tech preview with LLAP enabled.1. Log into the master node in you cluster where Hive is installed. If you used Hortonworks Cloud to create your instance, locate the node with a name ending in master. The ssh command is shown next to the master instance. If you are logging in from a linux host, click on the icon to the right of the ssh command to select the command text and copy the command. In the linux shell, change to the directory containing your AWS key .pem file and the run the copied command.If yo

Introduction In Cloudera Data Engineering, you may have a need to move the jobs from one virtual cluster to another. This can be for testing purposes or to move the jobs into the latest CDE versions. This can be easily achieved by the backup & restore process and is well documented here. The purpose of this article is to provide additional tools to simplify this and demonstrate the process. The example shown here demonstrates this process within the same CDE service, but you can use the same approach to move across CDE services. Steps Ensure that CDP CLI is set up on the system with the required access to list and describe CDE services & virtual clusters. Also, have your workload password handy as this is required for the export and import process. Download export & import scripts using the following commands.wget https://raw.githubusercontent.com/karthikeyanvijay/cdp-cde/main/export-import-cde-jobs/cde-vc-jobs-export.shwget https://raw.githubusercontent.com/karthikeyanvi

Overview: To replace an old Namenode hardware with new hardware with the same hostname/ IP address on a live production Hadoop cluster in a large-scale environment.   Problem:   When the hardware is due for refresh due to End of Service Life (EOSL) by the vendor, we will not get service support for this product such as repairs, limited tech support, parts availability, etc. It is time for replacing the servers with new systems. There is no straightforward method published by any Hadoop vendor for this task for in-place replacement or swap of hardware with the same hostname and IP address, so that cluster does not require any downtime or restart of Hadoop services.   Option #1 - Moving highly available NN, failover controller & JN roles using Migrate Roles wizardMigrate Roles wizard offers the capability to move roles of a highly available HDFS service from one host to another. It can be used to move Namenode, Journal Node, and Failover Controller roles to new h

Phone Tracking with OwnTracks and Apache NiFi 1.xOwnTracks is an Open Source project which provides an iOS and an Android app with which your smartphone records its current location.I installed the OwnTracks application for iOS and it let's you specify your own REST JSON server to receive calls. I added Apache NiFi. It can also send via MQTT direct to mosquittio on prem to NIFI or to CloudMQTT. You just need to enter your Apache NiFi address and port. Tell NiFi to listen for HTTP on port 9179 for the phone push, allow it to use GET, POST and PUT. No coding required.Respond to the phone with HTTP Status Code 200 and use the context map to connect the HTTP flow.We pull out the attributes from the JSON Flow File.Store our Phone Data in Apache Phoenix on HBaseupsert into phone (uuid,battery,longitude ,accelerator ,velocity,vac ,latitude , tvalue ,connection , tst , altitude , messagetype , tid, httpremotehost, useragent, filename, datetime)values ('${'uuid'}','${'battery'}','${'lo

Introduction In this article, we will learn how to query the Hive tables data by using column names with regular expressions in Spark. Assume we have a table with column names like col1, col2, col3, col4, col5, etc. If we want to select the data, we will use queries like select col1, col2, col3, col4, col5 from the table. Instead of specifying col1, col2, col3, col4, col5, we can use regular expressions while selecting columns like select `col.*` from table. From Hive, this feature is supported from Hive 0.13.0 onwards. By default, this feature is disabled, and in order to enable it, we need to use set hive.support.quoted.identifiers=none. From Spark, this feature is supported from Spark 2.3.0 onwards. By default this feature is disabled, and in order to enable it, we need use set spark.sql.parser.quotedRegexColumnNames=true. Steps Hive Create the Hive table and insert the data:beeline -u jdbc:hive2://host:port> create database if not exists regex_test;&g

Started working on the Cloudera Public Cloud in Azure and couldn't bare a thought that we would need to pay for all those resources that are not being used in non-prod environments. My goal was to be able to easily stop/start environment(s) based on the time, so resources (compute and money) can be utilized wisely. I haven't used Cloudera Manager API because of 2 reasons: I couldn't authenticate using the workload username and password. Kept getting empty responses. Only intercepted token from the browser allowed to make a successful request. With a multitude of clusters, it is daunting to invoke a multitude of separate APIs. Unfortunately, cdpcli is still a bit immature and doesn't have a good API in Python yet. I didn't want to use Java, which has what seems to be a slightly better API, for such a trivial task. I am running Azure Function App with a Timer trigger. I had to use the ElasticPremium tier because I wanted the function to run on the node in the same subnet as my Clo