Resources

Below are the high level requirements which are needed to connect to Secure Hbase cluster- hbase-client- Hbase config file- Kerberos config files and keytab for userPom file and Sample Code are given below:Java Class: (Change Paths for config files and kerberos related parameters):package com.hortonworks.hbase;import java.io.IOException;import org.apache.hadoop.conf.Configuration;import org.apache.hadoop.fs.Path;import org.apache.hadoop.hbase.HBaseConfiguration;import org.apache.hadoop.hbase.TableName;import org.apache.hadoop.hbase.client.Connection;import org.apache.hadoop.hbase.client.ConnectionFactory;import org.apache.hadoop.hbase.client.ResultScanner;import org.apache.hadoop.hbase.client.Scan;import org.apache.hadoop.hbase.client.Table;import org.apache.hadoop.security.UserGroupInformation;public class HBaseConnection { publicstatic void main(String ar[]) throws IOException { //System Properties (Change Path/Properties according to env) //copy krb5.conf from cluster System.set

In this article I'd like to focus on how to automatically remove Solr documents from a collection. I ran into a lot of questions and issues recently that all came back to TTL (time-to-live) of documents, e.g. nodes running out of space company policies requiring to remove old audit logs auto-purging etc. SOLR-5795 introduced a new UpdateProcesser called DocExpirationUpdateProcessorFactory, which allows us to add an expiration date to solr documents and making sure expired documents are removed automatically. How does it work? Every indexed document will get an expiration field, which contains the date and time at which the document will "expire". This field is calculated relative to the time it is indexed as well as by using the configuration provided by a field called _ttle_ (this is the default name, you can always rename it if you want). _ttl_ sets the lifetime of documents, e.g. +10DAYS, +2WEEKS,+4HOURS,... For example: Current Time is: 2016-10-26 20:14:00_ttl_ is define

  A number of customers want to run CDSW R sessions within the Jupyter Notebook.  This is unsuccessful within CDSW when a customer attempts to perform this and results in the following errors depending on how you execute Jupyter.   Execution haltedWARNING:root:kernel 8166ed20-6142-44d1-92b8-9a0ae11777a9 restartedError in ok_device(filename, ...) : X11 is not availableCalls: <Anonymous> ... evaluate -> dev.new -> do.call -> <Anonymous> -> ok_deviceExecution halted   By default the Jupyter R Kernel is not shipped with CDSW 1.5+.  In order to get this to work you will have to install the Jupyter R Kernel manually.  The instructions are below as follows:   - https://irkernel.github.io/installation/   Instructions on installing Jupyter R Kernel from source. 1/2) Installing from source ZMQ You'll need zmq development headers to compile pbdZMQ (Click your OS): LINUXOS XWINDOWS R packages Start R in the same te

Ingesting HL7 Data in NiFi Pre-requisites We will assume you have the following: Basic understanding of Linux CLI HDF cluster or sandbox with NiFi, Kafka, and HDFS installed and running Basic knowledge of connecting components in NiFi and configuring settings Overview Here's the finished product: This NiFi flow accomplishes the following: Read the data from a Kafka stream Format the data to comply with HL7 specifications Extract the HL7 attributes from the formatted content block Transform the attribute list into a JSON block Impose the data's natural hierarchy onto the JSON rather than leaving it as a flat-mapped structure Write the data to HDFS Here's an example of our input data: MSH|^~\&|XXXXXX||HealthOrg01||||ORU^R01|Q1111111111111111111|P|2.3|<cr>PID|||000000001||SMITH^JOHN||19700101|M||||||||||999999999999|123456789|<cr>PD1||||1234567890^LAST^FIRST^M^^^^^NPI|<cr>OBR|1|341856649^HNAM_ORDERID|000000000000000000|648088^Basic Metabolic Panel|||201501010001

Accessing AWS Cloudera Data Warehouse to query data on Azure Cloudera Data Warehouse Introduction Cloudera Data Platform enables in a single console to work with different public cloud providers. With this, you can have a true hybrid environment with only one admin console. Cloudera Data Warehouse is a public cloud service that allows fast analytics in your preferred cloud provider. In this article, I'll show how easy it is to connect between two Virtual Warehouses located in different cloud providers using Cloudera Data Warehouse. Scenario We're using two different cloud providers for Cloudera Data Warehouse: one in AWS with TPC-DS data and another in Azure with the same TPC-DS data. We'll use Hive ACID to update the customer table on Azure and merge it with the customer table in AWS. 1.Pre-Requisites 1.1 - Cloudera CDP Control Plane Access and Register two environments For this exercise, you will need access to the Cloudera Data Platform. More information can be accessed here. Also

Motivation:When Hadoop components (HDFS et. al.) are configured to connect to external sources like LDAP, the LDAP bind passwords need to be given in configuration file (core-site.xml) in clear text. For many enterprise environments, having password in clear text is not allowed and is often flagged as risk in Security Audits.This article teaches Hadoop administrators on how to secure these plaintext password in Hadoop configuration.Configuration Steps:1. Before starting, make sure that LDAP bind is working with plain text password. So, the HDFS should be configured with these settings in core-site.xml (the following values should be changed to match your LDAP/AD environment. These are working example values from my AD setup)hadoop.security.group.mapping=org.apache.hadoop.security.LdapGroupsMapping hadoop.security.group.mapping.ldap.base=ou=CorpUsers,dc=lab,dc=hortonworks,dc=nethadoop.security.group.mapping.ldap.bind.user=cn=ldap-reader,ou=ServiceUsers,dc=lab,dc=hortonworks,dc=neth

In order to convert JSON to XML, you can use ConvertRecord processor available as part of the NiFi tool (I worked on version: 1.11.4). Here are the steps to configure ConvertRecord processor to convert JSON input to XML output: Add a processor – ConvertRecord and right-click to “configure”: Click on the “Record Reader value” and select “Create New Service”. Select “JSONTreeReader” and click CREATE: Similarly, Click on the “Record Writer value” and select “Create New Service”. Select “XMLRecordSetWriter” and click CREATE: Final configuration of Convert Record should look like this: Click on “à to configure “JSONTreeReader” to configure details about the schema which should be read by the registry. The controller service form should open: Click on “configure icon” of “JSONTreeReader” to start configuring the JSON Reader and configure as per below screenshot: When you select “Create New Service” in the above screenshot, another form opens up. Fill it up based on below screenshot and cli

Following are the configurations for connecting Apache Ranger with LDAP/LDAPS. There's an important tool that will help identify some settings in your AD AD Explorer - Windows Sysinternals | Microsoft Docs. This configuration will sync LDAP users and link them with their LDAP groups every 12 hours, so later from Apache Ranger, you can give permission based on LDAP groups as well. For connecting using LDAPS, ensure you have the proper certificates added in the same server that contains the Ranger's UserSync service. Configuration Name Configuration Value Comment ranger.usersync.source.impl.class org.apache.ranger.ldapusersync.process.LdapUserGroupBuilder   ranger.usersync.sleeptimeinmillisbetweensynccycle 12 hour   ranger.usersync.ldap.url ldaps://myldapserver.example.com ldaps or ldap based on your LDAP security ranger.usersync.ldap.binddn myuser@example.com   ranger.usersync.ldap.ldapbindpassword mypassword   ranger.usersync.ldap.searchBase OU=

IntroductionFor a simple PySpark application, you can use `--py-files` to specify its dependencies. A large PySpark application will have many dependencies, possibly including transitive dependencies. Sometimes a large application needs a Python package that has C code to compile before installation. And, there are times when you might want to run different versions of python for different applications. For such scenarios with large PySpark applications, `--py-files` is inconvenient. Fortunately, in the Python world you can create a virtual environment as an isolated Python runtime environment. We recently enabled virtual environments for PySpark in distributed environments. This eases the transition from local environment to distributed environment with PySpark. In this article, I will talk about how to use virtual environment in PySpark. (This feature is currently only supported in yarn mode.)Prerequisites Hortonworks supports two approaches for setting up a virtual environment: vi

Note: Cloudera does not support Antivirus software of any kind. This article contains recommendations for excluding CDH components and directories from AV scans and monitoring. It is important to note that these recommendations do not apply to each service, and further, some services will have additional items to exclude which are unique to them. These details will be addressed in individual articles dedicated to the service in question. The three primary locations you will want to exclude from antivirus are: Data directories: These can be very large, and therefore, take a long time to scan; they can also be very write-heavy, and therefore suffer performance impacts or failures if the AV holds up writes. Log directories: These are write-heavy. Scratch directories: These are internal locations used by some services for writing temporary data, and can also cause performance impacts or failures if the AV holds up writes. In general, consider excluding the following directories and

Note: Cloudera does not support Antivirus software of any kind.This article contains recommendations for excluding CDP components and directories from AV scans and monitoring. It is important to note that these recommendations do not apply to each service, and further, some services will have additional items to exclude that are unique to them. These details will be addressed in individual articles dedicated to the service in question.The three primary locations you will want to exclude from Antivirus are:Data directories: These can be very large, and therefore, take a long time to scan; they can also be very write-heavy, and therefore suffer performance impacts or failures if the AV holds up writes.Log directories: These are write-heavy.Scratch directories: These are internal locations used by some services for writing temporary data, and can also cause performance impacts or failures if the AV holds up writes.In general, consider excluding the following directories and all of their s

How to read fsimage:We can use Offline Image Viewer tool to view the fsimage data in a human readable format. Sometimes, this becomes more essential to analyse the fsimage to understand the usage pattern, how many 0 bite files are created, what is the space consumption pattern and is the fsimage corrupt. Donwload the fsimage: hdfs dfsadmin –fetchImage /fsimage This will download the latest fsimage from Name node 16/12/27 05:40:43 INFO namenode.TransferFsImage: Opening connection to

The hive testbench consists of a data generator and a standard set of queries typically used for benchmarking hive performance. This article describes how to generate data and run a query in using beeline and Hive 2.0 with and without LLAP. It also shows how to use explain to see the difference in query plans.If you don't have a cluster already configured for LLAP, you can provision one in AWS using Hortonworks Cloud. See this article for instructions on how to provision a 2.5 tech preview with LLAP enabled.1. Log into the master node in you cluster where Hive is installed. If you used Hortonworks Cloud to create your instance, locate the node with a name ending in master. The ssh command is shown next to the master instance. If you are logging in from a linux host, click on the icon to the right of the ssh command to select the command text and copy the command. In the linux shell, change to the directory containing your AWS key .pem file and the run the copied command.If yo

Introduction In Cloudera Data Engineering, you may have a need to move the jobs from one virtual cluster to another. This can be for testing purposes or to move the jobs into the latest CDE versions. This can be easily achieved by the backup & restore process and is well documented here. The purpose of this article is to provide additional tools to simplify this and demonstrate the process. The example shown here demonstrates this process within the same CDE service, but you can use the same approach to move across CDE services. Steps Ensure that CDP CLI is set up on the system with the required access to list and describe CDE services & virtual clusters. Also, have your workload password handy as this is required for the export and import process. Download export & import scripts using the following commands.wget https://raw.githubusercontent.com/karthikeyanvijay/cdp-cde/main/export-import-cde-jobs/cde-vc-jobs-export.shwget https://raw.githubusercontent.com/karthikeyanvi

Overview: To replace an old Namenode hardware with new hardware with the same hostname/ IP address on a live production Hadoop cluster in a large-scale environment.   Problem:   When the hardware is due for refresh due to End of Service Life (EOSL) by the vendor, we will not get service support for this product such as repairs, limited tech support, parts availability, etc. It is time for replacing the servers with new systems. There is no straightforward method published by any Hadoop vendor for this task for in-place replacement or swap of hardware with the same hostname and IP address, so that cluster does not require any downtime or restart of Hadoop services.   Option #1 - Moving highly available NN, failover controller & JN roles using Migrate Roles wizardMigrate Roles wizard offers the capability to move roles of a highly available HDFS service from one host to another. It can be used to move Namenode, Journal Node, and Failover Controller roles to new h

Phone Tracking with OwnTracks and Apache NiFi 1.xOwnTracks is an Open Source project which provides an iOS and an Android app with which your smartphone records its current location.I installed the OwnTracks application for iOS and it let's you specify your own REST JSON server to receive calls. I added Apache NiFi. It can also send via MQTT direct to mosquittio on prem to NIFI or to CloudMQTT. You just need to enter your Apache NiFi address and port. Tell NiFi to listen for HTTP on port 9179 for the phone push, allow it to use GET, POST and PUT. No coding required.Respond to the phone with HTTP Status Code 200 and use the context map to connect the HTTP flow.We pull out the attributes from the JSON Flow File.Store our Phone Data in Apache Phoenix on HBaseupsert into phone (uuid,battery,longitude ,accelerator ,velocity,vac ,latitude , tvalue ,connection , tst , altitude , messagetype , tid, httpremotehost, useragent, filename, datetime)values ('${'uuid'}','${'battery'}','${'lo

Introduction In this article, we will learn how to query the Hive tables data by using column names with regular expressions in Spark. Assume we have a table with column names like col1, col2, col3, col4, col5, etc. If we want to select the data, we will use queries like select col1, col2, col3, col4, col5 from the table. Instead of specifying col1, col2, col3, col4, col5, we can use regular expressions while selecting columns like select `col.*` from table. From Hive, this feature is supported from Hive 0.13.0 onwards. By default, this feature is disabled, and in order to enable it, we need to use set hive.support.quoted.identifiers=none. From Spark, this feature is supported from Spark 2.3.0 onwards. By default this feature is disabled, and in order to enable it, we need use set spark.sql.parser.quotedRegexColumnNames=true. Steps Hive Create the Hive table and insert the data:beeline -u jdbc:hive2://host:port> create database if not exists regex_test;&g

Started working on the Cloudera Public Cloud in Azure and couldn't bare a thought that we would need to pay for all those resources that are not being used in non-prod environments. My goal was to be able to easily stop/start environment(s) based on the time, so resources (compute and money) can be utilized wisely. I haven't used Cloudera Manager API because of 2 reasons: I couldn't authenticate using the workload username and password. Kept getting empty responses. Only intercepted token from the browser allowed to make a successful request. With a multitude of clusters, it is daunting to invoke a multitude of separate APIs. Unfortunately, cdpcli is still a bit immature and doesn't have a good API in Python yet. I didn't want to use Java, which has what seems to be a slightly better API, for such a trivial task. I am running Azure Function App with a Timer trigger. I had to use the ElasticPremium tier because I wanted the function to run on the node in the same subnet as my Clo

How to test/ create the Livy interactive sessions The following session is an example of how we can create a Livy session and print out the Spark version: Create a session with the following command:curl -X POST --data '{"kind": "spark"}' -H "Content-Type: application/json" http://172.25.41.3:8998/sessions​ Wait for the application to spawn, replace the session ID:curl -X POST --data '{"kind": "spark","code":"sc.version"}' -H "Content-Type: application/json" http://172.25.41.3:8998/sessions/25/statements Replace the session ID and get the result:curl -X GET --data '{"kind": "spark"}' http://172.25.41.3:8998/sessions/25/statements *Livy objects properties for interactive sessions   How to test the Batch Applications Using the Livy API Following is the SparkPi test job submitted through Livy API: To submit the SparkPi job using Livy, you should upload the required jar files to HDFS before running the job. This is the main difference between the Livy API and spark-submit.

In this article, we will learn how to integrate Spark with Hive JdbcStorageHandler in CDP.   JdbcStorageHandler By using the JdbcStorageHandler, we can connect Apache Hive to JDBC Data Sources (MySQL, PostgreSQL, Oracle, DB2, or Derby etc). Currently writing to a JDBC data source is not supported. To use JdbcStorageHandler, we need to create an external table using JdbcStorageHandler.   Example:   CREATE EXTERNAL TABLE employee( id BIGINT, name STRING, age INT, salary DOUBLE)STORED BY 'org.apache.hive.storage.jdbc.JdbcStorageHandler'TBLPROPERTIES ( "hive.sql.database.type" = "MYSQL", "hive.sql.jdbc.driver" = "com.mysql.jdbc.Driver", "hive.sql.jdbc.url" = "jdbc:mysql://localhost/test", "hive.sql.dbcp.username" = "username", "hive.sql.dbcp.password" = "password", "hive.sql.table" = "EMPLOYEE", "hive.sql.dbcp.maxActive" = "1");   Supported Data Types: The column data type for a Hive JdbcStorageHandler table can be: Numeric data ty

In Beeline, the command-line options such as sslTrustStore, trustStorePassword, showDbInPromp etc are case sensitive.   For example, below is a working connection string from a test bed:  beeline -u "jdbc:hive2://host-A-fqdn:21051/default;principal=impala/host-A-fqdn@COE.CLOUDERA.COM;ssl=true;sslTrustStore=/opt/cloudera/security/truststore.jks"  In the above example, the common mistakes are principal mentioned as Principal and sslTrustStore mentioned as ssltruststore.   Here, if the case sensitivity is not followed, Beeline silently ignores the command line options and drops them:  //Sample stringbeeline -u "jdbc:hive2://host-A-fqdn:21051/default;Principal=impala/host-A-fqdn@COE.CLOUDERA.COM;ssl=true;ssltruststore=/opt/cloudera/security/truststore.jks" If you use the above connection string, at first, you will encounter a Kerberos issue as the property "principal" will be dropped and the actual Kerberos authentication w

For non-SSL enabled NiFi below should work: curl --tlsv1.2 -i -H'Content-Type: application/json' -XPUT -d'{"id":"cdb54c9a-0158-1000-5566-c45ca9692f85","state":"RUNNING"}'localhost:8080/nifi-api/flow/process-groups/cdb54c9a-0158-1000-5566-c45ca9692f85 Start Process Group for SSL enabled NiFi : Generate access token: [root@nifi-ambari-01~]# curl --tlsv1.2 https://000.00.00.000:9091/nifi-api/access/token--data 'username=awadhwani&password=password' -k O/P: eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJjbj1BcnRpIFdhZGh3YW5pLG91PVBlb3BsZSxkYz1zbWUsZGM9aHd4IiwiaXNzIjoiTGRhcFByb3ZpZGVyIiwiYXVkIjoiTGRhcFByb3ZpZGVyIiwicHJlZmVycmVkX3VzZXJuYW1lIjoiQXJ0aSBXYWRod2FuaSIsImtpZCI6MywiZXhwIjoxNDg5MjMyNDQyLCJpYXQiOjE0ODkxODkyNDJ9.17iHL3XX7Bw6dXv5lCByimu_asQOaSwW11o2IQEFO0s[root@nifi-ambari-01~]# Start PG by passing the above access token [root@nifi-ambari-01~]# curl --tlsv1.2 -ik -H 'Content-Type: application/json' -H 'Authorization:BearereyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJjbj1BcnRpIFdhZGh3YW5pLG91PVBlb3BsZSxkY

Configuring Ranger Usersync with AD/LDAP is highly dependent on the customer environment. This requires understanding of the customer environment as well as the specific requirements for syncing users and groups.In order to get some background on User management in Hadoop, please refer to this Dataworks Summit Presentation. Corresponding slides are available at: SlidesIn this article we will take a common use case and see how Ranger Usersync can be configured in that scenario. Ranger Usersync supports various configuration options with AD/LDAP and details are available hereUse case: Sync all the users who are members of a specific group(s)This use case can be handled in multiple ways and is also based on the AD/LDAP attributes available in the server. In case of Active Directory, users contain information of the group(s) they belong to using the “memberof” attribute. Similarly groups contain information about the user(s) that belong to them using the “member” attribute. Whereas in defa

slapd-conf.tar.gzEnvironment:Hortonworks Sandbox “HDP_2.3.2_virtualbox" Ranger Version: 0.5.0.2.3 LDAP Version: openldap-2.4.40-7.el6_7.x86_64LDAP setup url : LDAP setup tutorial 
[Note: The ldap setup here does not have ssl implemented. Pls find working “slapd.conf”attached]Steps:1.Configure openldap server on sandbox as mentioned in the Ldap setup url above.2.Login to Ambari console using “admin” user.3.Click on “Ranger” service -> “Configs”4.Go to “LDAP Settings” and set the below properties –ranger.ldap.user.searchfilter=(uid={0})ranger.ldap.user.dnpattern=cn=Manager,dc=hortonworks,dc=comranger.ldap.url=ldap://127.0.0.1:389ranger.ldap.referral=ignore ranger.ldap.group.roleattribute=uid ranger.ldap.bind.password=*****ranger.ldap.bind.dn=cn=Manager,dc=hortonworks,dc=comranger.ldap.base.dn=dc=hortonworks,dc=comNote: ldap.bind.password=<Admin password of openldap>5.Go to “Advanced ranger-admin-site” and set below properties -ranger.ldap.group.searchfilter=(member=uid={0},ou=Us

Security/Governance/GDPR Demo on CDP-Private Cloud Base 7.x Summary This article explains how to quickly set up Cloudera Security/Governance/GDPR (Worldwide Bank) demo using Cloudera Data Platform - Private Cloud Base (formerly known as CDP-Data Center). It can be deployed either on AWS using AMI or on your own setup via provided script What's included Single node CDP 7.1.7 including: Cloudera Manager (60-day trial license included) - for managing the services Kerberos - for authentication (via local MIT KDC) Ranger - for authorization (via both resource/tag-based policies for access and masking) Atlas - for governance (classification/lineage/search) Zeppelin - for running/visualizing Hive queries Impala/Hive 3 - for Sql access and ACID capabilities Spark/HiveWarehouseConnector - for running secure SparkSQL queries Worldwide Bank artifacts Demo hive tables Demo tags/attributes and lineage in Atlas Demo Zeppelin notebooks to walk through a demo scenario Ranger policies across HDF