Resources

Working with Hive in HDP 3.x is a challenge for Administrators and Developers because there is no Hive UI. The old Hive View has been removed in HDP 3.x and since then Developers were forced to use external 3rd party tools to access Hive.  The only other option for Administrators and Developers was the command line for Hive which certainly works, but is not as friendly as the old Hive View.  Both Beeline and the Hive View are limited when compared to the entire capabilities in the WebUI for Hue (HDFS, Hive, Zookeeper, YARN, Oozie, Sqoop, RDBMS, etc.).   First a little back story on Hue Ambari Service.  First created by Kyle Joe as a Hue Service for Ambari 2.4 the repo did not have any updates in three years. At that time, Hortonworks and Hue removed and changed the location of the 3.11 files, making Kyle's repo not operational. In this article, I am going to outline how I created an HDP 3 test cluster, installed Kyle's 2.4 repo, and began to debug. Next with w

The MiNiFi Java agent builds the nifi.properties file from default values and the ones exposed in the config.yml. To override the default values not exposed in the config.yml, a property must be added to the config.yml in the NiFi Properties Overrides section. For example, content archiving is disabled by default. To enable this, add the following in the config.yml file:   NiFi Properties Overrides: { nifi.content.repository.archive.max.retention.period: 1 hour, nifi.content.repository.archive.max.usage.percentage: 10%, nifi.content.repository.archive.enabled: 'true'}   Then, save the config.yml and restart the agent. If you look at the nifi.properties file, you will see the updated values.   NOTE:  This is a known issue (internal bug CEM-967) when using the Edge Flow Manager (EFM) to publish flows to the Java agent.  When a new flow is published, the agent does not retain the overridden properties. Hence, the properties have to be re-entere

Issue Unable to install NiFi 3.5.1 from an HDP 3.1.5 cluster. Background In a twelve node HDP 3.1.5 cluster, all services are installed and run successfully. The cluster has Kerberos and full SSL installed. The correct mpack for HDF NiFi has been installed. Ambari-server and agents are restarted. However, NiFi fails to install.  Error The following is the error displayed during the installation:  Traceback (most recent call last):File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi.py", line 312, in <module>Master().execute()File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 352, in executemethod(env)File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi.py", line 66, in installself.install_packages(env)File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 853, in install_packagesretry_count=agent_stack_retry_count)File "/usr/lib/ambari-agent/

These instructions assume you don't need to preserve Solr indexes. If you do, modify ZK commands to move nodes instead of removing them.Stop every SolrCloud node:# on every SolrCloud nodesu - solrcd /opt/lucidworks-hdpsearch/solr/bin/./solr stop -allConnect to your ZooKeeper quorum and run zkCli shell:su - zookeepercd /usr/hdp/current/zookeeper-client/bin/# point it to a ZK quorum (or just a single ZK server is ok, e.g. localhost)./zkCli.sh -server lake02:2181,lake03:2181,lake04:2181If you followed best practices and chroot'ed your SolrCloud already, then things are easy (and you probably are done by now):# in ZK cli shellrmr /solrMore often this wasn't the case, so perform the following operations on your ZK tree:# in ZK cli shellrmr /clusterstate.jsonrmr /aliases.jsonrmr /live_nodesrmr /overseerrmr /overseer_electrmr /collectionsNext, follow this best practices article, create a new ZK home for your SolrCloud cluster and start it up in chroot'ed mode.

Why Auto-Scale? There is great excitement about Cloud Computing and this is compounded by the Covid-19 pandemic.  Auto-Scaling in the cloud, from adding resources when needed to terminating them when finished, provides great flexibility for computing resources on demand.  It also makes cloud computing financially viable.  A recent survey showed spending 23% over budget (n=750, Flexera,  "State of the Cloud").    Cloudera lets you scale to the power you need and helps you avoid this over-budget spending with Auto-Scaling for Data Warehousing and Machine Learning using Kubernetes. We're going to dive into the details of how to set up an Auto-Scaling environment in Cloudera AI Setting up a Cloudera Machine Learning Workspace CDP provides self-service provisioning of Workspaces. A Workspace can run Python, Scala, R, Spark, Tensorflow, etc. This example will show a Spark Auto-Scale environment. You can run Spark code here and not do any data science if you cho

While installing HDP components using Ambari some times it fails with below errors.raise Fail("Cannot match package for regexp name {0}. Available packages: {1}".format(name, self.available_packages_in_repos)) resource_management.core.exceptions.Fail: Cannot match package for regexp name zookeeper_${stack_version}. Available packages: []What does this mean? basically Ambari is unable to find the packages. the possible reasons can be,Reason1: Is that host already have that binaries installed? If it is already installed using different repo-name then Ambari would not be able to determine those binaries. You can find out that by running yum list installed | grep zookeeperThis tells using which tag/repo it was installed (look for specific HDP version). If you find entry and it was installed different tag/repo then manually remove above packages and attempt to install again.Reason2: Its possible that provided URL does not have those binaries. you can determine by running below commandyum li

This is a step by step guide to test Kafka clients from a Windows machine that connects to an HDF/HDP environment.   We start with the review of the current Kafka broker listeners. In this case, we will cover the following: SASL_PLAINTEXT > Kerberized environments PLAINTEXT > Plain connections This can be done by using the Ambari console > Kafka > configs > Kafka Broker. After that, search for listeners and make sure either one or both protocols are enabled.   PLAINTEXT security protocol Go to your Windows machine and download the apache Kafka software. It is recommended to download the same version that it's running in your HDP/HDF cluster. Select the "Scala 2.12" link to avoid exceptions while running the Kafka clients. Extract the content of this folder in a preferred location in the Windows host. While connecting to Kafka through PLAINTEXT listener, Kafka does not have a way to identify you as a user. Hence, add Kafka ACLs and give permissi

Objective Apache NiFi provides the option of starting an embedded ZooKeeper server. However, NiFi can also be configured to run with an external ZooKeeper server. This article describes how to install and configure a 3 host ZooKeeper ensemble to work with a 2 node NiFi cluster.Environment This tutorial was tested using the following environment and components: Mac OS X 10.11.6 Apache ZooKeeper 3.4.6 Apache NiFi 1.3.0ZooKeeperZooKeeper Version The version of ZooKeeper chosen for this tutorial is Release 3.4.6. Note: ZooKeeper 3.4.6 is the version supported by the latest and previous versions of Hortonworks HDF as shown in the "Component Availability In HDF" table of the HDF 3.0.1.1 Release Notes.ZooKeeper Download Go to http://www.apache.org/dyn/closer.cgi/zookeeper/ to determine the best Apache mirror site to download a stable ZooKeeper distribution. From that mirror site, select the zookeeper-3.4.6 directory and download the zookeeper-3.4.6.tar.gz file. Unzip the tar.

Objective About a year ago, I wrote an article that detailed how to use the ConvertRecord processor and Record Reader/Writer controller services to easily convert a CVS file into various formats (JSON, Avro, XML): https://community.hortonworks.com/content/kbentry/115311/convert-csv-to-json-avro-xml-using-convertrecord-p.html. At the time, the CSV to XML conversion was done using a ScriptedRecordSetWriter. With the release of NiFi 1.7.0, the CSV to XML conversion can be done much more simply with the new XMLRecordSetWriter.Environment This tutorial was tested using the following environment and components: Mac OS X 10.11.6 Apache NiFi 1.7.0Convert CSV to XMLSupport Files Here is a template of the flow discussed in this tutorial:convert-cvs-to-xml.xml Here is the CSV file used in the flow:users.txtNote: Change the extension from .txt to .csv after downloading.Demo ConfigurationImport Template Start NiFi. Import the provided template and add it to the canvas. You should see the

The concept of Delegation token is introduced to avoid frequent authentication check against Kerberos(AD/MIT). After the initial authentication against Namenode using Keberos, any subsequent authentication can be done without Kerberos service ticket(ot TGT). Once the client authentication with Kerberos for Namenode is successfull, The client can get a delegation token from Namenode.This token has expiration and max issue date. But this can be renewed up to max issue date. In this article, We are going to see how delegation token can be created with the initial authentication and even if you delete initial TGT, You can still list the content with the help of delegation token1). List the ticket from Kerberos.root@hdptest-1 ~]# cd /etc/security/keytabs/[root@hdptest-1 keytabs]# klist -kt hdfs.headless.keytabKeytab name: FILE:hdfs.headless.keytabKVNO Timestamp Principal---- ----------------- -------------------------------------------------------- 0 07/23/16 02:44:25 hdfs-hdptest

Cloudera Machine Learning (and Cloudera Data Science Workbench) is built on a very robust and flexible framework to ease integration with third parties. In this article, I decided to explore the integration of Teradata with CML via ODBC.   A few notes before I dive in: I could have easily used JDBC via JayDeBeApi (see my previous article), but where is the fun in that This article could be generalized to more ODBC connections, provided proper parameters are setup To get a fully rounded tutorial, I will go through these 3 steps: Step 1 (optional if you already have a Teradata instance): Setting up Teradata Vantage in AWS Step 2: Create a custom image with Teradata driver Step 3: Configure and run python to Teradata ODBC Step 1: Setting up Teradata Vantage in AWS If you want to do development testing of Teradata, you can use Teradata Vantage Developer on the AWS Market Place. I am not going to dive too much into this as this configuration may change as Teradata evolves

Interacting with Hadoop HDFS using Python codesThis post will go through the following: Introducing python “subprocess” module Running HDFS commands with Python Examples of HDFS commands from Python1-Introducing python “subprocess” moduleThe Python “subprocess” module allows us to: spawn new Unix processes connect to their input/output/error pipes obtain their return codesTo run UNIX commands we need to create a subprocess that runs the command. The recommended approach to invoking subprocesses is to use the convenience functions for all use cases they can handle. Or we can use the underlying Popen interface can be used directly.2-Running HDFS commands with PythonWe will create a Python function called run_cmd that will effectively allow us to run any unix or linux commands or in our case hdfs dfs commands as linux pipe capturing stdout and stderr and piping the input as list of arguments of the elements of the native unix or HDFS command. It is passed as a Python list rather than a s

In this article, the requirement is to have simple tag names that will be applied to attributes in a Hive table. Here, tags such as “Hashed” and “Masked” will be used in many tables. Different groups may need access to the data in the clear, while others will not be granted access. The opposite may apply for a different table.   Table name atlas_tag_test has the columns ssn that will be shown for hr_users and masked for power_users.   The value will be a comma-separated list of groups, in this case, we only have a single group. Now in Ranger, we have a policy for masked. For each group we are adding a condition, we check if power_users are set in the group's attribute like the following:if ( ctx.getAttributeValue("Masked","groups").indexOf("power_users") !== -1 ){ ctx.result = false; } else { ctx.result = true; }     If it is, the data will be available to the user to see in the clear. It is not, the users will not be able to see the data.   We will add

Short Description: A quick tutorial on how to mask columns in Hive for regulatory purposes Abstract: This tutorial will cover how to apply tags to Atlas entities and subsequently leverage tagging policies in Ranger to mask Personally Identifiable Information (PII). Atlas serves as a common metadata store designed to exchange metadata both within and outside of the Hadoop stack. It features a simple user interface and a REST API for ease of access and integration. The Atlas-Ranger paradigm unites data classification with policy enforcement. Figures will be used as a graphical aid. Steps will be provided in between figures via bullet points. Note: This tutorial assumes the user has successfully installed Ranger, enabled Ranger audit to Solr, installed Atlas, installed Hive, configured Atlas to work with Hive, and configured Atlas to work with Ranger. It also assumes the user has dummy data for development purposes. For detailed instructions on how to accomplish these steps please

Masking of Hive columns could be achieved using Hive resource-based policies and masking policies for databases, tables, and columns. However, dynamic masking could be achieved using Atlas tags or collections (from HDP-3.x), which empowers users to regulate the visibility of sensitive data by leveraging Atlas tag-based policies in Ranger.   Prerequisites of this tutorial include a healthy HDP cluster with existing tables/databases in Hive, Atlas configuration with Hive and Ranger, and Audit to Solr enabled for Ranger.   Open the video on YouTube here   This video is based on the original article How to Mask Columns in Hive with Atlas and Ranger. Other references:Providing Authorization with Apache Ranger

These steps have been successfully tried and tested on Ambari-2.4.2.0 and HDP-2.5.3..When you install and configure Solr cloud via Ambari or embedded solr via Ambari Infra on a kerberized cluster, SPNEGO authentication gets enabled by default. There is not direct switch to disable only SPNEGO authentication..Please follow below method to disable it..Step 1 - Login to Ambari server.Step 2 - Please take backup of below script/var/lib/ambari-server/resources/common-services/SOLR/<version>/package/scripts/setup_solr_kerberos_auth.py.Step 3 - Edit above script and do the modifications as belowOriginal value: command += '\'{"authentication":{"class": "org.apache.solr.security.KerberosPlugin"}}\'' Recommended value: command += '\'{ }\''.Step 4 - Please replace cached script with below command and restart ambari-agent followed by Solr service(via Ambari)cp /var/lib/ambari-server/resources/mpacks/solr-ambari-mpack-5.5.2.2.5/common-services/SOLR/5.5.2.2.5/package/scripts/setup_solr_kerbe

1.Prerequisites Enable HDFS location access for the HIve table Enable HDFS Policy in Ranger Restrict POSIX access in HDFS File system Grant Hive table access in Ranger   2. Enable HDFS location access for the user Login to Ranger and select HDFS Policy Make sure to mention the Hive table location (In below example “Resource Path” points to default Hive Warehouse location) Validate the HDFS location access is restricted to ensure Ranger policy is working as expected 3. Create table policy in Hive Access if not exists Select the Hive Policy Add a new policy if not exists Grant table access to required users and validate the access by querying the table 4. Enable table policy in Hive Row Level Filter Select “Row level filter” under Hive policy Select “Add New Policy” Provide the required information in the Policy Details. Enter “Where” clause condition in “Row Level Filter” Below example grants access to “Alice” user only to rows that has a condition of “state = ca” Bef

CML provides some nice high-level session, model, and experiment auditing. But sometimes a situation arises where you would like audit logs at the level of file modification and user logins. This article provides a quick walkthrough of accessing the underlying RDBMS for this information. This page gives details on the tables and information available. This guide presupposes you are using AWS. Thanks to @fletch_jeff for the help.   Step 1 - Install aws-iam-authenticator Install according to your OS as per these instructions. Step 2 - Grab your User ARN From the command line:       ==> aws sts get-caller-identity{ "Account": "012345678910", "UserId": "ABBBBBBBBB BXXXXXXXXXX", "Arn": "arn:aws:iam::012345678910:user/cperro"}       From AWS Console Open the AWS Console. Navigate to IAM. Navigate to Users. Select your user name. Copy the ARN.   Step 3 - Add your ARN to your Workspace Navigate to Machine Learning Worksp

This article explains how to setup Hive Warehouse Connector (HWC), in CDP Public Cloud CML (tested with CDP Public Cloud runtime 7.1). Step 1 - Start a Virtual Warehouse Navigate to Data Warehouses in CDP. If you haven't activated your environment, ensure you have activated it first. Once activated, provision a new Hive Virtual Warehouse: Step 2 - Find your HiveServer2 JDBC URL Once your Virtual Warehouse has been created, find it in the list of Virtual Warehouses. Click the Options icon (three vertical dots) for your Virtual Warehouse. Select the Copy JDBC URL option and note what is copied to the clipboard: Step 3 - Find your Hive Metastore URI Navigate back to your environment Overview page and select the Data Lake tab. Click on the CM-UI Service: Click on the Options icon (three vertical dots) for the Hive Metastore Service and click Configuration: Click the Actions dropdown menu and select the Download Client Configuration item: Extrac

Go here for the latest Prototype - Fraud Detection See the webinar for this Prototype here: https://www.cloudera.com/about/events/webinars/build-a-customer-churn-insights-application-with-cdp.html   The complexity of creating an end-to-end machine learning workflow is one of the biggest hurdles data science and machine learning engineers are facing. It's not a running model.fit() that is hard, but it is ingesting data, getting the data in the right format for the model training process, deploying the model in a way that is accessible to other parts of the business, and running applications that consume the model that is hard. Machine Learning is useful when it's deployed with an end-to-end workflow.     We have been working to create Applied Machine Learning Prototypes for CML that will help you build a fully working machine learning example in CML. The Prototypes will include source data, and walk through various steps: Ingest data into a useful place in CDP

Introduction How do we quickly gain insight and start working with data in a secure, governed, and scalable environment in the cloud?   This article explains how to achieve this using the Cloudera Data Warehouse platform connected with Apache Superset.   Cloudera Data Warehouse in CDP (Cloudera Data Platform) is an enterprise solution for modern analytics. It's an auto-scaling, highly concurrent, and cost-effective hybrid, a multi-cloud analytics solution that ingests data anywhere, at massive scale, from structured, unstructured, and edge sources.   Apache Superset (incubating) is a modern, enterprise-ready business intelligence web application. Pre-Requisites This exercise is performed on a Mac OS. The versions below were tested at the moment of writing this article and may change in the future:   Python         Python 3.7.5 pip 20.0.2         After Python/pip installation, install the following packages/versions in Python

JSON Batch to Single Row PhoenixI grabbed open data on Crime from Philly's Open Data (https://www.opendataphilly.org/dataset/crime-incidents), after a free sign up you get access to JSON crime data (https://data.phila.gov/resource/sspu-uyfa.json) You can grab individual dates or ranges for thousands of records. I wanted to spool each JSON record as a separate HBase row. With the flexibility of Apache NiFi 1.0.0, I can specify run times via cron or other familiar setup. This is my master flow.First I use GetHTTP to retrieve the SSL JSON messages, I split the records up and store them as RAW JSON in HDFS as well as send some of them via Email, format them for Phoenix SQL and store them in Phoenix/HBase. All with no coding and in a simple flow. For extra output, I can send them to Reimann server for monitoring.Setting up SSL for accessing HTTPS data like Philly Crime, require a little configuration and knowing what Java JRE you are using to run NiFi. You can run service nifi s

Hey Everyone, This video will show you how to create a Cloudera AI Workspace. A Workspace in CML is where all of your data science work will take place and allow you to collaborate with your team easily. Creating a basic Workspace is quick and simple if you already have an environment set up. Also, I cover the different advanced options available to help admins and users manage costs and stay secure when creating a new Workspace.     More information: Cloudera Machine Learning – What You Should Know https://community.cloudera.com/t5/Community-Articles/Cloudera-Machine-Learning-What-You-Should-Know/ta-p/292935 More information on Cloudera AI https://www.cloudera.com/products/machine-learning.html Documentation on CDP https://docs.cloudera.com/

Short Description:This sample code helps to connect to Kerberos enabled HBase cluster from Java program. Code Walkthrough:Create HBaseConfiguration and pass HBase cluster parameters. Configuration hbaseConfig = HBaseConfiguration.create();hbaseConfig.addResource("/path_to_hbase_conf/hbase-site.xml");hbaseConfig.addResource("/path_to_hbase_conf/core-site.xml");hbaseConfig.set("hadoop.security.authentication", "Kerberos"); User principal and key tab file names. Please make sure key tab files are in the respective folder. String principal = System.getProperty("kerberosPrincipal", "hbaseuser@EXAMPLE.COM");String keytab = System.getProperty("kerberosKeytab", "/path_to_keytab/hbase-client.keytab"); The essential Kerberos configuration information is the default realm and the default KDC. As with most Kerberos installations, a Kerberos configuration file krb5.conf is consulted to determine such things as the default realm and KDC. The default location i

It's very easy to integrate Apache Flume with Apache NiFi, either as a source or destination (sink).An example of using ExecuteFlumeSource and ExecuteFlumeSink in Apache NiFi. We use a Flume source to read Netcat and send to Slack. So you can send messages from Linux to a Slack chat. For the Flume sink to HDFS, we read data from a web feed. We can also connect our Flume Source to Flume Sink.Configuring a Flume SourceIf you have configured Apache Flume configuration files, this is the same thing. Data Received in Slack (Join and See Messages Live)Checking Provenance EventConfigure The ExecuteFlumeSink to Store to HDFS-rw-r--r-- 3 flume hdfs 28 2016-07-25 16:14 /flume/events/16-07-25/1610/00/events-.1469463242392To monitor what's going on in the Flume side of the process: tail -f /var/log/flume/flume-a1.log25 Jul 2016 16:14:38,854 INFO [agent-shutdown-hook] (org.apache.flume.instrumentation.MonitoredCounterGroup.stop:177) - Shutdown Metric for type: SINK, name: k1. si