Community Articles

This article assumes that you're already configured to use Ranger authorization in HDF 3.0.x or earlier and that Ranger instance is syncing the available users/groups with your Directory Server. In order to leverage policies defined with these groups, we'll need to upgrade to HDF 3.1.0 or later and migrate your configuration to utilize a new Ranger authorizer in NiFi.In short, you’ll need to update your authorizers.xml file to configure a userGroupProvider and modify your authorizer to be a ManagedRangerAuthorizer. The idea is that previously an Authorizer was configured which was responsible for making access decisions, managing policies, and managing users/groups. Whether or not the Authorizer supported managing policies and users/groups was optional. The default File base authorizer supported this. The existing Ranger authorizer did not (it only made access decisions).For HDF 3.1 we decoupled these concepts so you can independently configure the Authorizer (makes access decisions),

Here are the best practices for writing an HBase client application for HDP. 1. Use the new HBase-1.0 API's instead of old interfaces. Instead of HTable, use Table, instead of HConnection, use Connection, etc. Also the Connection management has been changed so that the connection lifecycle management is best performed by the client application. Check out these slide decks for more examples:https://www.dropbox.com/s/v1x3djtlp1qg204/HBase%201.0%20API%20Changes%20-%20Meetup%2010_15_2014.pdf?dl=0http://www.slideshare.net/enissoz/meet-hbase-10As well as further examples here:https://github.com/hortonworks/hbase-release/tree/HDP-2.5.3.0-tag/hbase-examples2. Always close the Connection, Table, Scanner and Admin interfaces when you are done. These interfaces implement Closeable, and holds resources on the client side or on the server side (Scanner). Thus properly closing these is very important for best performance. Java's new try syntax is an easy way to auto-close these resources in your cod

Introduction This document describes the details of Knox SSO integration with a SAML identity provider (For the purpose of this article, the IDP is Ping Federate). Here, Ping Federate acts as the identity provider and it has the ability to connect to any authentication source such as AD, LDAP, etc.   The following diagram depicts the process at a high level:    The SAML authentication process has three components  Identity Provider: Identity provider has the flexibility to authenticate users against an authentication source, and sends a SAML response XML back to the service provider Service Provider: Service provider is an application that requires users to authenticate Authentication source:  A directory service such as AD, LDAP. The service provider uses the authentication token and does not need to know the authentication source. As part of this integration, both IDP (Ping Fed) and SP (Knox server) exchange information for the handshake to work successfu

Configure Ldap server on Redhat/Centos :- Check the ldap packages are installed or not on Server with following command #rpm –qa|grep openldap2. If packages are not installed then install the packages with yum command#yum install openldap-* -y 3. Once pacakge are installed then check with following command#rpm –qa |grep openldap4. Create Ldap password with following command#slappasswd [Enter the password and copy the md5 formate password for adding the password into thedatabase file]5. Edit database files for domain#vi /etc/openldap/slapd.d/cn=config/olcDatabase={2}bdb.ldifoldSuffix:dc=example,dc=comolcRootDN:cn=Manager,dc=example,dc=comolcRootPW:copy the password here which isgenerated after set the slpappasswd. #vi /etc/openldap/slpapd.d/cn=config/olcDatabase={1}monitor.ldifolcAccess: {0}to * by dn.base="gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth" read by dn.base="cn=manager,dc=example,dc=com" read by * none6.Run the updatedb command to initialize database. Create

Yesterday evening, in a moment of inspiration, I wondered if one could build a game in NiFi (a tool for moving data in realtime). Of course, building a game is quite unnecessary, but it is far from pointless as it can really show how flexible a tool can be with a bit of creativity. A few years ago I played ‘Worlds first game of chess… against PySpark’ but given that NiFi is meant for simpler logic, I decided to start with something different. Building Tic Tac Toe in NiFi My goals were to build something that allowed people across the globe to play in realtime, by using a board (no code). Fortunately, the Graphical User Interface of NiFi was made with realtime interaction of multiple users in mind, so I had a good foundation for this. Furthermore, I wanted NiFi to determine the winner automatically. For this, I leveraged its capabilities in applying logic/business rules. Of course, NiFi wants to apply the logic to something, so to ensure that the winner would be determined in a timely

Introduction This article assumes that KnoxSSO for NiFi UI is enabled and is working as expected.   To configure Knox topology in order to access NiFi Rest API using Knox based NiFi URL, do the following:     Access NiFi UI through Knox SSO based URL such as the following:https://<knox.fqdn>:8443/gateway/<nifi-topology>/nifi-app/nifi/​This helps the users to expose only the Knox host and port (not the NiFi hosts) and also authenticate users via SSO before they successfully log in to NiFi UI. Once the SSO is enabled on NiFi UI, NiFi URL for the UI or the Rest API will automatically redirect to the SSO page. Usually, the NiFi Rest API access involves obtaining Bearer token and using it in subsequent API calls. With Knox proxy-based NiFi endpoint, this Bearer token would not work as Knox does not recognize this token. On top of that, the URL always redirects to the SSO URL.    To establish access to Knox based NiFi URLs, both Knox and NiFi need

Working with Hive in HDP 3.x is a challenge for Administrators and Developers because there is no Hive UI. The old Hive View has been removed in HDP 3.x and since then Developers were forced to use external 3rd party tools to access Hive.  The only other option for Administrators and Developers was the command line for Hive which certainly works, but is not as friendly as the old Hive View.  Both Beeline and the Hive View are limited when compared to the entire capabilities in the WebUI for Hue (HDFS, Hive, Zookeeper, YARN, Oozie, Sqoop, RDBMS, etc.).   First a little back story on Hue Ambari Service.  First created by Kyle Joe as a Hue Service for Ambari 2.4 the repo did not have any updates in three years. At that time, Hortonworks and Hue removed and changed the location of the 3.11 files, making Kyle's repo not operational. In this article, I am going to outline how I created an HDP 3 test cluster, installed Kyle's 2.4 repo, and began to debug. Next with w

The MiNiFi Java agent builds the nifi.properties file from default values and the ones exposed in the config.yml. To override the default values not exposed in the config.yml, a property must be added to the config.yml in the NiFi Properties Overrides section. For example, content archiving is disabled by default. To enable this, add the following in the config.yml file:   NiFi Properties Overrides: { nifi.content.repository.archive.max.retention.period: 1 hour, nifi.content.repository.archive.max.usage.percentage: 10%, nifi.content.repository.archive.enabled: 'true'}   Then, save the config.yml and restart the agent. If you look at the nifi.properties file, you will see the updated values.   NOTE:  This is a known issue (internal bug CEM-967) when using the Edge Flow Manager (EFM) to publish flows to the Java agent.  When a new flow is published, the agent does not retain the overridden properties. Hence, the properties have to be re-entere

Issue Unable to install NiFi 3.5.1 from an HDP 3.1.5 cluster. Background In a twelve node HDP 3.1.5 cluster, all services are installed and run successfully. The cluster has Kerberos and full SSL installed. The correct mpack for HDF NiFi has been installed. Ambari-server and agents are restarted. However, NiFi fails to install.  Error The following is the error displayed during the installation:  Traceback (most recent call last):File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi.py", line 312, in <module>Master().execute()File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 352, in executemethod(env)File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi.py", line 66, in installself.install_packages(env)File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 853, in install_packagesretry_count=agent_stack_retry_count)File "/usr/lib/ambari-agent/

These instructions assume you don't need to preserve Solr indexes. If you do, modify ZK commands to move nodes instead of removing them.Stop every SolrCloud node:# on every SolrCloud nodesu - solrcd /opt/lucidworks-hdpsearch/solr/bin/./solr stop -allConnect to your ZooKeeper quorum and run zkCli shell:su - zookeepercd /usr/hdp/current/zookeeper-client/bin/# point it to a ZK quorum (or just a single ZK server is ok, e.g. localhost)./zkCli.sh -server lake02:2181,lake03:2181,lake04:2181If you followed best practices and chroot'ed your SolrCloud already, then things are easy (and you probably are done by now):# in ZK cli shellrmr /solrMore often this wasn't the case, so perform the following operations on your ZK tree:# in ZK cli shellrmr /clusterstate.jsonrmr /aliases.jsonrmr /live_nodesrmr /overseerrmr /overseer_electrmr /collectionsNext, follow this best practices article, create a new ZK home for your SolrCloud cluster and start it up in chroot'ed mode.

Why Auto-Scale? There is great excitement about Cloud Computing and this is compounded by the Covid-19 pandemic.  Auto-Scaling in the cloud, from adding resources when needed to terminating them when finished, provides great flexibility for computing resources on demand.  It also makes cloud computing financially viable.  A recent survey showed spending 23% over budget (n=750, Flexera,  "State of the Cloud").    Cloudera lets you scale to the power you need and helps you avoid this over-budget spending with Auto-Scaling for Data Warehousing and Machine Learning using Kubernetes. We're going to dive into the details of how to set up an Auto-Scaling environment in Cloudera AI Setting up a Cloudera Machine Learning Workspace CDP provides self-service provisioning of Workspaces. A Workspace can run Python, Scala, R, Spark, Tensorflow, etc. This example will show a Spark Auto-Scale environment. You can run Spark code here and not do any data science if you cho

While installing HDP components using Ambari some times it fails with below errors.raise Fail("Cannot match package for regexp name {0}. Available packages: {1}".format(name, self.available_packages_in_repos)) resource_management.core.exceptions.Fail: Cannot match package for regexp name zookeeper_${stack_version}. Available packages: []What does this mean? basically Ambari is unable to find the packages. the possible reasons can be,Reason1: Is that host already have that binaries installed? If it is already installed using different repo-name then Ambari would not be able to determine those binaries. You can find out that by running yum list installed | grep zookeeperThis tells using which tag/repo it was installed (look for specific HDP version). If you find entry and it was installed different tag/repo then manually remove above packages and attempt to install again.Reason2: Its possible that provided URL does not have those binaries. you can determine by running below commandyum li

This is a step by step guide to test Kafka clients from a Windows machine that connects to an HDF/HDP environment.   We start with the review of the current Kafka broker listeners. In this case, we will cover the following: SASL_PLAINTEXT > Kerberized environments PLAINTEXT > Plain connections This can be done by using the Ambari console > Kafka > configs > Kafka Broker. After that, search for listeners and make sure either one or both protocols are enabled.   PLAINTEXT security protocol Go to your Windows machine and download the apache Kafka software. It is recommended to download the same version that it's running in your HDP/HDF cluster. Select the "Scala 2.12" link to avoid exceptions while running the Kafka clients. Extract the content of this folder in a preferred location in the Windows host. While connecting to Kafka through PLAINTEXT listener, Kafka does not have a way to identify you as a user. Hence, add Kafka ACLs and give permissi

Objective Apache NiFi provides the option of starting an embedded ZooKeeper server. However, NiFi can also be configured to run with an external ZooKeeper server. This article describes how to install and configure a 3 host ZooKeeper ensemble to work with a 2 node NiFi cluster.Environment This tutorial was tested using the following environment and components: Mac OS X 10.11.6 Apache ZooKeeper 3.4.6 Apache NiFi 1.3.0ZooKeeperZooKeeper Version The version of ZooKeeper chosen for this tutorial is Release 3.4.6. Note: ZooKeeper 3.4.6 is the version supported by the latest and previous versions of Hortonworks HDF as shown in the "Component Availability In HDF" table of the HDF 3.0.1.1 Release Notes.ZooKeeper Download Go to http://www.apache.org/dyn/closer.cgi/zookeeper/ to determine the best Apache mirror site to download a stable ZooKeeper distribution. From that mirror site, select the zookeeper-3.4.6 directory and download the zookeeper-3.4.6.tar.gz file. Unzip the tar.

Objective About a year ago, I wrote an article that detailed how to use the ConvertRecord processor and Record Reader/Writer controller services to easily convert a CVS file into various formats (JSON, Avro, XML): https://community.hortonworks.com/content/kbentry/115311/convert-csv-to-json-avro-xml-using-convertrecord-p.html. At the time, the CSV to XML conversion was done using a ScriptedRecordSetWriter. With the release of NiFi 1.7.0, the CSV to XML conversion can be done much more simply with the new XMLRecordSetWriter.Environment This tutorial was tested using the following environment and components: Mac OS X 10.11.6 Apache NiFi 1.7.0Convert CSV to XMLSupport Files Here is a template of the flow discussed in this tutorial:convert-cvs-to-xml.xml Here is the CSV file used in the flow:users.txtNote: Change the extension from .txt to .csv after downloading.Demo ConfigurationImport Template Start NiFi. Import the provided template and add it to the canvas. You should see the

The concept of Delegation token is introduced to avoid frequent authentication check against Kerberos(AD/MIT). After the initial authentication against Namenode using Keberos, any subsequent authentication can be done without Kerberos service ticket(ot TGT). Once the client authentication with Kerberos for Namenode is successfull, The client can get a delegation token from Namenode.This token has expiration and max issue date. But this can be renewed up to max issue date. In this article, We are going to see how delegation token can be created with the initial authentication and even if you delete initial TGT, You can still list the content with the help of delegation token1). List the ticket from Kerberos.root@hdptest-1 ~]# cd /etc/security/keytabs/[root@hdptest-1 keytabs]# klist -kt hdfs.headless.keytabKeytab name: FILE:hdfs.headless.keytabKVNO Timestamp Principal---- ----------------- -------------------------------------------------------- 0 07/23/16 02:44:25 hdfs-hdptest

Cloudera Machine Learning (and Cloudera Data Science Workbench) is built on a very robust and flexible framework to ease integration with third parties. In this article, I decided to explore the integration of Teradata with CML via ODBC.   A few notes before I dive in: I could have easily used JDBC via JayDeBeApi (see my previous article), but where is the fun in that This article could be generalized to more ODBC connections, provided proper parameters are setup To get a fully rounded tutorial, I will go through these 3 steps: Step 1 (optional if you already have a Teradata instance): Setting up Teradata Vantage in AWS Step 2: Create a custom image with Teradata driver Step 3: Configure and run python to Teradata ODBC Step 1: Setting up Teradata Vantage in AWS If you want to do development testing of Teradata, you can use Teradata Vantage Developer on the AWS Market Place. I am not going to dive too much into this as this configuration may change as Teradata evolves

Interacting with Hadoop HDFS using Python codesThis post will go through the following: Introducing python “subprocess” module Running HDFS commands with Python Examples of HDFS commands from Python1-Introducing python “subprocess” moduleThe Python “subprocess” module allows us to: spawn new Unix processes connect to their input/output/error pipes obtain their return codesTo run UNIX commands we need to create a subprocess that runs the command. The recommended approach to invoking subprocesses is to use the convenience functions for all use cases they can handle. Or we can use the underlying Popen interface can be used directly.2-Running HDFS commands with PythonWe will create a Python function called run_cmd that will effectively allow us to run any unix or linux commands or in our case hdfs dfs commands as linux pipe capturing stdout and stderr and piping the input as list of arguments of the elements of the native unix or HDFS command. It is passed as a Python list rather than a s

In this article, the requirement is to have simple tag names that will be applied to attributes in a Hive table. Here, tags such as “Hashed” and “Masked” will be used in many tables. Different groups may need access to the data in the clear, while others will not be granted access. The opposite may apply for a different table.   Table name atlas_tag_test has the columns ssn that will be shown for hr_users and masked for power_users.   The value will be a comma-separated list of groups, in this case, we only have a single group. Now in Ranger, we have a policy for masked. For each group we are adding a condition, we check if power_users are set in the group's attribute like the following:if ( ctx.getAttributeValue("Masked","groups").indexOf("power_users") !== -1 ){ ctx.result = false; } else { ctx.result = true; }     If it is, the data will be available to the user to see in the clear. It is not, the users will not be able to see the data.   We will add

Short Description: A quick tutorial on how to mask columns in Hive for regulatory purposes Abstract: This tutorial will cover how to apply tags to Atlas entities and subsequently leverage tagging policies in Ranger to mask Personally Identifiable Information (PII). Atlas serves as a common metadata store designed to exchange metadata both within and outside of the Hadoop stack. It features a simple user interface and a REST API for ease of access and integration. The Atlas-Ranger paradigm unites data classification with policy enforcement. Figures will be used as a graphical aid. Steps will be provided in between figures via bullet points. Note: This tutorial assumes the user has successfully installed Ranger, enabled Ranger audit to Solr, installed Atlas, installed Hive, configured Atlas to work with Hive, and configured Atlas to work with Ranger. It also assumes the user has dummy data for development purposes. For detailed instructions on how to accomplish these steps please

Masking of Hive columns could be achieved using Hive resource-based policies and masking policies for databases, tables, and columns. However, dynamic masking could be achieved using Atlas tags or collections (from HDP-3.x), which empowers users to regulate the visibility of sensitive data by leveraging Atlas tag-based policies in Ranger.   Prerequisites of this tutorial include a healthy HDP cluster with existing tables/databases in Hive, Atlas configuration with Hive and Ranger, and Audit to Solr enabled for Ranger.   Open the video on YouTube here   This video is based on the original article How to Mask Columns in Hive with Atlas and Ranger. Other references:Providing Authorization with Apache Ranger

These steps have been successfully tried and tested on Ambari-2.4.2.0 and HDP-2.5.3..When you install and configure Solr cloud via Ambari or embedded solr via Ambari Infra on a kerberized cluster, SPNEGO authentication gets enabled by default. There is not direct switch to disable only SPNEGO authentication..Please follow below method to disable it..Step 1 - Login to Ambari server.Step 2 - Please take backup of below script/var/lib/ambari-server/resources/common-services/SOLR/<version>/package/scripts/setup_solr_kerberos_auth.py.Step 3 - Edit above script and do the modifications as belowOriginal value: command += '\'{"authentication":{"class": "org.apache.solr.security.KerberosPlugin"}}\'' Recommended value: command += '\'{ }\''.Step 4 - Please replace cached script with below command and restart ambari-agent followed by Solr service(via Ambari)cp /var/lib/ambari-server/resources/mpacks/solr-ambari-mpack-5.5.2.2.5/common-services/SOLR/5.5.2.2.5/package/scripts/setup_solr_kerbe

1.Prerequisites Enable HDFS location access for the HIve table Enable HDFS Policy in Ranger Restrict POSIX access in HDFS File system Grant Hive table access in Ranger   2. Enable HDFS location access for the user Login to Ranger and select HDFS Policy Make sure to mention the Hive table location (In below example “Resource Path” points to default Hive Warehouse location) Validate the HDFS location access is restricted to ensure Ranger policy is working as expected 3. Create table policy in Hive Access if not exists Select the Hive Policy Add a new policy if not exists Grant table access to required users and validate the access by querying the table 4. Enable table policy in Hive Row Level Filter Select “Row level filter” under Hive policy Select “Add New Policy” Provide the required information in the Policy Details. Enter “Where” clause condition in “Row Level Filter” Below example grants access to “Alice” user only to rows that has a condition of “state = ca” Bef

CML provides some nice high-level session, model, and experiment auditing. But sometimes a situation arises where you would like audit logs at the level of file modification and user logins. This article provides a quick walkthrough of accessing the underlying RDBMS for this information. This page gives details on the tables and information available. This guide presupposes you are using AWS. Thanks to @fletch_jeff for the help.   Step 1 - Install aws-iam-authenticator Install according to your OS as per these instructions. Step 2 - Grab your User ARN From the command line:       ==> aws sts get-caller-identity{ "Account": "012345678910", "UserId": "ABBBBBBBBB BXXXXXXXXXX", "Arn": "arn:aws:iam::012345678910:user/cperro"}       From AWS Console Open the AWS Console. Navigate to IAM. Navigate to Users. Select your user name. Copy the ARN.   Step 3 - Add your ARN to your Workspace Navigate to Machine Learning Worksp

This article explains how to setup Hive Warehouse Connector (HWC), in CDP Public Cloud CML (tested with CDP Public Cloud runtime 7.1). Step 1 - Start a Virtual Warehouse Navigate to Data Warehouses in CDP. If you haven't activated your environment, ensure you have activated it first. Once activated, provision a new Hive Virtual Warehouse: Step 2 - Find your HiveServer2 JDBC URL Once your Virtual Warehouse has been created, find it in the list of Virtual Warehouses. Click the Options icon (three vertical dots) for your Virtual Warehouse. Select the Copy JDBC URL option and note what is copied to the clipboard: Step 3 - Find your Hive Metastore URI Navigate back to your environment Overview page and select the Data Lake tab. Click on the CM-UI Service: Click on the Options icon (three vertical dots) for the Hive Metastore Service and click Configuration: Click the Actions dropdown menu and select the Download Client Configuration item: Extrac