Resources

The purpose of this document is how to leverage “R” to predict HDFS growth assuming we have access to the latest fsimage of a given cluster. This way we can forecast how much capacity would need to be added to the cluster ahead of time. In case of on-prem clusters, ordering H/W can be a lengthy process and for some organizations, it can take months till they can actually add capacity to their clusters. Giving teams managing Hadoop clusters an insight on HDFS growth rate can help with placing H/W orders ahead of timeA prereq. for this script, you would need access to a recent fsimage file, R and a machine with hadoop binaries installed on it to run OIV.Example: hdfs oiv -i fsimage_xxxxx -p Delimited > fsimage.outLoading required librarieslibrary(dplyr)library(anytime)library(lubridate)library(prophet)library(rmarkdown) --> only needed if you are using RStudio and planning on exporting the script to a PDF or Word formatData Filtering & manipulationThe only three columns we need

Launching Cloudbreak on GCPBefore launching Cloudbreak on GCP, you must meet the following prerequisites.Meet the PrerequisitesBefore launching Cloudbreak on GCP, you must meet the following prerequisites. GCP AccountService AccountSSH Key PairRegion and zoneGCP accountIn order to launch Cloudbreak on GCP, you must log in to your GCP account. If you don't have an account, you can create one at https://console.cloud.google.com. Once you log in to your GCP account, you must either create a project or use an existing project.To create a new project, provide name, choose organization or leave it as No Organization.Now Select this newly created Project.On the main dashboard page, you will find the Project ID. You will need this to define your credential in Cloudbbreak in a later stepGCP - APIs DashboardGo to the Service Accounts screen by (1) clicking the menu in the top left, (2) hovering over APIs and Services, and (3) Clicking on Dashboard.GCP - APIs & Services DashboardVerify that t

According to default Oozie log4j configuration in Ambari - Log get's rotated by every hour and retention is set to 30 days..log4j.appender.oozie=org.apache.log4j.rolling.RollingFileAppenderlog4j.appender.oozie.RollingPolicy=org.apache.oozie.util.OozieRollingPolicylog4j.appender.oozie.File=${oozie.log.dir}/oozie.loglog4j.appender.oozie.Append=truelog4j.appender.oozie.layout=org.apache.log4j.PatternLayoutlog4j.appender.oozie.layout.ConversionPattern=%d{ISO8601} %5p %c{1}:%L - SERVER[${oozie.instance.id}] %m%n# The FileNamePattern must end with "-%d{yyyy-MM-dd-HH}.gz" or "-%d{yyyy-MM-dd-HH}" and also start with the# value of log4j.appender.oozie.Filelog4j.appender.oozie.RollingPolicy.FileNamePattern=${log4j.appender.oozie.File}-%d{yyyy-MM-dd-HH}# The MaxHistory controls how many log files will be retained (720 hours / 24 hours per day = 30 days); -1 to disablelog4j.appender.oozie.RollingPolicy.MaxHistory={{oozie_log_maxhistory}}.If you want to configure DRFA to roll the log file daily, pl

Requirement: HDP 2.6.1 Fix version is knox: 0.12.0, according to KNOX-841 Method I: Using "solr" as data service {/usr/hdp/current/knox-server/data/services/solr}Add below in Ambari UI > KNOX > Configs > Advanced topology for Solr UI. default.xml <service> <role>SOLR</role> <url>http://<solr-hostname>:8983</url> </service> Make sure service def is in Upper Case. Restart KNOX Use below URL for access Solr UI using KNOX https://<knox-hostname>:8443/gateway/default/solrMethod II: Using "solrapi" as data service {/usr/hdp/current/knox-server/data/services/solrapi} for Banana UI KNOX-528Add below in Ambari UI > KNOX > Configs > Advanced topology for Solr UI.default.xml <service> <role>SOLRAPI</role> <url>http://<solr-hostname>:8983/solr</url> </service> Login into sorl server:Please make

Row and Columnar Storage For Hive Customers often ask about columnar storage formats for Hive tables and when to use them. It depends on your uses cases. If your data access patterns mostly involve selecting a few columns to perform aggregations, then using columnar storage will save disk space, reduce I/O when fetching data, and improve query execution time. In this article I tested the columnar ORC file format for Hive to quantify how it outperforms row based Text files when used for queries that aggregate and group on a few columns from a table with many columns. ORC is a columnar storage format used in Hadoop for Hive tables. It is an efficient file format for storing data in which records contain many columns. An example is Clickstream (web) data to analyze website activity and performance. Queries typically retrieve a subset of columns for each row.SUMMARY OF TEST RESULTS: Comparing ORC vs Text files for storing 22 million clickstream page view records in Hive tables

1.Prerequisites Enable HDFS location access for the HIve table Enable HDFS Policy in Ranger Restrict POSIX access in HDFS File system Grant Hive table access in Ranger 2. Enable HDFS location access for the user Login to Ranger and select HDFS Policy Make sure to mention the Hive table location (In below example “Resource Path” points to default Hive Warehouse location) Validate the HDFS location access is restricted to ensure Ranger policy is working as expected 3. Create table policy in Hive Access if not exists Select the Hive Policy Add a new policy if not exists Grant table access to required users and validate the access by querying the table 4. Enable table policy in Hive Column Masking Select “Masking” under Hive policy Select “Add New Policy” Provide the required information in the Policy Details. Select the Masking option as per the requirement Below example restricts access for “Sales1” user on column “ip” by masking it using Hash option Before applying Masking

1. Prerequisites Install Ranger Install Knox Test in lower environments Inform stakeholders or plan for short outage as few services requires a restart Identify the IP addresses and users to allow access 2. Enable Knox-Ranger Plugin in Ambari Login to Ambari > select Ranger > configs > Ranger Plugin Enable Knox Plugin by clicking the “off” button, and restart the required services Ambari suggests 3. Create a Knox Policy for Hive After enabling the Knox Plugin for Ranger in Ambari, Knox policy should be automatically displayed in Ranger. Select the Knox Policy Add a new policy, if not exists Grant groups/users access to required IP address 4. Connections Test Below are the IPs I considered for testing 172.25.39.156 172.25.40.41 Test Cases: Allow connections through any IPs for the group “sales” Allow only connection access through IP range for group “sales” Case 1: Allow connections through Knox → Hive for any IPs for the group “sales” Test connection from both the IP

ObjectiveUse Oozie's Hive 2 Action to create a workflow which will connect to Hive Serve 2 in a Kerberized environment. Execute a Hive query script which will sum the number of crimes in the crimes database table for a particular year - passed in as a parameter queryYear in thejob.properties file. Write the results to a new Hive table - crimenumbersProcedureLog into the edge server containing the Oozie client. Change users to the oozie user.[root@jyoung-hdp234-1 ~]# su - oozieAuthenticate to the KDC using the oozie service account kerberos keytab[oozie@jyoung-hdp234-1 ~]$ kinit -kt /etc/security/keytabs/oozie.service.keytab oozie/jyoung-hdp234-1.openstacklocal@EXAMPLE.COMDownload the City of Chicago crime data in CSV form.[oozie@jyoung-hdp234-1 ~]$ mkdir -p /tmp/crime[oozie@jyoung-hdp234-1 ~]$ cd /tmp/crime[oozie@jyoung-hdp234-1 crime]$ curl -o crime -L https://data.cityofchicago.org/api/views/ijzp-q8t2/rows.csv?accessType=DOWNLOADPut the crime csv into HDFS into the /tmp/crime folder.

If you looking to upload template using NIFI REST API, below command curl command upload the template as multipart form postcurl -XPOST -H "Authorization: Bearer {{ token }}" https://localhost:9091/nifi-api/process-groups/10c263d0-0160-1000-0000-00006a157654/templates/upload -k -v -F template=@NifiTemplate.xml

Sqoop is used for importing data from multiple sources onto HDFS. One of the most common use can is to use Hive imports sqoop import --connect jdbc:mysql://localhost:3306/sqoop --username root -P --split-by id --columns id,name --table customer --target-dir /user/cloudera/ingest/raw/customers --fields-terminated-by "," --hive-import --create-hive-table --hive-table sqoop_workspace.customersIf you want to specify a specific Queue for sqoop job -Dmapred.job.queuename=queueName needs to be immediately added after the import keyword. sqoop import -Dmapred.job.queuename=queueName --connect jdbc:mysql://localhost:3306/sqoop --username root -P --split-by id --columns id,name --table customer --target-dir /user/cloudera/ingest/raw/customers --fields-terminated-by "," --hive-import --create-hive-table --hive-table sqoop_workspace.customers
This will launch the sqoop job in the specific queu but the hive job will be launched in the default queue. To launch the hive job is specific queue

Perform below steps on the servers chosen for Datanodes having different OS version 1. Prerequisites Minimum Software Requirements: https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.2/bk_support-matrices/content/ch_matrices-ambari.html Check and set the maximum Open File descriptors on new host https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.2/bk_support-matrices/content/ch_matrices-ambari.html Check DNS and NSCD https://docs.hortonworks.com/HDPDocuments/Ambari-2.5.2.0/bk_ambari-installation/content/check_dns.html Disable THP Disable SELinux Enable NTP Swappiness Install same Python version Install same JDK Version   2. Manage Stack Version Go to Ambari Versions page:

If a user has partial access to a Hive table access via ranger policies, he/she won’t be able to run the describe statement against this table. As per screenshot below, user “yarn” has access to all columns of customer table except column “c_last_name”.If user “yarn” tries to run “describe customer;” a Permission Denied message will be displayed.To overcome this inconvenience, we can add “xasecure.hive.describetable.showcolumns.authorization.option=show-all” property in “Custom ranger-hive-security” section in Hive configs via ambari and restart Hive serviceOnce restart is done, try running the describe statement again. We should be able to see the table description.

This is a guide to walk a user through enabling high availability for Oozie in an Ambari-managed cluster in which Kerberos has been enabled. It is assume that Ambari 2.4.0 (or above) is installed and Kerberos has been previously set up. However, if Kerberos has not yet been set up, the Kerberos-related items can be performed when Kerberos is enabled. Also, it is expected that an Oozie HA supported database is configured. A database such as MySQL or Oracle needs to have been configured since the Derby database is not sufficient. MotivationMost documentation properly shows the steps to enable Oozie HA; however when Kerberos is involved, some configuration properties need to be set in an alternate way. According some documentation, when HA is setup in an environment where Kerberos is enabled, the following properties must be set:oozie-site/oozie.authentication.kerberos.principal = *oozie-site/oozie.authentication.kerberos.keytab = /etc/security/keytabs/oozie.ha.keytabIf this is fol

Starting with version 2.6.5 on the 2.x line and 3.0.0 on the 3.x line, HDP has added support for running workloads against Google Cloud Storage (GCS). This article talks about how to get started using GCS for workloads.The FileSystem Abstraction Apache Hadoop includes a ‘FileSystem’ abstraction, which is used to access data on HDFS. This is an abstraction, which can be used to talk not just to HDFS, but any other storage layer which has an implementation of the FileSystem abstract class for the specific storage layer. Existing applications don’t need to change, to make use of GCS. Instead, some configuration and path changes is all that is required for existing jobs to run against Cloud Storage.Google Storage Connector The Google Storage Connector provides an implementation of the FileSystem abstraction, and is available in HDP versions mentioned above, to facilitate access to GCS. “gs” is the filesystem prefix used for Google Cloud Storage. e.g. gs://tmpSetting up Credentials

While upgrading the Ambari server from 2.6.2 to Ambari 2.7.0.0-897 ,the ambari-server upgrade command fails with dependency unavailability.[root@c2140-node4 yum.repos.d]# ambari-server upgrade Using python /usr/bin/python Upgrading ambari-serverTraceback (most recent call last): File "/usr/sbin/ambari-server.py", line 35, in <module> from ambari_commons.os_utils import remove_file File "/usr/lib/ambari-server/lib/ambari_commons/os_utils.py", line 39, in <module> from ambari_commons.os_linux import os_change_owner, os_getpass, os_is_root, os_run_os_command, \ File "/usr/lib/ambari-server/lib/ambari_commons/os_linux.py", line 25, in <module> from ambari_commons import subprocess32 File "/usr/lib/ambari-server/lib/ambari_commons/subprocess32.py", line 146, in <module> import importlibImportError: No module named importlibSolution The simple solution is to install the dependency python-importlib.noarch[root@c2140-node4 yum.repos.d]# yum install pyth

This is applicable for CentOS7/RHEL6 - for CentOS7/RHEL7 please follow https://community.hortonworks.com/articles/188269/javapython-updates-and-ambari-agent-tls-settings.htmlUpgrading to jdk1.8.0_171 version disables some of the TLSv1_1 TLSv1 protocols and algorithms. With this only option is using TLS1_2 version but CentOS6/RHEL6 uses Python 2.6 and it does not support TLS1_2. Agent-server communication would fail with below error.WARNING 2018-04-24 16:35:10,989 NetUtil.py:124 - Server at https://***.***.***.***:8440 is not reachable, sleeping for 10 seconds...INFO 2018-04-24 16:35:20,990 NetUtil.py:70 - Connecting to https://***.***.***.***:8440/caERROR 2018-04-24 16:35:20,991 NetUtil.py:96 - EOF occurred in violation of protocol (_ssl.c:579)ERROR 2018-04-24 16:35:20,991 NetUtil.py:97 - SSLError: Failed to connect. Please check openssl library versions. To workaround this problem you can tweak /usr/jdk64/jdk1.8.0_112/jre/lib/security/java.security file in Ambari server host to enab

Nifi build on HDF 3.1.2 and HDF 3.1.0 fail with a depency issue when trying to push data to ADLS. This is because the new version of ADLS has some dependency on hadoop 2.8 feature, which is not available in 2.7.3 which is referenced by nifi.to fix this you can build nifi again. You could eith build it against hadoop 2.8 or againt hdp 2.6.x which should have the classes that ADLS depends on.to do thatgit clone nifi repositorycd <nif-repo-home>/nifi-nar-bundles/nifi-hadoop-libraries-bundle/nifi-hadoop-libraries-nar/vi pom.xmladd a hadoop.version property to the pom.xml as shown below. if already set, no change is needed.change nifi version to match the nifi version you are running for the parent<parent> <groupId>org.apache.nifi</groupId> <artifactId>nifi-hadoop-libraries-bundle</artifactId> <version>1.5.0.3.1.2.0-7</version> </parent> <artifactId>nifi-hadoop-libraries-nar</artifactId> <pac

Note : This feature is available from HDP 3.0 (Ambari 2.7) Ambari 2.7 has a cool new feature where it is integrated with Swagger and you can try and explore all the REST APIs. Steps to use Swagger Login to Ambari Hit this url ( http://{ambari-host}:8080/api-docs) This page takes you to the API explorer where you can try different APIs. Here are some of the screenshots. You can get all the supported endpoints from http://{ambari-host}:8080/api-docs/swagger.json) . Hope this helps 🙂

This tutorial will be designed to extent my other work, and the goal is to provide the reader a frame work for a complete end-to-end solution for sentiment analysis on streaming Twitter data. The solution is going to use many different components in the HDF/HDP stack including NiFi, Solr, Spark, and Zeppelin, and it will also utilize libraries that are available in the Open Source Community. I have included the Reference Architecture, which depicts the order of events for the solution. *Note: Before starting this tutorial, the reader would be best served reading my other tutorials listed below:Twitter Sentiment using Spark Core NLP in Apache ZeppelinConnecting Solr to Spark - Apache Zeppelin Notebook*PrerequisitesEstablished Apache Solr collections called "Tweets" and "TweetSentiment"Downloaded and installed Open Scoring Completed this Tutorial: Build and Convert a Spark NLP Pipeline into PMML in Apache ZeppelinStep 1 - Download and deploy NiFi flow.The NiFi Flow can be found here: t

In kafka there are three types of communication : 1. Between brokers2. Between client and broker. 3. Broker to zookeeperIn order to communicate in kerberos enabled cluster one needs to authenticate itself. So when broker will try to communicate with other broker in the cluster it will need to authenticate first. Same for the clients communicating to the brokers. In Kafka, JAAS files is used for authentication. Lets first understand what's there in JAAS file. In kafka there are two JAAS files : 1. kafka_jaas.conf 2. kafka_client_jaas.conf. Let's discuss about kafka_jaas.conf. This file will be used for authentication when a broker in a cluster tries to communicate with other brokers in cluster. Take a look at the content and understand its purpose : KafkaServer { com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true keyTab="/etc/security/keytabs/kafka.service.keytab" storeKey=true useTicketCache=false serviceName="kafka" principa

jline-sqlline-mysql-connector.tar.gzStep1:Download the attached tar file and untar ittar -xvf jline_sqlline__mysql_connector.tar Step2:Syntax: [root]# java -Djava.ext.dirs=/path/to/jline_sqlline__mysql_connector/ sqlline.SqlLineExample:[root@mi3 sqlline]# java -Djava.ext.dirs=/root/sqlline/jline-1.0/ sqlline.SqlLinesqlline version 1.0.2 by Marc Prud'hommeaux Step3:Syntax:sqlline> !connect jdbc:mysql://<mysql hostname>:<port>/<db> <username> <password>Example:sqlline> !connect jdbc:mysql://hostabc:3306/hue hue hueConnecting to jdbc:mysql://mi1.openstacklocal:3306/hueConnected to: MySQL (version 5.1.73)Driver: MySQL-AB JDBC Driver (version mysql-connector-java-5.1.17-SNAPSHOT ( Revision: ${bzr.revision-id} ))Autocommit status: trueTransaction isolation: TRANSACTION_REPEATABLE_READ0: jdbc:mysql://mi1.openstacklocal:3306/hue>

We have created a quickstart guide to walk you through step by step going from Google Cloud Platform (GCP) account to provisioning big data workloads on GCP using Hortonworks Cloudbreak. In this overview, we walk through the high level tasks to be performed and point you to the documentation with the step by step instructions.Overview There are 4 main steps involved to get started with GCP.Cloudbreak GCP Prerequisites: Setting up your GCP Environment to provision Cloudbreak and HDP/HDF workloads. Install Cloudbreak on GCP: Setting up Cloudbreak on GCP.Create Cloudbreak Credentials: Setting up your credentials in Cloudbreak to deploy HDP/HDF workloads on GCP.Creating a Cluster on GCP: Step by step walkthrough creating a big data workload on GCP.Cloudbreak GCP PrerequisitesBefore we install Cloudbreak we need to setup your working environment and GCP account for Cloudbreak. In this step, we will setup the following prerequisites.Install Google Cloud SDK: Setting up the Google Cloud SDK a

Introduction & Context There is a reason why I spent my life studying and working in computer science: understanding a computer's psychology is usually fairly straight forward. Indeed, when presented with a specific input, computer programs tend to respond in a very predictable way, as opposed to our fellow human beings. Of course, this observation goes out of the window as our algorithms become increasingly more complex and capable of learning. Regardless, as much as I love computer science, I always had a keen interest in human sciences. Personality psychology is a fascinating subject that has seen its ups and downs as any science topic. At the center of personality psychology reside the big five personality traits: Openness to Experience Conscientiousness Extraversion Agreeableness Neuroticism (or Emotional Stability) This taxonomy was determined by applying statistical models to personality surveys, essentially clustering results of surveys of

Currently Ambari (as of 2.1.2.1) does not support Postgres HA.

What are Ambari Blueprints ?Ambari Blueprints are definition of your HDP cluster in “JSON” format, it contents information about all the hosts in your cluster, their components, mapping of stack components with each hosts or hostgroups and other cool stuff. Using Blueprints we can call Ambari APIs to completely automate HDP installation process. Interesting stuff, isn’t it ?Lets get started with single node cluster installation. Below are the steps to setup single-node HDP cluster with Ambari Blueprints..Note - For Ambari 2.6.X onwards, we will have to register VDF to register internal repository, or else Ambari will pick up latest version of HDP and use the public repos. please see below document for more information. For Ambari version less than 2.6.X, this guide will work without any modifications.Document - https://docs.hortonworks.com/HDPDocuments/Ambari-2.6.0.0/bk_ambari-release-notes/content/ambari_relnotes-2.6.0.0-behavioral-changes.html.Step 1: Install Ambari server using step