Community Articles

IntroductionHow many times we've heard that a problem can't be solved if it isn't completely understood? However, all of us, even when having a lot of experience and technical skill, we still tend to jump to the solution workspace by-passing key steps in the scientific approach. We just don’t have enough patience and, quite often, the business is on our back to deliver a quick fix. We just want to get it done! We found ourselves like that squirrel in the Ice Age chasing for the acorn just using our experience and minimal experiment, dreaming that the acorn will fall on our lap. This article is meant to re-emphasize a repeatable approach that can be applied to tune Storm topologies starting from understanding the problem, forming a hypothesis, testing that hypothesis, collecting the data, analyzing the data, drawing conclusions based on facts and less on empirical. Hopefully it helps to catch more acorns!Understand the ProblemUnderstand the functional problem resolved by the topology; q

SYMPTOM : Immediately after exporting HDFS directories via NFS , some of the directories start throwing permission denied errors to authorized users added in Ranger policies.ROOT CAUSE : NFS neither honors Ranger policies nor HDFS ACLs. If a directory has HDFS permission bits such as 000 and access is controlled fully via Ranger, this directory won’t be exported at all. Messages such as below can be seen in NFS gateway logs :- 2016-07-27 17:35:19,071 INFO mount.RpcProgramMountd (RpcProgramMountd.java:mnt(127)) - Path /test1 is not shared. 2016-07-27 17:35:37,297 INFO mount.RpcProgramMountd (RpcProgramMountd.java:mnt(127)) - Path /test2 is not shared. 2016-07-27 17:39:34,581 INFO mount.RpcProgramMountd (RpcProgramMountd.java:mnt(144)) - Giving handle (fileId:12345) to client for export /Even if the directory gets exported due to some available permissions, effective permission bits are only from HDFS and not from Ranger policies.

SYMPTOMS : Errors such as "KeeperErrorCode = NoAuth for /config/topics"ROOT CAUSE : Errors such as above are reported while trying to create or delete topic from an ordinary user because only the process owner of Kafka service such as root can write to zookeeper znodes i.e. /configs/topics.Ranger policies do not get enforced when a non privileged user creates a topic is because kafka-topics.sh script talks directly to zookeeper in order to create the topics. It will add entries into the zookeeper nodes and the watchers on the broker side will monitor and create topics accordingly. Due to this process involving zookeeper, the authorization cannot be done through the ranger plugin. NEXT STEPS : If one would want to allow users to be able to create topics, We have a script called kafka-acls.sh which would help allow or deny users on topics and many such options. The details are elaborated in the document mentioned below :- http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.0/bk_secure-

SizingThere's no simple rule of thumb for this, it's as much an art as it is a science, as it depends on the workloads and how chatty they are with your current ZKs.One way to look at this is:If you have 3 ZKs you can afford to lose one, if you have 5 you can afford to lose two.If your IT is aggressively applying security patches and other upgrades, like firmware, kernel, Java, other packages used by Hadoop tools, and taking nodes down to do the job, then during those upgrades with 3 ZKs, you ZK runs with only two nodes, and if you are unlucky and one of them goes down, then your whole cluster will go down. So, in this case 5 are better.Warning: the more ZK nodes you have, the slower the ZK becomes for writes.PlacementZookeeper is a master node, as such it can be collocated with other master services. Ideally, you would not want to collocate it with an HA service. It is quite light on memory and CPU requirements, but since is disk intensive, don't collocate it with disk-intensive servi

Disable Transparent Huge Pages (THP)Transparent Huge Pages (THP) is a Linux memory management system that reduces the overhead of Translation Lookaside Buffer (TLB) lookups on machines with large amounts of memory by using larger memory pages.However THP feature is known to perform poorly in Hadoop cluster and results in excessively high CPU utilization.Disable THP to reduce the amount of system CPU utilization on your worker nodes. This can be done by ensuring that both proc entries are set to [never] instead of [always].Use Recommended File System TypesSome file systems offer better performance and stability than others. As such, the HDFS dfs.datanode.data.dir and YARN yarn.nodemanager.local-dirs should be configured to use mount points that are not formatted with the most optimal file systems. Take a look at this article on file system choices: https://community.hortonworks.com/articles/14508/best-practices-linux-file-systems-for-hdfs.htmlDisable Host SwappinessThe Linux kernel pr

Linux ACLs are implemented in such a way that setting default ACLs on parent directory shall automatically get inherited to child directories and umask shall have no influence in this behavior. However HDFS ACLs have slightly different approach here, they do take into account umask set in hdfs-site.xml in parameter "fs.permissions.umask-mode" and enforce ACLs on child folders based on these two parameter with umask taking precedence over the other.Let's try and reproduce this case :- [gaurav@test ~]$ fs -mkdir /tmp/acltest [gaurav@test ~]$ fs -setfacl -m default:mask::rwx /tmp/acltest [gaurav@test ~]$ fs -setfacl -m mask::rwx /tmp/acltest [gaurav@test ~]$ fs -setfacl -m default:user:adam:rwx /tmp/acltest [gaurav@test ~]$ fs -setfacl -m user:adam:rwx /tmp/acltest Let's see what ACLs are implemented :- [gaurav@test~]$ fs -getfacl /tmp/acltest # file: /tmp/acltest # owner: gaurav# group: hdfs user::rwx user:adam:rwx group::r-x mask::rwx other::r-x default:user::rwx default:user:adam:rwx d

Below are the steps to run Hive(TEZ) query in a shell script using Oozie shell action.1. Configure job.propertiesExample:#*************************************************# job.properties#*************************************************nameNode=hdfs://<namenode-fqdn>:8020jobTracker=<resourcemanager-host-fqdn>:8050queueName=defaultexamplesRoot=examplesoozie.use.system.libpath=trueoozie.wf.application.path=${nameNode}/user/${user.name}/.2. Configure Workflow.xmlExample:<?xml version="1.0" encoding="UTF-8"?><workflow-app xmlns="uri:oozie:workflow:0.4"name="test-shell-with-kerberos-wf"> <global> <job-tracker>${jobTracker}</job-tracker> <name-node>${nameNode}</name-node> <configuration> <property> <name>mapred.job.queue.name</name> <value>${queueName}</value> </property> <property> <na

HiveServer2 does not expose jmx or equivalent metrics but often we require to monitor the Hiveserver2 process to know cpu usage, heap usage and state of various thread along with some other params like no of class loaded by hs2 jvm. it is also very useful if you want to run some cpu/memory sampling to profile hiveserver2.Step 1: login into Ambari and open hive advanced configurationStep 2: expand advanced hive-env settings and add the following configuration in hive-env templateif [ "$SERVICE" = "hiveserver2" ]; then export HADOOP_CLIENT_OPTS="$HADOOP_CLIENT_OPTS -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=8008"fiStep 3: Restart HiveServer2Step 4: on client machine open jvisualvm($JAVA_HOME/bin), once open go to File -> Add JMX Connection -> enter hostname and port no(it's 8008 in this case) and you are good to go.

If you are not using any Hive Authorization like SQL Standard Authorization or Ranger and you don't want other to read/write data and allow the only owner to read & write, and allowing the group to read.Changing the warehouse directory permission.1.From the command line in the Ambari server node, edit the file/var/lib/ambari-server/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive.pySearch for hive_apps_whs_dir which should go to this block: params.HdfsResource(params.hive_apps_whs_dir, type="directory", action="create_on_execute", owner=params.hive_user, group=params.user_group, mode=0755 ) Modify the value for mode from 0755 to the desired permission, for example, 0777.Save and close the file.Restart the Ambari server to propagate the change to all nodes in the cluster:ambari-server restartIt may take a few seconds to update the file in the Ambari agents on all nodes. To verify if the change has been applied on a particular node, check the content of

I recently ran into a situation where I had enabled HDFS HA and later had to change the value of dfs.nameservices. So basically during HA setup I set the value for dfs.nameservices to "MyHorton", but a couple hours later realized I should have used "MyCluster" instead.This article explains how you can change the dfs.nameservices value after HDFS HA has been enabled already. Background: What is the purpose of dfs.nameservices? Its the logical name of your HDFS nameservice. Its important to remember that there are several configuration parameters that have a key, which includes the actual value of dfs.nameservices, e.g. dfs.namenode.rpc-address.[nameservice id].nn1Preparation: Put your HDFS in safemode and backup the namespace (https://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-hdfs/HDFSCommands.html#dfsadmin; dfsadmin -safemode enter; dfsadmin -saveNamespace); Stop Namenode service Backup Hive Metastore (mysqldump hive > /tmp/mydir/backup_hive.sql) Change Configuration

Although it is a simple solution, there is not much reference to this exact problem in the www.We have seen people doing multiple projection, flattening and group by to get the data into shape for storing into the final table using pig after the aggregation and group operations.Source Table:CREATE EXTERNAL TABLE IF NOT EXISTS source(pk1 string,pk2 string,agg1 INT,agg2 INT)STORED AS ORC tblproperties("orc.compress"="ZLIB");Data:Target Table:CREATE EXTERNAL TABLE IF NOT EXISTS target_aggregated(pk1 string,pk2 string,sum_agg1 BIGINT,sum_agg2 BIGINT)STORED AS ORC tblproperties("orc.compress"="ZLIB");Pig Script:--Load the data in pig relationsstaging = LOAD 'DEFAULT.SOURCE' USING org.apache.hive.hcatalog.pig.HCatLoader();--Group the datagroup_staging = group staging BY (pk1,pk2);--Flatten the grouped data and generate aggregates with same attribute names as the target tablecalculate_group_staging = FOREACH group_staging GENERATE FLATTEN(group) AS(pk1,pk2),SUM(staging.agg1) as sum_agg1, SU

Pre-requisite :Remove the incubator-ranger project from eclipse if it's already there and close the eclipse. Clone incubator-ranger repo in the local machine from GitHub. Ignore this step if you have already cloned the required ranger branch. To check out the code from Apache Incubator-ranger GIT repository refer below command. For example :git clone https://github.com/apache/incubator-ranger.git Set JAVA_HOME in the environment variable.For example : export JAVA_HOME=/usr/lib/jvm/java-1.7.0-openjdk.x86_64 Install maven 3.3.9 or higher. Ignore this step if required version of maven is already installed. To get the installed maven version try command : mvn -versionFor example : cd /usr/local wget http://mirror.fibergrid.in/apache/maven/maven-3/3.3.9/binaries/apache-maven-3.3.9-bin.tar.gz tar xvf apache-maven-3.3.9-bin.tar.gzSet maven command line tool in the system path environment variable:For example : export M2_HOME=/usr/local/apache-maven-3.3.9export M2=$M2_HOME/binexport PATH=$M2:

To start pulling jmx metrics from hadoop you need to first enable it via jvm parameters. Go to ambari-->Yarn-->config-->yarn-envEnable jmx by adding the following params to YARN_RESOURCEMANAGER_OPTS"-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=8001"Set the port to whatever port is available on your clusterSave the new config. This will require restart of yarn. Now you have jmx enabled but you need to client to start pulling jmx metrics. Go to your data node and and download (from here http://wiki.cyclopsgroup.org/jmxterm/download.html) the latest jxmterm-xxx-xxx-uber.jar. For this article I use jmxterm-1.0-alpha-4-uber.jarOnce you have the jmxterm client downloaded lets connect to jxm using java -jar jmxterm-1.0-alpha-4-uber.jar -l service:jmx:rmi:///jndi/rmi://localhost:<YOURPORT>/jmxrmiIn this example I am set port to 8012.java -jar jmxterm-1.0-alpha-4-u

Sometimes you have Java messages that you would like to easily ingest into HDFS or perhaps HDFS as raw files, Phoenix, Hive and other destinations. You can do that pretty easy with Apache NiFi 1.0.0 as part of HDF 2.0. For this simple example, I also added a REST gateway for bulk loading, testing and to provide another way to easily send JMS messages. ListenHTTP accepts HTTP POSTS on port 8099, which I made the listener port for that processor. It takes what you send and publishes that to a JMS queue. I am using ActiveMQ.I have a little Python 2.7 script that I found on github that makes fake log records and modified it to send 1,000 JSON messages via REST to our REST to JMS gateway in NIFI for testing. You can easily do this with shell script and CURL, Apache JMeter, Java code, Go script and many other open source REST testers and clients.url = 'http://server.com:8099/contentListener'r = requests.post(url, json={"rtimestamp": timestamp, "ip": random_ip(), "country": country,

To make it easier for you to find the information that you need, we significantly reorganized SmartSense 1.3.0 documentation: SmartSense Installation introduces the new SmartSense 1.3.0 architecture, and guides you through the installation and upgrade process.SmartSense User Guide guides you as you’re using SmartSense functionality, including accessing and using the new Activity Explorer.ReleaseNotes informs you about new features, fixed issues, and known issues.Check out these updates and help us improve the documentation by providing your feedback on HCC.Thanks! Hortonworks Technical Documentation Team

GoalsUse custom kerberos Principal for configuring HAWQ loginVerifying the usage by creating, loading and query an object with non-default principalNOTE HAWQ does not allow these parameters to be changed during the installation as configuration files are only generated after initialization of the HAWQ init process, so this is applicable only when you have a HAWQ cluster up and running with kerberos enabled. Steps Part 1: KerberosEnsure that the HAWQ is up and runningLogin to kerberos server and create a custom headless principalkadmin.local: addprinc -randkey custom-postgres@HORTONWORKS.COMWARNING: no policy specified for custom-postgres@HORTONWORKS.COM; defaulting to no policyPrincipal "custom-postgres@HORTONWORKS.COM" created.Export the key into a keytab file, ensure that the name remains as "hawq.service.keytab" or else this will be required to change kadmin.local: xst -k hawq.service.keytab custom-postgres@HORTONWORKS.COM....kadmin.local: quitCopy over the keytab file over to al

Configuring HTTPFSTo configure HTTPFS you can refer below post, it clearly explains the steps. We can install httpfs in any host where Hadoop clients are installed and we call it <HTTPFS_HOST>.HTTPFS - Configure and Run with HDPCreate topology fileIn Knox Gateway host under /etc/knox/conf/topologies directory create a new topology file (copy default.xml) say example.xml After copying it looks like below [root@test-3 topologies]# ls -l /etc/knox/conf/topologiestotal 20-rw-r--r--. 1 knox knox 89 Oct 4 11:25 README-rw-r--r--. 1 knox knox 4422 Oct 4 11:25 admin.xml-rw-r--r--. 1 knox knox 3026 Oct 12 10:22 default.xml-rw-r--r--. 1 knox knox 3026 Oct 12 10:35 example.xmlIn example.xml topology file for WEBHDFS role change host to <HTTPFS_HOST> and port to 14000 . By default HTTPFS_HTTP_PORT is 14000 <service> <role>WEBHDFS</role> <url>http://<HTTPFS_HOST>:14000/webhdfs</url> </service&

Often lines of business, individual users or shared teams will use online Google Sheets to share spreadsheet and tabular data amongst teams or without outside vendors. It's quick and easy to add sheets and store your data in Google Drive as spreadsheets.Often you will want to consolidate, federate, analyze, enrich and use this data for reporting and dashboards throughout your organization.An easy way to do that is to read in the data using Google's Sheet API. This is a standard SSL HTTP REST API that returns clean JSON data. I created a simple Google Sheet to test ingesting a Google Sheet with HDF. You will need to enable Google Sheets API in the Google APIs Console. You must be logged into Google and have a Google Account (use the one where you created your Google Spreadsheets).Google DocumentationGoogle provides a few Quick starts that you can use to ingest this data: https://developers.google.com/sheets/quickstart/jsor https://developers.google.com/sheets/quickstart/python.

This topic describes installing and configuring Splice Machine on a Hortonworks Ambari-managed cluster. Follow these steps: Verify Prerequisites Download and Install Splice Machine Configure Hadoop Services Make any needed Optional Configuration Modifications Restart the Cluster Verify your Splice Machine InstallationVerify PrerequisitesBefore starting your Splice Machine installation, please make sure that your cluster contains the prerequisite software components:A cluster running HDP 2.4.2HBase installedHDFS installedYARN installedZooKeeper installedDownload and Install Splice MachinePerform the following steps on each node in your cluster:1.Create the splice installation directory:sudo mkdir -p /opt/splice2.Download the Splice Machine package into the splice directory on the node:sudo curl 'https://s3.amazonaws.com/20snapshot/installer/2.0.1.23/hdp2.4.2/SPLICEMACHINE-2.0.1.23-SNAPSHOT.hdp2.4.2.p0.108.tar.gz' -o /opt/splice/SPLICEMACHINE-2.0.1.23-SNAPSHOT.hdp2.4.2.p0.108

As part of a live Drone ingest, I want to identify what in the image. The metadata provides a ton of information on GPS, altitude and image characteristics, but not what's in the image.IBM, Microsoft and Google all have APIs that do a good job of this and they for the most part of "free" tiers. I wanted to run something locally using libraries installed on my cluster. For my first option, I used TensorFlow Inception-v3 Image Recognition. In future articles I will cover PaddlePaddle, OpenCV and some other Deep Learning and non-deep learning options for Image Recognition. Also I will show the entire Drone to Front-End flow including Phoenix, Spring Boot, Zeppelin, LeafletJS and more. This will be done as part of a meetup presentation with a certified drone pilot.To Run My TensorFlow Binary From HDF 2.0 I use the ExecuteStreamCommand to run a shell script containing the information below:source /usr/local/lib/bazel/bin/bazel-complete.bashexport JAVA_HOME=/opt/jdk1.8.0_101/hdfs df

If you haven't already done so first download the Sandbox from the Hortonworks Website and import it to Virtualbox/VMware. Once you start the Virtual Machine you are directed to the landing page for your new sandbox. http://localhost:8888/Here you have 2 Paths to choose from. If you are completely new to HDP I encourage you to browse the "New to HDP" section. Click on Advanced HDP to get to the links for the components we will be using in this tutorial. If you look at the Atlas link you see the Note: Off by default. Enable by logging in as raj_ops into AmbariYou will need to start in Ambari as user ... raj_opsGo ahead and click the link for Ambari and login. If you see a page that looks like this you are doing great! Atlas has dependencies on other Apache projects. We need to make sure HDFS, Hbase, Kafka and Ambari Infra are all started. Click on each service and then under the service actions drop down select start. After verifying all components are up and running go ahead and follo

Enable a Ranger Plugin and audit to HDFS for a Hadoop component say in this case HiveServer2.Audit files will be stored in folder structure defined in the audit configuration file of the respective ranger plugin. Default format is /ranger/audit/<hadoop-component>/<YYYYMMDD>/<component>_ranger_audit_<hosname>.<count>.logExample audit log for HiveServer2 ranger plugin:hdfs dfs -ls -R /ranger/audit/hiveServer2 /ranger/audit/hiveServer2/20160315/ranger/audit/hiveServer2/20160315/hive_ranger_audit_.1.log/ranger/audit/hiveServer2/20160315/hive_ranger_audit_.2.log/ranger/audit/hiveServer2/20160316/ranger/audit/hiveServer2/20160316/hive_ranger_audit_.1.log/ranger/audit/hiveServer2/20160316/hive_ranger_audit_.2.log/ranger/audit/hiveServer2/20160317/ranger/audit/hiveServer2/20160317/hive_ranger_audit_.1.log/ranger/audit/hiveServer2/20160317/hive_ranger_audit_.2.logProcedure to Create Store Ranger Audit Log in HIVE Create a Hive External table with a dummy location

In HDF 2.0, administrators can secure access to individual NiFi components in order to support multi-tenant authorization. This provides organizations the ability to create least privilege policies for distinct groups of users. For example, let's imagine we have a NiFi Team and a Hadoop Team at our company and the Hadoop Team can only access dataflows they've created, whereas the NiFi Team can access all dataflows. NiFi 1.0 in HDF 2.0 can use different authorizers, such as file-based policies (managed within NiFi) and Ranger-based policies (managed within Ranger), as well as custom, pluggable authorizers. In this example, we'll use Ranger. For more detail on configuring Ranger as the authorizer for NiFi, please see this article. To separate the different teams' dataflows, we'll create separate process groups for each team. In NiFi, access policies are inheritable, supporting simpler policy management with the flexibility of overriding access at the component level. This means tha

Administrator Operations The operations described in this section require superuser privileges. Allow Snapshots: Allowing snapshots of a directory to be created. If the operation completes successfully, the directory becomes snapshottable. Command: hadoop dfsadmin -allowSnapshot $path Arguments: path – The path of the snapshottable directory. See also the corresponding Java API void allowSnapshot(Path path) in HdfsAdmin. Disallow Snapshots: Disallowing snapshots of a directory to be created. All snapshots of the directory must be deleted before disallowing snapshots. Command: hadoop dfsadmin -disallowSnapshot $path Arguments: path – The path of the snapshottable directory. See also the corresponding Java API void disallowSnapshot(Path path) in HdfsAdmin. User Operations The section describes user operations. Note that HDFS superuser can perform all the operations without satisfying the permiss

HDFS Snapshots Overview HDFS Snapshots are read-only point-in-time copies of the file system. Snapshots can be taken on a subtree of the file system or the entire file system. Some common use cases of snapshots are data backup, protection against user errors and disaster recovery. The implementation of HDFS Snapshots is efficient in the following ways: 1) Snapshot creation is instantaneous. The cost is O(1) excluding the inode lookup time. 2) Additional memory is used only when modifications are made relative to a snapshot. Memory usage is O(M), where M is the number of modified files/directories. 3) Blocks in datanodes are not copied. The snapshot files record the block list and the file size. 4) Snapshots do not adversely affect regular HDFS operations, and there is a minor performance impact from accessing snapshotted data depending on the number of modifications. The snapshot data is computed by subtracting the modifications from the current dat