Resources

I was looking for a way to access HDFS extended attributes metadata in a Hive query. Extended attributes can be used to tag HDFS files with metadata such as file source, import time... They can then be used in queries for all records contained in such files, whatever their format is.After asking the community and digging further into it, I came up with the following Groovy UDF based on using Hive virtual column INPUT__FILE__NAME:compile `import org.apache.hadoop.hive.ql.exec.UDF \;import org.apache.hadoop.io.Text \;import org.apache.hadoop.conf.Configuration \;import org.apache.hadoop.fs.FileSystem \;import org.apache.hadoop.fs.Path \;import java.net.URI \;public class XAttr extends UDF {public Text evaluate(Text uri, Text attr){if (uri == null || attr == null) return null \; URI myURI = URI.create(uri.toString()) \;Configuration myConf = new Configuration() \;FileSystem fs = FileSystem.get(myURI, myConf) \;return new Text(fs.getXAttr(new Path(myURI), attr.toString())) \;}} ` AS GROOVY

Here are some key things that will help an HDInsight cluster manageable and perform better. The following best practices items should be noted. Do not use only one storage account for a given HDInsight cluster. For a 48 node cluster, Microsoft is recommending 4-8 storage accounts. Not because of the storage space but what each storage account provides additional networking bandwidth that opens up the pipe as wide s possible for the compute nodes to finish their jobs faster. Make the naming convention of the storage account as random as possible, no prefix. This is to reduce the chances that you hit storage bottlenecks or common mode failures in storage across all storage accounts at the same time. This type of storage partitioning in WASB is meant to avoid storage throttling. Use D13 for head nodes, D12 for worker nodes.When containers are created, make sure to only have one container per storage account. This yields better performance.The Hive metastore that comes by default when HDI

Description - When we start the oozie/hive from ambari it checks the connectivity with db first before starting using below command /usr/java/jdk64-1.7.0_79/bin/java -cp /usr/lib/ambari-agent/DBConnectionVerification.jar:/usr/hdp/current/oozie-server/libext/ojdbc.jar org.apache.ambari.server.DBConnectionVerification '<<DB URL>>Error - Ambari was not able to load the class due to permission issue on the jar file Current Permission —> -rw-r----- 1 root root 1931 Jan 29 01:02 /usr/lib/ambari-agent/DBConnectionVerification.jar(640) Required Permission —> -rw-r--r-- 1 root root 1931 Jan 29 01:02 /usr/lib/ambari-agent/DBConnectionVerification.jar(644)Change the permission from 640 to 644.

Incorporating the Apache NiFi Receiver into your Spark application is pretty easy. First, you'll need to add the Receiver to your application's POM: <dependency> <groupId>org.apache.nifi</groupId> <artifactId>nifi-spark-receiver</artifactId> <version>0.4.1</version> </dependency>That's all that is needed in order to be able to use the NiFi Receiver. So now we'll look at how to use the Receiver in your code.The NiFi Receiver is a Reliable Java Receiver. This means that if we lose a node after it pulls the data from NiFi, the data will not be lost. Instead, another node will simply pull and process the data. In order to create a NiFi Receiver, we need to first create a configuration that tells the Receiver where to pull the data from. The simplest form is to just tell the config where NiFi is running and which Port to pull the data from:SiteToSiteClientConfig config = new SiteToSiteClient.Builder() .url("http://localhost:8080/nifi"

I'll use iperf tool, which is available in the EPEL repo if you're using RH/CentOS.Installation is pretty straightforward :# yum -y install iperfFor use, you'll have to execute one instance on the first machine as the server, and another on the other machine as the client.Execute as a server (-s) on port (-p) 2000 :[root@machine01 ~]# iperf -s -p 2000------------------------------------------------------------Server listening on TCP port 2000TCP window size: 85.3 KByte (default)Now execute client on the other machine for 25 seconds (-t 25) with 5 seconds (-i5) between each periodic report, so we'll have 5 reports[root@machine02 ~]# iperf -c machine01 -p 2000 -i5 -t25------------------------------------------------------------Client connecting to machine01, TCP port 2000TCP window size: 23.2 KByte (default)------------------------------------------------------------[ 3] local 10.195.196.18 port 45284 connected with 10.195.196.48 port 2000[ ID] Interval Transfer Bandwidth[ 3]

Kylin Pronounced “KEY LIN” / “CHI LIN” - This project brings OLAP (Online Analytical Processing) to Big Data. It is a top-level project in Apache. Through it’s UI, you can create a logical model (dimensions/measures) from a star schema in Hive. Kylin will then create cube aggregates using MR and put the aggregates and cube metadata into HBase. Users can then query the cube data through the Kylin UI or a BI tool that uses the Kylin odbc driver. A good video from the committers overviewing the project: https://www.youtube.com/watch?v=7iDcF7pNhV4 DefinitionsCube - A data structure containing dimensions and measures for quickly accessing aggregated information (measures) across many axis's (dimensions)Cuboid - A "slice" or subset of a cubeDimensions - Think of these as alphanumeric columns that sit in a group by clause of SQL. i.e. Location, Department, Time, etc.Measure - Think of these as metric/numerical values that sit in a select clause of SQL. i.e. Sum(value), Max(bonus), Min(e

Fix: HDP “Unauthorized connection for super-user: oozie from IP 127.0.0.1” Recently have been playing with HortonWorks HDP 2.2. Was starting to configure some oozie workflows and when submitting the job the first step’s Hive script failed with this error and stack. JA002: Unauthorized connection for super-user: oozie from IP 127.0.0.1 Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException): Unauthorized connection for super-user: oozie from IP 127.0.0.1at org.apache.hadoop.ipc.Client.call(Client.java:1468) at org.apache.hadoop.ipc.Client.call(Client.java:1399) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232) at com.sun.proxy.$Proxy39.getDelegationToken(Unknown Source) at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getDelegationToken(ApplicationClientProtocolPBClientImpl.java:306)... 30 moreI was found the solution here.

Node labels enable you partition a cluster into sub-clusters so that jobs can be run on nodes with specific characteristics. For example, you can use node labels to run memory-intensive jobs only on nodes with a larger amount of RAM. Node labels can be assigned to cluster nodes, and specified as exclusive or shareable. You can then associate node labels with capacity scheduler queues. Each node can have only one node label. Demo: Use case 2 node labels : node1 & node2 + Default & Spark queue Submit job to node1 Node labels added : yarn rmadmin -addToClusterNodeLabels "node1(exclusive=true),node2(exclusive=false)" Label assigned: yarn rmadmin -replaceLabelsOnNode "phdns02.cloud.hortonworks.com=node2 phdns01.cloud.hortonworks.com=node1" Job Submission: Job send to node1 only and assign to queue spark hadoop jar /usr/hdp/current/hadoop-yarn-client/hadoop-yarn-applications-distributedshell.jar -shell_command "sleep 100" -jar /usr/hdp/current/hadoop-yarn

Hi Everyone,I came across a Kerberos cache issue and wanted to share and possible have more ideas. I understand few of us had some issues in the past and hope this article might help.One of our clients have RedHat IDM (supported version of freeIpa) and when you install sssd along with krb5 by IDM the default cache setting is 'KEYRING' than 'File'You will still be able to get the tickets but will have GSSException error on Cluster.KEYRING persistent cache setting works with many applications however not with HADOOP Cluster, I'm sure HDO engineering team must be looking into this for a solution, since KEYRING is the future for kerberos cache and the stuff you can do with it like keylist and etc.To solve the errors you can comment out the "default_ccache_name=KEYRING....." on krb.conf or change it to "default_ccache_name = FILE:/tmp/krb5cc_%{uid}"Logout and log in again - destroy the previous tickets and you should have something like "Ticket cache: FILE:/tmp/krb5cc_" in your klist outpu

dfs.datanode.max.xcievers / dfs.datanode.max.transfer.threads = 4096 (use 16k if running HBase) 
dfs.datanode.balance.max.concurrent.moves = 500 (can go to 1000 if needed) 
/* Each data node has a limited bandwidth for rebalancing. The default value for the bandwidth is 5MB/s. In the worst case, each data transfer has a limited bandwidth of 1MB/s. Default is dfs.datanode.balance.bandwidthPerSec = 5242880 */dfs.datanode.balance.bandwidthPerSec = 104857600 /* 100 MB/s */ hdfs balancer -Dfs.defaultFS=hdfs://<NN_HOSTNAME>:8020 -Ddfs.balancer.movedWinWidth=5400000 -Ddfs.balancer.moverThreads=1000 -Ddfs.balancer.dispatcherThreads=200 -Ddfs.datanode.balance.max.concurrent.moves=5 -Ddfs.balance.bandwidthPerSec=100000000 -Ddfs.balancer.max-size-to-move=10737418240 -threshold 5

First 15 seconds - Queue MappingParent article CS view

Question:I am about to initiate the cluster install wizard on a new Ambari install. I reviewed the information on service users at http://docs.hortonworks.com/HDPDocuments/Ambari-2.2.0.0/bk_ambari_reference_guide/content/_defining_service_users_and_groups_for_a_hdI am wondering whether I should take the "Skip Group Modifications" option.The doc states "Choosing this option is typically required if your environment manages groups using LDAP and not on the local Linux machines". In our environment, users and groups are managed via Active Directory (via Centrify). We are planning to enable security on the cluster after it's installed, and that will include a host of new users being created, after which many of the initial users and groups will be orphaned.What does that "Skip group modifications" option actually do? Should it be used in this case?Answer:I believe the answer lies in the fact that we do a groupmod hadoop statement and there is no group called hadoop, or this is not a

Original1) Setup Azure account2) Setup CloudBreak accountVery important steps : Applies to Azure onlyCreate a test network in Azure before you start creating cloudbreak credentials. In your local machine, run the following and accept default values. openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout azuretest.key -out azuretest.pem You will see 2 files as listed below.-rw-r--r-- 1 nsabharwal staff 1346 May 7 17:00 azuretest.pem --> We need this file to create credentials in cloudbreak.-rw-r--r-- 1 nsabharwal staff 1679 May 7 17:00 azuretest.key --> We need this to login into the host after cluster deployment.chmod 400 azuretest.key --> otherwise, you will receiver bad permission error for example: ssh -i azuretest.key ubuntu@<server>Very important: check your openssl version and if it's latest version then run the following and use azuretest_login.key to login openssl rsa -in azuretest.key-out azuretest_login.keyhw11326:jumk nsabharwal$ openssl vers

1OverviewTraditionally enterprises have been dealing with data flows or data movement within their data centers. But as the world has become more flattened and global presence of companies has become a norm, enterprises are faced with the challenge of collecting and connecting data from their global footprint. This problem was daunting NSA a decade ago and they came up with a solution for this using a product which was later named as Apache Nifi. Apache nifi is a easy to use, powerful, and reliable system to process and distribute data. Within Nifi, as you will see, I will be able to build a global data flow with minimal to no Coding. You can learn the details about Nifi from Apache Nifi website. This is one of most well documented Apache projects. The focus of this article to just look at one specific feature within Nifi that I believe no other software product does it as well as Nifi. And this feature is “site to site” protocol data transfer. 2Business use caseOne of the classi

Below are steps to replace disk in slave nodes or to perform maintenance of slave nodes servers.1. Decommission the Datanode and all services running on it (i.e. NodeManager, HBase RegionServer, Datanode etc). Refer below docs for the same.https://docs.hortonworks.com/HDPDocuments/Ambari-2.4.1.0/bk_ambari-user-guide/content/decommissioning_masters_and_slaves.html https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.0/bk_administration/content/ch_slave_nodes.html 2. Replace the disks or perform any other tasks for server maintenance.3. Recommission the node.4. Start all services components on the node.5. Run Fsck utility for HDFS to ensure that HDFS is in healthy state. FSCK reports usually show a few over replicated blocks after a datanode is recommissioned which would automatically be fixed over time.

This is part 2 of the Dataflow Optimization tutorial. Part 1 can be found here: NiFi/HDF Dataflow Optimization part 1-----------------------------------------------------------------------------------------------------------------------------------------How do I prevent my system/processors from becoming overwhelmed?Another aspect of optimization is preventing the dataflows from overwhelming the underlying system and affecting NiFi software stability and/or performance. If allowed to run unconstrained, it is more likely to see a severe performance hit. You are more likely to see severe performance hit if dataflows are unbounded and the content repo fills 100%For example, if you have 50MB for the content_repository partition and data normally is delivered via an input port in groups of 10MB. What would happen if the data source was down for a few hours and then came back online with a backlog of 75MB? Once the content_repository filled, the input port would start generating errors wh

NiFi/HDF Dataflow OptimizationWho is This Guide For?This guide is for Dataflow Managers (DFM) very comfortable with NiFi and a good knowledge of the underlying system on which NiFi is running. This guide also assumes that the user has applied the recommend changes in the Configuration Best Practices section of the Admin Guide https://nifi.apache.org/docs.html and applied appropriate changes recommended in the setup best practices here: https://community.hortonworks.com/content/kbentry/7882/hdfnifi-best-practices-for-setting-up-a-high-perfo.html. This guide is not intended to be an exhaustive manual for Dataflow Optimization, but more of points to consider when optimizing their own dataflow.What is meant by dataflow optimization?Dataflow Optimization isn’t an exact science with hard and fast rules that will always apply. But more of a balance between system resources (memory, network, disk space, disk speed and CPU), the number of files and size of those files, the types of processor

There are new Data Visualization, Data Explorer tabs that come part of Hive view which are really nice. I found this by accident so thought there may be others who are probably not aware of this either.Environment details: Should work in vanilla cluster/sandbox as well but in my case the env is as below (setup using steps here😞 Kerborized HDP 2.3.2 w/ Ranger installed Secure Ambari 2.1.2 (authenticating to users in IPA LDAP)The sample salary data from sandbox has also been imported and was used for visualization1. After logging to Ambari 2.1.2, open the Hive view (in this case I had to create a new instance of the view configured for kerberos). 2. Click the Data Visualization tab on the right and drag/drop description and salary fields onto x, y fields3. Select from the various chart options to change the chart 4. Click transpose button:5. Navigate to Data Explorer tab to explore your data using the different fieldsSo all in all: visualizations look good, are responsive (at least on

Symptoms  "Permission denied" errors can present in a variety of use cases and from nearly any application that utilizes CDH.   For example, when attempting to start the jobtracker using this command   service hadoop-0.20-mapreduce-jobtracker start    You may see this error, or one similar   org.apache.hadoop.security.AccessControlException: Permission denied: user=mapred, access=WRITE, inode="/":hdfs:supergroup:drwxr-xr-xat org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:224)at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:204)at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:149)at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:4891)at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:4873)at org.apache.hadoop.hdfs.server.namenode.FSNamesy

Please refer to below forked project for details of benchmark commands ,configuration and results, was able to achieve 3 Million records/sec Read , 1.5 Million records/sec and 5 msec 99.9 Percentile for End-to-end Latency.https://gist.github.com/saurabhmishra/6ed428be4c00003dd926

NiFi can be deployed to run as any non-root user. By default NiFi runs as user "nifi", but this can be modified.For instructions, https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html then go to 'Admin Guide' then go to 'Bootstrap Properties’. In there it talks about 'run.as'

**** This article only pertains to HDF 1.x or Apache NiFi 0.x baseline versions. HDF 2.x+ and Apache NiFi 1.x went through a major infrastructure change in the NiFi core.Converting a Standalone NiFi instance into a NiFi Cluster The purpose of this article is to cover the process of taking standalone NiFi instance and converting it into a NiFi cluster. Often times as a NiFi user we start simple by creating a new dataflow on a single installed instance of NiFi. We may find ourselves doing this to save on resources during the build and test phase or because at the time a single server or VM provide all the resources required. Later we want to add in the fault tolerance that a NiFi cluster provides or resources become constrained as our dataflow grows requiring additional resources that a NiFi cluster also provides. It is possible to take all that work done on that single Instance and convert it for use in NiFi Cluster. Your single NiFi instance could easily become one of the

In event there is an error while executing any oozie action during a workflow processing, those root cause errors are wrapped within an Oozie error and some times you have to dig a little deeper to find the root cause of the issue. Once you look in the right place (screen/log) the error gives a direct indication of what the issue is but that error is buried a couple of layers below the Oozie UI. Follow these steps to get more details on Oozie issues - 1) Open Oozie UI 2) Locate the job instance that failed and double click on it 3) Locate the step that failed and double click on it 4) From the Action pop-up, (which also has the cryptic error message in Error Message) click on the small magnifying glass on the right hand side of the Console URL. 5) From the next screen, click on the link that reads "logs" and view the details. (TBD - Add Screen snapshot)

Groovy UDF exampleCan be compiled at run timeCurrently only works in "hive" shell, does not work in beeline<code>su guesthive paste the following code into the hive shellthis will use Groovy String replace function to replace all instances of lower case 'e' with 'E'<code>compile `import org.apache.hadoop.hive.ql.exec.UDF \;import org.apache.hadoop.io.Text \;public class Replace extends UDF { public Text evaluate(Text s){ if (s == null) return null \; return new Text(s.toString().replace('e', 'E')) \; }} ` AS GROOVY NAMED Replace.groovy; now create a temporary function to leverage the Groovy UDF<code>CREATE TEMPORARY FUNCTION Replace as 'Replace'; now you can use the function in your SQL<code>SELECT Replace(description) FROM sample_08 limit 5; full example<code>hive> compile `import org.apache.hadoop.hive.ql.exec.UDF \; > import org.apache.hadoop.io.Text \; > public class Replace extends UDF { > public Text evaluate(Text s){