Community Articles

Using ambari metrics we are able to determine queue utilization. Here is how I did it.Determine which queue you need to run analysis againstcreate a widgetAdd this widget to any dash board..For exampleYou can view the utilization by time using the drop down:

I just completed my first Express upgrade (EU) using Ambari-2.2.0, from HDP-2.2.8 to HDP-2.3.4 and here are my observations and issues I encountered. The cluster has 12 nodes, 2 masters and 10 workers with configured Namenode HA and RM HA running on RHEL-6.5 using Java-7. Installed Hadoop components: HDFS, MR2, Yarn, Hive, Tez, HBase, Pig, Sqoop, Oozie, ZooKeeper, and AmbariMetrics. About 2 weeks before this EU, the cluster was upgraded from HDP-2.1.10 and Ambari-1.7.1. Please use this as a reference: based on cluster settings and previous history (previous upgrade or fresh install), the issues will differ, and the problems I had should by no means considered to be representative, and taking place during every EU. It's good idea to backup all cluster supporting data-bases in advance, in my case Ambari, Hive metastore, Oozie and Hue (although Hue cannot be upgraded by Ambari)There is no need to prepare or download HDP.repo file in advance, Ambari will crate the file, now called HDP-2.3

Use case: There is lot of data in the locally managed table and we want to convert those table into external table because we are working on a use case where our spark and home grown application has trouble reading locally managed tables. Solution: alter table table_name SET TBLPROPERTIES('EXTERNAL'='TRUE');Test:Locally managed table droppedhive> drop table sample_07;OKTime taken: 0.602 secondsData deleted as it's locally managed table. hive> dfs -ls /apps/hive/warehouse/sample_07;ls: `/apps/hive/warehouse/sample_07': No such file or directoryCommand failed with exit code = 1Query returned non-zero code: 1, cause: nullConverting sample_07_test from locally managed to external to keep the data there after drop.hive> alter table sample_07_test SET TBLPROPERTIES('EXTERNAL'='TRUE');OKTime taken: 0.463 secondsTable dropped after converting into external table. hive> drop table sample_07_test;OKTime taken: 0.552 secondsData is still on HDFS hive> dfs -ls /apps/hive/warehouse/

Requirement:Currently we have /hadoop/hdfs/data and /hadoop/hdfs/data1 as datanode directories. I have new mountpoint (/hadoop/hdfs/data/datanew) with faster disk and I want to keep only this mountpoint as datanode directory. Steps:Stop the cluster.Go to the ambari HDFS configuration and edit the datanode directory configuration: Remove /hadoop/hdfs/data and /hadoop/hdfs/data1. Add /hadoop/hdfs/datanew save.Login into each datanode VM and copy the contents of /data and /data1 into /datanewChange the ownership of /datanew and everything under it to “hdfs”.Start the cluster.

The cluster has 1 management node (Bright Cluster Manager and Ambari server), 2 NameNodes (1 active, 1 passive) and 17 DataNodes and is running Hortonworks HDP 2.3.2 and Ambari 2.1.2.Each node has a 2 10GbE NICs which are bonded together and jumbo frames (MTU=9000) is enabled on the interfaces.There are sporadic NodeManager Web UI alerts in Ambari. For all 17 DataNodes we get connection timeouts throughout the day. These timeouts are not correlated with any sort of load on the system, they happen no matter what.When the connection to port 8042 is successful the connection is is around 5-7ms but when the connections fails I get response times of 5 seconds. Never 3 seconds or 6 seconds, always 5 seconds. For example...[root@XXXX ~]# python2.7 YARN_response.pyTesting response time at http://XXXX:8042Output is written if http response is > 1 second.Press Ctrl-C to exit!2016-02-08 07:19:17.877947 Host: XX23:8042 conntime - 5.0073 seconds, HTTP response - 2002016-02-08 07:19:22.889430 Hos

Customer: Is Hadoop Enterprise Ready?Me: Standing next to the whiteboard, Yes and that's why we use the term "Enterprise Ready Data Lake" Imagine that there are 3 pointsPoint 1 -> You need to prove your identity to get access to Lake and then need permissions or authority to access data.Point 2 -> Once you proved your authenticity then demands comes to manage the lifecycle of data from it's requirement to retirement "Automated process"Point 3 -> Life Cycle Management process needs to be integrated with a Governance solution to manage data of data "metadata" , data lineage, auditing and more to fullfil security and compliance requirement.Point 1 --> Entry Point: You must have strong Authentication in place to get into the system and more users will be coming in to access data as we move away from silos of data to a centralized repository. The access management must be easier to manage i,e Security solution should have a centralized place toAdmin (create, define and manage) s

Problem:File"/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 140,in _call_wrapper result = _call(command,**kwargs_copy)File"/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 291,in _callraiseFail(err_msg)resource_management.core.exceptions.Fail:Execution of 'yarn resourcemanager -format-state-store' returned 255.15/10/2616:11:16 INFO resourcemanager.ResourceManager: STARTUP_MSG:15/10/2616:11:17 INFO recovery.ZKRMStateStore: org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore$VerifyActiveStatusThread thread interrupted!Exiting!15/10/2616:11:17 INFO zookeeper.ZooKeeper:Session:0x150a4b3429b0002 closed15/10/2616:11:17 FATAL resourcemanager.ResourceManager:Error starting ResourceManagerorg.apache.zookeeper.KeeperException$NotEmptyException:KeeperErrorCode=Directorynot empty for/rmstore/ZKRMStateRoot/RMAppRoot at org.apache.zookeeper.KeeperException.create(KeeperException.java:125) at org.ap

osquery allows you to easily ask questions about your Linux and OSX infrastructure. Whether your goal is intrusion detection, infrastructure reliability, or compliance, osquery gives you the ability to empower and inform a broad set of organizations within your company. Download https://osquery.readthedocs.org/en/latest/installation/install-linux/ [root@phdns02 ~]# sudo rpm -ivh https://osquery-packages.s3.amazonaws.com/centos6/noarch/osquery-s3-centos6-repo-1-0.0.noarch.rpm Retrieving https://osquery-packages.s3.amazonaws.com/centos6/noarch/osquery-s3-centos6-repo-1-0.0.noarch.rpm warning: /var/tmp/rpm-tmp.rCrgXh: Header V4 RSA/SHA1 Signature, key ID c9d8b80b: NOKEY Preparing... ########################################### [100%] 1:osquery-s3-centos6-repo########################################### [100%] [root@phdns02 ~]# yum install osquery Loaded plugins: fastestmirror Setting up Install Process Loading mirror speeds from cached hostfile epel/metalink | 12 kB 00:00 * base: mirrors.ab

OLAP (Online Analytical Processing) is the technology behind many Business Intelligence (BI) applications. OLAP is a powerful technology for data discovery, including capabilities for limitless report viewing, complex analytical calculations, and predictive “what if” scenario (budget, forecast) planning.OLAP is an acronym for Online Analytical Processing. OLAP performs multidimensional analysis of business data and provides the capability for complex calculations, trend analysis, and sophisticated data modeling. It is the foundation for may kinds of business applications for Business Performance Management, Planning, Budgeting, Forecasting, Financial Reporting, Analysis, Simulation Models, Knowledge Discovery, and Data Warehouse Reporting. OLAP enables end-users to perform ad hoc analysis of data in multiple dimensions, thereby providing the insight and understanding they need for better decision making. SourceOLAP solutions Open source Apache Kylin http://kylin.apache.org/Apache Kyli

After installing zeppelin all goes fine, but when you try to connect you may get a "disconnected" status on the notebook tab and you're unable to create any new note, while the interpreter tab works fine. If you checked you can reach the port on which zeppelin is listen, it may be an issue with your content filter firewall and you should see this log in your firewall application:2016-02-04 15:47:58 Deny 192.168.0.128 40.112.76.49 http/tcp 57772 9995 1-Ecube 0-Internet ProxyDeny: HTTP Invalid Request-Line Format (TCP-UDP-isoardi OUT-00) HTTP-Client.isoardi proc_id="http-proxy" rc="594" msg_id="1AFF-0005" proxy_act="HTTP-Client.isoardi" line="\x81\x8d\xf2\x9eW\xfe\x89\xbc8\x8e\xd0\xa4u\xae\xbb\xd0\x10\xdc\x8f\x81\x8d\xf8]\xb8z\x83\x7f\xd7\x0a" TrafficThe solution is to disable the content filter for the domain on which zeppelin is running

ISSUE:Choosing the appropriate Linux file system for HDFS deploymentSOLUTION:The Hadoop Distributed File System is platform independent and can function on top of any underlying file system and Operating System. Linux offers a variety of file system choices, each with caveats that have an impact on HDFS.As a general best practice, if you are mounting disks solely for Hadoop data, disable ‘noatime’. This speeds up reads for files.There are three Linux file system options that are popular to choose from: Ext3Ext4XFSYahoo uses the ext3 file system for its Hadoop deployments. ext3 is also the default filesystem choice for many popular Linux OS flavours. Since HDFS on ext3 has been publicly tested on Yahoo’s cluster it makes for a safe choice for the underlying file system.ext4 is the successor to ext3. ext4 has better performance with large files. ext4 also introduced delayed allocation of data, which adds a bit more risk with unplanned server outages while decreasing fragmentatio

A customer wants to use Cloudbreak for deploying Hadoop Clusters. They want to scale up and down the hadoop storage nodes.a) How does HDFS detect Scale down of nodes and will it kick in HDFS Rebalance - Cloudbreak instructs Ambari via decommission REST Api call to decommission a DataNode and NodeManager. - Ambari triggers the decomission on the HDP cluster, from birds perspective this is what happens: https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.2/bk_Sys_Admin_Guides/content/ref-a179736c-eb7c-4dda-b3b4-6f3a778bd8c8.1.html but in automated way. - Since the decommission of DataNodes can take a long time if you have a lot of blocks; HDFS needs to replicate blocks belonging to decommissioning DataNodes to other live DataNodes to reach the replication factor that you have specified via dfs.replication in hdfs-site.xml. The default value of this replication factor is 3. - You can get feedback from the decomission process e.g from NameNode UI: http://ip of_namenode:50070/dfshealth.

I was looking for a way to access HDFS extended attributes metadata in a Hive query. Extended attributes can be used to tag HDFS files with metadata such as file source, import time... They can then be used in queries for all records contained in such files, whatever their format is.After asking the community and digging further into it, I came up with the following Groovy UDF based on using Hive virtual column INPUT__FILE__NAME:compile `import org.apache.hadoop.hive.ql.exec.UDF \;import org.apache.hadoop.io.Text \;import org.apache.hadoop.conf.Configuration \;import org.apache.hadoop.fs.FileSystem \;import org.apache.hadoop.fs.Path \;import java.net.URI \;public class XAttr extends UDF {public Text evaluate(Text uri, Text attr){if (uri == null || attr == null) return null \; URI myURI = URI.create(uri.toString()) \;Configuration myConf = new Configuration() \;FileSystem fs = FileSystem.get(myURI, myConf) \;return new Text(fs.getXAttr(new Path(myURI), attr.toString())) \;}} ` AS GROOVY

Here are some key things that will help an HDInsight cluster manageable and perform better. The following best practices items should be noted. Do not use only one storage account for a given HDInsight cluster. For a 48 node cluster, Microsoft is recommending 4-8 storage accounts. Not because of the storage space but what each storage account provides additional networking bandwidth that opens up the pipe as wide s possible for the compute nodes to finish their jobs faster. Make the naming convention of the storage account as random as possible, no prefix. This is to reduce the chances that you hit storage bottlenecks or common mode failures in storage across all storage accounts at the same time. This type of storage partitioning in WASB is meant to avoid storage throttling. Use D13 for head nodes, D12 for worker nodes.When containers are created, make sure to only have one container per storage account. This yields better performance.The Hive metastore that comes by default when HDI

Description - When we start the oozie/hive from ambari it checks the connectivity with db first before starting using below command /usr/java/jdk64-1.7.0_79/bin/java -cp /usr/lib/ambari-agent/DBConnectionVerification.jar:/usr/hdp/current/oozie-server/libext/ojdbc.jar org.apache.ambari.server.DBConnectionVerification '<<DB URL>>Error - Ambari was not able to load the class due to permission issue on the jar file Current Permission —> -rw-r----- 1 root root 1931 Jan 29 01:02 /usr/lib/ambari-agent/DBConnectionVerification.jar(640) Required Permission —> -rw-r--r-- 1 root root 1931 Jan 29 01:02 /usr/lib/ambari-agent/DBConnectionVerification.jar(644)Change the permission from 640 to 644.

Incorporating the Apache NiFi Receiver into your Spark application is pretty easy. First, you'll need to add the Receiver to your application's POM: <dependency> <groupId>org.apache.nifi</groupId> <artifactId>nifi-spark-receiver</artifactId> <version>0.4.1</version> </dependency>That's all that is needed in order to be able to use the NiFi Receiver. So now we'll look at how to use the Receiver in your code.The NiFi Receiver is a Reliable Java Receiver. This means that if we lose a node after it pulls the data from NiFi, the data will not be lost. Instead, another node will simply pull and process the data. In order to create a NiFi Receiver, we need to first create a configuration that tells the Receiver where to pull the data from. The simplest form is to just tell the config where NiFi is running and which Port to pull the data from:SiteToSiteClientConfig config = new SiteToSiteClient.Builder() .url("http://localhost:8080/nifi"

I'll use iperf tool, which is available in the EPEL repo if you're using RH/CentOS.Installation is pretty straightforward :# yum -y install iperfFor use, you'll have to execute one instance on the first machine as the server, and another on the other machine as the client.Execute as a server (-s) on port (-p) 2000 :[root@machine01 ~]# iperf -s -p 2000------------------------------------------------------------Server listening on TCP port 2000TCP window size: 85.3 KByte (default)Now execute client on the other machine for 25 seconds (-t 25) with 5 seconds (-i5) between each periodic report, so we'll have 5 reports[root@machine02 ~]# iperf -c machine01 -p 2000 -i5 -t25------------------------------------------------------------Client connecting to machine01, TCP port 2000TCP window size: 23.2 KByte (default)------------------------------------------------------------[ 3] local 10.195.196.18 port 45284 connected with 10.195.196.48 port 2000[ ID] Interval Transfer Bandwidth[ 3]

Kylin Pronounced “KEY LIN” / “CHI LIN” - This project brings OLAP (Online Analytical Processing) to Big Data. It is a top-level project in Apache. Through it’s UI, you can create a logical model (dimensions/measures) from a star schema in Hive. Kylin will then create cube aggregates using MR and put the aggregates and cube metadata into HBase. Users can then query the cube data through the Kylin UI or a BI tool that uses the Kylin odbc driver. A good video from the committers overviewing the project: https://www.youtube.com/watch?v=7iDcF7pNhV4 DefinitionsCube - A data structure containing dimensions and measures for quickly accessing aggregated information (measures) across many axis's (dimensions)Cuboid - A "slice" or subset of a cubeDimensions - Think of these as alphanumeric columns that sit in a group by clause of SQL. i.e. Location, Department, Time, etc.Measure - Think of these as metric/numerical values that sit in a select clause of SQL. i.e. Sum(value), Max(bonus), Min(e

Fix: HDP “Unauthorized connection for super-user: oozie from IP 127.0.0.1” Recently have been playing with HortonWorks HDP 2.2. Was starting to configure some oozie workflows and when submitting the job the first step’s Hive script failed with this error and stack. JA002: Unauthorized connection for super-user: oozie from IP 127.0.0.1 Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException): Unauthorized connection for super-user: oozie from IP 127.0.0.1at org.apache.hadoop.ipc.Client.call(Client.java:1468) at org.apache.hadoop.ipc.Client.call(Client.java:1399) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232) at com.sun.proxy.$Proxy39.getDelegationToken(Unknown Source) at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getDelegationToken(ApplicationClientProtocolPBClientImpl.java:306)... 30 moreI was found the solution here.

Node labels enable you partition a cluster into sub-clusters so that jobs can be run on nodes with specific characteristics. For example, you can use node labels to run memory-intensive jobs only on nodes with a larger amount of RAM. Node labels can be assigned to cluster nodes, and specified as exclusive or shareable. You can then associate node labels with capacity scheduler queues. Each node can have only one node label. Demo: Use case 2 node labels : node1 & node2 + Default & Spark queue Submit job to node1 Node labels added : yarn rmadmin -addToClusterNodeLabels "node1(exclusive=true),node2(exclusive=false)" Label assigned: yarn rmadmin -replaceLabelsOnNode "phdns02.cloud.hortonworks.com=node2 phdns01.cloud.hortonworks.com=node1" Job Submission: Job send to node1 only and assign to queue spark hadoop jar /usr/hdp/current/hadoop-yarn-client/hadoop-yarn-applications-distributedshell.jar -shell_command "sleep 100" -jar /usr/hdp/current/hadoop-yarn

Hi Everyone,I came across a Kerberos cache issue and wanted to share and possible have more ideas. I understand few of us had some issues in the past and hope this article might help.One of our clients have RedHat IDM (supported version of freeIpa) and when you install sssd along with krb5 by IDM the default cache setting is 'KEYRING' than 'File'You will still be able to get the tickets but will have GSSException error on Cluster.KEYRING persistent cache setting works with many applications however not with HADOOP Cluster, I'm sure HDO engineering team must be looking into this for a solution, since KEYRING is the future for kerberos cache and the stuff you can do with it like keylist and etc.To solve the errors you can comment out the "default_ccache_name=KEYRING....." on krb.conf or change it to "default_ccache_name = FILE:/tmp/krb5cc_%{uid}"Logout and log in again - destroy the previous tickets and you should have something like "Ticket cache: FILE:/tmp/krb5cc_" in your klist outpu

dfs.datanode.max.xcievers / dfs.datanode.max.transfer.threads = 4096 (use 16k if running HBase) 
dfs.datanode.balance.max.concurrent.moves = 500 (can go to 1000 if needed) 
/* Each data node has a limited bandwidth for rebalancing. The default value for the bandwidth is 5MB/s. In the worst case, each data transfer has a limited bandwidth of 1MB/s. Default is dfs.datanode.balance.bandwidthPerSec = 5242880 */dfs.datanode.balance.bandwidthPerSec = 104857600 /* 100 MB/s */ hdfs balancer -Dfs.defaultFS=hdfs://<NN_HOSTNAME>:8020 -Ddfs.balancer.movedWinWidth=5400000 -Ddfs.balancer.moverThreads=1000 -Ddfs.balancer.dispatcherThreads=200 -Ddfs.datanode.balance.max.concurrent.moves=5 -Ddfs.balance.bandwidthPerSec=100000000 -Ddfs.balancer.max-size-to-move=10737418240 -threshold 5

First 15 seconds - Queue MappingParent article CS view

Question:I am about to initiate the cluster install wizard on a new Ambari install. I reviewed the information on service users at http://docs.hortonworks.com/HDPDocuments/Ambari-2.2.0.0/bk_ambari_reference_guide/content/_defining_service_users_and_groups_for_a_hdI am wondering whether I should take the "Skip Group Modifications" option.The doc states "Choosing this option is typically required if your environment manages groups using LDAP and not on the local Linux machines". In our environment, users and groups are managed via Active Directory (via Centrify). We are planning to enable security on the cluster after it's installed, and that will include a host of new users being created, after which many of the initial users and groups will be orphaned.What does that "Skip group modifications" option actually do? Should it be used in this case?Answer:I believe the answer lies in the fact that we do a groupmod hadoop statement and there is no group called hadoop, or this is not a

Original1) Setup Azure account2) Setup CloudBreak accountVery important steps : Applies to Azure onlyCreate a test network in Azure before you start creating cloudbreak credentials. In your local machine, run the following and accept default values. openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout azuretest.key -out azuretest.pem You will see 2 files as listed below.-rw-r--r-- 1 nsabharwal staff 1346 May 7 17:00 azuretest.pem --> We need this file to create credentials in cloudbreak.-rw-r--r-- 1 nsabharwal staff 1679 May 7 17:00 azuretest.key --> We need this to login into the host after cluster deployment.chmod 400 azuretest.key --> otherwise, you will receiver bad permission error for example: ssh -i azuretest.key ubuntu@<server>Very important: check your openssl version and if it's latest version then run the following and use azuretest_login.key to login openssl rsa -in azuretest.key-out azuretest_login.keyhw11326:jumk nsabharwal$ openssl vers