Resources

Here are some lessons learned while trying to deploy the latest version of Cloudbreak (1.1.0) on Azure. Please refer to the latest Hortonworks Cloudbreak documentation for the detailed steps and use this article as a supplement until the documents are updated.Logging into the Deployer VMThere is a prebuilt image for Azure Cloudbreak deployer. This image does not require a ssh key; however, record the username and password you've specified at setup in the Azure portal (see the section in green highlight below). Those are your credentials to login to the deployer VM and setup Cloudbreak services later. Once the VM deploys (refer to the Azure portal for status), grab the the Public IP specified in the Azure portal and ssh into the VM. Ssh into the Deployer VM by,ssh <your specified username>@<public IP specified in Azure portal>password: <enter you specified password>Deployer VM run initiation stepsOnce you've logged into the Deployer VM, run these initiation steps which

Origin: http://henning.kropponline.de/2015/09/27/storm-serialization-with-avro-using-kryo-serializer/Working with complex data events can be a challenge designing Storm topologies for real-time data processing. In such cases emitting single values for multiple and varying event characteristics soon reveals it's limitations. For message serialization Storm leverages the Kryo serialization framework used by many other projects. Kryo keeps a registry of serializers being used for corresponding Class types. Mappings in that registry can be overridden or added making the framework extendable to diverse type serializations.On the other hand Avro is a very popular "data serialization system" that bridges between many different programming languages and tools. While the fact that data objects can be described in JSON makes it really easy to use, Avro is often being used for it's support of schema evolution. With support for schema evolution the same implementation (Storm topology)

There are 2 different ways of accessing HDFS over http.Using WebHDFShttp://<active-namenode-server>:<namenode-port>/webhdfs/v1/<file-path>?op=OPENUsing HttpFshttp://<hadoop-httpfs-server>:<httpfs-port>/webhdfs/v1/<file-path>?op=OPENWebHDFS: Pros: Built-in with default Hadoop installation Efficient as load is streamed from each data nodeCons: Does not work if high availability is enabled on cluster, Active namenode needs to be specified to use webHdfsHttpFsPros: Works with HA enabled clusters.Cons: Needs to be installed as additional service. Impacts performance because data is streamed from single node.Creates single point of failureAdditional performance implications of webHDFS vs HttpFshttps://www.linkedin.com/today/post/article/20140717115238-176301000-accessing-hdfs-using-the-webhdfs-rest-api-vs-httpfsWebHDFS vs HttpFs Major difference between WebHDFS and HttpFs: WebHDFS needs access to all nodes of the cluster and when some data is read it is tr

The current state of Hive ACID is as follows:Technical Alert sent out on 2015-09-10:Subject: Hortonworks Technical Alert: Hive ACID featureDear HDP Customer, Hortonworks Technical Support would like to make you aware about a potential issue you may encounter.Below are the details:COMPONENT: HiveVERSION:HDP 2.2.*, HDP 2.3.0PROBLEM:Hive ACID feature not ready for production deploymentIMPACT:Hive ACID supports two primary use cases:Hive Streaming IngestInserts/Updates/Deletes at low concurrencyAs the first set of users are developing their ACID solutions and deploying them in their staging environments, they’ve run into a series of technical issues which are indicate insufficient test case coverage for this particular feature. Hortonworks is currently working closely with these customers to provide fixes for these issues and make them successful in production. In addition, we are going back to our engineering processes and identifying the missing test cases and increasing our test case c

Visualize near-real-time stock price changes using Solr and Banana UI The goal of this tutorial is to create a moving chart that shows the changes in price of a few stock symbols, similar to Google Finance or Yahoo Finance. Summary of steps Download and install the HDP Sandbox Download and install the latest NiFi release Create a Solr dashboard to visualize the results Create a new NiFi flow to pull from Google Finance API, transform, and store in HBase and Solr Step-by-step 1. Download and install the HDP Sandbox Download the latest (2.3 as of this writing) HDP Sandbox here. Import it into VMware or VirtualBox, start the instance, and update the DNS entry on your host machine to point to the new instance’s IP. On Mac, edit /etc/hosts, on Windows, edit %systemroot%\system32\drivers\etc\ as administrator and add a line similar to the below: 192.168.56.102 sandbox sandbox.hortonworks.com   2. Download and install the latest NiFi release Follow the directions here. These were the

There are certain times where we need to change the priority of the hadoop jobs. Due to some business criticality, we want some jobs to have high priority and some jobs to have low priority. So, that the important jobs are completed early.If Hadoop cluster is using the Capacity Scheduler with priorities enabled for queues, then we can set priority of our hadoop jobs. This article explain to set the priority of hadoop jobs and explained how to change the priority of Hadoop Jobs.1)Set the priority in Map Reduce Program: In Map/Reduce program we can set the job priority using following way.Configuration conf = new Configuration(); // set the priority to VERY_HIGH conf.set("mapred.job.priority", JobPriority.VERY_HIGH.toString());Allowed priority values are:VERY_HIGH, HIGH, NORMAL, LOW, VERY_LOW2)Set the priority in Pig Program: We can set priority of Pig job using below property, This property is used to set the job priority is Pig Programming :job.priorityFor example:grunt> SET job.

Use case: User want to map ad group hdpadmin using Yarn queue manager view. Environment: HDP 2.3.4 and Ambari 2.2.0Originial request was made by one of HCC users. Thread linkQuestion : How to assigned capacity scheduler queue based on AD group.Solution/Demo: [root@phdns02 scripts]# id neeraj uid=29800018(neeraj) gid=29800018(neeraj) groups=29800018(neeraj),29800017(hdpadmin) [root@phdns02 scripts]#

Bug : https://issues.apache.org/jira/browse/AMBARI-14466[root@phdns01 ~]# ambari-server startUsing python /usr/bin/python2Starting ambari-serverAmbari Server running with administrator privileges.Organizing resource files at /var/lib/ambari-server/resources...WARNING: setpgid(31734, 0) failed - [Errno 13] Permission deniedServer PID at: /var/run/ambari-server/ambari-server.pidServer out at: /var/log/ambari-server/ambari-server.outServer log at: /var/log/ambari-server/ambari-server.logWaiting for server start....................Ambari Server 'start' completed successfully.[root@phdns01 ~]#wget https://issues.apache.org/jira/secure/attachment/12779059/AMBARI-14466.patchpatch -p1 < AMBARI-14466.patchFile to patch: /usr/sbin/ambari_server_main.pyIssue resolved.

Folks often ask about best practice for setting replication factor, evidently wondering if the default value of 3 is supported by factual data. The cool answer is, yes it is!Rob Chansler, an excellent engineering manager and contributor to Hadoop at Yahoo for several years, posted the best material in 2011. The hard-core math is in a spreadsheet attached to Apache Jira https://issues.apache.org/jira/browse/HDFS-2535, "A Model for Data Durability", where he uses reasonable assumptions and experience from Yahoo to calculate the probable rate of data loss events at a single site due to node failures when replication is set to 3, as 0.021 events per century. See "Attachments" : "LosingBlocks.xlsx" in the Jira ticket.Rob also published an article in Usenix titled Data Availability and Durability with the Hadoop Distributed File System, and did a related presentation at Hadoop Summit 2011, Data Integrity and Availability of HDFS (video).Sanjay Radia summarized this information in Hortonwo

Error:2016-02-11 08:57:07,482 ERROR [Thread-9]: client.RangerAdminRESTClient (RangerAdminRESTClient.java:getServicePoliciesIfUpdated(81)) - Error getting policies. request=http://phdns01.cloud.hortonworks.com:6080/service/plugins/policies/download/HDPTEST_hive?lastKnownVersion=11&pluginId=hiveServer2@phdns02.cloud.hortonworks.com-HDPTEST_hive, response={"httpStatusCode":400,"statusCode":1,"msgDesc":"Serivce:HDPTEST_hive not found","messageList":[{"name":"DATA_NOT_FOUND","rbKey":"xa.error.data_not_found","message":"Data not found"}]}, serviceName=HDPTEST_hiveIssue:Ranger was deleted and delete process ended up causing issues at the database level as shown above.Delete was done using the following commandcurl --user admin:admin -i -H "X-Requested-By: ambari" -X DELETE http://`hostname -f`:8080/api/v1/clusters/CLUSTERNAME/services/RANGERNow, No Hive repo showed up in Ranger console after adding Ranger and enabling Hive plugin.Fix:As you can seen in the error "HDPTEST_hive" Data Not

When you upgrade your stack from 2.2 to 2.3 then you will have issue using hue or hive due to a bug.In hive config file you may have wrong value in templeton.libjars property = /usr/hdp/${hdp.version}/zookeeper,/usr/hdp/${hdp.version}/hive/lib/hive-common.jar/zookeeper.jar So change it to correct one. Actually this is a bug which occurs with upgrade from HDP 2.2 to 2.3. correct value is below /usr/hdp/${hdp.version}/zookeeper/zookeeper.jar,/usr/hdp/${hdp.version}/hive/lib/hive-common.jar

When I played with Kafka-Storm streaming analysis, I often met the problem java.lang.RuntimeException: Could not find leader nimbus from seed hosts And I even cannot kill the related topology through StormUI or command line. I think it is because I got stale topology data in storm and zookeeper. My solution is to do directory clean up manually. This idea is borrowed from the bash files in our amazing "trucking demo". Step 1: Clean up files in storm folder #paths through Ambari install may start with /mnt if [ -d "/mnt/hadoop/storm" ]; then rm -rf /mnt/hadoop/storm/supervisor/isupervisor/* rm -rf /mnt/hadoop/storm/supervisor/localstate/* rm -rf /mnt/hadoop/storm/supervisor/stormdist/* rm -rf /mnt/hadoop/storm/supervisor/tmp/* rm -rf /mnt/hadoop/storm/workers/* rm -rf /mnt/hadoop/storm/workers-users/* fi #paths in sandbox may start with /hadoop if [ -d "/hadoop/storm" ]; then rm -rf /hadoop/storm/supervisor/isupervisor/* rm -rf /hadoop/storm/supervisor/localstate/* rm -rf /hadoo

Command: curl -s -d execute="select+*+from+nstest;" -d statusdir="1p" 'http://localhost:50111/templeton/v1/hive?user.name=hive'Let's insert some data and test againWait for job to finish and the directory will show up on HDFSYou can monitor the job from RM UIMore details: https://cwiki.apache.org/confluence/display/Hive/HiveWebInterface

Using ambari metrics we are able to determine queue utilization. Here is how I did it.Determine which queue you need to run analysis againstcreate a widgetAdd this widget to any dash board..For exampleYou can view the utilization by time using the drop down:

I just completed my first Express upgrade (EU) using Ambari-2.2.0, from HDP-2.2.8 to HDP-2.3.4 and here are my observations and issues I encountered. The cluster has 12 nodes, 2 masters and 10 workers with configured Namenode HA and RM HA running on RHEL-6.5 using Java-7. Installed Hadoop components: HDFS, MR2, Yarn, Hive, Tez, HBase, Pig, Sqoop, Oozie, ZooKeeper, and AmbariMetrics. About 2 weeks before this EU, the cluster was upgraded from HDP-2.1.10 and Ambari-1.7.1. Please use this as a reference: based on cluster settings and previous history (previous upgrade or fresh install), the issues will differ, and the problems I had should by no means considered to be representative, and taking place during every EU. It's good idea to backup all cluster supporting data-bases in advance, in my case Ambari, Hive metastore, Oozie and Hue (although Hue cannot be upgraded by Ambari)There is no need to prepare or download HDP.repo file in advance, Ambari will crate the file, now called HDP-2.3

Use case: There is lot of data in the locally managed table and we want to convert those table into external table because we are working on a use case where our spark and home grown application has trouble reading locally managed tables. Solution: alter table table_name SET TBLPROPERTIES('EXTERNAL'='TRUE');Test:Locally managed table droppedhive> drop table sample_07;OKTime taken: 0.602 secondsData deleted as it's locally managed table. hive> dfs -ls /apps/hive/warehouse/sample_07;ls: `/apps/hive/warehouse/sample_07': No such file or directoryCommand failed with exit code = 1Query returned non-zero code: 1, cause: nullConverting sample_07_test from locally managed to external to keep the data there after drop.hive> alter table sample_07_test SET TBLPROPERTIES('EXTERNAL'='TRUE');OKTime taken: 0.463 secondsTable dropped after converting into external table. hive> drop table sample_07_test;OKTime taken: 0.552 secondsData is still on HDFS hive> dfs -ls /apps/hive/warehouse/

Requirement:Currently we have /hadoop/hdfs/data and /hadoop/hdfs/data1 as datanode directories. I have new mountpoint (/hadoop/hdfs/data/datanew) with faster disk and I want to keep only this mountpoint as datanode directory. Steps:Stop the cluster.Go to the ambari HDFS configuration and edit the datanode directory configuration: Remove /hadoop/hdfs/data and /hadoop/hdfs/data1. Add /hadoop/hdfs/datanew save.Login into each datanode VM and copy the contents of /data and /data1 into /datanewChange the ownership of /datanew and everything under it to “hdfs”.Start the cluster.

The cluster has 1 management node (Bright Cluster Manager and Ambari server), 2 NameNodes (1 active, 1 passive) and 17 DataNodes and is running Hortonworks HDP 2.3.2 and Ambari 2.1.2.Each node has a 2 10GbE NICs which are bonded together and jumbo frames (MTU=9000) is enabled on the interfaces.There are sporadic NodeManager Web UI alerts in Ambari. For all 17 DataNodes we get connection timeouts throughout the day. These timeouts are not correlated with any sort of load on the system, they happen no matter what.When the connection to port 8042 is successful the connection is is around 5-7ms but when the connections fails I get response times of 5 seconds. Never 3 seconds or 6 seconds, always 5 seconds. For example...[root@XXXX ~]# python2.7 YARN_response.pyTesting response time at http://XXXX:8042Output is written if http response is > 1 second.Press Ctrl-C to exit!2016-02-08 07:19:17.877947 Host: XX23:8042 conntime - 5.0073 seconds, HTTP response - 2002016-02-08 07:19:22.889430 Hos

Customer: Is Hadoop Enterprise Ready?Me: Standing next to the whiteboard, Yes and that's why we use the term "Enterprise Ready Data Lake" Imagine that there are 3 pointsPoint 1 -> You need to prove your identity to get access to Lake and then need permissions or authority to access data.Point 2 -> Once you proved your authenticity then demands comes to manage the lifecycle of data from it's requirement to retirement "Automated process"Point 3 -> Life Cycle Management process needs to be integrated with a Governance solution to manage data of data "metadata" , data lineage, auditing and more to fullfil security and compliance requirement.Point 1 --> Entry Point: You must have strong Authentication in place to get into the system and more users will be coming in to access data as we move away from silos of data to a centralized repository. The access management must be easier to manage i,e Security solution should have a centralized place toAdmin (create, define and manage) s

Problem:File"/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 140,in _call_wrapper result = _call(command,**kwargs_copy)File"/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 291,in _callraiseFail(err_msg)resource_management.core.exceptions.Fail:Execution of 'yarn resourcemanager -format-state-store' returned 255.15/10/2616:11:16 INFO resourcemanager.ResourceManager: STARTUP_MSG:15/10/2616:11:17 INFO recovery.ZKRMStateStore: org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore$VerifyActiveStatusThread thread interrupted!Exiting!15/10/2616:11:17 INFO zookeeper.ZooKeeper:Session:0x150a4b3429b0002 closed15/10/2616:11:17 FATAL resourcemanager.ResourceManager:Error starting ResourceManagerorg.apache.zookeeper.KeeperException$NotEmptyException:KeeperErrorCode=Directorynot empty for/rmstore/ZKRMStateRoot/RMAppRoot at org.apache.zookeeper.KeeperException.create(KeeperException.java:125) at org.ap

osquery allows you to easily ask questions about your Linux and OSX infrastructure. Whether your goal is intrusion detection, infrastructure reliability, or compliance, osquery gives you the ability to empower and inform a broad set of organizations within your company. Download https://osquery.readthedocs.org/en/latest/installation/install-linux/ [root@phdns02 ~]# sudo rpm -ivh https://osquery-packages.s3.amazonaws.com/centos6/noarch/osquery-s3-centos6-repo-1-0.0.noarch.rpm Retrieving https://osquery-packages.s3.amazonaws.com/centos6/noarch/osquery-s3-centos6-repo-1-0.0.noarch.rpm warning: /var/tmp/rpm-tmp.rCrgXh: Header V4 RSA/SHA1 Signature, key ID c9d8b80b: NOKEY Preparing... ########################################### [100%] 1:osquery-s3-centos6-repo########################################### [100%] [root@phdns02 ~]# yum install osquery Loaded plugins: fastestmirror Setting up Install Process Loading mirror speeds from cached hostfile epel/metalink | 12 kB 00:00 * base: mirrors.ab

OLAP (Online Analytical Processing) is the technology behind many Business Intelligence (BI) applications. OLAP is a powerful technology for data discovery, including capabilities for limitless report viewing, complex analytical calculations, and predictive “what if” scenario (budget, forecast) planning.OLAP is an acronym for Online Analytical Processing. OLAP performs multidimensional analysis of business data and provides the capability for complex calculations, trend analysis, and sophisticated data modeling. It is the foundation for may kinds of business applications for Business Performance Management, Planning, Budgeting, Forecasting, Financial Reporting, Analysis, Simulation Models, Knowledge Discovery, and Data Warehouse Reporting. OLAP enables end-users to perform ad hoc analysis of data in multiple dimensions, thereby providing the insight and understanding they need for better decision making. SourceOLAP solutions Open source Apache Kylin http://kylin.apache.org/Apache Kyli

After installing zeppelin all goes fine, but when you try to connect you may get a "disconnected" status on the notebook tab and you're unable to create any new note, while the interpreter tab works fine. If you checked you can reach the port on which zeppelin is listen, it may be an issue with your content filter firewall and you should see this log in your firewall application:2016-02-04 15:47:58 Deny 192.168.0.128 40.112.76.49 http/tcp 57772 9995 1-Ecube 0-Internet ProxyDeny: HTTP Invalid Request-Line Format (TCP-UDP-isoardi OUT-00) HTTP-Client.isoardi proc_id="http-proxy" rc="594" msg_id="1AFF-0005" proxy_act="HTTP-Client.isoardi" line="\x81\x8d\xf2\x9eW\xfe\x89\xbc8\x8e\xd0\xa4u\xae\xbb\xd0\x10\xdc\x8f\x81\x8d\xf8]\xb8z\x83\x7f\xd7\x0a" TrafficThe solution is to disable the content filter for the domain on which zeppelin is running

ISSUE:Choosing the appropriate Linux file system for HDFS deploymentSOLUTION:The Hadoop Distributed File System is platform independent and can function on top of any underlying file system and Operating System. Linux offers a variety of file system choices, each with caveats that have an impact on HDFS.As a general best practice, if you are mounting disks solely for Hadoop data, disable ‘noatime’. This speeds up reads for files.There are three Linux file system options that are popular to choose from: Ext3Ext4XFSYahoo uses the ext3 file system for its Hadoop deployments. ext3 is also the default filesystem choice for many popular Linux OS flavours. Since HDFS on ext3 has been publicly tested on Yahoo’s cluster it makes for a safe choice for the underlying file system.ext4 is the successor to ext3. ext4 has better performance with large files. ext4 also introduced delayed allocation of data, which adds a bit more risk with unplanned server outages while decreasing fragmentatio

A customer wants to use Cloudbreak for deploying Hadoop Clusters. They want to scale up and down the hadoop storage nodes.a) How does HDFS detect Scale down of nodes and will it kick in HDFS Rebalance - Cloudbreak instructs Ambari via decommission REST Api call to decommission a DataNode and NodeManager. - Ambari triggers the decomission on the HDP cluster, from birds perspective this is what happens: https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.2/bk_Sys_Admin_Guides/content/ref-a179736c-eb7c-4dda-b3b4-6f3a778bd8c8.1.html but in automated way. - Since the decommission of DataNodes can take a long time if you have a lot of blocks; HDFS needs to replicate blocks belonging to decommissioning DataNodes to other live DataNodes to reach the replication factor that you have specified via dfs.replication in hdfs-site.xml. The default value of this replication factor is 3. - You can get feedback from the decomission process e.g from NameNode UI: http://ip of_namenode:50070/dfshealth.