Community Articles

Here my findings on how to increase sandbox disk space. Please if anyone has a easier solution, update this:1- Backup sandox files2- Convert .vmdk disk to .vdi (virtuabox does not allow resizing on vmdk images): VBoxManage clonehd "/Users/gbraccialli/VirtualBox VMs/HDP 2.3_1.2_B/Hortonworks Sandbox with HDP 2.3-disk1.vmdk" "/Users/gbraccialli/VirtualBox VMs/HDP 2.3_1.2_B/Hortonworks Sandbox with HDP 2.3-disk1.vdi" --format VDI --variant Standard3- Increase .vdi sizeVBoxManage modifyhd "/Users/gbraccialli/VirtualBox VMs/HDP 2.3_1.2_B/Hortonworks Sandbox with HDP 2.3-disk1.vdi" --resize 4000004- Open virtualbox, remove .vmdk image and add new .vdi image file5- Download gparted iso imagehttp://sourceforge.net/projects/gparted/?source=typ_redirect6- Add gparted iso image as cdrom drive on virtualbox, make cdrom as first device on boot order and boot virtualbox using gparted image7- Using gparted gui, resize the partition, save changes, shutdown virtualbox and remove gparted cdrom iso image

To disable SSLv3 for old Hue version, do the following:1.) open file "/usr/lib/hue/desktop/core/src/desktop/lib/wsgiserver.py"2.) add the following line, after line no. 1669: ctx.set_options(SSL.OP_NO_SSLv2 | SSL.OP_NO_SSLv3)3.) restart HueTo check if Hue replies on SSLv3 or not you can use this one:openssl s_client -connect <hue-server-name>:<hue-port> -ssl3Here the corresponding GitHub patch, thanks to @cdraper:https://github.com/cloudera/hue/commit/0060abf9aae0049c082c9948658eea7df848ab6e

1. If using Ambari versions before 2.1.2, Disable HBase per region metrics. On Ambari server host. Edit the following files under /var/lib/ambari-server/resources/:common-services/HBASE/0.96.0.2.0/package/templates/HBASE/hadoop-metrics2-hbase.properties-GANGLIA-MASTER.j2common-services/HBASE/0.96.0.2.0/package/templates/HBASE/hadoop-metrics2-hbase.properties-GANGLIA-RS.j2and add the following lines at the end before '% endif %}':*.source.filter.class=org.apache.hadoop.metrics2.filter.GlobFilterhbase.*.source.filter.exclude=*Regions*Do a rolling restart of the HBase RegionServers. Note: This does not disable RS metrics. It just disables the per region / per table metrics collected at Region level. This is disabled by default from Ambari 2.1.2. 2. Tune AMS configs: Find out the Heap available to AMS collector host. Change the following settings based on available memory:ams-hbase-env_ :: hbase_master_heapsize = 8192m (Or 16384m if available)ams-hbase-env_ :: hbase_master_xmn_size =

Tutorial LinkSandboxVersion: HDP 2.3.2AmbariVersion2.1.2Hadoop stack version:Hadoop2.7.1.2.3.2.0-2950Issue 1: Error initializing SparkContext when executing spark-shell commandWhen you issue the command as rootspark-shell --master yarn-client --driver-memory 512m --executor-memory 512mYou would would receive the error:ERROR SparkContext: Error initializing SparkContext.org.apache.hadoop.security.AccessControlException: Permission denied: user=root, access=WRITE, inode="/user/root/.sparkStaging/application_1451921066894_0001":hdfs:hdfs:drwxr-xr-x Resolutionsudo su - hdfshdfs dfs -mkdir /user/roothdfs dfs -chown root:hdfs /user/rootexit--------------------------------------------------------Issue 2

Tutorial LinkSandbox Version: HDP 2.3.2Ambari Version 2.1.2Hadoop stack version: Hadoop 2.7.1.2.3.2.0-2950Issue 1: Permission DeniedCopy the data over to HDFS on Sandbox with the following command will result in a Permission denied error.hadoop fs -put ~/Hortonworks /user/guest/HortonworksResolution: See Hands-on Spark Tutorial: Permission Denied------------------------------

With recent improvements in Ambari, upgrading can be done easily either using Rolling Upgrade or Express Upgrade. But there are times when upgrade/downgrade gets stuck either since all the precautions were not followed or due to product issues. When upgrade gets stuck, it is typically in an Upgrade Paused status or there aren't any status as such. At this juncture, care needs to be taken to ensure that the ambari-server is not restarted without consulting with Technical Support.Current status of the upgrade can be checked using multiple method: Ambari Log files Ambari API URL's Ambari Databases1. Ambari Log files Review Ambari Log files for any errors / exception when the upgrade is in progress. 2. Use Ambari URL's to find what are the failures http://<ambari-server>:8080/api/v1/clusters/c1/upgrades For eg http://vcert1.novalocal:8080/api/v1/clusters/VCertify/upgrades This would show all the upgrade or downgrade attempts. for eg if this is a upgrade failure, identify the latest

Getting started with Nifi expression language and custom Nifi processors on HDP sandboxThis tutorial is part of a webinar for partners on Hortonworks DataFlow. The recording will be made available at http://hortonworks.com/partners/learn/Background For a primer on HDF, you can refer to the materials here to get a basic backgroundA basic tutorial on using Nifi on HDP sandbox is also available hereGoals Build Nifi flow to analyze Nifi's network traffic using tcpdump. Use Expression Language to extract out source/target IPs/portsBuild and use custom tcpdump processor to filter Nifi's source/target IPs/ports on HDP sandboxNote that: Nifi can be installed independent of HDPThe custom processor also can be built on any machine where Java and eclipse are installedSandbox is being used for demo purposes, to have everything in one placePre-Requisites: Install Nifi on sandbox The lab is designed for the HDP Sandbox. Download the HDP Sandbox here, import into VMWare Fusion and start the VMAfter

This document is an informal guide to setting up a test cluster on Amazon AWS, specifically the EC2 service. This is not a best practice guide nor is it suitable for a full PoC or production install of HDP.Please refer to Hortonworks documentation online to get a complete set of documentation.Create InstancesCreated the following RHEL 6.3 64bit instances:m1.medium ambarimasterm1.large hdpmaster1m1.large hdpmaster2m1.medium hdpslave1m1.medium hdpslave2m1.medium hdpslave3Note: when instantiating instances, I increased the root partition to 100Gb on each of them. For long term use, you may want to create separate volumes for your each of the datanodes to store larger amounts of data. Typical raw storage per node is 12-24Tb per slave node.Note: I edit the Name column in the EC2 Instances screen to the names mentioned above so I know which box I’m dealing withConfigure Security GroupsUsed the following security group rules: ICMP Port (Service) Source Action ALL sg-79c54511 (had

Apache JIRA HDFS-9552 is a documentation patch that clarifies exactly what kinds of permission checks are enforced by HDFS for each kind of user operation on a file system path. This documentation is not yet live on hadoop.apache.org. I am posting the same content here in the interim until HDFS-9552 ships in an official Apache release.Permission ChecksEach HDFS operation demands that the user has specific permissions (some combination of READ, WRITE and EXECUTE), granted through file ownership, group membership or the other permissions. An operation may perform permission checks at multiple components of the path, not only the final component. Additionally, some operations depend on a check of the owner of a path.All operations require traversal access. Traversal access demands the EXECUTE permission on all existing components of the path, except for the final path component. For example, for any operation accessing /foo/bar/baz, the caller must have EXECUTE permission on /, /foo and

By default, Ranger uses the log4j DailyRollingFileAppender to manage rotation of log files. This is the appender used by many HDP components, and it does not have a concept of max number of files to keep around. The advantage of using this appender is that it is easier to locate a log entry since files are split up on date boundaries. The main disadvantage of this appender is that there is no way to limit the amount of log data that is kept.There are two common options for dealing with this.1. Keep the DailyRollingFileAppender, and trim logs using a cron script. The following script would only keep 30 days worth of logs:#!/bin/bashfind /var/log/ranger -mtime +30 | xargs --no-run-if-empty rmYou can change the +30 to be +number_of_days.This script could be installed, for example, in a file called /etc/cron.daily/trim_ranger_logs and it would run each morning and remove logs older than 30 days.2. Change to the RollingFileAppender. To do this you need to modify the log4j.xml file for the a

Q: What are the use cases for Centrify?A: Integrate AD with Linux and there are no local user deployment similar to how SSSD is configured. The size of the cluster and/or domains is big enough that it's hard to manage with SSSD. Centrify greatly simplifies the management of this type of environment. Centrify ldapproxy also abstracts the complexity around integrating with other filers/appliances like Isilon, NetApp, etc. ldapproxy fronts Hadoop and doesn't expose internals of AD but only give information about a zone. It is usually used for machine to machine type authentication. Q: If there are multiple domains in a forest, how does Centrify know which domain controller to use to authenticate a user?A: Centrify walk the forest tree and figure out what domain controller to use to authenticate the user. It utilizes Domain Controller Service to perform this action. It doesn't use krb5.conf file. The Centrify agent knows what forest or domain controller it belongs to. It is PAM and sit

The release of HDP 2.3.4 offers a newer version of the ATS which is version 1.5. Part of configuring this new ATS version includes providing the following property if Tez is used:<property> <name>yarn.timeline-service.entity-group-fs-store.group-id-plugin-classes</name> <value>org.apache.tez.dag.history.logging.ats.TimelineCachePluginImpl</value> <description>Plugins that can translate a timeline entity read request into a list of timeline cache ids, separated by commas. </description> </property>If this property is included but the tez-client is not present, a message will be seen in the ATS log stating ClassNotFound and the ATS server will fail shortly after issuing a startup.The full documentation for configuring ATS 1.5 is located here:http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.4/bk_yarn_resource_mgt/content/ch08s02.htmlA documentation bug has been filed as the only reference to this in the configuration section

Here are some lessons learned while trying to deploy the latest version of Cloudbreak (1.1.0) on Azure. Please refer to the latest Hortonworks Cloudbreak documentation for the detailed steps and use this article as a supplement until the documents are updated.Logging into the Deployer VMThere is a prebuilt image for Azure Cloudbreak deployer. This image does not require a ssh key; however, record the username and password you've specified at setup in the Azure portal (see the section in green highlight below). Those are your credentials to login to the deployer VM and setup Cloudbreak services later. Once the VM deploys (refer to the Azure portal for status), grab the the Public IP specified in the Azure portal and ssh into the VM. Ssh into the Deployer VM by,ssh <your specified username>@<public IP specified in Azure portal>password: <enter you specified password>Deployer VM run initiation stepsOnce you've logged into the Deployer VM, run these initiation steps which

Origin: http://henning.kropponline.de/2015/09/27/storm-serialization-with-avro-using-kryo-serializer/Working with complex data events can be a challenge designing Storm topologies for real-time data processing. In such cases emitting single values for multiple and varying event characteristics soon reveals it's limitations. For message serialization Storm leverages the Kryo serialization framework used by many other projects. Kryo keeps a registry of serializers being used for corresponding Class types. Mappings in that registry can be overridden or added making the framework extendable to diverse type serializations.On the other hand Avro is a very popular "data serialization system" that bridges between many different programming languages and tools. While the fact that data objects can be described in JSON makes it really easy to use, Avro is often being used for it's support of schema evolution. With support for schema evolution the same implementation (Storm topology)

There are 2 different ways of accessing HDFS over http.Using WebHDFShttp://<active-namenode-server>:<namenode-port>/webhdfs/v1/<file-path>?op=OPENUsing HttpFshttp://<hadoop-httpfs-server>:<httpfs-port>/webhdfs/v1/<file-path>?op=OPENWebHDFS: Pros: Built-in with default Hadoop installation Efficient as load is streamed from each data nodeCons: Does not work if high availability is enabled on cluster, Active namenode needs to be specified to use webHdfsHttpFsPros: Works with HA enabled clusters.Cons: Needs to be installed as additional service. Impacts performance because data is streamed from single node.Creates single point of failureAdditional performance implications of webHDFS vs HttpFshttps://www.linkedin.com/today/post/article/20140717115238-176301000-accessing-hdfs-using-the-webhdfs-rest-api-vs-httpfsWebHDFS vs HttpFs Major difference between WebHDFS and HttpFs: WebHDFS needs access to all nodes of the cluster and when some data is read it is tr

The current state of Hive ACID is as follows:Technical Alert sent out on 2015-09-10:Subject: Hortonworks Technical Alert: Hive ACID featureDear HDP Customer, Hortonworks Technical Support would like to make you aware about a potential issue you may encounter.Below are the details:COMPONENT: HiveVERSION:HDP 2.2.*, HDP 2.3.0PROBLEM:Hive ACID feature not ready for production deploymentIMPACT:Hive ACID supports two primary use cases:Hive Streaming IngestInserts/Updates/Deletes at low concurrencyAs the first set of users are developing their ACID solutions and deploying them in their staging environments, they’ve run into a series of technical issues which are indicate insufficient test case coverage for this particular feature. Hortonworks is currently working closely with these customers to provide fixes for these issues and make them successful in production. In addition, we are going back to our engineering processes and identifying the missing test cases and increasing our test case c

Visualize near-real-time stock price changes using Solr and Banana UI The goal of this tutorial is to create a moving chart that shows the changes in price of a few stock symbols, similar to Google Finance or Yahoo Finance. Summary of steps Download and install the HDP Sandbox Download and install the latest NiFi release Create a Solr dashboard to visualize the results Create a new NiFi flow to pull from Google Finance API, transform, and store in HBase and Solr Step-by-step 1. Download and install the HDP Sandbox Download the latest (2.3 as of this writing) HDP Sandbox here. Import it into VMware or VirtualBox, start the instance, and update the DNS entry on your host machine to point to the new instance’s IP. On Mac, edit /etc/hosts, on Windows, edit %systemroot%\system32\drivers\etc\ as administrator and add a line similar to the below: 192.168.56.102 sandbox sandbox.hortonworks.com   2. Download and install the latest NiFi release Follow the directions here. These were the

There are certain times where we need to change the priority of the hadoop jobs. Due to some business criticality, we want some jobs to have high priority and some jobs to have low priority. So, that the important jobs are completed early.If Hadoop cluster is using the Capacity Scheduler with priorities enabled for queues, then we can set priority of our hadoop jobs. This article explain to set the priority of hadoop jobs and explained how to change the priority of Hadoop Jobs.1)Set the priority in Map Reduce Program: In Map/Reduce program we can set the job priority using following way.Configuration conf = new Configuration(); // set the priority to VERY_HIGH conf.set("mapred.job.priority", JobPriority.VERY_HIGH.toString());Allowed priority values are:VERY_HIGH, HIGH, NORMAL, LOW, VERY_LOW2)Set the priority in Pig Program: We can set priority of Pig job using below property, This property is used to set the job priority is Pig Programming :job.priorityFor example:grunt> SET job.

Use case: User want to map ad group hdpadmin using Yarn queue manager view. Environment: HDP 2.3.4 and Ambari 2.2.0Originial request was made by one of HCC users. Thread linkQuestion : How to assigned capacity scheduler queue based on AD group.Solution/Demo: [root@phdns02 scripts]# id neeraj uid=29800018(neeraj) gid=29800018(neeraj) groups=29800018(neeraj),29800017(hdpadmin) [root@phdns02 scripts]#

Bug : https://issues.apache.org/jira/browse/AMBARI-14466[root@phdns01 ~]# ambari-server startUsing python /usr/bin/python2Starting ambari-serverAmbari Server running with administrator privileges.Organizing resource files at /var/lib/ambari-server/resources...WARNING: setpgid(31734, 0) failed - [Errno 13] Permission deniedServer PID at: /var/run/ambari-server/ambari-server.pidServer out at: /var/log/ambari-server/ambari-server.outServer log at: /var/log/ambari-server/ambari-server.logWaiting for server start....................Ambari Server 'start' completed successfully.[root@phdns01 ~]#wget https://issues.apache.org/jira/secure/attachment/12779059/AMBARI-14466.patchpatch -p1 < AMBARI-14466.patchFile to patch: /usr/sbin/ambari_server_main.pyIssue resolved.

Folks often ask about best practice for setting replication factor, evidently wondering if the default value of 3 is supported by factual data. The cool answer is, yes it is!Rob Chansler, an excellent engineering manager and contributor to Hadoop at Yahoo for several years, posted the best material in 2011. The hard-core math is in a spreadsheet attached to Apache Jira https://issues.apache.org/jira/browse/HDFS-2535, "A Model for Data Durability", where he uses reasonable assumptions and experience from Yahoo to calculate the probable rate of data loss events at a single site due to node failures when replication is set to 3, as 0.021 events per century. See "Attachments" : "LosingBlocks.xlsx" in the Jira ticket.Rob also published an article in Usenix titled Data Availability and Durability with the Hadoop Distributed File System, and did a related presentation at Hadoop Summit 2011, Data Integrity and Availability of HDFS (video).Sanjay Radia summarized this information in Hortonwo

Error:2016-02-11 08:57:07,482 ERROR [Thread-9]: client.RangerAdminRESTClient (RangerAdminRESTClient.java:getServicePoliciesIfUpdated(81)) - Error getting policies. request=http://phdns01.cloud.hortonworks.com:6080/service/plugins/policies/download/HDPTEST_hive?lastKnownVersion=11&pluginId=hiveServer2@phdns02.cloud.hortonworks.com-HDPTEST_hive, response={"httpStatusCode":400,"statusCode":1,"msgDesc":"Serivce:HDPTEST_hive not found","messageList":[{"name":"DATA_NOT_FOUND","rbKey":"xa.error.data_not_found","message":"Data not found"}]}, serviceName=HDPTEST_hiveIssue:Ranger was deleted and delete process ended up causing issues at the database level as shown above.Delete was done using the following commandcurl --user admin:admin -i -H "X-Requested-By: ambari" -X DELETE http://`hostname -f`:8080/api/v1/clusters/CLUSTERNAME/services/RANGERNow, No Hive repo showed up in Ranger console after adding Ranger and enabling Hive plugin.Fix:As you can seen in the error "HDPTEST_hive" Data Not

When you upgrade your stack from 2.2 to 2.3 then you will have issue using hue or hive due to a bug.In hive config file you may have wrong value in templeton.libjars property = /usr/hdp/${hdp.version}/zookeeper,/usr/hdp/${hdp.version}/hive/lib/hive-common.jar/zookeeper.jar So change it to correct one. Actually this is a bug which occurs with upgrade from HDP 2.2 to 2.3. correct value is below /usr/hdp/${hdp.version}/zookeeper/zookeeper.jar,/usr/hdp/${hdp.version}/hive/lib/hive-common.jar

When I played with Kafka-Storm streaming analysis, I often met the problem java.lang.RuntimeException: Could not find leader nimbus from seed hosts And I even cannot kill the related topology through StormUI or command line. I think it is because I got stale topology data in storm and zookeeper. My solution is to do directory clean up manually. This idea is borrowed from the bash files in our amazing "trucking demo". Step 1: Clean up files in storm folder #paths through Ambari install may start with /mnt if [ -d "/mnt/hadoop/storm" ]; then rm -rf /mnt/hadoop/storm/supervisor/isupervisor/* rm -rf /mnt/hadoop/storm/supervisor/localstate/* rm -rf /mnt/hadoop/storm/supervisor/stormdist/* rm -rf /mnt/hadoop/storm/supervisor/tmp/* rm -rf /mnt/hadoop/storm/workers/* rm -rf /mnt/hadoop/storm/workers-users/* fi #paths in sandbox may start with /hadoop if [ -d "/hadoop/storm" ]; then rm -rf /hadoop/storm/supervisor/isupervisor/* rm -rf /hadoop/storm/supervisor/localstate/* rm -rf /hadoo

Command: curl -s -d execute="select+*+from+nstest;" -d statusdir="1p" 'http://localhost:50111/templeton/v1/hive?user.name=hive'Let's insert some data and test againWait for job to finish and the directory will show up on HDFSYou can monitor the job from RM UIMore details: https://cwiki.apache.org/confluence/display/Hive/HiveWebInterface