Community Articles

When submitting issues in the Sandbox forums, make sure to provide the following information: Describe the problem: Provide stack traces, screen shots and error messages. For instance, "Receiving error 'permission denied' when trying to access file in the File Browser View"Steps to reproduce the issue: Here just provide steps of how you reproduce the behavior you are seeing. What have you done to analyze the issue: For instance, reinstalled Sandbox, cleared cache on my browser etc..Version: Get the Sandbox Version sandbox-version

HDP 2.2 and Isilon OneFS 7.2.0.4 are now officially certified by Hortonworks and EMC Isilon and ready for Hadoop deployment. Hortonworks and EMC Isilon have a close engineering relation that started in September of 2014, to ensure that Hortonworks Data Platform (HDP) is integrated with the Isilon OneFS filesystem. The HDP test suite employed to certify the Isilon OneFS filesystem has over 10,000 individual tests. The gives assurance to customers that both EMC and Hortonworks have done the due diligence to ensure that all Hadoop workloads work on this integrated platform, with security and operational ease.HDP with Isilon reference architectureIsilon OneFS has implemented the HDFS API as an over the wire protocol consistent with its multi-protocol support for NFS, SMB and others. This is different from implementations of Hadoop Compatible File Systems (HCFS) in that OneFS mimics the HDFS behavior for the subset of features that it supports. The deployment model for HDP with Isil

Environment Best practices dictate that, where possible, a Hadoop cluster should be maintained behind a firewall to minimize any potential security vulnerabilities that may arise from exposed ports and web interfaces. A common approach to enabling user access in this situation is to open up SSH into a set of gateway/edge nodes. This ensures that users must authenticate prior to accessing any pieces of the Hadoop ecosystem and implicitly encrypts all data sent between the client and the cluster. This is a common setup for vanilla cloud-based installations. The problem with this setup is that, by default, all access is limited to the CLI on the gateway machines. Users outside of the cluster firewall cannot access valuable features such as web UIs, and JDBC/ODBC connections. There are a few options to securely enable these capabilities:Enable Kerberos+SPNEGO and Knox. Then open up the appropriate ports in the firewall. Implement firewall rules to expose specific ports and ho

Connections among nodes in Hadoop cluster should not be restricted.Many ports used within the cluster by various components are dynamic and are not even known until install occurs.If you want to set firewall rules for external access to the cluster, and you want to know all the ports to restrict see the following: For HDP 2.3: http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.2/bk_HDP_Reference_Guide/content/reference_chap2.htmlFor Ambari 2.1.2: http://docs.hortonworks.com/HDPDocuments/Ambari-2.1.2.1/bk_ambari_reference_guide/content/ch_configuring_network_port_numbers.htmlFor HDP 2.2: http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.2.9/bk_HDP_Reference_Guide/content/reference_chap2.html For Ambari 2.1: http://docs.hortonworks.com/HDPDocuments/Ambari-2.1.1.0/bk_ambari_reference_guide/content/_default_network_port_numbers_-_ambari.htmlIf you employ Kerberos which is a must for truly secure clusters for authentication, kerberos already identifies users, services and machines.I fo

Sentiment Analysis of Live Twitter Stream Using Apache Spark ============================================================This application analyzes live tweets and predicts if they are positive, or negative. The application works by connecting to the Twitter stream, and applying a model built offline using Spark's machine learning library (Mllib) to classify the tweet's sentiment. Using the instructions on this page, you will be able to build the model on HDP Sandbox and then apply it to a live twitter stream.Prerequisites-------------* Download the HDP 2.3 Sandbox from [here](http://hortonworks.com/products/hortonworks-sandbox/#install)* Start the Sandbox, and add its IP address into your local machine's /etc/hosts file:```bash$ sudo echo "172.16.139.139 sandbox.hortonworks.com" >> /etc/hosts```* Log into the sandbox, and clone this repository: ```bash$ ssh root@sandbox.hortonworks.com$ cd$ git clone https://github.com/DhruvKumar/spark-twitter-sentiment```* Download the labeled

Some interesting talks for Hadoop Summit 2016 (EMEA). Please vote: the deadline is Dec 15 and you can vote for as many talks as you wantHadoop governance, security, deployment, operations1. Rolling & Express Upgrades of the Stack in Ambarihttps://hadoopsummit.uservoice.com/forums/332061-hadoop-governance-security-deployment-and-operat/suggestions/10848390-rolling-express-upgrades-of-the-stack-in-ambariData science0. Analyzing Hollywood with Spark and Hadoop:https://hadoopsummit.uservoice.com/forums/332055-data-science-applications-for-hadoop/suggestions/10846959-analyzing-hollywood-with-spark-and-hadoop1. Magellan (how to scale geospatial joins using Magellan):https://hadoopsummit.uservoice.com/forums/332079-the-future-of-apache-hadoop/suggestions/10848510-magellan-spark-as-a-geospatial-analytics-engine2. Online Learning (how to do machine learning online and fast while being accurate):https://hadoopsummit.uservoice.com/forums/332082-hadoop-and-the-internet-of-things/suggestions/1

Problem:When trying to distcp to AWS, this error is reported:2015-12-03 09:50:01,132 FATAL [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster java.util.ServiceConfigurationError: org.apache.hadoop.fs.FileSystem: Provider org.apache.hadoop.fs.s3a.S3AFileSystem could not be instantiated at java.util.ServiceLoader.fail(ServiceLoader.java:224) and also:Caused by: java.lang.ClassNotFoundException: com.amazonaws.event.ProgressListener Solution:In ambari, under yarn, configs please add: /usr/hdp/2.3.2.0-2950/hadoop/* and /usr/hdp/2.3.2.0-2950/hadoop/lib* to the yarn.application.classpath Then under mapred, configs, do the same for mapreduce.application.classpath restart all affected services and it should work.

Caused by: java.lang.OutOfMemoryError: Java heap spaceThis particular case is related to "Reducer tasks of hive job fails with Out Of Memory error during shuffle fetcher stage"Fix: Increase hive.tez.container.size if it is set too low.tez.runtime.shuffle.memory.limit.percent from default value 0.7 Changed to 0.4 Decrease tez.runtime.shuffle.fetch.buffer.percent from default 0.25 to .15 if needed. (Different values were tested between the range of 0.25 to 0.10)

The Hadoop Ecosystem Tablehttps://hadoopecosystemtable.github.io/

Use caseThere are 2 groups Analytics and DW. We want to split the cluster resources between these 2 groups. User - neeraj belongs to Analytics group.User - dwuser belongs to DW groupUser neeraj is not allowed to use Default and dwuser queue. Be default, all the jobs submitted by user neeraj must go to it's assigned queue. User dwuser is not allowed to use Default and Analytics queue. By default, all the jobs submitted by user dwuser must go to it's assigned queue. Environment HDP 2.3 (Hortonworks Data Platform) and Ambari 2.1This tutorial completely independent of Hadoop distribution. Yarn is must i,e Hadoop 2.x I will be using Capacity Scheduler view to configure queues. yarn.scheduler.capacity.maximum-am-resource-percent=0.2 yarn.scheduler.capacity.maximum-applications=10000 yarn.scheduler.capacity.node-locality-delay=40 yarn.scheduler.capacity.queue-mappings=u:neeraj:Analytics,u:dwuser:DW yarn.scheduler.capacity.queue-mappings-override.enable=true yarn.scheduler.capacity.root.acces

yum install expect*#!/usr/bin/expect spawn ambari-server sync-ldap --existingexpect "Enter Ambari Admin login:"send "admin\r"expect "Enter Ambari Admin password:"send "admin\r"expect eof

This article will shield light on the HP Big Data Reference Architecture. Hadoop Architectures have been traditionally build to move the math to the data. This has been the basis for Hadoop architectures since the initial implementations back at Yahoo. Let's take a look at traditional verse HP's Big Data Architecture approach.Current traditional Big Data approach• Compute and storage are always collocated • All servers are identical • Data is partitioned across servers on direct-attached storage (DAS)tradhadoop.pngNew HP Big Data approach• Separate compute and storage tiers connected by Ethernet networking • Standard Hadoop installed asymmetrically with storage components on the storage servers and yarn applications on the compute servershpbdra.pngWith this new approach, HP engineers have challenged the conventional wisdom that compute should always be co-located with data.HP has been aggressively benchmarking with this new archiecture, and feel they have designed an architecture th

Use OpenTSDB Ambari service to store/visualize stock data on HDP sandbox Goal: OpenTSDB (Scalable Time Series DB) allows you to store and serve massive amounts of time series data without losing granularity (more details here). In this tutorial we will install it on Hbase on HDP sandbox using the Ambari sevice and use it to import and visualize stock data.Steps:Setup VM and install Ambari service Download HDP latest sandbox VM image (.ova file) from Hortonworks websiteImport ova file into VMWare and ensure the VM memory size is set to at least 8GBNow start the VMAfter it boots up, find the IP address of the VM and add an entry into your machines hosts file e.g.192.168.191.241 sandbox.hortonworks.com sandbox Connect to the VM via SSH (password hadoop)ssh root@sandbox.hortonworks.comStart HBase service from Ambari and ensure Hbase is up and root has authority to create tables. You can do this by trying to create a test tablehbase shellcreate 't1', 'f1', 'f2', 'f3'If this fails with th

Step 1 - Goto ambari UI and add/modify below propertiesGoto service hive → configs and change autherization to SQLStdAuthStep 2 - In Hive-site.xml, make sure you have set below properties:hive.server2.enable.doAs --> falsehive.users.in.admin.role --> root (comma separated list of users)Step 3 - Make sure that you have below properties set in Hiveserver2-site.xmlhive.security.authorization.manager --> org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdConfOnlyAuthorizerFactoryhive.security.authorization.enabled --> truehive.security.authenticator.manager --> org.apache.hadoop.hive.ql.security.ProxyUserAuthenticatorStep 4 - Restart hive services from UIStep 5 - Testing our setup, in this we will create one readonly user and try to drop table. 5.1 Login to beeline using root ( as we have added root in hive.users.in.admin.role ) 0: jdbc:hive2://localhost:10010> !connect jdbc:hive2://localhost:10010Connecting to jdbc:hive2://localhost:10010Enter username

Note: Credit for the key piece of information to solve this problem goes to Phil D’Amore. A customer had a problem writing a Java application that used the Hive client libraries to talk to two secure Hadoop clusters that resided in different Kerberos realms. The same problem could be encountered by a client connecting to a single secure Hadoop cluster that happened not to be in the Kerberos “default_realm” as specified in the client host’s krb5.conf file. The same problem could occur for any Hadoop ecosystem client, not just Hive clients.In order to communicate with two different secure Hadoop clusters, in different Kerberos realms, the client application did the following things correctly:It harvested the needed configuration files (in this case, core-site.xml, hdfs-site.xml, and hive-site.xml) from each target cluster, and used the appropriate configuration when communicating with each respective cluster.Its application user id had two Kerberos principals, one registered and authent

The MergeContent processor certainly can be challenging to understand its inner workings. If you are running into the nifi.queue.swap.threshold limit of MergeContent as described in NIFI-697, then you should increase that value in the nifi.properties file and restart your NiFi process. A multiple of 10000 is recommended. You will also likely have to increase your Java memory settings in bootstrap.conf.MergeContent works like this.When a FlowFile arrives at MergeContent, it is assigned to a bin based on Merge Strategy and Correlation Attribute Name. Maximum Number of Bins controls resource usage such that if all bins have FlowFiles in them and another FlowFile arrives that doesn't fit into one of those bins, then the oldest bin is automatically marked as complete, and the new FlowFile starts its own new bin.A bin will be complete once (number of files in bin) >= Minimum Number of Entries AND (number of bytes in bin) >= Minimum Group Size OR the bin has existed for Max Bin Age.

IntroductionApache Zeppelin (Incubator at the time of writing this post) is one of my favourite tools that I try to position and present to anyone interested in Analytics, Its 100% open source with an intelligent international team behind it in Korea (NFLABS) (Moving to San Francisco soon), its mainly based on interpreter concept that allows any language/data-processing-backend to be plugged into Apache Zeppelin.Very similar to IPython/Jupyter except that the UI is probably more appealing and the amount of interpreters supported are richer, at the time of writing this Blog Zeppelin supported: Apache Hive QLApacheSpark (SQL, Scala and Python)ApacheFlinkPostgresPivotal HAWQShellApache TajoAngularJSApache CassandraApache igniteApache PhoenixApache GeodeApache KylinApache Lenswith this rich set of interpreters provided, it makes on boarding platforms like Apache Hadoop or Data Lake concepts much easier where data is sitting and consolidated somewhere and different organizational units wit

HDP deployment in Azure and configuring wasb in ambariLessons learned: All VMs needs to be under virtual private network and hostnames needs to be changed.1) Create virtual network for your cluster Note: screen shot has name hdptest01 but it's test2) Create VMs and choose network created in the above stepThis is very important step and we need to pay attention to the hostname. Added endpoint for 8080.There is a step to add disks to vm. You can follow this blog At this point all the hosts are created and we will make changes to hostnames.for example: the following needs to be modified. The command hostname should show internal hostname or same output as hostname -f so modify /etc/sysconfig/network and reboot all the nodes or change hostname using hostname command.[root@hdpmaster01 ~]# cat /etc/sysconfig/networkHOSTNAME=hdpmaster01.hdpmaster01.j3.internal.cloudapp.netNETWORKING=yes#hostnamehdpmaster01.hdpmaster01.j3.internal.cloudapp.netFollow HDP Docs to install cluster using AmbariUse

Initially I had installed hdp2.1 with ambari 1.7, then I upgraded ambari to 2.1.2 and upgraded hdp stack to 2.2.0.0 as per this documentation. As per below note I had to delete STORM_UI_SERVER component:"In HDP 2.2, STORM_REST_API component was deleted because the service was moved into STORM_UI_SERVER. When upgrading from HDP 2.1 to 2.2, you must delete this component using the API as follows"I tried deleting it using ambari REST APIs: First stop the component using below command: curl -u admin:admin -X PUT -H 'X-Requested-By:1' -d '{"RequestInfo":{"context":"Stop Component"},"Body":{"HostRoles":{"state":"INSTALLED"}}}' http://hdpambari.hortonworks.com:8080/api/v1/clusters/c1/hosts/hdpambari.hortonworks.com/host_components/STORM_REST_API Then Delete using below curl call.curl -u admin:admin -X DELETE -H 'X-Requested-By:1' http://hdpambari.hortonworks.com:8080/api/v1/clusters/hdpambari/services/STORM/components/STORM_REST_API<br>;It did not work because stop command did not com

Authorization Models applicable to the Hive CLI Hive provides a few different authorization models plus Apache Ranger, as described in the Hive Authorization section of the HDP System Administration Guide. Hive CLI is subject to the following two models--Hive default (Insecure) - Any user can run GRANT statements - DO NOT USEStorage-based (Secure) - Authorization at the level of databases/tables/partitions, based on HDFS permissions (and ACLs in HDP 2.2.0+) Frequently Asked Questions about Hive CLI Security Can I set restrictive permissions on the hive executable (shell wrapper script) and hive-cli jar?No, components such as Sqoop and Oozie may fail. Additionally, a user can run their own copy of the hive client from anywhere they can set execution privileges. To avoid this limitation, migrate to the Beeline CLI and utilize HiveServer2, and restrict access to the cluster through a gateway such as Knox.Can Ranger be used to enforce permissions for Hive CLI users?HDFS policies can be cre

container-executor.cfgYARN containers in a secure cluster use the operating system facilities to offer execution isolation for containers. Secure containers execute under the credentials of the job user. The operating system enforces access restriction for the container. The container must run as the user that submitted the application. Therefore it is recommended to never submit jobs from a superuser account (HDFS or Linux) when LinuxContainerExecutor is used.To prevent superusers from submitting jobs, the container executor configuration (/etc/hadoop/conf/container-executor.cfg) includes the properties banned.users and min.user.id. Attempting to submit a job that violates either of these settings will result in an error indicating the AM container failed to launch: INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: Application application_1234567890123_4567 failed 2 times due to AM Container for appattempt_1234567890123_4567_000002 exited with exitCode: -1000Followed

One of the search use cases that I’ve been introduced to would require the ability to index text such as scanned text in png files. I set out to figure out how to do this with SOLR. I came across a couple pretty good blog posts, but as usual, you have to put together what you learn from multiple sources before you can get things to work correctly (or at least that’s what usually happens for me). So I thought I would put together the steps I took to get it to work.I used HDP Sandbox 2.3.Step-by-step guideInstall dependencies - this will provide you support for processing pngs, jpegs, and tiffs yum install autoconf automake libtoolyum install libpng-develyum install libjpeg-develyum install libtiff-develyum install zlib-develDownload Leptonica, an image processing librarywget http://www.leptonica.org/source/leptonica-1.69.tar.gzDownload Tesseract, an Optical Character Recognition engine wget http://tesseract-ocr.googlecode.com/files/tesseract-ocr-3.02.02.tar.gzEnsure proper variables and

Recently I met a HBase connection problem from Storm. The error message is:Error message: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.ipc.controller.ServerRpcControllerFactoryIt is because when we turn on Phoenix in HBase, HBase will add some additional properties (e.g. org.apache.hadoop.hbase.ipc.controller.ServerRpcControllerFactory) related to Phoenix in the hbase-site.xml. Users suppose to add these new dependencies in their pom.xml file, otherwise the hbase connection will fail, even if user does not use Phoenix at all. In order to avoid this "class not found error", the easiest way is to make sure you turn Phoenix off when connecting HBase from Storm. This solution is still valid in HDP 2.5.

Configuring a separate service RPC port can improve the responsiveness of the NameNode by allowing DataNode and client requests to be processed via separate RPC queues.Adding a service RPC port to an HA cluster with automatic failover via ZKFCs requires a workaround step.The steps to configure a service RPC port are as follows:Add the following settings to hdfs-site.xml. <property> <name>dfs.namenode.servicerpc-address.mycluster.nn1</name> <value>nn1.example.com:8040</value> </property> <property> <name>dfs.namenode.servicerpc-address.mycluster.nn2</name> <value>nn2.example.com:8040</value> </property> 2. Restart NameNodes. 3. Stop the ZKFC processes on both NameNodes 4. Run the following command to reset the ZKFC state in ZooKeeperhdfs zkfc -formatZK Without this step you will see the following exception after ZKFC restart.java.lang.RuntimeException: Mismatched address stored in ZK for Name

I had setup Transparent data encryption some time back for HDP2.2 by referring steps mentioned at http://hortonworks.com/kb/hdfs-transparent-data-encryption/I tested my setup, everything worked perfectly except one thing, superuser was able to access the contents from encrypted zone transparently which I think should not work that way, so to restrict your encrypted zone you need to modify below property in /usr/kms-demo/hadoop/etc/hadoop/kms-acls.xml<property><name>hadoop.kms.acl.DECRYPT_EEK</name><value>user1,user2</value><description> ACL for decryptEncryptedKey CryptoExtension operations.</description></property>Add list of users for whom you want to give access to your encrypted zone.Hope this is useful 🙂