Community Articles

ISSUE:While configuring NFS mounts to access HDFS as a part of local FS, we do tend to control the access using nfs proxies as shown below,<property> <name>hadoop.proxyuser.nfsserver.groups</name> <value>nfs-users1,nfs-users2</value> <description> The 'nfsserver' user is allowed to proxy all members of the 'nfs-users1' and 'nfs-users2' groups. Set this to '*' to allow nfsserver user to proxy any group. </description></property><property> <name>hadoop.proxyuser.nfsserver.hosts</name> <value>nfs-client-host1.com</value> <description> This is the host where the nfs gateway is running. Set this to '*' to allow requests from any hosts to be proxied. </description></property>However, a user who has access to NFS server would be able to access (view) the HDFS file system even if they are not part of "hadoop.proxyuser.nfsserver.groups" and "hadoop.proxyuser.nfsserver.hosts" .

This is applicable in a Kerberos enabled HDP 2.5.x cluster with Zeppelin, Livy & Spark.Post successful Kerberos setup, log in to Zeppelin and run Spark note, the note runs file.But running simple sc.version from livy interpreter gives "Cannot start spark" in the Zeppelin UI.In the Livy log at /var/log/livy/livy-livy-server.out you may find a message similar to the following.INFO: 17/05/25 21:24:12 INFO metastore: Trying to connect to metastore with URI thrift://vinay-hdp25-2.field.hortonworks.com:9083May 25, 2017 9:24:12 PM org.apache.spark.launcher.OutputRedirector redirectINFO: 17/05/25 21:24:12 ERROR TSaslTransport: SASL negotiation failureMay 25, 2017 9:24:12 PM org.apache.spark.launcher.OutputRedirector redirectINFO: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]This happens when Livy tried to connect to Hive Metastore and fails with above message.The fix is to c

Currently ranger does not support defining password of default admin users(rangerusersync and rangertagsync) during installation. This is a security concern if you miss to update the password post installation as default password for these users is same as username, thereby compromising admin access to ranger. There is plan to address this in future HDP release by allowing to define the passwords for admin users during installation, however for the time being below workaround can be used to enforce password policy. Once ranger installation is successful one can use below rest api get and put command to change the password of any user. This is helpful in case of automated cluster installation using ambari blueprint. To change the password of the user first get the user details using below call:curl -s -u admin:admin -H "Accept: application/json" -H "Content-Type: application/json" -X GET http://h2.openstacklocal:6080/service/xusers/users/2 You will get something like this.{"id":2,"cre

Sharing the steps to make Hive UDF/UDAF/UDTF to work natively with SparkSQL1- Open spark-shell with hive udf jar as parameter:spark-shell --jars path-to-your-hive-udf.jar2- From spark-shell, open declare hive context and create functionsval sqlContext = new org.apache.spark.sql.hive.HiveContext(sc);sqlContext.sql("""create temporary function balance as 'com.github.gbraccialli.hive.udf.BalanceFromRechargesAndOrders'""");3- From spark-shell, use your UDFs directly in SparkSQL:sqlContext.sql("""create table recharges_with_balance_array as select reseller_id, phone_number, phone_credit_id, date_recharge, phone_credit_value, balance(orders,'date_order', 'order_value', reseller_id, date_recharge, phone_credit_value) as balancefrom orders""");PS: I found some issues using UDTFs with spark 1.3, which was fixed on spark 1.4.1. I tested all, UDF, UDAF and UDTF, all of them worked properly, same sql statements and same results.

Note:This article is an extension of the instructions at this link.This article uses the example of a 12-node CentOS 7 VM cluster on Openstack environment and with HDP 2.5 as base stack for Metron ver > 0.3.x (or HDP 2.4 for Metron < 0.2.x).Please be aware that Metron deployment using Ambari management packs is actively being enhanced and worked upon. Some of the steps in this article and/or the behavior might be altered or become obsolete. Refer to this link for the current set of limitations with the Ambari Mpack installation and setup.The selection of nodes, services, slaves and clients in the Ambari cluster wizard are an indicative example. Optimization for performance and scale requirements are out of scope of this article.PrerequisitesFor Generating RPMs and Management PackLocal system (Mac or Linux) with build tools installed viz. Maven and docker. Refer here for more details on the tool pre-requisites.Docker service has been installed and is running.A cloned copy of the M

NOTE: This issue was found in the HDF Sandbox. A new Sandbox will be posted soon. Until then, please use the instructions below.After importing the HDP 2.6 Sandbox hosted in Oracle VirtualBox and starting the VM, I saw the message "Connectivity issues detected!" in the console and when I tried to connect to Ambari in the web browser, I was not able to connect.To correct the connectivity:1. Go to the Oracle VM VirtualBox Manager. Select the sandbox virtual machine. Right click and select Close > Power off to shut down the VM. 2. Right click on the sandbox virtual machine and select Setting. The VM Settings dialog displays.3. Click on Network from the list on the left side of the dialog.4. Click on Advanced to unfold the Advanced network settings.5. Check Cable Connected. 6. Click OK to save the setting.7. Restart the VM.8. You should now be able to connect to Ambari by entering the url http://127.0.0.1:8080 in your browser.

HDP 2.6/Ambari 2.5 GA! Now available!Key HDP 2.6 and Ambari 2.5 offerings: Hive LLAPSpark 2.1Ambari Views Workflow ManagerAmbari Views Hue-to-View Migration ToolApache Phoenix and HBase Storage Quota SupportApache Ranger Policy Import/ExportMacros/shortcuts in policy authoringPerformance enhancements for policy executionPolicy Sync status pagesApache Atlas Improved metadata catalog search experienceImproved Lineage and Impact analysis user experienceTag based policies for Kafka, HDFS, HBaseManual entity create/updateNew V2 API which significantly improve consumability, extensibility and partner integrationsNumerous performance and scalability improvementsApache Ambari Service Auto-startSmartSense Recommendations in Apache AmbariAmbari Views: File Browser: Display files inside an S3 bucketApache HDFS Device Behavior AnalyticsGeneration 1 Garbage Collector (Technical Preview)Apache YARN Queue PrioritiesEnhanced Fault Tolerance, Diagnostics & Log CollectionSupported Platforms: Cent

This tutorial will walk you through the process of using Cloudbreak recipes to install TensorFlow for Anaconda Python on an HDP 2.6 cluster during cluster provisioning. We'll then update Zeppelin to use the newly install version of Anaconda and run a quick TensorFlow test. Prerequisites You should already have a Cloudbreak v1.14.4 environment running. You can follow this article to create a Cloudbreak instance using Vagrant and Virtualbox: HCC Article You should already have created a blueprint that deploys HDP 2.6 with Spark 2.1. You can follow this article to get the blueprint setup. Do not create the cluster yet, as we will do that in this tutorial: HCC Article You should already have credentials created in Cloudbreak for deploying on AWS (or Azure). This tutorial does not cover creating credentials. Scope This tutorial was tested in the following environment: Cloudbreak 1.14.4 AWS EC2 HDP 2.6 Spark 2.1 Anaconda 2.7.13 TensorFlow 1.1.0 Steps Create Recipe Before you can use a re

This tutorial will walk you through the process of using Cloudbreak to deploy an HDP 2.6 cluster with Spark 2.1. We'll copy and edit the existing hdp-spark-cluster blueprint which deploys Spark 1.6 to create a new blueprint which installs Spark 2.1. This tutorial is part one of a two-part series. The second tutorial walks you through using Zeppelin to verify the Spark 2.1 installation. You can find that tutorial here: HCC Article Prerequisites You should already have a Cloudbreak v1.14.0 environment running. You can follow this article to create a Cloudbreak instance using Vagrant and Virtualbox: HCC Article You should already have updated Cloudbreak to support deploying HDP 2.6 clusters. You can follow this article to enable that functionality: HCC Article Scope This tutorial was tested in the following environment: Cloudbreak 1.14.4 AWS EC2 HDP 2.6 Spark 2.1 Steps Create Blueprint Before we can deploy a Spark 2.1 cluster using Cloudbreak, we need to create a blueprint that specif

Server 1 :- 192.168.154.111 (centos)Server 2 :- 192.168.154.113 (centos2) 1) First install the client on server from where you want to connect Configure Repository and install client [root@centos2 ~]# yum install mysql-client 2) My Sql software installed on centos2[root@centos2 ~]# yum install mysql-server 3)Connected to Mysql on Centos2, Following user exist before creating Hive user and Database.#mysql -u root -p passwordmysql> Select host,user from mysql.user;+-----------------+-------+| host | user |+-----------------+-------+| localhost | root |+-----------------+-------+8 rows in set (0.00 sec)Creating hive user and database mysql> create user 'hive'@'192.168.154.111' identified by 'hive';Query OK, 0 rows affected (0.05 sec)mysql> create user 'hive'@'localhost' identified by 'hive';Query OK, 0 rows affected (0.00 sec)mysql> create database hive;Query OK, 1 row affected (0.00 sec)mysql> grant ALL ON hive.* TO 'hive'@'192.168.154.111';Query OK, 0

Ingesting JMS Data into HiveA company has a lot of data transmitting around the enterprise asynchronously with Apache ActiveMQ, they want to tap into it and convert JSON messages coming from web servers and store into Hadoop.I am storing the data into Apache Phoenix / HBase via SQL. And also since it's so easy, I am storing the data into ORC files in HDFS for Apache Hive access. Apache NiFi 1.2 generates DDL for a Hive Table for ushive.ddlCREATE EXTERNAL TABLE IF NOT EXISTS meetup (id INT, first_name STRING, last_name STRING, email STRING, ip_address STRING, company STRING, macaddress STRING, cell_phone STRING) STORED AS ORCLOCATION '/meetup'insert into meetup(id , first_name , last_name , email , ip_address , company , macaddress , cell_phone)values(?,?,?,?,?,?,?,?) HDF Flow ConsumeJMSPath 1: Store in Hadoop as ORC with a Hive Table InferAvroSchema: get a schema from the JSON dataConvertJSONtoAVRO: build an AVRO file from JSON dataMergeContent: build a larger chunk of AVRO d

Backup Files from Hadoop ListHDFS set parameters, pick a high level directory and work down. /etc/hadoop/conf/core-site.xmlFetchHDFS ${path}/${filename}PutFileStore in localBackup Hive TablesSelectHiveQL Output format AVRO, with SQL: select * from beaconstatusConvertAVROtoORC generic for all the tablesUpdateAttribute tablename ${hive.ddl:substringAfter('CREATE EXTERNAL TABLE IF NOT EXISTS '):substringBefore(' (')}PutFile Use replace directories and create missing directories with directory: /Volumes/Transcend/HadoopBackups/hive/${tablename}For Phoenix tables, I use the same ConvertAvroToORC, UpdateAttribute and PutFile boxes and just add ExecuteSQL to ingest Phoenix data.For every new table, I add one box and link it to ConvertAvroToORC. Done!This is enough of a backup so if I need to rebuild and refill my development cluster, I can do so easily. Also I have them schedule for once a day to rewrite everything.This is Not For Production or Extremely Large Data! This works grea

Apart from checking the topologies from a Storm WebUI, we can also list the active topologies from one of the cluster nodes,We can use the following command to list the same,/usr/hdp/<HDP-version>/storm/bin/storm listIf there are no topologies running, we would get an output as follows,No topologies running.

TOAD is a freeware tool available for OSX and Windows from Dell. It supports Kerberos, Hive, HDFS Explorer, SQL, export to CSV/XLS, charting and logging. The documentation mentions support for up to HDP 2.3, but I had most features work well and fast with HDP 2.4.2 Sandbox. The tool is very smart to auto detect settings using Ambari.AMBARI URL for Sandbox: sandbox.hortonworks.com:8080With Ambari User.Once configured, you can query Hive Tables through Spark SQL or Hive. The performance is quite good.The tool provides a nice view of Charts and Logs.You can can also browse HDFS files.

With Apache Spark 2.1, SparkR supports virtual environment. With VirtualEnv when you deploy Spark on YARN, each SparkR job will have its own library directory which binds to the executor’s YARN container. All necessary third-party libraries are installed into that directory. After the Spark job finishes, the directory is deleted and there is no interference with other Spark job’s environment.SparkR now supports the following two scenarios:1) Internet enabled cluster allows packages to be installed (say from CRAN) on the executors so that R packages can be used in the tasks.2) For isolated cluster, the driver first downloads the required packages or download packages ahead of time and then add them to the cluster. Spark will distribute the packages to executors and we can install them when that executor runs.Both of these scenarios install R packages in a separate local directory on executor for each user, and will not pollute that executor’s native R environment. This directory will b

With Apache Spark 2.1, SparkR supports virtual environment. With VirtualEnv when you deploy Spark on YARN, each SparkR job will have its own library directory which binds to the executor’s YARN container. All necessary third-party libraries are installed into that directory. After the Spark job finishes, the directory is deleted and there is no interference with other Spark job’s environment.SparkR now supports the following two scenarios:1) Internet enabled cluster allows packages to be installed (say from CRAN) on the executors so that R packages can be used in the tasks.2) For isolated cluster, the driver first downloads the required packages or download packages ahead of time and then add them to the cluster. Spark will distribute the packages to executors and we can install them when that executor runs.Both of these scenarios install R packages in a separate local directory on executor for each user, and will not pollute that executor’s native R environment. This directory will b

Ingest data with NIFI to Hive LLAP and DruidSetting up Hive LLAPSetting up DruidConfiguring the dimensionsSetting up SupersetConnection to DruidCreating a dashboard with visualisationsDifferences with ES/Kibana and SOLR/Banana

Background Hadoop and its ecosystem projects are distributed systems. Hadoop provides foundational services,such as distributed file system, cluster wide resource management, that many other projects (e.g OOzie, MR Jobs, Tez, Hive Queries, Spark Jobs) leverage. With this layered approach a Hive Query, or a Spark job leads to sequence of operations that span multiple systems. For example, a Spark job may leverage YARN to get containers to run, it may also read from HDFS. Tracing and debugging problems in distributed systems is hard since it involves tracing call across many systems. Hadoop resolved this problem. Hadoop has implemented a feature of log tracing – caller context (HDFS-9184 andYARN-4349). It can help users to better diagnose and understand how specific applications impact parts of the Hadoop system and potential problems they may be creating (e.g. overloading NN). As HDFS mentioned in HDFS-9184 , for a given HDFS operation, it's very helpful to track which upper level

Prerequisites You should already have a Cloudbreak v1.14.4 environment running. You can follow this article to create a Cloudbreak instance using Vagrant and Virtualbox: HCC Article You should already have credentials created in Cloudbreak for deploying on AWS (or Azure). Scope This tutorial was tested in the following environment: macOS Sierra (version 10.12.4) Cloudbreak 1.14.4 AWS EC2 NOTE: Cloudbreak 1.14.0 (TP) had a bug which caused HDP 2.6 clusters installs to fail. You should upgrade your Cloudbreak deployer instance to 1.14.4. Steps Create application.yml file UPDATE 05/24/2017: The creation of a custom application.yml file is not required with Cloudbreak 1.14.4. This version of Cloudbreak includes support for HDP 2.5 and HDP 2.6. This step remains for educational purposes for future HDP updates. You need to create an application.yml file in the etc directory within your Cloudbreak deployment directory. This file will contain the repo information for HDP 2.6. If you follow

It is often the case that we need to install Hortonworks in environments with strict requirements. One such requirement may be that all http traffic must go through a dedicated proxy server.When installing Hortoworks HDP using Ambari, you can find instructions for configuring Ambari to use the proxy on the docs.hortonworks.com website. For example, here is the page for configuring Ambari 2.2http://docs.hortonworks.com/HDPDocuments/Ambari-2.2.2.0/bk_ambari_reference_guide/content/ch_setting_up_an_internet_proxy_server_for_ambari.htmlNotice that the instructions mention that you must also configure yum to use the proxy. It’s important to note that the above instructions for yum will configure all repositories to use the proxy, and you may not want this behavior.So while it is great to set the proxy at yum’s global level, you should review any existing repository configurations to determine if they should not use the proxy. If any repositories should not use the proxy, then you can update

ENVIRONMENTThis problem has been replicated and fixed on HDP 2.5.xPROBLEMI have followed up the https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.0/bk_zeppelin-component-guide/content/zepp-with-spark.html to add additional jar for livy - copy jar file into /usr/hdp/<version>/livy/repl-jars folder. Restarted livy service afterwards. When running: %livy import de.xxx.statistik.program.model.LineModel I am getting: <console>:25: error: not found: value de import de.xxx.statistik.program.model.LineModel SOLUTIONAdded the following into Ambari -> MapReduce2 -> mapreduce.application.classpath - :$PWD/*Before Change$PWD/mr-framework/hadoop/share/hadoop/mapreduce/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/*:$PWD/mr-framework/hadoop/share/hadoop/common/*:$PWD/mr-framework/hadoop/share/hadoop/common/lib/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/lib/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/*:$PWD/mr-framework/ha

If you haven't already see the 1st tutorial I made which guides you through the setup of rapidminer to read from hive. We will pick up where it leaves off. Read-from-hive-using-rapidmineradd a "set role" operator next to the your "retrieve from hive operator" that is located in the "Radoop Nest".This allows you to select the column you wish to use in the model.In this case I set the name field to the category column in my dataset. You can obtain this dataset here: dataThis is just for illustrative purposes so if you have data that has labels already feel free to use in place of this. Now add a "split validation" operator and connect the ports.Then double click the validation operator. Add a "decision tree" operator on the left pane and add an "apply model" and "performance" operator and connect them all. For performance select accuracy or whatever you wish to check. If you are using the sample data provided in this tutorial you will see some errors. Click on the error icons on each ope

First you need to have Rapidminer downloaded and installed on your machine. https://my.rapidminer.com/nexus/account/index.html#downloadsOnce installed open Rapidminer and look at the list of operators. There is a link at the bottom left to "Get More Operators"Click the link then search for "Radoop"Select both packages and click install. After Rapidminer restarts you will see in the extensions folder the new operators we downloaded. Now we need to configure the connection. In the toolbar select "connections" then "Manage Radoop Connections" Select "+ New Connection".If you have your Hadoop config files available you can use those to set the properties otherwise select "manual" Select the hadoop version you have. In my case "Hortonworks 2.x" and supply the master url... If you have multiple masters select the check box and provide the details. Click "OK"Now click ">> Quick Test" If successful you are all set to read from Hive. Drag an "Radoop Nest" operator onto the canvas. Select

Ambari user sync will fail to map or import users when trying to pull from groups with 1,500+ members.What we see when we use ldapsearch to query a large group is: <snip> member;range=0-1499: CN=Elgine Metzger,OU=users,OU=test,DC=j4ck3l,DC=net member;range=0-1499: CN=Friedolf Welter,OU=users,OU=test,DC=j4ck3l,DC=net </snip> This is seems to come from the LDAP Policy value: MaxValRange "MaxValueRange controls the number of values that are returned on a single attribute on a single object. Default"1500 Hard Limit: 5000" -- http://ldapwiki.com/wiki/MaxValRange To fix this: Go to the domain controller that we’re connecting to for the sync Find the file ntdsutil.exe (most likely under c:\windows\system32 or c:\winnt\system32 Run the ntdsutil.exe Type “ldap policies” and enter Type "connections" and enter Type "Connect to server [YourDCName]" and enter Type "q" and enter Type "Show Values" to see the current settings Type “Set MaxValRange to 2500” and enter Type “Commit Change

We tried 3 scenarios and all failed.1) Creating Hive Table on HBase. (proper hive/hbase polices for user) hive> CREATE TABLE hbase_table_1(key int, value string) STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,cf1:val") TBLPROPERTIES ("hbase.table.name" = "xyz", "hbase.mapred.output.outputtable" = "xyz"); FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:org.apache.hadoop.hbase.security.AccessDeniedException: org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient permissions for user 'abc@ABC.COM' (action=create) at org.apache.ranger.authorization.hbase.AuthorizationSession.publishResults(AuthorizationSession.java:254) at org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor.authorizeAccess(RangerAuthorizationCoprocessor.java:595) at org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor.requirePermission(RangerAuth