Resources

With Apache Spark 2.1, SparkR supports virtual environment. With VirtualEnv when you deploy Spark on YARN, each SparkR job will have its own library directory which binds to the executor’s YARN container. All necessary third-party libraries are installed into that directory. After the Spark job finishes, the directory is deleted and there is no interference with other Spark job’s environment.SparkR now supports the following two scenarios:1) Internet enabled cluster allows packages to be installed (say from CRAN) on the executors so that R packages can be used in the tasks.2) For isolated cluster, the driver first downloads the required packages or download packages ahead of time and then add them to the cluster. Spark will distribute the packages to executors and we can install them when that executor runs.Both of these scenarios install R packages in a separate local directory on executor for each user, and will not pollute that executor’s native R environment. This directory will b

In this tutorial, we will set up the Hortonworks Data Cloud for AWS 1.16 (released in June 2017) using the advanced template, which requires you to configure a VPC and an RDS instance and database prior to launching the cloud controller.We will go through the following steps: Subscribing to HDCloud servicesLaunching an RDS instanceLaunching the cloud controller using the advanced templateAccessing the cloud controller UIThis tutorial assumes no prior experience with AWS. If you encounter errors while performing the steps, refer to the Troubleshooting documentation. Let's get started!Prerequisites The prerequisites for this tutorial are:1. You have an AWS account.2. You know in which AWS region you want to launch.3. You have an SSH key pair in the selected region.If you need help with the steps required to meet these prerequisites, refer to the following post:How to set up Hortonworks Data Cloud (HDCloud) for AWSSubscribe to HDCloud ServicesIn order to use the product, you need to subsc

This tutorial has been successfully tried on HDP-2.4.0.0 and Ambari 2.2.1.0.I have my HDP Cluster Kerberized and Ambari has been configured for SSL.Note - Steps are same for Ambari with or without SSL..Please follow below steps for Configuring Hive View on Kerberized HDP Cluster. .Step 1 - Please configure your Ambari Server for Kerberos with the steps mentioned in below article. Please follow steps 1 to 5.https://community.hortonworks.com/articles/40635/configure-tez-view-for-kerberized-hdp-cluster.html.Step 2 - Please add below properties to core-site.xml via Ambari UI and restart required services..Note - If you are running Ambari Server as root user then add below properties hadoop.proxyuser.root.groups=*hadoop.proxyuser.root.hosts=*.If you are running Ambari server as non-root user then please add below properties in core-site.xmlhadoop.proxyuser.<ambari-server-user>.groups=*hadoop.proxyuser.<ambari-server-user>.hosts=*Please replace <ambari-server-user> with use

Issue:On Adding Oozie service to cluster, sometimes its possible user could see an error with simple message Failed to create new war file as below and on top of it Oozie server wouldn't start. INFO: Adding extension: /usr/hdp/current/oozie-server/libext/mysql-jdbc-driver.jar Failed: creating new Oozie WARSolution:Oozie deployment script is trying to unzip the existing Oozie war file and then zip it back up and place it in a sub directory of /tmp/... . This unzip and zip process doubles the storage space required for the Oozie war file. Make sure there is enough /tmp storage for the above process to take place, logs wouldn't provide this info.

Working With S3 Compatible Data Stores (and handling single source failure)With the major outage of S3 in my region, I decided I needed to have an alternative file store. I found a great open source server called Minio that I run on a miniPC running Centos 7. We could also use this solution for connecting to other S3 compatible stores such as RiakCS and Google Cloud Storage. I like to remain cloud and location neutral.In Apache NiFi, it's really easy. You can have two sources and two destinations, instead of just your regular AWS S3, you can have one for AWS S3 and one for another. Or you can use the second as a disaster recovery data backup. Since my Minio box is local, I can store data locally. It's pretty affordable to get a few terabytes connected to a small Linux box to hold some backups. With Apache NiFi, you have queues to buffer a potentially slower ingest/egress. Minio Setupwget https://dl.minio.io/server/minio/release/linux-amd64/minio chmod 755 minionohup ./min

{"authentication":{"class": "org.apache.solr.security.KerberosPlugin", "blockUnknown": true},"authorization":{"class":"solr.RuleBasedAuthorizationPlugin","permissions":[{"name":"security-edit","role":"admin"},{"name":"collection-admin-edit", "role" : "dev"},{"name":"read", "role" : "read_coll_data","path":"/select"}, {"name":"update", "role" : "put_coll_data"}],"user-role":{"a1@EXAMPLE.COM":"admin", "a2@EXAMPLE.COM":"dev", "a3@EXAMPLE.COM":"read_coll_data", "a4@EXAMPLE.COM":"put_coll_data", "a5@EXAMPLE.COM": "read_coll_data","rangeradmin@EXAMPLE.COM":["read_coll_data", "put_coll_data"]}}}This file has two parts, the authentication part and the authorization part. The authentication part stores information about the class being used for authentication. The authorization part is not related to Basic authentication, but is a separate authorization plugin designed to support fine-grained user access control. Each role is comprised of one or more permissions which define what the user is al

ISSUE:While configuring NFS mounts to access HDFS as a part of local FS, we do tend to control the access using nfs proxies as shown below,<property> <name>hadoop.proxyuser.nfsserver.groups</name> <value>nfs-users1,nfs-users2</value> <description> The 'nfsserver' user is allowed to proxy all members of the 'nfs-users1' and 'nfs-users2' groups. Set this to '*' to allow nfsserver user to proxy any group. </description></property><property> <name>hadoop.proxyuser.nfsserver.hosts</name> <value>nfs-client-host1.com</value> <description> This is the host where the nfs gateway is running. Set this to '*' to allow requests from any hosts to be proxied. </description></property>However, a user who has access to NFS server would be able to access (view) the HDFS file system even if they are not part of "hadoop.proxyuser.nfsserver.groups" and "hadoop.proxyuser.nfsserver.hosts" .

This is applicable in a Kerberos enabled HDP 2.5.x cluster with Zeppelin, Livy & Spark.Post successful Kerberos setup, log in to Zeppelin and run Spark note, the note runs file.But running simple sc.version from livy interpreter gives "Cannot start spark" in the Zeppelin UI.In the Livy log at /var/log/livy/livy-livy-server.out you may find a message similar to the following.INFO: 17/05/25 21:24:12 INFO metastore: Trying to connect to metastore with URI thrift://vinay-hdp25-2.field.hortonworks.com:9083May 25, 2017 9:24:12 PM org.apache.spark.launcher.OutputRedirector redirectINFO: 17/05/25 21:24:12 ERROR TSaslTransport: SASL negotiation failureMay 25, 2017 9:24:12 PM org.apache.spark.launcher.OutputRedirector redirectINFO: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]This happens when Livy tried to connect to Hive Metastore and fails with above message.The fix is to c

Currently ranger does not support defining password of default admin users(rangerusersync and rangertagsync) during installation. This is a security concern if you miss to update the password post installation as default password for these users is same as username, thereby compromising admin access to ranger. There is plan to address this in future HDP release by allowing to define the passwords for admin users during installation, however for the time being below workaround can be used to enforce password policy. Once ranger installation is successful one can use below rest api get and put command to change the password of any user. This is helpful in case of automated cluster installation using ambari blueprint. To change the password of the user first get the user details using below call:curl -s -u admin:admin -H "Accept: application/json" -H "Content-Type: application/json" -X GET http://h2.openstacklocal:6080/service/xusers/users/2 You will get something like this.{"id":2,"cre

Sharing the steps to make Hive UDF/UDAF/UDTF to work natively with SparkSQL1- Open spark-shell with hive udf jar as parameter:spark-shell --jars path-to-your-hive-udf.jar2- From spark-shell, open declare hive context and create functionsval sqlContext = new org.apache.spark.sql.hive.HiveContext(sc);sqlContext.sql("""create temporary function balance as 'com.github.gbraccialli.hive.udf.BalanceFromRechargesAndOrders'""");3- From spark-shell, use your UDFs directly in SparkSQL:sqlContext.sql("""create table recharges_with_balance_array as select reseller_id, phone_number, phone_credit_id, date_recharge, phone_credit_value, balance(orders,'date_order', 'order_value', reseller_id, date_recharge, phone_credit_value) as balancefrom orders""");PS: I found some issues using UDTFs with spark 1.3, which was fixed on spark 1.4.1. I tested all, UDF, UDAF and UDTF, all of them worked properly, same sql statements and same results.

Note:This article is an extension of the instructions at this link.This article uses the example of a 12-node CentOS 7 VM cluster on Openstack environment and with HDP 2.5 as base stack for Metron ver > 0.3.x (or HDP 2.4 for Metron < 0.2.x).Please be aware that Metron deployment using Ambari management packs is actively being enhanced and worked upon. Some of the steps in this article and/or the behavior might be altered or become obsolete. Refer to this link for the current set of limitations with the Ambari Mpack installation and setup.The selection of nodes, services, slaves and clients in the Ambari cluster wizard are an indicative example. Optimization for performance and scale requirements are out of scope of this article.PrerequisitesFor Generating RPMs and Management PackLocal system (Mac or Linux) with build tools installed viz. Maven and docker. Refer here for more details on the tool pre-requisites.Docker service has been installed and is running.A cloned copy of the M

NOTE: This issue was found in the HDF Sandbox. A new Sandbox will be posted soon. Until then, please use the instructions below.After importing the HDP 2.6 Sandbox hosted in Oracle VirtualBox and starting the VM, I saw the message "Connectivity issues detected!" in the console and when I tried to connect to Ambari in the web browser, I was not able to connect.To correct the connectivity:1. Go to the Oracle VM VirtualBox Manager. Select the sandbox virtual machine. Right click and select Close > Power off to shut down the VM. 2. Right click on the sandbox virtual machine and select Setting. The VM Settings dialog displays.3. Click on Network from the list on the left side of the dialog.4. Click on Advanced to unfold the Advanced network settings.5. Check Cable Connected. 6. Click OK to save the setting.7. Restart the VM.8. You should now be able to connect to Ambari by entering the url http://127.0.0.1:8080 in your browser.

HDP 2.6/Ambari 2.5 GA! Now available!Key HDP 2.6 and Ambari 2.5 offerings: Hive LLAPSpark 2.1Ambari Views Workflow ManagerAmbari Views Hue-to-View Migration ToolApache Phoenix and HBase Storage Quota SupportApache Ranger Policy Import/ExportMacros/shortcuts in policy authoringPerformance enhancements for policy executionPolicy Sync status pagesApache Atlas Improved metadata catalog search experienceImproved Lineage and Impact analysis user experienceTag based policies for Kafka, HDFS, HBaseManual entity create/updateNew V2 API which significantly improve consumability, extensibility and partner integrationsNumerous performance and scalability improvementsApache Ambari Service Auto-startSmartSense Recommendations in Apache AmbariAmbari Views: File Browser: Display files inside an S3 bucketApache HDFS Device Behavior AnalyticsGeneration 1 Garbage Collector (Technical Preview)Apache YARN Queue PrioritiesEnhanced Fault Tolerance, Diagnostics & Log CollectionSupported Platforms: Cent

This tutorial will walk you through the process of using Cloudbreak recipes to install TensorFlow for Anaconda Python on an HDP 2.6 cluster during cluster provisioning. We'll then update Zeppelin to use the newly install version of Anaconda and run a quick TensorFlow test. Prerequisites You should already have a Cloudbreak v1.14.4 environment running. You can follow this article to create a Cloudbreak instance using Vagrant and Virtualbox: HCC Article You should already have created a blueprint that deploys HDP 2.6 with Spark 2.1. You can follow this article to get the blueprint setup. Do not create the cluster yet, as we will do that in this tutorial: HCC Article You should already have credentials created in Cloudbreak for deploying on AWS (or Azure). This tutorial does not cover creating credentials. Scope This tutorial was tested in the following environment: Cloudbreak 1.14.4 AWS EC2 HDP 2.6 Spark 2.1 Anaconda 2.7.13 TensorFlow 1.1.0 Steps Create Recipe Before you can use a re

This tutorial will walk you through the process of using Cloudbreak to deploy an HDP 2.6 cluster with Spark 2.1. We'll copy and edit the existing hdp-spark-cluster blueprint which deploys Spark 1.6 to create a new blueprint which installs Spark 2.1. This tutorial is part one of a two-part series. The second tutorial walks you through using Zeppelin to verify the Spark 2.1 installation. You can find that tutorial here: HCC Article Prerequisites You should already have a Cloudbreak v1.14.0 environment running. You can follow this article to create a Cloudbreak instance using Vagrant and Virtualbox: HCC Article You should already have updated Cloudbreak to support deploying HDP 2.6 clusters. You can follow this article to enable that functionality: HCC Article Scope This tutorial was tested in the following environment: Cloudbreak 1.14.4 AWS EC2 HDP 2.6 Spark 2.1 Steps Create Blueprint Before we can deploy a Spark 2.1 cluster using Cloudbreak, we need to create a blueprint that specif

Server 1 :- 192.168.154.111 (centos)Server 2 :- 192.168.154.113 (centos2) 1) First install the client on server from where you want to connect Configure Repository and install client [root@centos2 ~]# yum install mysql-client 2) My Sql software installed on centos2[root@centos2 ~]# yum install mysql-server 3)Connected to Mysql on Centos2, Following user exist before creating Hive user and Database.#mysql -u root -p passwordmysql> Select host,user from mysql.user;+-----------------+-------+| host | user |+-----------------+-------+| localhost | root |+-----------------+-------+8 rows in set (0.00 sec)Creating hive user and database mysql> create user 'hive'@'192.168.154.111' identified by 'hive';Query OK, 0 rows affected (0.05 sec)mysql> create user 'hive'@'localhost' identified by 'hive';Query OK, 0 rows affected (0.00 sec)mysql> create database hive;Query OK, 1 row affected (0.00 sec)mysql> grant ALL ON hive.* TO 'hive'@'192.168.154.111';Query OK, 0

Ingesting JMS Data into HiveA company has a lot of data transmitting around the enterprise asynchronously with Apache ActiveMQ, they want to tap into it and convert JSON messages coming from web servers and store into Hadoop.I am storing the data into Apache Phoenix / HBase via SQL. And also since it's so easy, I am storing the data into ORC files in HDFS for Apache Hive access. Apache NiFi 1.2 generates DDL for a Hive Table for ushive.ddlCREATE EXTERNAL TABLE IF NOT EXISTS meetup (id INT, first_name STRING, last_name STRING, email STRING, ip_address STRING, company STRING, macaddress STRING, cell_phone STRING) STORED AS ORCLOCATION '/meetup'insert into meetup(id , first_name , last_name , email , ip_address , company , macaddress , cell_phone)values(?,?,?,?,?,?,?,?) HDF Flow ConsumeJMSPath 1: Store in Hadoop as ORC with a Hive Table InferAvroSchema: get a schema from the JSON dataConvertJSONtoAVRO: build an AVRO file from JSON dataMergeContent: build a larger chunk of AVRO d

Backup Files from Hadoop ListHDFS set parameters, pick a high level directory and work down. /etc/hadoop/conf/core-site.xmlFetchHDFS ${path}/${filename}PutFileStore in localBackup Hive TablesSelectHiveQL Output format AVRO, with SQL: select * from beaconstatusConvertAVROtoORC generic for all the tablesUpdateAttribute tablename ${hive.ddl:substringAfter('CREATE EXTERNAL TABLE IF NOT EXISTS '):substringBefore(' (')}PutFile Use replace directories and create missing directories with directory: /Volumes/Transcend/HadoopBackups/hive/${tablename}For Phoenix tables, I use the same ConvertAvroToORC, UpdateAttribute and PutFile boxes and just add ExecuteSQL to ingest Phoenix data.For every new table, I add one box and link it to ConvertAvroToORC. Done!This is enough of a backup so if I need to rebuild and refill my development cluster, I can do so easily. Also I have them schedule for once a day to rewrite everything.This is Not For Production or Extremely Large Data! This works grea

Apart from checking the topologies from a Storm WebUI, we can also list the active topologies from one of the cluster nodes,We can use the following command to list the same,/usr/hdp/<HDP-version>/storm/bin/storm listIf there are no topologies running, we would get an output as follows,No topologies running.

TOAD is a freeware tool available for OSX and Windows from Dell. It supports Kerberos, Hive, HDFS Explorer, SQL, export to CSV/XLS, charting and logging. The documentation mentions support for up to HDP 2.3, but I had most features work well and fast with HDP 2.4.2 Sandbox. The tool is very smart to auto detect settings using Ambari.AMBARI URL for Sandbox: sandbox.hortonworks.com:8080With Ambari User.Once configured, you can query Hive Tables through Spark SQL or Hive. The performance is quite good.The tool provides a nice view of Charts and Logs.You can can also browse HDFS files.

With Apache Spark 2.1, SparkR supports virtual environment. With VirtualEnv when you deploy Spark on YARN, each SparkR job will have its own library directory which binds to the executor’s YARN container. All necessary third-party libraries are installed into that directory. After the Spark job finishes, the directory is deleted and there is no interference with other Spark job’s environment.SparkR now supports the following two scenarios:1) Internet enabled cluster allows packages to be installed (say from CRAN) on the executors so that R packages can be used in the tasks.2) For isolated cluster, the driver first downloads the required packages or download packages ahead of time and then add them to the cluster. Spark will distribute the packages to executors and we can install them when that executor runs.Both of these scenarios install R packages in a separate local directory on executor for each user, and will not pollute that executor’s native R environment. This directory will b

With Apache Spark 2.1, SparkR supports virtual environment. With VirtualEnv when you deploy Spark on YARN, each SparkR job will have its own library directory which binds to the executor’s YARN container. All necessary third-party libraries are installed into that directory. After the Spark job finishes, the directory is deleted and there is no interference with other Spark job’s environment.SparkR now supports the following two scenarios:1) Internet enabled cluster allows packages to be installed (say from CRAN) on the executors so that R packages can be used in the tasks.2) For isolated cluster, the driver first downloads the required packages or download packages ahead of time and then add them to the cluster. Spark will distribute the packages to executors and we can install them when that executor runs.Both of these scenarios install R packages in a separate local directory on executor for each user, and will not pollute that executor’s native R environment. This directory will b

Ingest data with NIFI to Hive LLAP and DruidSetting up Hive LLAPSetting up DruidConfiguring the dimensionsSetting up SupersetConnection to DruidCreating a dashboard with visualisationsDifferences with ES/Kibana and SOLR/Banana

Background Hadoop and its ecosystem projects are distributed systems. Hadoop provides foundational services,such as distributed file system, cluster wide resource management, that many other projects (e.g OOzie, MR Jobs, Tez, Hive Queries, Spark Jobs) leverage. With this layered approach a Hive Query, or a Spark job leads to sequence of operations that span multiple systems. For example, a Spark job may leverage YARN to get containers to run, it may also read from HDFS. Tracing and debugging problems in distributed systems is hard since it involves tracing call across many systems. Hadoop resolved this problem. Hadoop has implemented a feature of log tracing – caller context (HDFS-9184 andYARN-4349). It can help users to better diagnose and understand how specific applications impact parts of the Hadoop system and potential problems they may be creating (e.g. overloading NN). As HDFS mentioned in HDFS-9184 , for a given HDFS operation, it's very helpful to track which upper level

Prerequisites You should already have a Cloudbreak v1.14.4 environment running. You can follow this article to create a Cloudbreak instance using Vagrant and Virtualbox: HCC Article You should already have credentials created in Cloudbreak for deploying on AWS (or Azure). Scope This tutorial was tested in the following environment: macOS Sierra (version 10.12.4) Cloudbreak 1.14.4 AWS EC2 NOTE: Cloudbreak 1.14.0 (TP) had a bug which caused HDP 2.6 clusters installs to fail. You should upgrade your Cloudbreak deployer instance to 1.14.4. Steps Create application.yml file UPDATE 05/24/2017: The creation of a custom application.yml file is not required with Cloudbreak 1.14.4. This version of Cloudbreak includes support for HDP 2.5 and HDP 2.6. This step remains for educational purposes for future HDP updates. You need to create an application.yml file in the etc directory within your Cloudbreak deployment directory. This file will contain the repo information for HDP 2.6. If you follow