Community Articles

DISCLAIMER: it was tested on HDP 2.3.2 only. There are two blocking JIRAs preventing usage of blob storage as primary filesystem on HDP 2.3.0. For HBase, you need to use page blob instead of block blob.First things first, install Azure CLI for Mac or use Azure portal. The steps below are for CLI.azure loginenter usernameenter passwordazure storage account create storageaccountname --type LRSazure storage account keys list storageaccountnamenote the account keys, you will need them in the next stepazure storage container create storagecontainername --account-name storageaccountname --account-key accountkeystringjust to validate it was createdazure storage blob list storagecontainernae --account-name storageaccountname --account-key Once the previous steps have been completed, go to Ambari UI and edit the core-site.xmlIn addition to these properties, you need to replace fs.defaultFS property with the wasb path.These properties and their descriptions are discussed in hadoop-azure document

Once one moves beyond trivial flow design a visual image like the one below becomes common. Surely, it becomes more and more messy over time and harder to glance over.Here's a great tip - double-click on the connection line that you want to bend:A new yellow anchor will appear which one can then drag around to organize things nicely.Bonus tip - one can add multiple bend points to a connection.To remove a specific anchor, simply double-click on it again, repeat for each yellow point.And an extra bonus courtesy of @mgilman@hortonworks.com: a connection label can be moved along the connection and snaps to a bend point, see below:

Consider the following flow where records are inserted in the RDBMS (or Phoenix/HBase in this instance):The PutSQL and similar processor depends on a database connection, provided by a DBCPConnectionPool controller service (Controller Settings -> Controller Services):The best part: when one creates a template, all linked services will automatically be included. When template is rehydrated in another NiFi instance, services like the one below will automatically be added to the new NiFi flow:

You have generated a certificate file:openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout test.key -out test.pem After deploying the key, you try to ssh into the instance but get prompted for a password. ssh -vvv -i test.pem <user>@<host>—————This is an issue with an updated openssl version,> openssl versionOpenSSL 1.0.1k 8 Jan 2015This is a new version of openssl. The new version does not create the key with RSA at the begin and end. So you have to use a separate command to convert the key file to old version of sshopenssl rsa -in test.key -out test_new.keyOnce that is done, use the new file for ssh. ssh -vv -i test_new.key <user>@<host>

One of the most interesting facts about Oracle is that many of their products are agnostic when it comes with working with Hadoop distributions. Hortonworks and Oracle have been working closely for over two years on product certifications and building a strong partnership from both the PM and Engineering organizations. Below are four key products in the Oracle portfolio that are certified and supported with Hortonworks. Oracle Data Integration (Oracle Data Integrator, Oracle GoldenGate) Oracle Data Integrator is certified on HDP 2.1 and 2.2 Oracle GoldenGate Big Data Edition is certified with HDP 2.2Both are currently working on HDP 2.3 certification (as of Oct '15)Blogs and helpful articles Oracle Big Data Integration Certified on HDP 2.2 Oracle Big Data Integration with Hortonworks SeriesOracle Big Data Connector (BDC) Certified for HDP 2.2, less R Analytics. Working with the BDC PM/Dev team to complete certification Blogs and helpful articles Oracle Big Data Connector Certif

Please find below a sample code which write to a kerberised HBase using keytab. package hbase; import java.io.IOException;import java.util.ArrayList;import java.util.List; import org.apache.hadoop.conf.Configuration;import org.apache.hadoop.hbase.HBaseConfiguration;import org.apache.hadoop.hbase.HColumnDescriptor;import org.apache.hadoop.hbase.HTableDescriptor;import org.apache.hadoop.hbase.KeyValue;import org.apache.hadoop.hbase.MasterNotRunningException;import org.apache.hadoop.hbase.ZooKeeperConnectionException;import org.apache.hadoop.hbase.client.Delete;import org.apache.hadoop.hbase.client.Get;import org.apache.hadoop.hbase.client.HBaseAdmin;import org.apache.hadoop.hbase.client.HTable;import org.apache.hadoop.hbase.client.Result;import org.apache.hadoop.hbase.client.ResultScanner;import org.apache.hadoop.hbase.client.Scan;import org.apache.hadoop.hbase.client.Put;import org.apache.hadoop.hbase.util.Bytes;import org.apache.hadoop.security.UserGroupInformation; public class HBas

There have been a number of questions recently on using AD/IPA with HDP 2.3 security:How to setup Active Directory/IPA?How to setup cluster OS to recognize users from AD using SSSD?How to enable kerberos for authentication?How to install Ranger for authorization/audit and setup plugins for HDFS, Hive, HBase, Kafka, Storm, Yarn, Knox and test these components on kerborized cluster?How to sync Ranger user/group sync with AD/IPA?How to integrate Knox with AD/IPA?How to setup encryption at rest with Ranger KMS?To help answer some of these questions, the partner team have prepared cheatsheets on security workshops. These are living materials with sample code snippets which are being updated/enhanced per the feedback from the field so rather than replicate the materials here, the latest materials can be referenced at the GitHub repo linked from here:https://community.hortonworks.com/repos/4465/workshops-on-how-to-setup-security-on-hadoop-using.htmlTo help get started with security, we have a

I attended a panel discussion at Hadoop World last week (September 2015) and I’d like to share my notes for others. The topics of interest were: the future of data governance, who are the actors in data governance, and what are the tools to use. Full details of the talk are here: http://strataconf.com/big-data-conference-ny-2015/public/schedule/detail/43493One early question was on what the future holds for data governance in Hadoop. An interesting answer was that collecting data in the future will require greater consent from individuals. This echoed the morning’s keynote speaker who urged the same practice. The theme of that keynote was that too much data is being collected right now, mostly without the knowledge of the individual, and eventually there will be negative consequences for all. Back on the panel, another answer was that data governance of the future will help data analysts make sense of the data inside of their data lake. For example, audit logs will be automatica

In some situations, Ranger is not an option but a Hive Authorization scheme is advisable. It is possible to use SQLStdAuth for this. However it comes with a couple caveats.Hive Wiki1. Configuring SQLStdAuth ( HDP 2.3 ) - In Ambari, select Authorization SQLStdAuth. This will set all related configuration parameters like hive.enable.authorization and disable doAs.- Add admin users to admin role. If you have a user you want to be admin add him to hive.users.in.admin.role=hive,hue,myadmin2. Prepare HDFSSince all queries now run as the hive user he needs to have read/write rights on all files in HDFS. This includes the load directories for external tables. Ideally change the owner to the warehousing folder to hive and set access right 700. I also added hive to an ETL group and made all load folders read AND writable to this group. 3. Create rolesIn Hive as an admin user- Become Admin: SET ROLE ADMIN;- Create roles:CREATE ROLE BI; ( should have read rights to all tables ) CREATE ROLE ETL; (

PROBLEM: The dfs.namenode.accesstime.precision value cannot be set from Ambari 2.1.0 when using the HDP-2.2 stack. Symptoms include: Searching for this config using the 'Filter...' box from Ambari shows 'No properties to display.'Adding the property to 'Custom hdfs-site' (in the 'Add Property' popup) tells the user that 'This property is already defined'. Attempts at clicking on the 'Find property' link again says 'No properties to display.' The property is visible on all cluster nodes in /etc/hadoop/conf/hdfs-site.xml but it is set to 0. Trying to edit this property manually on each node only reverts the setting back to 0 whenever the HDFS is restarted. ROOT CAUSE: Known bug: https://issues.apache.org/jira/browse/AMBARI-13006 https://hortonworks.jira.com/browse/BUG-43478Ambari 2.1.0 added the ability to manage HDFS NFS Gateway settings, but, this only works for HDP stacks running at version 2.3.0 and higher. RESOLUTION: A fix will be included in Ambari 2.1.2. WORKAROUND: Ambari's co

What is the YARN's Capacity Scheduler?YARN's CapacityScheduler is designed to run Hadoop applications in a shared, multi-tenant cluster while maximizing the throughput and the utilization of the cluster.Traditionally each organization has it own private set of compute resources that have sufficient capacity to meet the organization's SLA. This generally leads to poor average utilization. Also there is heavy overhead of managing multiple independent clusters.Sharing clusters between organizations allows economies of scale . However, organizations are concerned about sharing a cluster in the fear of not getting enaough available resources that are critical to meet their SLAs.The CapacityScheduler is designed to allow sharing a large cluster while giving each organization capacity guarantees. There is an added benefit that an organization can access any excess capacity not being used by others. This provides elasticity for the organizations in a cost-effective manner.Sharing clusters acro

The following provide directions to swap out the default Derby DB in Ambari for Ooize to Mysql. followed *some* steps from this for preparing to create the MySQL db creation - http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.1-latest/bk_installing_manually_book/content/rpm-chap8-3.htmlSetup databse (dropped the existing DB first) $ mysqldump -u oozie -p -h <host> oozie > oozie.dump.sql$ mysql -u oozie -p -h <host> -e "drop database oozie"$ sudo mysqlmysql> create database oozie; Query OK, 1 row affected (0.03 sec)mysql> grant all privileges on oozie.* to 'oozie'@'localhost' identified by 'oozie' Query OK, 0 rows affected (0.03 sec)mysql> grant all privileges on oozie.* to 'oozie'@'%' identified by 'oozie'; Query OK, 0 rows affected (0.03 sec)mysql> exitbackup oozie-site.xmlmodify the following in oozie-site.xmlchange db info to mysql (note: use the proper password for the grants: pass=oozie). This is used for creating the database. We need to manually c

Apache Ranger delivers a comprehensive approach to security for a Hadoop cluster. It provides central security policy administration across the core enterprise security requirements of authorization, accounting and data protection.Apache Ranger already extends baseline features for coordinated enforcement across Hadoop workloads from batch, interactive SQL and real–time in Hadoop.In this tutorial, we cover using Apache Ranger for HDP 2.3 to secure your Hadoop environment. We will walkthrough the following topics: Support for Knox authorization and auditCommand line policies in HiveCommand line policies in HBaseREST APIs for policy managerPrerequisiteThe only prerequisite for this tutorial is that you have Hortonworks Sandbox.Once you have Hortonworks Sandbox, login through SSH:Starting Knox Service and Demo LDAP ServiceFrom the Ambari console at http://localhost:8080/ (username and password is admin and adminrespectively), select Knox from the list of Services on the left-hand side of