Resources

This article will shield light on the HP Big Data Reference Architecture. Hadoop Architectures have been traditionally build to move the math to the data. This has been the basis for Hadoop architectures since the initial implementations back at Yahoo. Let's take a look at traditional verse HP's Big Data Architecture approach.Current traditional Big Data approach• Compute and storage are always collocated • All servers are identical • Data is partitioned across servers on direct-attached storage (DAS)tradhadoop.pngNew HP Big Data approach• Separate compute and storage tiers connected by Ethernet networking • Standard Hadoop installed asymmetrically with storage components on the storage servers and yarn applications on the compute servershpbdra.pngWith this new approach, HP engineers have challenged the conventional wisdom that compute should always be co-located with data.HP has been aggressively benchmarking with this new archiecture, and feel they have designed an architecture th

Use OpenTSDB Ambari service to store/visualize stock data on HDP sandbox Goal: OpenTSDB (Scalable Time Series DB) allows you to store and serve massive amounts of time series data without losing granularity (more details here). In this tutorial we will install it on Hbase on HDP sandbox using the Ambari sevice and use it to import and visualize stock data.Steps:Setup VM and install Ambari service Download HDP latest sandbox VM image (.ova file) from Hortonworks websiteImport ova file into VMWare and ensure the VM memory size is set to at least 8GBNow start the VMAfter it boots up, find the IP address of the VM and add an entry into your machines hosts file e.g.192.168.191.241 sandbox.hortonworks.com sandbox Connect to the VM via SSH (password hadoop)ssh root@sandbox.hortonworks.comStart HBase service from Ambari and ensure Hbase is up and root has authority to create tables. You can do this by trying to create a test tablehbase shellcreate 't1', 'f1', 'f2', 'f3'If this fails with th

Step 1 - Goto ambari UI and add/modify below propertiesGoto service hive → configs and change autherization to SQLStdAuthStep 2 - In Hive-site.xml, make sure you have set below properties:hive.server2.enable.doAs --> falsehive.users.in.admin.role --> root (comma separated list of users)Step 3 - Make sure that you have below properties set in Hiveserver2-site.xmlhive.security.authorization.manager --> org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdConfOnlyAuthorizerFactoryhive.security.authorization.enabled --> truehive.security.authenticator.manager --> org.apache.hadoop.hive.ql.security.ProxyUserAuthenticatorStep 4 - Restart hive services from UIStep 5 - Testing our setup, in this we will create one readonly user and try to drop table. 5.1 Login to beeline using root ( as we have added root in hive.users.in.admin.role ) 0: jdbc:hive2://localhost:10010> !connect jdbc:hive2://localhost:10010Connecting to jdbc:hive2://localhost:10010Enter username

Note: Credit for the key piece of information to solve this problem goes to Phil D’Amore. A customer had a problem writing a Java application that used the Hive client libraries to talk to two secure Hadoop clusters that resided in different Kerberos realms. The same problem could be encountered by a client connecting to a single secure Hadoop cluster that happened not to be in the Kerberos “default_realm” as specified in the client host’s krb5.conf file. The same problem could occur for any Hadoop ecosystem client, not just Hive clients.In order to communicate with two different secure Hadoop clusters, in different Kerberos realms, the client application did the following things correctly:It harvested the needed configuration files (in this case, core-site.xml, hdfs-site.xml, and hive-site.xml) from each target cluster, and used the appropriate configuration when communicating with each respective cluster.Its application user id had two Kerberos principals, one registered and authent

The MergeContent processor certainly can be challenging to understand its inner workings. If you are running into the nifi.queue.swap.threshold limit of MergeContent as described in NIFI-697, then you should increase that value in the nifi.properties file and restart your NiFi process. A multiple of 10000 is recommended. You will also likely have to increase your Java memory settings in bootstrap.conf.MergeContent works like this.When a FlowFile arrives at MergeContent, it is assigned to a bin based on Merge Strategy and Correlation Attribute Name. Maximum Number of Bins controls resource usage such that if all bins have FlowFiles in them and another FlowFile arrives that doesn't fit into one of those bins, then the oldest bin is automatically marked as complete, and the new FlowFile starts its own new bin.A bin will be complete once (number of files in bin) >= Minimum Number of Entries AND (number of bytes in bin) >= Minimum Group Size OR the bin has existed for Max Bin Age.

IntroductionApache Zeppelin (Incubator at the time of writing this post) is one of my favourite tools that I try to position and present to anyone interested in Analytics, Its 100% open source with an intelligent international team behind it in Korea (NFLABS) (Moving to San Francisco soon), its mainly based on interpreter concept that allows any language/data-processing-backend to be plugged into Apache Zeppelin.Very similar to IPython/Jupyter except that the UI is probably more appealing and the amount of interpreters supported are richer, at the time of writing this Blog Zeppelin supported: Apache Hive QLApacheSpark (SQL, Scala and Python)ApacheFlinkPostgresPivotal HAWQShellApache TajoAngularJSApache CassandraApache igniteApache PhoenixApache GeodeApache KylinApache Lenswith this rich set of interpreters provided, it makes on boarding platforms like Apache Hadoop or Data Lake concepts much easier where data is sitting and consolidated somewhere and different organizational units wit

HDP deployment in Azure and configuring wasb in ambariLessons learned: All VMs needs to be under virtual private network and hostnames needs to be changed.1) Create virtual network for your cluster Note: screen shot has name hdptest01 but it's test2) Create VMs and choose network created in the above stepThis is very important step and we need to pay attention to the hostname. Added endpoint for 8080.There is a step to add disks to vm. You can follow this blog At this point all the hosts are created and we will make changes to hostnames.for example: the following needs to be modified. The command hostname should show internal hostname or same output as hostname -f so modify /etc/sysconfig/network and reboot all the nodes or change hostname using hostname command.[root@hdpmaster01 ~]# cat /etc/sysconfig/networkHOSTNAME=hdpmaster01.hdpmaster01.j3.internal.cloudapp.netNETWORKING=yes#hostnamehdpmaster01.hdpmaster01.j3.internal.cloudapp.netFollow HDP Docs to install cluster using AmbariUse

Initially I had installed hdp2.1 with ambari 1.7, then I upgraded ambari to 2.1.2 and upgraded hdp stack to 2.2.0.0 as per this documentation. As per below note I had to delete STORM_UI_SERVER component:"In HDP 2.2, STORM_REST_API component was deleted because the service was moved into STORM_UI_SERVER. When upgrading from HDP 2.1 to 2.2, you must delete this component using the API as follows"I tried deleting it using ambari REST APIs: First stop the component using below command: curl -u admin:admin -X PUT -H 'X-Requested-By:1' -d '{"RequestInfo":{"context":"Stop Component"},"Body":{"HostRoles":{"state":"INSTALLED"}}}' http://hdpambari.hortonworks.com:8080/api/v1/clusters/c1/hosts/hdpambari.hortonworks.com/host_components/STORM_REST_API Then Delete using below curl call.curl -u admin:admin -X DELETE -H 'X-Requested-By:1' http://hdpambari.hortonworks.com:8080/api/v1/clusters/hdpambari/services/STORM/components/STORM_REST_API<br>;It did not work because stop command did not com

Authorization Models applicable to the Hive CLI Hive provides a few different authorization models plus Apache Ranger, as described in the Hive Authorization section of the HDP System Administration Guide. Hive CLI is subject to the following two models--Hive default (Insecure) - Any user can run GRANT statements - DO NOT USEStorage-based (Secure) - Authorization at the level of databases/tables/partitions, based on HDFS permissions (and ACLs in HDP 2.2.0+) Frequently Asked Questions about Hive CLI Security Can I set restrictive permissions on the hive executable (shell wrapper script) and hive-cli jar?No, components such as Sqoop and Oozie may fail. Additionally, a user can run their own copy of the hive client from anywhere they can set execution privileges. To avoid this limitation, migrate to the Beeline CLI and utilize HiveServer2, and restrict access to the cluster through a gateway such as Knox.Can Ranger be used to enforce permissions for Hive CLI users?HDFS policies can be cre

Hi All,   i want to get the Cloudera Licence for my cluster inorder to upgrade my server. how can i get the latest licence.I am not able to find any contact details in the cloudera site. Please help me getting the contact perso from where  can get the licence. thanks.

container-executor.cfgYARN containers in a secure cluster use the operating system facilities to offer execution isolation for containers. Secure containers execute under the credentials of the job user. The operating system enforces access restriction for the container. The container must run as the user that submitted the application. Therefore it is recommended to never submit jobs from a superuser account (HDFS or Linux) when LinuxContainerExecutor is used.To prevent superusers from submitting jobs, the container executor configuration (/etc/hadoop/conf/container-executor.cfg) includes the properties banned.users and min.user.id. Attempting to submit a job that violates either of these settings will result in an error indicating the AM container failed to launch: INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: Application application_1234567890123_4567 failed 2 times due to AM Container for appattempt_1234567890123_4567_000002 exited with exitCode: -1000Followed

One of the search use cases that I’ve been introduced to would require the ability to index text such as scanned text in png files. I set out to figure out how to do this with SOLR. I came across a couple pretty good blog posts, but as usual, you have to put together what you learn from multiple sources before you can get things to work correctly (or at least that’s what usually happens for me). So I thought I would put together the steps I took to get it to work.I used HDP Sandbox 2.3.Step-by-step guideInstall dependencies - this will provide you support for processing pngs, jpegs, and tiffs yum install autoconf automake libtoolyum install libpng-develyum install libjpeg-develyum install libtiff-develyum install zlib-develDownload Leptonica, an image processing librarywget http://www.leptonica.org/source/leptonica-1.69.tar.gzDownload Tesseract, an Optical Character Recognition engine wget http://tesseract-ocr.googlecode.com/files/tesseract-ocr-3.02.02.tar.gzEnsure proper variables and

Recently I met a HBase connection problem from Storm. The error message is:Error message: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.ipc.controller.ServerRpcControllerFactoryIt is because when we turn on Phoenix in HBase, HBase will add some additional properties (e.g. org.apache.hadoop.hbase.ipc.controller.ServerRpcControllerFactory) related to Phoenix in the hbase-site.xml. Users suppose to add these new dependencies in their pom.xml file, otherwise the hbase connection will fail, even if user does not use Phoenix at all. In order to avoid this "class not found error", the easiest way is to make sure you turn Phoenix off when connecting HBase from Storm. This solution is still valid in HDP 2.5.

Configuring a separate service RPC port can improve the responsiveness of the NameNode by allowing DataNode and client requests to be processed via separate RPC queues.Adding a service RPC port to an HA cluster with automatic failover via ZKFCs requires a workaround step.The steps to configure a service RPC port are as follows:Add the following settings to hdfs-site.xml. <property> <name>dfs.namenode.servicerpc-address.mycluster.nn1</name> <value>nn1.example.com:8040</value> </property> <property> <name>dfs.namenode.servicerpc-address.mycluster.nn2</name> <value>nn2.example.com:8040</value> </property> 2. Restart NameNodes. 3. Stop the ZKFC processes on both NameNodes 4. Run the following command to reset the ZKFC state in ZooKeeperhdfs zkfc -formatZK Without this step you will see the following exception after ZKFC restart.java.lang.RuntimeException: Mismatched address stored in ZK for Name

I had setup Transparent data encryption some time back for HDP2.2 by referring steps mentioned at http://hortonworks.com/kb/hdfs-transparent-data-encryption/I tested my setup, everything worked perfectly except one thing, superuser was able to access the contents from encrypted zone transparently which I think should not work that way, so to restrict your encrypted zone you need to modify below property in /usr/kms-demo/hadoop/etc/hadoop/kms-acls.xml<property><name>hadoop.kms.acl.DECRYPT_EEK</name><value>user1,user2</value><description> ACL for decryptEncryptedKey CryptoExtension operations.</description></property>Add list of users for whom you want to give access to your encrypted zone.Hope this is useful 🙂

DISCLAIMER: it was tested on HDP 2.3.2 only. There are two blocking JIRAs preventing usage of blob storage as primary filesystem on HDP 2.3.0. For HBase, you need to use page blob instead of block blob.First things first, install Azure CLI for Mac or use Azure portal. The steps below are for CLI.azure loginenter usernameenter passwordazure storage account create storageaccountname --type LRSazure storage account keys list storageaccountnamenote the account keys, you will need them in the next stepazure storage container create storagecontainername --account-name storageaccountname --account-key accountkeystringjust to validate it was createdazure storage blob list storagecontainernae --account-name storageaccountname --account-key Once the previous steps have been completed, go to Ambari UI and edit the core-site.xmlIn addition to these properties, you need to replace fs.defaultFS property with the wasb path.These properties and their descriptions are discussed in hadoop-azure document

Once one moves beyond trivial flow design a visual image like the one below becomes common. Surely, it becomes more and more messy over time and harder to glance over.Here's a great tip - double-click on the connection line that you want to bend:A new yellow anchor will appear which one can then drag around to organize things nicely.Bonus tip - one can add multiple bend points to a connection.To remove a specific anchor, simply double-click on it again, repeat for each yellow point.And an extra bonus courtesy of @mgilman@hortonworks.com: a connection label can be moved along the connection and snaps to a bend point, see below:

Consider the following flow where records are inserted in the RDBMS (or Phoenix/HBase in this instance):The PutSQL and similar processor depends on a database connection, provided by a DBCPConnectionPool controller service (Controller Settings -> Controller Services):The best part: when one creates a template, all linked services will automatically be included. When template is rehydrated in another NiFi instance, services like the one below will automatically be added to the new NiFi flow:

You have generated a certificate file:openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout test.key -out test.pem After deploying the key, you try to ssh into the instance but get prompted for a password. ssh -vvv -i test.pem <user>@<host>—————This is an issue with an updated openssl version,> openssl versionOpenSSL 1.0.1k 8 Jan 2015This is a new version of openssl. The new version does not create the key with RSA at the begin and end. So you have to use a separate command to convert the key file to old version of sshopenssl rsa -in test.key -out test_new.keyOnce that is done, use the new file for ssh. ssh -vv -i test_new.key <user>@<host>

One of the most interesting facts about Oracle is that many of their products are agnostic when it comes with working with Hadoop distributions. Hortonworks and Oracle have been working closely for over two years on product certifications and building a strong partnership from both the PM and Engineering organizations. Below are four key products in the Oracle portfolio that are certified and supported with Hortonworks. Oracle Data Integration (Oracle Data Integrator, Oracle GoldenGate) Oracle Data Integrator is certified on HDP 2.1 and 2.2 Oracle GoldenGate Big Data Edition is certified with HDP 2.2Both are currently working on HDP 2.3 certification (as of Oct '15)Blogs and helpful articles Oracle Big Data Integration Certified on HDP 2.2 Oracle Big Data Integration with Hortonworks SeriesOracle Big Data Connector (BDC) Certified for HDP 2.2, less R Analytics. Working with the BDC PM/Dev team to complete certification Blogs and helpful articles Oracle Big Data Connector Certif

Please find below a sample code which write to a kerberised HBase using keytab. package hbase; import java.io.IOException;import java.util.ArrayList;import java.util.List; import org.apache.hadoop.conf.Configuration;import org.apache.hadoop.hbase.HBaseConfiguration;import org.apache.hadoop.hbase.HColumnDescriptor;import org.apache.hadoop.hbase.HTableDescriptor;import org.apache.hadoop.hbase.KeyValue;import org.apache.hadoop.hbase.MasterNotRunningException;import org.apache.hadoop.hbase.ZooKeeperConnectionException;import org.apache.hadoop.hbase.client.Delete;import org.apache.hadoop.hbase.client.Get;import org.apache.hadoop.hbase.client.HBaseAdmin;import org.apache.hadoop.hbase.client.HTable;import org.apache.hadoop.hbase.client.Result;import org.apache.hadoop.hbase.client.ResultScanner;import org.apache.hadoop.hbase.client.Scan;import org.apache.hadoop.hbase.client.Put;import org.apache.hadoop.hbase.util.Bytes;import org.apache.hadoop.security.UserGroupInformation; public class HBas

When I try to download some of the protected stuff, such as Cloudera Express or Quickstart VM, I have to submit some form. The thing is form submission is broken due to the absence of some geoplugin. More details on the attached image (I erased unnecessary info). Is this a bug or I'm doing something wrong?

There have been a number of questions recently on using AD/IPA with HDP 2.3 security:How to setup Active Directory/IPA?How to setup cluster OS to recognize users from AD using SSSD?How to enable kerberos for authentication?How to install Ranger for authorization/audit and setup plugins for HDFS, Hive, HBase, Kafka, Storm, Yarn, Knox and test these components on kerborized cluster?How to sync Ranger user/group sync with AD/IPA?How to integrate Knox with AD/IPA?How to setup encryption at rest with Ranger KMS?To help answer some of these questions, the partner team have prepared cheatsheets on security workshops. These are living materials with sample code snippets which are being updated/enhanced per the feedback from the field so rather than replicate the materials here, the latest materials can be referenced at the GitHub repo linked from here:https://community.hortonworks.com/repos/4465/workshops-on-how-to-setup-security-on-hadoop-using.htmlTo help get started with security, we have a

I attended a panel discussion at Hadoop World last week (September 2015) and I’d like to share my notes for others. The topics of interest were: the future of data governance, who are the actors in data governance, and what are the tools to use. Full details of the talk are here: http://strataconf.com/big-data-conference-ny-2015/public/schedule/detail/43493One early question was on what the future holds for data governance in Hadoop. An interesting answer was that collecting data in the future will require greater consent from individuals. This echoed the morning’s keynote speaker who urged the same practice. The theme of that keynote was that too much data is being collected right now, mostly without the knowledge of the individual, and eventually there will be negative consequences for all. Back on the panel, another answer was that data governance of the future will help data analysts make sense of the data inside of their data lake. For example, audit logs will be automatica

In some situations, Ranger is not an option but a Hive Authorization scheme is advisable. It is possible to use SQLStdAuth for this. However it comes with a couple caveats.Hive Wiki1. Configuring SQLStdAuth ( HDP 2.3 ) - In Ambari, select Authorization SQLStdAuth. This will set all related configuration parameters like hive.enable.authorization and disable doAs.- Add admin users to admin role. If you have a user you want to be admin add him to hive.users.in.admin.role=hive,hue,myadmin2. Prepare HDFSSince all queries now run as the hive user he needs to have read/write rights on all files in HDFS. This includes the load directories for external tables. Ideally change the owner to the warehousing folder to hive and set access right 700. I also added hive to an ETL group and made all load folders read AND writable to this group. 3. Create rolesIn Hive as an admin user- Become Admin: SET ROLE ADMIN;- Create roles:CREATE ROLE BI; ( should have read rights to all tables ) CREATE ROLE ETL; (