Community Articles

"Druid is fast column-oriented distributed data store".Druid is an open source data store designed for OLAP queries on event data.Architecture Historical nodes are the workhorses that handle storage and querying on "historical" data (non-realtime). Historical nodes download segments from deep storage, respond to the queries from broker nodes about these segments, and return results to the broker nodes. They announce themselves and the segments they are serving in Zookeeper, and also use Zookeeper to monitor for signals to load or drop new segments.Coordinator nodes monitor the grouping of historical nodes to ensure that data is available, replicated and in a generally "optimal" configuration. They do this by reading segment metadata information from metadata storage to determine what segments should be loaded in the cluster, using Zookeeper to determine what Historical nodes exist, and creating Zookeeper entries to tell Historical nodes to load and drop new segments.Broker nodes receiv

I follow the examples from 3 Pillar Global Post and Apache Hbase Blog Post and then updated for newer versions.To write an HBase Coprocessor you need Google Protocol Buffers. To use recent versions, on a Mac you need to install v2.50 as that works with HBase:brew tap homebrew/versionsbrew install protobuf250Check your version protoc --versionSource Code with Maven Build (pom.xml) is here. You will need Maven and Java 7 (or newer) JDK for compilation.Testing on Hadoopexport HADOOP_CLASSPATH=`hbase classpath`hadoop jar hbasecoprocessor-1.0.jar com.dataflowdeveloper.hbasecoprocessor.SumEndPointUpload your Jar to HDFShadoop fs -mkdir /user/tspann hadoop fs -ls /user/tspann hadoop fs -put hbasecoprocessor-1.0.jar /user/tspann hadoop fs -chmod 777 /user/tspann/hbasecoprocessor-1.0.jarInstall Dynamicallydisable 'stocks'alter 'stocks','coprocessor'=>'hdfs://sandbox.hortonworks.com/user/tspann/hbasecoprocessor-1.0.jar|com.dataflowdeveloper.hbasecoprocessor.SumEndPoint|1001|arg1=1'enable

I have put together few videos for general community help for people who wish to deploy HDP 2.3 on Amazon EC2. Hope this helps.HDP 2.3 Installation on RHEL 7 with Amazon EC2https://www.youtube.com/watch?v=6-RY4Ll6ABU&list=P...Hue 3.9 Installation on the same Clusterhttps://www.youtube.com/watch?v=UZoKXSsz5cw&list=P...

AbstractThis article will focus on creating a custom HIVE UDF in the Scala programming language. Intellij IDEA 2016 was used to create the project and artifacts. Creation and testing of the UDF was performed on the Hortonworks Sandbox 2.4 using Oracle Virtual Box. The full source code for the project can be found here.Create project.Using Intellij IDEA, create a new project with the following configuration.Project type: ScalaProject name: shoehornProject SDK: 1.8 (java version 1.8)Scala SDK: scala-sdk-2.11.8Create packageOnce the project is created, add a new package under /shoehorn/src/ named: udfSet dependenciesEdit your project structure and add the hive-exec-1.2.1.jar file to the module dependencies.Create ArtifactEdit your project structure and add and artifact of type JAR>module with dependencies. For this example, I am adding the artifact to the shoehorn module.Create Scala classThe first thing we need to do is create a Scala class referencing the org.apache.hadoop.hive.ql.ex

Alluxio is available to install on HDP and works with your existing HDFS. http://www.alluxio.org/documentation/en/Configuring-Alluxio-with-HDFS.html http://www.alluxio.com/2016/04/getting-started-with-alluxio-and-spark/ To use on HDP, you need to edit the /opt/alluxio-1.0.1/conf/alluxio-env.sh after you create it and change the port from 9000 to 8020. You can also override at the command line: exportALLUXIO_UNDERFS_ADDRESS=hdfs://localhost:8020 To Setup Alluxio. First download it. bin/alluxio format[root@sandboxalluxio-1.0.1]# bin/alluxio formatConnecting tolocalhost as root...Warning: Permanentlyadded 'localhost' (RSA) to the list of known hosts.Formatting AlluxioWorker @ sandbox.hortonworks.comConnection tolocalhost closed.Formatting AlluxioMaster @ localhost[root@sandboxalluxio-1.0.1]# bin/alluxio-start.sh localKilled 0 processeson sandbox.hortonworks.comKilled 0 processeson sandbox.hortonworks.comConnecting tolocalhost as root...Killed 0 processeson sandbox.hortonworks.comCon

Security is a key element when discussing Big Data. A common requirement with security is data encryption. By following the instructions below, you'll be able to setup transparent data encryption in HDFS on defined directories otherwise known as encryption zones "EZ". Before starting this step-by-step tutorial, there are three HDP services that are essential (must be installed): 1) HDFS 2) Ranger 3) Ranger KMSStep 1: Prepare environmentAs explained in the HDFS "Data at Rest" Encryption manuala) If using Oracle JDK, verify JCE is installed (OpenJDK has JCE installed by default)If the server running Ranger KMS is using Oracle JDK, you must install JCE (necessary for Ranger KMS to run) instructions on installing JCE can be found hereb) CPU Support for AES-NI optimizationAES-NI optimization requires an extended CPU instruction set for AES hardware acceleration.There are several ways to check for this; for example:cat /proc/cpuinfo | grep aesLook for output with flags and 'aes'.c) Librar

PROBLEM: When opening Zeppelin Ambari View on Sandbox (Virtualbox version), the page is blank. ROOT CAUSE: sandbox.hortonworks.com is what the view tries to redirect the browser to, which is not found if not explicitly set. SOLUTION: On your machine, modify the host file so that 127.0.0.1 sandbox.hortonworks.com is an entry. An alternative is to simply go directly to the web page 127.0.0.1:9995. Operating Systems have different location for hosts file and hence suggest reviewing your operating system's documentation of how to modify the hosts file.

How to get a docker image up and running which encapulates a PyCharm IDE integrated with spark and pybuilder. The IDE reside on the docker container and will be display on your laptop/machine. This is to isolate your development enviorment with has spark integrated with spark.Why? I am a spark developer and spend significant time trying to build a integrated environment. I am spending way too much time on integration before doing what I get paid to do --- Develop! Creating a isolated environment which is integrated with spark and a CIT, easily spun up and down, and repeatable is something which would accelerate my efficiency. Download latest virtualbox from here.To run docker containers or build images a docker machine is required. Download docker machine from here.Download xQuartz to display the IDE on your laptop.View my docker page for information on the docker image here.Clone my PyCharm github repo. You are doing this bootstrap code sample code I have built to your docker co

When using a postgresql DB for Hue, you might have encoutered [root@hue ~]# cd /usr/lib/hue [root@hue hue]# source ./build/env/bin/activate (env)[root@hue hue]# hue syncdbTraceback (most recent call last): File "/usr/lib/hue/build/env/bin/hue", line 9, in <module> load_entry_point('desktop==2.6.1', 'console_scripts', 'hue')() File "/usr/lib/hue/desktop/core/src/desktop/manage_entry.py", line 60, in entry execute_manager(settings) File "/usr/lib/hue/build/env/lib/python2.6/site-packages/Django-1.2.3-py2.6.egg/django/core/management/__init__.py", line 438, in execute_manager utility.execute() File "/usr/lib/hue/build/env/lib/python2.6/site-packages/Django-1.2.3-py2.6.egg/django/core/management/__init__.py", line 379, in execute self.fetch_command(subcommand).run_from_argv(self.argv) File "/usr/lib/hue/build/env/lib/python2.6/site-packages/Django-1.2.3-py2.6.egg/django/core/management/__init__.py", line 261, in fetch_command klass = load_command_class(app_name, s

Before completing this tutorial, it is important to understand data lineage.   What is Data Lineage Data lineage is defined as a data life cycle that conveys data origin and where data moves over time. In Apache Hive, if I create a table (TableA) and then insert data (from another table TableB), the data lineage will display TableA as the target and Table B as the source/origin. These two tables are linked together by a process "insert into Table..", allowing a user to understand the data life cycle. In a Hadoop ecosystem, Apache Atlas contains the data lineage for various systems like Apache Hive, Apache Falcon and Apache Sqoop. What is Apache Atlas Apache Atlas is a centralized governance framework that supports the Hadoop ecosystem as a metastore repository. To add metadata to Atlas, libraries called ‘hooks’ are enabled in various systems which automatically capture metadata events in the respective systems and propagate those events to Atlas. (More on Atlas' Architecture). Pre

The Hortonworks Hive ODBC Driver for OSX 10.11 (El Capitan) fails to install."This package is incompatible with this version of OS X and may fail to install"This issue is being tracked on (with full installer log) here:https://hortonworks.jira.com/browse/BUG-47990Support/Engineering will be reaching out to Simba Tech for a new Hive ODBC Driver.

Big Data Wrangling on HDP with Trifacta - How to Get startedData Preparation is a constant challenge for any enterprise and the speed, diversity and volumes of data introduced by Big Data simply amplify this problem substantially. Trifacta with HDP helps introduce a new approach to organizing, cleansing, enriching and structuring your data, Data Wrangling, where business users are able to connect and engage with the data to drive out high quality data sets for analytics.Step 1: Download VMTrifacta would like to provide Hortonworks partners and SI's with an opportunity to test drive Data Wrangling on HDP. Here is a link to a pre-configured virtual machine containing Trifacta Enterprise and HDP 2.3. Feel free to download and try Wrangling today:ftp://download.trifacta.com/Hortonworks/Trifacta_3.0_HDP_2.3.2_sandbox.ovauser: hortonworkspw: wranglerStep 2: Start VM and access consolesThe Trifacta Enterprise Wrangler on HDP is built on HDP 2.3 and the demo/sandbox instance of Centos. To acce

This tutorial has been successfully tried on HDP-2.4.0.0 and Ambari 2.2.1.0.I have my HDP Cluster Kerberized and Ambari has been configured for SSL.Note - Steps are same for Ambari with or without SSL..Please follow below steps for Configuring Pig View on Kerberized HDP Cluster..Step 1 - Please configure your Ambari Server for Kerberos with the steps mentioned in below article. Please follow steps 1 to 5.https://community.hortonworks.com/articles/40635/configure-tez-view-for-kerberized-hdp-cluster.html.Step 2 - Please add below properties to core-site.xml via Ambari UI and restart required services..Note - If you are running Ambari Server as root user then add below propertieshadoop.proxyuser.root.groups=*hadoop.proxyuser.root.hosts=*.If you are running Ambari server as non-root user then please add below properties in core-site.xmlhadoop.proxyuser.<ambari-server-user>.groups=*hadoop.proxyuser.<ambari-server-user>.hosts=*Please replace <ambari-server-user> with user r

How to make Mysql Database as Hive’s instance:Install Mysql if not available:brew updatebrew doctorbrew upgradebrew install mysqlmysql.server restartmysql_secure_installationlogin to mysql -> mysql –u root –pEnter password:Happy Mysql learning….Mysql is already installed on Hortonworks sandbox. Steps:Confirm with mysql –u root –pImport an already available database into Mysql:Ref: https://dev.mysql.com/doc/employee/en/employees-installation.htmlshell> tar -xjf $HOME/Downloads/employees_db-full-1.0.6.tar.bz2 shell> cd employees_db/shell> mysql -t < employees.sqlWith this installation of employee db in mysql is complete.Configuration of Mysql Instance with Hive:From HIVE create Mysql metastore[root@host]# mysqladmin -u root create hivedbmysql> USE hivedb;mysql> CREATE USER 'hive'@'localhost' IDENTIFIED BY 'hive';mysql> GRANT ALL PRIVILEGES ON *.* TO 'hive'@'localhost';With this we confirm that mysql database is the Hive’s new metastore.Suppose to perform a full im

This article was inspired by How to configure knox with existing ssl certificate and a client engagement to give more clarity into the actual commands that you need run to setup Apache Knox for SSL and LDAP with existing certificates.Pre-requisitesYou need to get procure the following items from your security department before you begin:Your LDAP or AD Digital Certificate: <ldap>.crtThe company's Digital CA Cert: <company_ca>.crtThe certificate/ key pair for the gateway node: <gateway_node>.crt and <gateway_node>.pemThe passphrase for the above gateway node key.Your Knox Master Secret--------------------------------------------------------------------What if?You don't know the Knox Master Secret? Then you can change it as follows cd $gatewaybin/knoxcli.cmd create-master --force-- Delete Keystores and restart Knox> rm data/security/keystores/gateway.jks> rm data/security/keystores__gateway-credentials.jceks You don't have a signed cert from a trusted CA

Documentation referred: https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.2/bk_installing_manually_book/content/configure-ranger_policy_admin_ha.htmlEnvironment Information: [ hivenv ] >> Cluster namehivenv-ambari-server.hwxblr.com >> 10.0.1.26 >> Ambari Server hn1.hwxblr.com >> 10.0.1.21 >> Existing Ranger Admin hn3.hwxblr.com >> 10.0.1.23 >> Load Balancer hn2.hwxblr.com >> 10.0.1.25 >> Additional Ranger Admin # hadoop version Hadoop 2.7.1.2.3.2.0-2950 Subversion git@github.com:hortonworks/hadoop.git -r5cc60e0003e33aa98205f18bccaeaf36cb193cc Compiled by jenkins on 2015-09-30T18:08Z Compiled with protoc 2.5.0 From source with checksum 69a3bf8c667267c2c252a54fbbf23d This command was run using /usr/hdp/2.3.2.0-2950/hadoop/lib/hadoop-common-2.7.1.2.3.2.0-2950.jar # uname -a Linux hn1.hwxblr.com 3.10.0-327.13.1.el7.x86_64 #1 SMP Thu Mar 31 16:04:38 UTC 2016x86_64 x86_64 x86_64 GNU/Linux # cat /etc/redhat-releas

One of the most common questions I come across when trying to help debug MapReduce jobs is: "How do I change the Log4j level for my job?" Many times, a user has a JAR with a class that implements Tool that they invoke using the hadoop jar command. The desire is to change the log level without changing any code or global configuration files:hadoop jar MyApplication.jar com.myorg.application.ApplicationJob <args ...>There is an extremely large amount of misinformation because how to do this has drastically changed from the 0.20.x and 1.x Apache Hadoop days. Most posts will inform you of some solution involving environment variables or passing Java opts to the mappers/reducers. In practice, there is actually a very straightforward solution.To change the Mapper Log4j level, set mapreduce.map.log.level. To change the Reducer Log4j level, set mapreduce.reduce.log.level. If for some reason you need to change the Log4j level on the MapReduce ApplicationMaster (e.g. to debug InputSplit

Sqoop can be used to bring data from RDBMS, but a limitation of sqoop is that data in HDFS is stored in one folder. If a partitioned table needs to be created in Hive for further queries, users need to create Hive script to distribute data to appropriate partitions. There is no direct option of creating partition tables based in Hive directly from sqoop.However, we can use sqoop features of putting output in a specific directory to simulate a partitioned table structure in HDFS. Since any partitioned table has a HDFS structure where each partition is <table name>/<partition column name=value> , we can use following sqoop structure to select appropriate data for each partition and move it to correct HDFS structure. sqoop --table <table1> --where <where clause for pt=0> --target-dir /home/user1/table1/pt=0 sqoop --table <table1> --where <where clause for pt=1> --target-dir /home/user1/table1/pt=1Now, an external HIVE table can be created that is poin

SQOOP CONNECTIONS:Sqoop is a tool designed to transfer data between Hadoop and relational databases. You can use Sqoop to import data from a relational database management system (RDBMS) such as MySQL or Oracle into the Hadoop Distributed File System (HDFS), transform the data in Hadoop MapReduce, and then export the data back into an RDBMS.Reference: sqoop user guide:https://sqoop.apache.org/docs/1.4.0-incubating/SqoopUserGuide.htmlJDBC ORACLE: Examples for Import:sqoop-import --connect jdbc:oracle:thin:@db.test.com:PORT:INSTANCE_NAME --table DW_DATAMART.HCM_EMPLOYEE_D --fields-terminated-by '\t' --lines-terminated-by '\n' --username SSANKU -Psqoop-import --connect jdbc:oracle:thin:@db.test.com:PORT:INSTANCE_NAME --table DW_DATAMART.HCM_EMPLOYEE_D --fields-terminated-by '\t' --lines-terminated-by '\n' --username SSANKU -PJDBC ORACLE: Example for Select:The eval tool allows users to quickly run simple SQL queries against a database; results are printed to the console. This allows user

Goals Get familiar with psql command line CLI interface for HAWQ Understanding how to access and list the options available for command line interface Accessing help from within the CLI NOTES Requirement for above is to ensure that there is either a sandbox configured with HAWQ A SSH interface to the sandbox either via HOST/remote OSX Or Putty when working with Windows Connecting to the Guest Sandbox Machine Depending upon the setup i.e., VirtualBox or VMWare Fusion for HAWQ & HDP, you can choose the connection method. For connecting via VMWare Fusion, you need to be aware of the IP address allocated to your virtual box. Connecting via VMWare Fusion HW13382:ODBC srai$ ssh root@172.16.105.137 -p 22root@172.16.105.137's password: Last login: Mon Jun 20 15:31:20 2016 from 172.16.105.1[root@sandbox ~]# Connecting via Virtualbox HW13382:ODBC srai$ ssh root@localhost -p 2222root@172.16.105.137's password: Last login: Mon Jun 20 15:31:29 2016 from 172.16.105.1[root@sandbox ~]#

Often times, maintaining a hadoop cluster for longer periods on cloud is an expensive task. Sometimes engineers might also encounter situations of not having immediate access to cloud environment, in order to quickly spin up their own cluster and play arround. As an easy alternative, vagrant with virtual box as a provider, HDP cluster can be set up on your own laptop.Step 1 - Install prerequisites Download and install from here Vagrant. Download and install Oracle VirtualBox as the Vagrant Provider.Step 2 - Generation of Vagrantfile Create a working directory for Vagrant file generation and initiating the deployment vis vagrant$ mkdir hdp22$ cd hdp22Following command will generate the Vagrantfile in current directory. This file will define VMs that are to be on the cluster.$ vagrant init $ vi VagrantfileStep 3 - Configuration of VMs in VagrantfileLets configure Vagrant to use CentOS 6.6/ CentOS 6.7 as the base box$ config.vm.box = "bento/centos-6.7" or $ config.vm.box = "chef/cento

WebHCat# this will execute a hive query and save result to hdfs file in your home directory called outputcurl -s -d execute="select+*+from+sample_08;" \ -d statusdir="output" \ 'http://localhost:50111/templeton/v1/hive?user.name=root'# if you ls on the directory, it will have two files, stderr and stdouthdfs dfs -ls output# if the job succeeded, you can cat the stdout file and view the resultshdfs dfs -cat output/stdout WebHDFS# list the output directory, notice the webhdfs portcurl -i "http://sandbox.hortonworks.com:50070/webhdfs/v1/user/root/output/?op=LISTSTATUS"# read the output filecurl -i -L "http://sandbox.hortonworks.com:50070/webhdfs/v1/user/root/output/stdout?op=OPEN"# rename a file, if you get dr. who error, add &user.name=root or any other user in the contextcurl -i -X PUT "sandbox.hortonworks.com:50070/webhdfs/v1/user/root/output/stdout?op=RENAME&user.name=root&destination=/user/root/newname"# read the output of the new filecurl -i -L "http://sandbox.horto

Choosing an approach for Kerberos implementation on Hadoop cluster is critical from a long term maintenance point. Enterprises have their own security policies and guidelines and a successful kerberos implementation needs to adhere to enterprise security architecture. There are multiple guides available on how to implement Kerberos but I couldn't find information on which approach to choose and Pros and Cons associated with each approach.In a Hortonworks Hadoop cluster, there are 3 different ways of generating keytabs and principals and managing them.a. Use an MIT KDC specific to Hadoop cluster - automated keytab management using AmbariKDC specific to Hadoop cluster can be installed and maintained on one of the Hadoop nodes. All users/keytabs required for kerberos implementation are automatically managed using Ambari.Pros:Enterprise security teams not involved with KDC setup. Hadoop administrators have complete control of KDC installation.Automated keytab management using Ambari. No ne

In this case, after we set up Hive and other MySQL-dependent services, we decided to hand MySQL responsibilities over to the DBA team so they could carry out specialized management including configuring an HA set-up. We therefore no longer wanted Ambari to run the MySQL server.The MySQL server was shut down and uninstalled from the host outside of Ambari, before we could do this cleanly via Ambari. Because of this, we had to carry out some extra steps to delete the component from Ambari. The following is mostly based on this HCC article by @Kuldeep Kulkarni. The version of Ambari in this case is 2.2.2.1.Deletion of the component from the host appears to have worked. localhost:~ nl$ curl -u admin:admin -X DELETE -H 'X-Requested-By:admin' http://localhost:8080/api/v1/clusters/mycluster/hosts/host005.hadoop.example.com/host_components/MYSQL_SERVERlocalhost:~ nl$ However, MySQL being stopped and uninstalled outside of Ambari appears to have caused an inconsistency, preventing the second AP

In order to distcp between two HDFS HA cluster (for example A and B), using nameservice id or to setup falcon clusters having namenode ha, these settings are needed. Assuming nameservice for cluster A and B is HAA and HAB respectively. One need to set following properties in hdfs-site.xml Add value of the nameservices of both clusters in dfs.nameservices. This needs to be done in both the clusters.dfs.nameservices=HAA,HABAdd property dfs.internal.nameservices In cluster A: dfs.internal.nameservices = HAA In cluster B: dfs.internal.nameservices = HAB Add dfs.ha.namenodes.<nameservice>.dfs.ha.namenodes.HAB=nn1,nn2dfs.ha.namenodes.HAA=nn1,nn2Add property dfs.namenode.rpc-address.<nameservice>.<nn>. dfs.namenode.rpc-address.HAB.nn1 = <NN1_fqdn>:8020dfs.namenode.rpc-address.HAB.nn2 = <NN2_fqdn>:8020dfs.namenode.rpc-address.HAA.nn1 = <NN1_fqdn>:8020 dfs.namenode.rpc-address.HAA.nn2 = <NN2_fqdn>:8020 Add property dfs.client.failover.proxy.provid