Resources

Disaster recovery plan or a business process contingency plan is a set of well-defined process or procedures that needs to be executed so that the effects of a disaster will be minimized and the organization will be able to either maintain or quickly resume mission-critical operations.Disaster usually comes in several forms and need to planned for recovery accordingly:Catastrophic failure at the data center level, requiring failover to a backup locationNeeding to restore a previous copy of your data due to user error or accidental deletionThe ability to restore a point-in-time copy of your data for auditing purposesDisclaimer: 1. This article is solely my personal take on disaster recovery in a Hadoop cluster 2. Disaster Recovery is specialized subject in itself. Do not Implement something based on this article in production until you have a good understanding on what you are implementing.Key objectives: Minimal or no downtime for production cluster Ensure High Availability

There're already several articles describing how to setup NiFi to connect Kafka or HDFS. However, following scattered piece of documentations to complete Kerberizing HDP, and setting ACL of Zookeeper, Kafka, HDFS, and Kerberos was not an easy task for me. So, the motivation of this article is to share rough but throughout operations needed to set it up right. You can elaborate each step by digging related documentation further. Example configuration and NiFi template are available, too. Some of these steps may not be required, and some were forgotten to be here, but these were the steps to Kerberize my HDP sandbox VM: Install by downloading the latest HDF Start Kafka, stop maintenance mode. Restart all affected services Install a new MIT KDC [1] KDC Enable Kerberos from Ambari Proceed with Ambari Kerberos wizard Check Pig failed to pass the test, but continued with Complete button anyway Start services manually that didn't start automatically Configured Kafka for Kerberos

Chronos is a replacement for cron. A fault tolerant job scheduler for Mesos which handles dependencies and ISO8601 based schedules Marathon is a framework for Mesos that is designed to launch long-running applications, and, in Mesosphere, serves as a replacement for a traditional system In Mesosphere, Chronos compliments Marathon as it provides another way to run applications, according to a schedule or other conditions, such as the completion of another job. It is also capable of scheduling jobs on multiple Mesos slave nodes, and provides statistics about job failures and successes. Source Install https://mesos.github.io/chronos/docs/ and gist Demo Part 1 - https://www.linkedin.com/pulse/data-center-operating-system-dcos-part-1-neeraj-sabharwalPart 2 - https://www.linkedin.com/pulse/apache-marathon-part-2-neeraj-sabharwal

You need Mesos to run this - Post 1 What is Apache Marathon? Marathon is a production-grade container orchestration platform for Mesosphere'sDatacenter Operating System (DCOS) and Apache Mesos. I am launching multiple applications using Marathon and Mesos is providing the framework to launch those applications. Demo More readinghttps://mesosphere.github.io/marathon/Gist & Application example

DC/OS - a new kind of operating system that spans all of the servers in a physical or cloud-based datacenter, and runs on top of any Linux distribution. Source Projects More details https://docs.mesosphere.com/overview/components/ Let's cover Mesos in this post Frameworks (Application running on mesos) http://mesos.apache.org/documentation/latest/frameworks/ I used http://mesos.apache.org/gettingstarted/ to install Mesos in my local machine. I am launching c++, java and python framework in this demo. Mesos demo More reading

This tutorial is great:https://github.com/hortonworks-gallery/ambari-vnc-serviceEclipse Pluginhttps://github.com/winghc/hadoop2x-eclipse-pluginJDK 7 is best for most use case and Scala 2.10.Maven and SBT are necessary as well.Setup your Environmenthttps://dzone.com/articles/spark-and-scala-resourceshttps://dzone.com/articles/whats-on-your-laptopLots of options:This is an eclipse project for Hbase Coprocessorhttps://github.com/tspannhw/hbasecoprocessorArtem has a great project for testinghttps://github.com/dbist/HBaseUnitTestOnce all the ports are open and not firewalled it’s usually straight forward.Eclipse to Sparkhttps://community.hortonworks.com/questions/36354/eclipse-to-sandbox-1.htmlhttps://community.hortonworks.com/questions/32567/scala-with-hive-in-ecplipse-scala.htmlHadoop Eclipse Pluginhttps://community.hortonworks.com/questions/10404/hadoop-eclipse-plugin.htmlIntelliJ Project for Sparkhttps://github.com/agilemobiledev/sparkworkshophttps://community.hortonworks.com/questions/

Hive: Apache Hive is a data warehouse infrastructure built on top of Hadoop for providing data summarization, query, and analysis. HBase: Apache HBase™ is the Hadoop database, a distributed, scalable, big data store Hawq: http://hawq.incubator.apache.org/ PXF: PXF is an extensible framework that allows HAWQ to query external system data Let's learn Query federation This topic describes how to access Hive data using PXF. Link Previously, in order to query Hive tables using HAWQ and PXF, you needed to create an external table in PXF that described the target table's Hive metadata. Since HAWQ is now integrated with HCatalog, HAWQ can use metadata stored in HCatalog instead of external tables created for PXF. HCatalog is built on top of the Hive metastore and incorporates Hive's DDL. This provides several advantages: You do not need to know the table schema of your Hive tables You do not need to manually enter information about Hive table location or format If Hive table metadat

I found this article "For mobile analytics, Yahoo is in the process of replacing HBase with Druid" History : 24th Oct, 2012 To test out the setup, I have deployed druid in 2 clusters. first deployment is in my multi node cluster and 2nd deployment is using this repo. Details are on this blog Demo - PS: It's 10 minutes demo We are loading pageviews and then executing queries. See links at the bottom to download the git and code. Gif Download I use this to control gif movement Links: Page view queries and data Spin up the environment in your mac or windows "not sure about windows" Git link . This will spin up Druid, ZK, Hadoop, Postgres Gist Happy Hadooping!!!!

"Druid is fast column-oriented distributed data store".Druid is an open source data store designed for OLAP queries on event data.Architecture Historical nodes are the workhorses that handle storage and querying on "historical" data (non-realtime). Historical nodes download segments from deep storage, respond to the queries from broker nodes about these segments, and return results to the broker nodes. They announce themselves and the segments they are serving in Zookeeper, and also use Zookeeper to monitor for signals to load or drop new segments.Coordinator nodes monitor the grouping of historical nodes to ensure that data is available, replicated and in a generally "optimal" configuration. They do this by reading segment metadata information from metadata storage to determine what segments should be loaded in the cluster, using Zookeeper to determine what Historical nodes exist, and creating Zookeeper entries to tell Historical nodes to load and drop new segments.Broker nodes receiv

I follow the examples from 3 Pillar Global Post and Apache Hbase Blog Post and then updated for newer versions.To write an HBase Coprocessor you need Google Protocol Buffers. To use recent versions, on a Mac you need to install v2.50 as that works with HBase:brew tap homebrew/versionsbrew install protobuf250Check your version protoc --versionSource Code with Maven Build (pom.xml) is here. You will need Maven and Java 7 (or newer) JDK for compilation.Testing on Hadoopexport HADOOP_CLASSPATH=`hbase classpath`hadoop jar hbasecoprocessor-1.0.jar com.dataflowdeveloper.hbasecoprocessor.SumEndPointUpload your Jar to HDFShadoop fs -mkdir /user/tspann hadoop fs -ls /user/tspann hadoop fs -put hbasecoprocessor-1.0.jar /user/tspann hadoop fs -chmod 777 /user/tspann/hbasecoprocessor-1.0.jarInstall Dynamicallydisable 'stocks'alter 'stocks','coprocessor'=>'hdfs://sandbox.hortonworks.com/user/tspann/hbasecoprocessor-1.0.jar|com.dataflowdeveloper.hbasecoprocessor.SumEndPoint|1001|arg1=1'enable

I have put together few videos for general community help for people who wish to deploy HDP 2.3 on Amazon EC2. Hope this helps.HDP 2.3 Installation on RHEL 7 with Amazon EC2https://www.youtube.com/watch?v=6-RY4Ll6ABU&list=P...Hue 3.9 Installation on the same Clusterhttps://www.youtube.com/watch?v=UZoKXSsz5cw&list=P...

AbstractThis article will focus on creating a custom HIVE UDF in the Scala programming language. Intellij IDEA 2016 was used to create the project and artifacts. Creation and testing of the UDF was performed on the Hortonworks Sandbox 2.4 using Oracle Virtual Box. The full source code for the project can be found here.Create project.Using Intellij IDEA, create a new project with the following configuration.Project type: ScalaProject name: shoehornProject SDK: 1.8 (java version 1.8)Scala SDK: scala-sdk-2.11.8Create packageOnce the project is created, add a new package under /shoehorn/src/ named: udfSet dependenciesEdit your project structure and add the hive-exec-1.2.1.jar file to the module dependencies.Create ArtifactEdit your project structure and add and artifact of type JAR>module with dependencies. For this example, I am adding the artifact to the shoehorn module.Create Scala classThe first thing we need to do is create a Scala class referencing the org.apache.hadoop.hive.ql.ex

Alluxio is available to install on HDP and works with your existing HDFS. http://www.alluxio.org/documentation/en/Configuring-Alluxio-with-HDFS.html http://www.alluxio.com/2016/04/getting-started-with-alluxio-and-spark/ To use on HDP, you need to edit the /opt/alluxio-1.0.1/conf/alluxio-env.sh after you create it and change the port from 9000 to 8020. You can also override at the command line: exportALLUXIO_UNDERFS_ADDRESS=hdfs://localhost:8020 To Setup Alluxio. First download it. bin/alluxio format[root@sandboxalluxio-1.0.1]# bin/alluxio formatConnecting tolocalhost as root...Warning: Permanentlyadded 'localhost' (RSA) to the list of known hosts.Formatting AlluxioWorker @ sandbox.hortonworks.comConnection tolocalhost closed.Formatting AlluxioMaster @ localhost[root@sandboxalluxio-1.0.1]# bin/alluxio-start.sh localKilled 0 processeson sandbox.hortonworks.comKilled 0 processeson sandbox.hortonworks.comConnecting tolocalhost as root...Killed 0 processeson sandbox.hortonworks.comCon

Security is a key element when discussing Big Data. A common requirement with security is data encryption. By following the instructions below, you'll be able to setup transparent data encryption in HDFS on defined directories otherwise known as encryption zones "EZ". Before starting this step-by-step tutorial, there are three HDP services that are essential (must be installed): 1) HDFS 2) Ranger 3) Ranger KMSStep 1: Prepare environmentAs explained in the HDFS "Data at Rest" Encryption manuala) If using Oracle JDK, verify JCE is installed (OpenJDK has JCE installed by default)If the server running Ranger KMS is using Oracle JDK, you must install JCE (necessary for Ranger KMS to run) instructions on installing JCE can be found hereb) CPU Support for AES-NI optimizationAES-NI optimization requires an extended CPU instruction set for AES hardware acceleration.There are several ways to check for this; for example:cat /proc/cpuinfo | grep aesLook for output with flags and 'aes'.c) Librar

PROBLEM: When opening Zeppelin Ambari View on Sandbox (Virtualbox version), the page is blank. ROOT CAUSE: sandbox.hortonworks.com is what the view tries to redirect the browser to, which is not found if not explicitly set. SOLUTION: On your machine, modify the host file so that 127.0.0.1 sandbox.hortonworks.com is an entry. An alternative is to simply go directly to the web page 127.0.0.1:9995. Operating Systems have different location for hosts file and hence suggest reviewing your operating system's documentation of how to modify the hosts file.

How to get a docker image up and running which encapulates a PyCharm IDE integrated with spark and pybuilder. The IDE reside on the docker container and will be display on your laptop/machine. This is to isolate your development enviorment with has spark integrated with spark.Why? I am a spark developer and spend significant time trying to build a integrated environment. I am spending way too much time on integration before doing what I get paid to do --- Develop! Creating a isolated environment which is integrated with spark and a CIT, easily spun up and down, and repeatable is something which would accelerate my efficiency. Download latest virtualbox from here.To run docker containers or build images a docker machine is required. Download docker machine from here.Download xQuartz to display the IDE on your laptop.View my docker page for information on the docker image here.Clone my PyCharm github repo. You are doing this bootstrap code sample code I have built to your docker co

When using a postgresql DB for Hue, you might have encoutered [root@hue ~]# cd /usr/lib/hue [root@hue hue]# source ./build/env/bin/activate (env)[root@hue hue]# hue syncdbTraceback (most recent call last): File "/usr/lib/hue/build/env/bin/hue", line 9, in <module> load_entry_point('desktop==2.6.1', 'console_scripts', 'hue')() File "/usr/lib/hue/desktop/core/src/desktop/manage_entry.py", line 60, in entry execute_manager(settings) File "/usr/lib/hue/build/env/lib/python2.6/site-packages/Django-1.2.3-py2.6.egg/django/core/management/__init__.py", line 438, in execute_manager utility.execute() File "/usr/lib/hue/build/env/lib/python2.6/site-packages/Django-1.2.3-py2.6.egg/django/core/management/__init__.py", line 379, in execute self.fetch_command(subcommand).run_from_argv(self.argv) File "/usr/lib/hue/build/env/lib/python2.6/site-packages/Django-1.2.3-py2.6.egg/django/core/management/__init__.py", line 261, in fetch_command klass = load_command_class(app_name, s

Before completing this tutorial, it is important to understand data lineage.   What is Data Lineage Data lineage is defined as a data life cycle that conveys data origin and where data moves over time. In Apache Hive, if I create a table (TableA) and then insert data (from another table TableB), the data lineage will display TableA as the target and Table B as the source/origin. These two tables are linked together by a process "insert into Table..", allowing a user to understand the data life cycle. In a Hadoop ecosystem, Apache Atlas contains the data lineage for various systems like Apache Hive, Apache Falcon and Apache Sqoop. What is Apache Atlas Apache Atlas is a centralized governance framework that supports the Hadoop ecosystem as a metastore repository. To add metadata to Atlas, libraries called ‘hooks’ are enabled in various systems which automatically capture metadata events in the respective systems and propagate those events to Atlas. (More on Atlas' Architecture). Pre

The Hortonworks Hive ODBC Driver for OSX 10.11 (El Capitan) fails to install."This package is incompatible with this version of OS X and may fail to install"This issue is being tracked on (with full installer log) here:https://hortonworks.jira.com/browse/BUG-47990Support/Engineering will be reaching out to Simba Tech for a new Hive ODBC Driver.

Big Data Wrangling on HDP with Trifacta - How to Get startedData Preparation is a constant challenge for any enterprise and the speed, diversity and volumes of data introduced by Big Data simply amplify this problem substantially. Trifacta with HDP helps introduce a new approach to organizing, cleansing, enriching and structuring your data, Data Wrangling, where business users are able to connect and engage with the data to drive out high quality data sets for analytics.Step 1: Download VMTrifacta would like to provide Hortonworks partners and SI's with an opportunity to test drive Data Wrangling on HDP. Here is a link to a pre-configured virtual machine containing Trifacta Enterprise and HDP 2.3. Feel free to download and try Wrangling today:ftp://download.trifacta.com/Hortonworks/Trifacta_3.0_HDP_2.3.2_sandbox.ovauser: hortonworkspw: wranglerStep 2: Start VM and access consolesThe Trifacta Enterprise Wrangler on HDP is built on HDP 2.3 and the demo/sandbox instance of Centos. To acce

This tutorial has been successfully tried on HDP-2.4.0.0 and Ambari 2.2.1.0.I have my HDP Cluster Kerberized and Ambari has been configured for SSL.Note - Steps are same for Ambari with or without SSL..Please follow below steps for Configuring Pig View on Kerberized HDP Cluster..Step 1 - Please configure your Ambari Server for Kerberos with the steps mentioned in below article. Please follow steps 1 to 5.https://community.hortonworks.com/articles/40635/configure-tez-view-for-kerberized-hdp-cluster.html.Step 2 - Please add below properties to core-site.xml via Ambari UI and restart required services..Note - If you are running Ambari Server as root user then add below propertieshadoop.proxyuser.root.groups=*hadoop.proxyuser.root.hosts=*.If you are running Ambari server as non-root user then please add below properties in core-site.xmlhadoop.proxyuser.<ambari-server-user>.groups=*hadoop.proxyuser.<ambari-server-user>.hosts=*Please replace <ambari-server-user> with user r

How to make Mysql Database as Hive’s instance:Install Mysql if not available:brew updatebrew doctorbrew upgradebrew install mysqlmysql.server restartmysql_secure_installationlogin to mysql -> mysql –u root –pEnter password:Happy Mysql learning….Mysql is already installed on Hortonworks sandbox. Steps:Confirm with mysql –u root –pImport an already available database into Mysql:Ref: https://dev.mysql.com/doc/employee/en/employees-installation.htmlshell> tar -xjf $HOME/Downloads/employees_db-full-1.0.6.tar.bz2 shell> cd employees_db/shell> mysql -t < employees.sqlWith this installation of employee db in mysql is complete.Configuration of Mysql Instance with Hive:From HIVE create Mysql metastore[root@host]# mysqladmin -u root create hivedbmysql> USE hivedb;mysql> CREATE USER 'hive'@'localhost' IDENTIFIED BY 'hive';mysql> GRANT ALL PRIVILEGES ON *.* TO 'hive'@'localhost';With this we confirm that mysql database is the Hive’s new metastore.Suppose to perform a full im

This article was inspired by How to configure knox with existing ssl certificate and a client engagement to give more clarity into the actual commands that you need run to setup Apache Knox for SSL and LDAP with existing certificates.Pre-requisitesYou need to get procure the following items from your security department before you begin:Your LDAP or AD Digital Certificate: <ldap>.crtThe company's Digital CA Cert: <company_ca>.crtThe certificate/ key pair for the gateway node: <gateway_node>.crt and <gateway_node>.pemThe passphrase for the above gateway node key.Your Knox Master Secret--------------------------------------------------------------------What if?You don't know the Knox Master Secret? Then you can change it as follows cd $gatewaybin/knoxcli.cmd create-master --force-- Delete Keystores and restart Knox> rm data/security/keystores/gateway.jks> rm data/security/keystores__gateway-credentials.jceks You don't have a signed cert from a trusted CA

Documentation referred: https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.2/bk_installing_manually_book/content/configure-ranger_policy_admin_ha.htmlEnvironment Information: [ hivenv ] >> Cluster namehivenv-ambari-server.hwxblr.com >> 10.0.1.26 >> Ambari Server hn1.hwxblr.com >> 10.0.1.21 >> Existing Ranger Admin hn3.hwxblr.com >> 10.0.1.23 >> Load Balancer hn2.hwxblr.com >> 10.0.1.25 >> Additional Ranger Admin # hadoop version Hadoop 2.7.1.2.3.2.0-2950 Subversion git@github.com:hortonworks/hadoop.git -r5cc60e0003e33aa98205f18bccaeaf36cb193cc Compiled by jenkins on 2015-09-30T18:08Z Compiled with protoc 2.5.0 From source with checksum 69a3bf8c667267c2c252a54fbbf23d This command was run using /usr/hdp/2.3.2.0-2950/hadoop/lib/hadoop-common-2.7.1.2.3.2.0-2950.jar # uname -a Linux hn1.hwxblr.com 3.10.0-327.13.1.el7.x86_64 #1 SMP Thu Mar 31 16:04:38 UTC 2016x86_64 x86_64 x86_64 GNU/Linux # cat /etc/redhat-releas

One of the most common questions I come across when trying to help debug MapReduce jobs is: "How do I change the Log4j level for my job?" Many times, a user has a JAR with a class that implements Tool that they invoke using the hadoop jar command. The desire is to change the log level without changing any code or global configuration files:hadoop jar MyApplication.jar com.myorg.application.ApplicationJob <args ...>There is an extremely large amount of misinformation because how to do this has drastically changed from the 0.20.x and 1.x Apache Hadoop days. Most posts will inform you of some solution involving environment variables or passing Java opts to the mappers/reducers. In practice, there is actually a very straightforward solution.To change the Mapper Log4j level, set mapreduce.map.log.level. To change the Reducer Log4j level, set mapreduce.reduce.log.level. If for some reason you need to change the Log4j level on the MapReduce ApplicationMaster (e.g. to debug InputSplit