Community Articles

As the extras features are not shipped with the default log4j implementation hence the users will need to download the "Apache Extras™ for Apache log4j" (Like: apache-log4j-extras) : https://logging.apache.org/log4j/extras/download.htmlExample: For example download the jar "apache-log4j-extras-1.2.17.jar" and place it inside the location: /etc/hadoop/conf/secure/2. Add the property -Dhadoop.root.logger=INFO,ZIPRFA to export AMBARI_JVM_ARGS parameter in /var/lib/ambari-server/ambari-env.sh:export AMBARI_JVM_ARGS=$AMBARI_JVM_ARGS' -Xms512m -Xmx2048m -Djava.security.auth.login.config=/etc/ambari-server/conf/krb5JAASLogin.conf -Djava.security.krb5.conf=/etc/krb5.conf -Djavax.security.auth.useSubjectCredsOnly=false -Dhadoop.root.logger=INFO,ZIPRFA'3. Uncomment/Add the server classpath in /var/lib/ambari-server/ambari-env.sh:export SERVER_CLASSPATH=/etc/hadoop/conf/secure4. Added the following property to log4j:#log4j.rootLogger=INFO,filelog4j.rootLogger=INFO,ZIPRFAComment out the following

The ambari-server script used to wait a fixed amount of time before it gave up and thought the server wasn't going to start. On slower systems or test systems this limit was really easy to hit (50 seconds). Thanks to JIRA AMBARI-20220 this timeout has been parameterized. By adding the server.startup.web.timeout property (measured in seconds) to /etc/ambari-server/conf/ambari.properties you can change the timeout to something like this:...server.startup.web.timeout=120... so you don't get errors like this:# ambari-server start...Waiting for server start............................................................DB configs consistency check: no errors and warnings were found.ERROR: Exiting with exit code 1.REASON: Server not yet listening on http port 8080 after 50 seconds. Exiting.Note: You need at least Ambari 2.5 for this functionality to be supported

Issue:When you notice RM UI page truncated & in the log it shows NPE like belowscreenshot-2017-07-12-000551.png2017-06-22 11:18:05,507 ERROR webapp.Dispatcher (Dispatcher.java:service(171)) - error handling URI: /cluster/schedulerjava.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.yarn.webapp.Dispatcher.service(Dispatcher.java:162) at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:263) at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:178) at com.google.inject.servlet.ManagedServletPipeline.service(ManagedSe

Issue:Sometimes ambari might throw the above alert which puts you in some confusion how all 50 DN's can be active and stale.Firstly, I suggest you to go through this article from HCC and understand how to identify stale datanodes Resolution:Increase the value of the property from 30000(default) to 60000 milli seconds or 180000 milli seconds dfs.namenode.stale.datanode.interval=60000 or 180000

Steps to replicate:hdfs dfs -ls /apps/hive/warehouse/testraj.db/testtable/filename=test.csv.gzFound 1 items -rw-rw-rw- 1 hive hive 38258 2017-06-27 21:04 /apps/hive/warehouse/testraj.db/testtable/filename=test.csv.gz/000000_0 USING hive -f scriptcat /tmp/test.txtALTER TABLE testraj.testtable PARTITION (filename="test.csv.gz") SET LOCATION "hdfs://ip-1-1-1-1.us-west-2.compute.internal:8020/apps/hive/warehouse/testraj.db/testtable/filename=test.csv.gz"; Error from hive -f scriptname: [hive@ip-1-1-1-1 rbiswas]$ hive -f /tmp/test.txt Logging initialized using configuration in file:/etc/hive/2.5.3.0-37/0/hive-log4j.properties FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. Unable to alter partition. alter is not possible [hive@ip-1-1-1-1 rbiswas]$ Error from Beeline: 0: jdbc:hive2://ip-1-1-1-1.us-west-2.com> ALTER TABLE testraj.testtable PARTITION (filename="test.csv.gz") SET LOCATION 'hdfs://ip-1-1-1-1.us-west-2.compute.internal:8020/apps/hive/warehou

By default, the file container-executor.cfg under /etc/hadoop/conf/ is overwritten in every nodemanager by /var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/templates/container-executor.cfg.j2When you have LinuxContainerExecutor , yarn execute jobs as the end user, in this case It's not recommended to change banned.users and allowed.system.userWhy you should ban super user from running Yarn jobs ? this is because anyone can run the job as "super-user" within hadoop group , hadoop trust what user you say you are when submitting jobs - if you pass kerberos wall with the keytab.( which can easily be found and used in the job ) - then any user can basically have full super user permissions on job submission.

Note: A newer version of this article is available here: https://community.hortonworks.com/articles/194076/using-vagrant-and-virtualbox-to-create-a-local-ins.htmlObjectives This tutorial is designed to walk you through the process of using Vagrant and Virtualbox to create a local instance of Cloudbreak. This will allow you to start your Cloudbreak deployer when you want to spin up an HDP cluster on the cloud without incurring costs associated with hosting your Cloudbreak instance on the cloud itself.Prerequisites You should already have installed VirtualBox 5.1.x. Read more here: VirtualBox You should already have installed Vagrant 1.9.x. Read more here: Vagrant You should already have installed the vagrant-vbguest plugin. This plugin will keep the VirtualBox Guest Additions software current as you upgrade your kernel and/or VirtualBox versions. Read more here: vagrant-vbguest You should already have installed the vagrant-hostmanager plugin. This plugin will automatically manage the

After upgrading Ambari (not entirely sure which version but think it was Ambari 2.4.2 to 2.5.0.3), Smartsense View came up unavailable with these messages:"Service Unavailable""The SmartSense service is currently unavailable. Please make sure the SmartSense Server is up and running."See screenshotIn the /var/log/ambari-server/ambari-server.log log you many see entries like this when you click on the Smartsense view.14 Jul 2017 11:14:05,523 ERROR [ambari-client-thread-467377] ServerProxy:162 - Failed to execute GET /recommendations. Reason: Connection refused14 Jul 2017 11:14:14,734 ERROR [ambari-client-thread-467506] ServerProxy:162 - Failed to execute GET /context. Reason: Connection refused14 Jul 2017 11:14:14,900 ERROR [ambari-client-thread-467507] ServerProxy:162 - Failed to execute GET /checkconfig. Reason: Connection refuseddiff'ing the hst-server.ini config file against previous saved config versions yielded this:# diff conf/hst-server.ini conf_13_04_17_04_17.save/2,4d1< enab

This article is based on one of my blog posts. It is specifically about how to troubleshoot and debug an application behind Knox and ultimately get it up and running.Start Small First try to access the service directly before you go over Knox. In many cases, there’s nothing wrong with your Knox setup, but with either the way you setup and configured the service behind Knox or the way you try to access that service. When you are familiar on how to access your service directly and when you have verified that it works as intended, try to do the same call on Knox.Example: You want to check if webhdfs is reachable so you first verify directly at the service and try to get the home directory of the service. curl --negotiate -u : http://webhdfs-host.field.hortonworks.com:50070/webhdfs/v1/?op=GETHOMEDIRECTORY If above request gives a valid 200 response and a meaningful answer you can safely check your Knox setup. curl -k -u myUsername:myPassword https://knox-host.field.hortonworks.c

When a Kafka cluster is over-subscribed, the loss of a single broker can be a jarring experience for the cluster as a whole. This is especially true when trying to bring a previously failed broker back into a cluster.In order to help mitigate some of the impact of returning a broker to a cluster when that broker has been out of the cluster for a number of days, removing the broker ID of the broker ready to re-enter the cluster from the Replicas list of all partitions can help. Generally, you want a Kafka cluster that is sized properly in order to handle single node failures, but as is often the case the size of the use case on the Kafka cluster can quickly start to exceed the physical limitations. In those situations when you're waiting for new hardware to arrive to augment your cluster, you still need to keep the existing cluster working as well as possible.To that end, there are some AWK scripts that are available on Github that help create the JSON files needed to essentially spo

When using post-user-creation-hook.sh script to create home directories for users we can edit the script to set a quota too.For information on enabling HDFS home dir creation see:https://docs.hortonworks.com/HDPDocuments/Ambari-2.5.0.3/bk_ambari-administration/content/create_user_home_directory.htmlIf you want to set a quota on this dir you can edit:/var/lib/ambari-server/resources/scripts/post-user-creation-hook.sh# the default implementation creates user home folders; the first argument must be the usernameambari_sudo "yarn jar /var/lib/ambari-server/resources/stacks/HDP/2.0.6/hooks/before-START/files/fast-hdfs-resource.jar $JSON_INPUT"#ADD THESE LINESwhile read -r LINEdo USR_NAME=$(echo "$LINE" | awk -F, '{print $1}')hdfs dfsadmin -setSpaceQuota 10 /user/$USR_NAME > /tmp/posthook.tmpdone <"$CSV_FILE"#END ADD QUOTAif [ "$DEBUG" -gt "0" ]; then echo "Switch debug OFF";set -x;unset DEBUG; else echo "debug: OFF"; fiunset DEBUG}main "$@"Add the lines between the comments and save.

Virtual Integration of Hadoop with External SystemsThe challenge of merging data from disparate systems has been a leading driver behind investments in data warehousing systems, as well as, in Hadoop. While data warehousing solutions are ready-built for RDBMS integration, Hadoop adds the benefits of infinite and economical scale – not to mention the variety of structured and non-structured formats that it can handle. Whether using a data warehouse or Hadoop or both, physical data movement and consolidation is the primary method of integration.However, there will be data in external systems that need to remain siloed for a variety of reasons (legal, political, economic, etc.). There may also be challenges with synchronizing rapidly changing data from a system of record to a consolidated platform (DW or Hadoop). This introduces the need for “data federation” or “data virtualization”, where data is integrated without copying data between systems. Best stated in the wiki, “A federated

I. Environment details- HDP 2.5.x- HDP 2.6.x- Kerberos enabledII. Steps to followa) JDBC interpreterSet JDBC interpreter in Zeppelin UI like- JDBC interpreter configphoenix.driver org.apache.phoenix.jdbc.PhoenixDriverphoenix.hbase.client.retries.number 1phoenix.password phoenix.url jdbc:phoenix:dkhdp262.openstacklocal,dkhdp261.openstacklocal,dkhdp263.openstacklocal:/hbase-securephoenix.user phoenixuserzeppelin.jdbc.auth.type KERBEROSzeppelin.jdbc.keytab.location /etc/security/keytabs/zeppelin.server.kerberos.keytabzeppelin.jdbc.principal zeppelin-dkhdp26@SUPPORT.COMARTIFACTS /usr/hdp/current/phoenix-client/phoenix-client.jar b) zeppelin notebook

Design approachThe designs depend on the work done in the below Jira, where data node is conceptualized as a collection of heterogeneous storage with different durability and performance requirements. https://issues.apache.org/jira/browse/HDFS-2832Design 11) Hot data with partitions that are wholly hosted by HDFS.2) Cold data with partitions that are wholly hosted by S3.3) A view that unions these two tables which is the live table that we expose to end users.Design 21) Hot data with partitions that are wholly hosted by HDFS.2) Cold data with partitions that are wholly hosted by S3.3) Both hot and cold data are in the same tableDesign 2 is chosen over Design 1 because Design 1 is not transparent to the application layer. The change from old table to the view would inherently transfer some level of porting/integration extra work to the application.Architecture DiagramHigh Level DesignAutomation Flow DiagramCodeAutomation tool codebasehttps://github.com/RajdeepBiswas/HybridArchiveStorag

IntroductionTraditional data warehouse archive strategy involves moving the old data into offsite tapes.This does not quite fit the size for modern analytics applications since the data is unavailable for business analytics in real time need. Mature Hadoop clusters need a modern data archival strategy to keep the storage expense at check when data volume increases exponentially.The term hybrid here designates an archival solution which is always available as well as completely transparent to the application layerThis document will cover Use caseRequirementStorage cost analysisDesign ApproachArchitecture diagramCodeHow to Setup and Run the codeUse caseEntire business data is in HDFS (HDP clusters) backed by Amazon EBS.Disaster recovery solution is in place. Amazon claims S3 storage delivers 99.999999999% durability. In the case of data loss from S3 we have to recover the data from disaster recovery site.Requirement Decrease storage costs.Archived data should be available to perform anal

The following process is assuming that you are installing your HDP cluster using a configuration management tool such as Ansible, Puppet or Chef and that you are deploying the cluster using Ambari blueprint. If you are looking into automating your deployment you might be interested to use the great from @Alex Bush with Ansible. https://github.com/bushnoh/ansible-hadoop-asapThis process could be used for : - Migrating from HDP 2.2 to 2.4 using a full reinstallation - Upgrading OS from RHEL 6 to RHEL 7 - OS boot from network and reinstallation of HDP The following process has been tested to migrate from HDP 2.3 to HDP 2.4 on a kerberised cluster and to reinstall an HDP 2.3 cluster. It should also work for HDP 2.5 as HDFS version is consistent across those versions.step 1:- Make a backup of your metastore DB ( Ranger, Hive and Oozie ). step 2:- Check that the Namenode is having a folder called namenode-formatted under dfs.namenode.name.dir. If you are using namenode HA, you need to chec

HDF OverviewOverviewHortonworks DataFlow (HDF) powered by Apache NiFi, Kafka and Storm, collects, curates, analyzes and delivers real-time data from the IoAT to data stores both on-premises and in the cloud. This is the quick installation guide to install Apache NiFi on AWS EC2 instance. Please refer this document as supplement guide to official Hortonworks HDF documentation.PrerequisitesBefore you install Apache NiFi on AWS, make sureYou have AWS account. (https://aws.amazon.com/)Amazon key pair to access EC2 instance to run HDP platform. (http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-key-pairs.html#having-ec2-create-your-key-pair)Installation Steps.The screenshots in this section detail the setup and configuration of Apache NiFi on EC2 instance.Refer the NiFi Admin Guide for the System requirements. This document covers installation on a Redhat linux (64 bit) EC2 instance. Login to AWS and launch the EC2 instance and OS of your choice. (Please make sure the selected OS is

Hybrid Procedural SQL On Hadoop (HPL/SQL) is a tool that implements procedural SQL for Hive.Lately, many people have investigated running HPL/SQL on HDP 2.6 and have encountered problems. These instructions tell you how to work around these problems so that you can experiment with HPL/SQL.In HDP 2.6, HPL/SQL is considered a "technical preview" and is not supported by Hortonworks support subscriptions. There are known limitations in HPL/SQL that may make it unsuitable for your needs, so test thoroughly before you decide HPL/SQL is right for you.These instructions require cluster changes which are not appropriate for production clusters and should only be done on development clusters or sandboxes.We'll cover 2 ways of using HPL/SQL: Using HiveServer Interactive (Preferred)Using an embedded metastoreIn either approach, you need to edit /etc/hive2/conf/hive-env.sh and change line 30 from:export HIVE_CONF_DIR=/usr/hdp/current/hive-server2-hive2/conf/conf.servertoexport HIVE_CONF_DIR=${HIVE_

If the mysqldump is for different version other than Hive 2.1.1000 in HDP 2.6. Then do the following:1. Stop Hive services from Ambari.2. Create new database under MySQL as say hive2:mysql> create database hive2;Query OK,1 row affected (0.00 sec)mysql> grant all privileges on hive2.* to 'hive'@'%' identified by'hive';Query OK,0 rows affected (0.00 sec)3. Restore database as:mysql -u hive -phive hive2 < dumpfilename.sql4. Update database connection string for mysql under Ambari -> Hive configs.5. Save configurations and try restarting. Since there is different in VERSION, service startup would fail.6. Run Hive schematool command to upgrade the schema as below:hive@ssnode260 bin]$ /usr/hdp/2.6.0.3-8/hive2/bin/schematool -upgradeSchema -dbType mysql7. Restart Hive services from Ambari.If the Hive metadata version is same as Hive 2.1.1000 in HDP 2.6, then follow steps 1 through 5.

When configuring LDAPS in HDP its common to see wrong certificates used or certificates without the correct chain.To ensure the correct chain of certificates is used when configuring LDAPS you can use openssl to read the certificate from the server and save it to a file. This file can them be imported into, for example, the Ambari truststore.echo -n | openssl s_client -connect <ad-server>:636 | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > /root/ldaps-cert.pem

This is a cool HAT that attaches without soldering to a Raspberry Pi 3. The main use is with Google's Android Things which is an IoT OS based on Android that you can run on Raspberry Pi and other devices.You can also use this HAT with regular Raspian and Python.This HAT is cool because it has input and output. As part of our Python script, we ingest data from it's sensor, but we also allow for input to it's A, B and C buttons clearly marked on the top. By selecting one of the buttons you can display the current temperature in Celsius, Fahrenheit or the current altitude. The altitude is some code I found commented out in the Pimoroni Rainbow HAT interface Python script. It determines altitude based on a starting value and pressure readings. Looks kind of cool and is relatively accurate. For point of reference, Princeton New Jersey is pretty close to sea level.My script is heavily copied and customized from my previous IoT Python scripts and from the RainbowHAT examples.This h

I am always ingesting social data, IoT and mobile streams into my HDFS cluster. Unfortunately my cluster is a ephemeral cloud based Hortonworks HDP 2.6 Hadoop cluster, so I don't have a permanent store for my data. I have my processes run for a few weeks and then they are destroyed. I wanted to quick way to save all my ORC files. Enter NiFi. Backup First we list from some top level directories in HDFS to capture all the files and sub-directories we want to backup. Each processor maintains a timestamp to know what files it processed already, as new files are added they will be assimilated into the flow. For massive data migration, we can run this on many nodes and use the Distributed Cache service to maintain the state. Restore The flow is very simple to restore, read from the local file system and write to HDFS. For HDFS, I use /${path} as the directory so each file is written to the correct sub-directory for it's file group. Easy it's like

So I might be late in posting this article, but I want to publicize the changes in HDP 2.6.0 documentation and blog postings for Hive, Hive Views, and Ambari that were released in early April 2017. After all, I don't think all of you out there are upgraded to HDP 2.6.0 yet.To complement the G.A. release of Hive low-latency analytical processing (LLAP), the "Hive Performance Tuning Guide" has been completely rewritten to focus on how to optimize your environment and queries for interactive processing. While the guide is tailored to run a Hive data warehouse on the Tez framework, the new edition still has relevant tips and links to information for users who do batch processing on a MapReduce framework. A couple of LLAP blogs to consider reading are "Fast Analytics in the Cloud with Hive LLAP" and "Top 5 Performance Boosters with Apache Hive LLAP".Important debugging and optimization tools to get the most out of your Hive queries and your architecture are documented in Hive View 2.0 and T

Incremental Backup of Data from HDP to Azure using Falcon for Disaster Recovery and Burst capacity Introduction Apache Falcon simplifies the configuration of data motion with: replication; lifecycle management; lineage and traceability. This provides data governance consistency across Hadoop components. Scenario In this tutorial we will walk through a scenario where email data gets processed on multiple HDP 2.2 clusters around the country then gets backed up hourly on a cloud hosted cluster . In our example: This cluster is hosted on Windows Azure. Data arrives from all the West Coast production servers. The input data feeds are often late for up to 4 hrs. The goal is to clean the raw data to remove sensitive information like credit card numbers and make it available to our marketing data science team for customer churn analysis. To simulate this scenario, we have a pig script grabbing the freely available Enron emails from the internet and feeding it into the pipeline.

Define and Process Data Pipelines in Hadoop With Apache Falcon Introduction Apache Falcon is a framework to simplify data pipeline processing and management on Hadoop clusters. It makes it much simpler to onboard new workflows/pipelines, with support for late data handling and retry policies. It allows you to easily define relationships between various data and processing elements and integrate with metastore/catalog such as Apache Hive/HCatalog. Finally it also lets you capture lineage information for feeds and processes. In this tutorial we are going to walkthrough the process of: Defining the feeds and processes Defining and executing a data pipeline to ingest, process and persist data continuously Prerequisites Download Hortonworks Sandbox Complete the Learning the Ropes of the Hortonworks Sandbox tutorial, you will need it for logging into ambari as an administrator user. Complete the Creating Falcon Cluster tutorial to start the falcon service, prepare HDFS dir