Resources

Issue:Running the below sqoop on a client machine might lead to the reported errorsqoop export -Dmapreduce.map.log.level=DEBUG -Dteradata.db.output.fastload.socket.port=8678 \-Dteradata.db.output.method=internal.fastload \--connect jdbc:teradata://mrt1.openstacklocal/Database=testdb \--connection-manager org.apache.sqoop.teradata.TeradataConnManager --username COOXP -P \--export-dir /user/karthick/fastload_test \--table test_table --input-fields-terminated-by ',' \--null-string '\N' --null-non-string '\N' --num-mappers 5 INFO mapreduce.Job: Task Id : attempt_1497611885470_0832_m_000000_0, Status : FAILEDError: com.teradata.connector.common.exception.ConnectorException: java.net.SocketException: Socket is not connected at java.net.Socket.getInputStream(Socket.java:905) at com.teradata.connector.teradata.TeradataInternalFastloadOutputFormat.getRecordWriter(TeradataInternalFastloadOutputFormat.java:356) at com.teradata.connector.common.ConnectorOutputFormat$ConnectorF

OverviewThe following versions of Apache Kafka have been incorporated in HDP 2.2.0 to 2.6.1: 0.8.1, 0.8.2, 0.9.0, 0.10.0, 0.10.1. Apache Kafka is now at 0.11.Hortonworks is working to make Kafka easier for enterprises to use. New focus areas include creation of a Kafka Admin Panel to create/delete topics and manage user permissions, easier and safer distribution of security tokens and support for multiple ways of publishing/consuming data via a Kafka REST server/API. Here are a few areas of strong contribution:Operations:Rack awareness for Increased resilience and availability such that replicas are isolated so they are guaranteed to span multiple racks or availability zones.Automated replica leader election for automated, even distribution of leaders in a cluster capability by detecting uneven distribution with some brokers serving more data compared to others and makes adjustments.Message Timestamps so every message in Kafka now has a timestamp field that indicates the time at which

With HDF 2.x, Ambari can be used to deploy a NiFi cluster. Lets say you deployed a 2 node cluster and want to go back at a later time and add an additional NiFi node to the cluster. While the process is very straight forward when your NiFi cluster has been setup non-secure (http), the same is not true if your existing NiFi cluster has been secured (https).Below you will see an existing 2 node secured NiFi cluster that was installed via Ambari:STEP 1: Add new host through Ambari. You can skip this step if the host you want to install the additional NiFi node on is already managed by your Ambari.STEP 2: Under "Hosts" in Ambari click on the host form the list where you want to install the new NiFi node.The NiFi component will be in a "stopped" state after it is installed on this new host. *** DO NOT START NIFI YET ON NEW HOST OR IT WILL FAIL TO JOIN CLUSTER. ***STEP 3: (This step only applies if NiFi's file based authorizer is being used) Before starting this new node we need to clea

If cluster size is large and it is more than 1 year then Ambari gets slows down bit - so it is recommended to purge historical operational data.Note: You would loose upgrade history as well - Ambari does not use the upgrade history in any way. Note: (Updated in August 2017) :From Ambari 2.5.x there is a product utility to do the clean up - "db-cleanup"Here is list of queries to delete more than 1 month operational data. please do this with caution.Take a database backup before attempting to perform this.CREATE TEMPORARY TABLE IF NOT EXISTS tmp_request_id AS SELECT MAX(request_id) AS request_id FROM request WHERE create_time <= (SELECT (UNIX_TIMESTAMP(NOW()) - 2678400) * 1000 as epoch_1_month_ago_times_1000);CREATE TEMPORARY TABLE IF NOT EXISTS tmp_task_id AS SELECT MAX(task_id) AS task_id FROM host_role_command WHERE request_id <= (SELECT request_id FROM tmp_request_id);CREATE TEMPORARY TABLE IF NOT EXISTS tmp_upgrade_ids AS SELECT upgrade_id FROM upgrade WHERE request_id <

As the extras features are not shipped with the default log4j implementation hence the users will need to download the "Apache Extras™ for Apache log4j" (Like: apache-log4j-extras) : https://logging.apache.org/log4j/extras/download.htmlExample: For example download the jar "apache-log4j-extras-1.2.17.jar" and place it inside the location: /etc/hadoop/conf/secure/2. Add the property -Dhadoop.root.logger=INFO,ZIPRFA to export AMBARI_JVM_ARGS parameter in /var/lib/ambari-server/ambari-env.sh:export AMBARI_JVM_ARGS=$AMBARI_JVM_ARGS' -Xms512m -Xmx2048m -Djava.security.auth.login.config=/etc/ambari-server/conf/krb5JAASLogin.conf -Djava.security.krb5.conf=/etc/krb5.conf -Djavax.security.auth.useSubjectCredsOnly=false -Dhadoop.root.logger=INFO,ZIPRFA'3. Uncomment/Add the server classpath in /var/lib/ambari-server/ambari-env.sh:export SERVER_CLASSPATH=/etc/hadoop/conf/secure4. Added the following property to log4j:#log4j.rootLogger=INFO,filelog4j.rootLogger=INFO,ZIPRFAComment out the following

The ambari-server script used to wait a fixed amount of time before it gave up and thought the server wasn't going to start. On slower systems or test systems this limit was really easy to hit (50 seconds). Thanks to JIRA AMBARI-20220 this timeout has been parameterized. By adding the server.startup.web.timeout property (measured in seconds) to /etc/ambari-server/conf/ambari.properties you can change the timeout to something like this:...server.startup.web.timeout=120... so you don't get errors like this:# ambari-server start...Waiting for server start............................................................DB configs consistency check: no errors and warnings were found.ERROR: Exiting with exit code 1.REASON: Server not yet listening on http port 8080 after 50 seconds. Exiting.Note: You need at least Ambari 2.5 for this functionality to be supported

Issue:When you notice RM UI page truncated & in the log it shows NPE like belowscreenshot-2017-07-12-000551.png2017-06-22 11:18:05,507 ERROR webapp.Dispatcher (Dispatcher.java:service(171)) - error handling URI: /cluster/schedulerjava.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.yarn.webapp.Dispatcher.service(Dispatcher.java:162) at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:263) at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:178) at com.google.inject.servlet.ManagedServletPipeline.service(ManagedSe

Issue:Sometimes ambari might throw the above alert which puts you in some confusion how all 50 DN's can be active and stale.Firstly, I suggest you to go through this article from HCC and understand how to identify stale datanodes Resolution:Increase the value of the property from 30000(default) to 60000 milli seconds or 180000 milli seconds dfs.namenode.stale.datanode.interval=60000 or 180000

Steps to replicate:hdfs dfs -ls /apps/hive/warehouse/testraj.db/testtable/filename=test.csv.gzFound 1 items -rw-rw-rw- 1 hive hive 38258 2017-06-27 21:04 /apps/hive/warehouse/testraj.db/testtable/filename=test.csv.gz/000000_0 USING hive -f scriptcat /tmp/test.txtALTER TABLE testraj.testtable PARTITION (filename="test.csv.gz") SET LOCATION "hdfs://ip-1-1-1-1.us-west-2.compute.internal:8020/apps/hive/warehouse/testraj.db/testtable/filename=test.csv.gz"; Error from hive -f scriptname: [hive@ip-1-1-1-1 rbiswas]$ hive -f /tmp/test.txt Logging initialized using configuration in file:/etc/hive/2.5.3.0-37/0/hive-log4j.properties FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. Unable to alter partition. alter is not possible [hive@ip-1-1-1-1 rbiswas]$ Error from Beeline: 0: jdbc:hive2://ip-1-1-1-1.us-west-2.com> ALTER TABLE testraj.testtable PARTITION (filename="test.csv.gz") SET LOCATION 'hdfs://ip-1-1-1-1.us-west-2.compute.internal:8020/apps/hive/warehou

By default, the file container-executor.cfg under /etc/hadoop/conf/ is overwritten in every nodemanager by /var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/templates/container-executor.cfg.j2When you have LinuxContainerExecutor , yarn execute jobs as the end user, in this case It's not recommended to change banned.users and allowed.system.userWhy you should ban super user from running Yarn jobs ? this is because anyone can run the job as "super-user" within hadoop group , hadoop trust what user you say you are when submitting jobs - if you pass kerberos wall with the keytab.( which can easily be found and used in the job ) - then any user can basically have full super user permissions on job submission.

Note: A newer version of this article is available here: https://community.hortonworks.com/articles/194076/using-vagrant-and-virtualbox-to-create-a-local-ins.htmlObjectives This tutorial is designed to walk you through the process of using Vagrant and Virtualbox to create a local instance of Cloudbreak. This will allow you to start your Cloudbreak deployer when you want to spin up an HDP cluster on the cloud without incurring costs associated with hosting your Cloudbreak instance on the cloud itself.Prerequisites You should already have installed VirtualBox 5.1.x. Read more here: VirtualBox You should already have installed Vagrant 1.9.x. Read more here: Vagrant You should already have installed the vagrant-vbguest plugin. This plugin will keep the VirtualBox Guest Additions software current as you upgrade your kernel and/or VirtualBox versions. Read more here: vagrant-vbguest You should already have installed the vagrant-hostmanager plugin. This plugin will automatically manage the

After upgrading Ambari (not entirely sure which version but think it was Ambari 2.4.2 to 2.5.0.3), Smartsense View came up unavailable with these messages:"Service Unavailable""The SmartSense service is currently unavailable. Please make sure the SmartSense Server is up and running."See screenshotIn the /var/log/ambari-server/ambari-server.log log you many see entries like this when you click on the Smartsense view.14 Jul 2017 11:14:05,523 ERROR [ambari-client-thread-467377] ServerProxy:162 - Failed to execute GET /recommendations. Reason: Connection refused14 Jul 2017 11:14:14,734 ERROR [ambari-client-thread-467506] ServerProxy:162 - Failed to execute GET /context. Reason: Connection refused14 Jul 2017 11:14:14,900 ERROR [ambari-client-thread-467507] ServerProxy:162 - Failed to execute GET /checkconfig. Reason: Connection refuseddiff'ing the hst-server.ini config file against previous saved config versions yielded this:# diff conf/hst-server.ini conf_13_04_17_04_17.save/2,4d1< enab

This article is based on one of my blog posts. It is specifically about how to troubleshoot and debug an application behind Knox and ultimately get it up and running.Start Small First try to access the service directly before you go over Knox. In many cases, there’s nothing wrong with your Knox setup, but with either the way you setup and configured the service behind Knox or the way you try to access that service. When you are familiar on how to access your service directly and when you have verified that it works as intended, try to do the same call on Knox.Example: You want to check if webhdfs is reachable so you first verify directly at the service and try to get the home directory of the service. curl --negotiate -u : http://webhdfs-host.field.hortonworks.com:50070/webhdfs/v1/?op=GETHOMEDIRECTORY If above request gives a valid 200 response and a meaningful answer you can safely check your Knox setup. curl -k -u myUsername:myPassword https://knox-host.field.hortonworks.c

When a Kafka cluster is over-subscribed, the loss of a single broker can be a jarring experience for the cluster as a whole. This is especially true when trying to bring a previously failed broker back into a cluster.In order to help mitigate some of the impact of returning a broker to a cluster when that broker has been out of the cluster for a number of days, removing the broker ID of the broker ready to re-enter the cluster from the Replicas list of all partitions can help. Generally, you want a Kafka cluster that is sized properly in order to handle single node failures, but as is often the case the size of the use case on the Kafka cluster can quickly start to exceed the physical limitations. In those situations when you're waiting for new hardware to arrive to augment your cluster, you still need to keep the existing cluster working as well as possible.To that end, there are some AWK scripts that are available on Github that help create the JSON files needed to essentially spo

When using post-user-creation-hook.sh script to create home directories for users we can edit the script to set a quota too.For information on enabling HDFS home dir creation see:https://docs.hortonworks.com/HDPDocuments/Ambari-2.5.0.3/bk_ambari-administration/content/create_user_home_directory.htmlIf you want to set a quota on this dir you can edit:/var/lib/ambari-server/resources/scripts/post-user-creation-hook.sh# the default implementation creates user home folders; the first argument must be the usernameambari_sudo "yarn jar /var/lib/ambari-server/resources/stacks/HDP/2.0.6/hooks/before-START/files/fast-hdfs-resource.jar $JSON_INPUT"#ADD THESE LINESwhile read -r LINEdo USR_NAME=$(echo "$LINE" | awk -F, '{print $1}')hdfs dfsadmin -setSpaceQuota 10 /user/$USR_NAME > /tmp/posthook.tmpdone <"$CSV_FILE"#END ADD QUOTAif [ "$DEBUG" -gt "0" ]; then echo "Switch debug OFF";set -x;unset DEBUG; else echo "debug: OFF"; fiunset DEBUG}main "$@"Add the lines between the comments and save.

Virtual Integration of Hadoop with External SystemsThe challenge of merging data from disparate systems has been a leading driver behind investments in data warehousing systems, as well as, in Hadoop. While data warehousing solutions are ready-built for RDBMS integration, Hadoop adds the benefits of infinite and economical scale – not to mention the variety of structured and non-structured formats that it can handle. Whether using a data warehouse or Hadoop or both, physical data movement and consolidation is the primary method of integration.However, there will be data in external systems that need to remain siloed for a variety of reasons (legal, political, economic, etc.). There may also be challenges with synchronizing rapidly changing data from a system of record to a consolidated platform (DW or Hadoop). This introduces the need for “data federation” or “data virtualization”, where data is integrated without copying data between systems. Best stated in the wiki, “A federated

I. Environment details- HDP 2.5.x- HDP 2.6.x- Kerberos enabledII. Steps to followa) JDBC interpreterSet JDBC interpreter in Zeppelin UI like- JDBC interpreter configphoenix.driver org.apache.phoenix.jdbc.PhoenixDriverphoenix.hbase.client.retries.number 1phoenix.password phoenix.url jdbc:phoenix:dkhdp262.openstacklocal,dkhdp261.openstacklocal,dkhdp263.openstacklocal:/hbase-securephoenix.user phoenixuserzeppelin.jdbc.auth.type KERBEROSzeppelin.jdbc.keytab.location /etc/security/keytabs/zeppelin.server.kerberos.keytabzeppelin.jdbc.principal zeppelin-dkhdp26@SUPPORT.COMARTIFACTS /usr/hdp/current/phoenix-client/phoenix-client.jar b) zeppelin notebook

Design approachThe designs depend on the work done in the below Jira, where data node is conceptualized as a collection of heterogeneous storage with different durability and performance requirements. https://issues.apache.org/jira/browse/HDFS-2832Design 11) Hot data with partitions that are wholly hosted by HDFS.2) Cold data with partitions that are wholly hosted by S3.3) A view that unions these two tables which is the live table that we expose to end users.Design 21) Hot data with partitions that are wholly hosted by HDFS.2) Cold data with partitions that are wholly hosted by S3.3) Both hot and cold data are in the same tableDesign 2 is chosen over Design 1 because Design 1 is not transparent to the application layer. The change from old table to the view would inherently transfer some level of porting/integration extra work to the application.Architecture DiagramHigh Level DesignAutomation Flow DiagramCodeAutomation tool codebasehttps://github.com/RajdeepBiswas/HybridArchiveStorag

IntroductionTraditional data warehouse archive strategy involves moving the old data into offsite tapes.This does not quite fit the size for modern analytics applications since the data is unavailable for business analytics in real time need. Mature Hadoop clusters need a modern data archival strategy to keep the storage expense at check when data volume increases exponentially.The term hybrid here designates an archival solution which is always available as well as completely transparent to the application layerThis document will cover Use caseRequirementStorage cost analysisDesign ApproachArchitecture diagramCodeHow to Setup and Run the codeUse caseEntire business data is in HDFS (HDP clusters) backed by Amazon EBS.Disaster recovery solution is in place. Amazon claims S3 storage delivers 99.999999999% durability. In the case of data loss from S3 we have to recover the data from disaster recovery site.Requirement Decrease storage costs.Archived data should be available to perform anal

The following process is assuming that you are installing your HDP cluster using a configuration management tool such as Ansible, Puppet or Chef and that you are deploying the cluster using Ambari blueprint. If you are looking into automating your deployment you might be interested to use the great from @Alex Bush with Ansible. https://github.com/bushnoh/ansible-hadoop-asapThis process could be used for : - Migrating from HDP 2.2 to 2.4 using a full reinstallation - Upgrading OS from RHEL 6 to RHEL 7 - OS boot from network and reinstallation of HDP The following process has been tested to migrate from HDP 2.3 to HDP 2.4 on a kerberised cluster and to reinstall an HDP 2.3 cluster. It should also work for HDP 2.5 as HDFS version is consistent across those versions.step 1:- Make a backup of your metastore DB ( Ranger, Hive and Oozie ). step 2:- Check that the Namenode is having a folder called namenode-formatted under dfs.namenode.name.dir. If you are using namenode HA, you need to chec

I'm making this post solely to see what happens. Hey, you were warned. 😛

HDF OverviewOverviewHortonworks DataFlow (HDF) powered by Apache NiFi, Kafka and Storm, collects, curates, analyzes and delivers real-time data from the IoAT to data stores both on-premises and in the cloud. This is the quick installation guide to install Apache NiFi on AWS EC2 instance. Please refer this document as supplement guide to official Hortonworks HDF documentation.PrerequisitesBefore you install Apache NiFi on AWS, make sureYou have AWS account. (https://aws.amazon.com/)Amazon key pair to access EC2 instance to run HDP platform. (http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-key-pairs.html#having-ec2-create-your-key-pair)Installation Steps.The screenshots in this section detail the setup and configuration of Apache NiFi on EC2 instance.Refer the NiFi Admin Guide for the System requirements. This document covers installation on a Redhat linux (64 bit) EC2 instance. Login to AWS and launch the EC2 instance and OS of your choice. (Please make sure the selected OS is

Hybrid Procedural SQL On Hadoop (HPL/SQL) is a tool that implements procedural SQL for Hive.Lately, many people have investigated running HPL/SQL on HDP 2.6 and have encountered problems. These instructions tell you how to work around these problems so that you can experiment with HPL/SQL.In HDP 2.6, HPL/SQL is considered a "technical preview" and is not supported by Hortonworks support subscriptions. There are known limitations in HPL/SQL that may make it unsuitable for your needs, so test thoroughly before you decide HPL/SQL is right for you.These instructions require cluster changes which are not appropriate for production clusters and should only be done on development clusters or sandboxes.We'll cover 2 ways of using HPL/SQL: Using HiveServer Interactive (Preferred)Using an embedded metastoreIn either approach, you need to edit /etc/hive2/conf/hive-env.sh and change line 30 from:export HIVE_CONF_DIR=/usr/hdp/current/hive-server2-hive2/conf/conf.servertoexport HIVE_CONF_DIR=${HIVE_

If the mysqldump is for different version other than Hive 2.1.1000 in HDP 2.6. Then do the following:1. Stop Hive services from Ambari.2. Create new database under MySQL as say hive2:mysql> create database hive2;Query OK,1 row affected (0.00 sec)mysql> grant all privileges on hive2.* to 'hive'@'%' identified by'hive';Query OK,0 rows affected (0.00 sec)3. Restore database as:mysql -u hive -phive hive2 < dumpfilename.sql4. Update database connection string for mysql under Ambari -> Hive configs.5. Save configurations and try restarting. Since there is different in VERSION, service startup would fail.6. Run Hive schematool command to upgrade the schema as below:hive@ssnode260 bin]$ /usr/hdp/2.6.0.3-8/hive2/bin/schematool -upgradeSchema -dbType mysql7. Restart Hive services from Ambari.If the Hive metadata version is same as Hive 2.1.1000 in HDP 2.6, then follow steps 1 through 5.

When configuring LDAPS in HDP its common to see wrong certificates used or certificates without the correct chain.To ensure the correct chain of certificates is used when configuring LDAPS you can use openssl to read the certificate from the server and save it to a file. This file can them be imported into, for example, the Ambari truststore.echo -n | openssl s_client -connect <ad-server>:636 | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > /root/ldaps-cert.pem