Community Articles

Below are the steps to move Namenode from one machine to another via Ambari.These steps have been successfully tested for below version of Ambari and HDPAmbari - 2.2.2.XHDP - 2.2.X/2.3.X/2.4.X.Step 1: Ensure that all the HDFS Components are up and running.Step 2: Select HDFS Services à Click on Service Actions on the top right à Click on Move Namenode ..Step 3: Read the instructions carefully and click on Next .Step 4: Select new target for your Standby NN à Click Next ..Step 5: Review the changes and click on Deploy .Step 6: This wizard will stop required services, setup new NN + ZFKC, Disable HDFS on earlier Namenode and Start services on New Namenode. Once this step is done à Click Next ..Step 7: Follow the Manual steps given in ‘Manual Commands’ section..Step 8 – Confirm that you have performed all the manual steps and click on OK .Step 9 – Once confirmed, Ambari will delete earlier Standby NN and Start all the Services ...Please comment if you have any feedback/questions/suggest

This article will cover debugging MiNiFi C++. This will be the first of a series of debugging, memory leak detection, and profiling MiNiFi C++. In my development of MiNiFi C++ I spent the bulk of my time using common GNU tools for debugging and ensuring we don't have memory leaks. Profiling allows us to baseline. This guide will focus on debugging using the GNU Debugger, GDBDebugging MiNiFi C++ uses CMAKE to generate builds. A precondition for debugging C++ tools is including debug symbols into the executable. To do this you will need to specify a build type of DEBUG or RELWITHDEBINFO, both of which instruct CMAKE to include those symbols.$ cmake -DCMAKE_BUILD_TYPE=RelWithDebInfo ..$ cmake -DCMAKE_BUILD_TYPE=DEBUG .. When you do this, CMAKE will generate compilation code to include debug symbols (-g). When you build the project debug symbols will be included, and allow you to run GDB. GDB is the GNU debugger. When you've installed MiNiFi you can debug minifi in place by running:gdb .

Note that this is a re-post of the original article that was posted at https://blogs.apache.org/nifi/entry/record-oriented-data-with-nifi.INTRO - THE WHATApache NiFi is being used by many companies and organizations to power their data distribution needs. One of NiFi's strengths is that the framework is data agnostic. It doesn't care what type of data you are processing. There are processors for handling JSON, XML, CSV, Avro, images and video, and several other formats. There are also several general-purposes processors, such as RouteText and CompressContent. Data can be hundreds of bytes or many gigabytes. This makes NiFi a powerful tool for pulling data from external sources; routing, transforming, and aggregating it; and finally delivering it to its final destinations.Note that thisWhile this ability to handle any arbitrary data is incredibly powerful, we often see users working with record-oriented data. That is, large volumes of small "records," or "messages," or "events." Everyon

As I was reading over Pig mailing lists, I found a plugin that allows one to turn Eclipse IDE into a Pig editor. There are alternatives like Ambari Pig view and Hue but if you're comfortable using an IDE, then you will feel right at home with this plugin. Setup is fairly straightforward and surprisingly supports Pig from 0.11 to 0.16, which means you can even use it with the latest HDP 2.5 release! The best part, I didn't need to start a Sandbox, download a binary distribution of Pig or mess with any config files, once plugin is installed you can start scripting as if you were working on a cluster. It is still work in progress but feature list is impressive nonetheless. It supports the following features as of 9/13/16: supports Apache Pig Latin 0.11 - 0.16syntax highlightingopen declaration (F3) - for macros, defines, and UDF'sauto complete (ctrl+space) - defines, relations, reserved words, built in functionstoggle comment (ctrl+shift+c) - to comment/uncomment a block using --find refe

ENVIRONMENT HDP-2.6.0.3Ambari 2.5.0.3 SOLUTION1. Install R on each DN$ yum install R-devel libcurl-devel openssl-devel2. Run on each DN$ R> install.packages("knitr")3. Test R from CLI[root@dghdp255 ~]# R -e "print(1+1)"R version 3.3.3 (2017-03-06) -- "Another Canoe"Copyright (C) 2017 The R Foundation for Statistical ComputingPlatform: x86_64-redhat-linux-gnu (64-bit)R is free software and comes with ABSOLUTELY NO WARRANTY.You are welcome to redistribute it under certain conditions.Type 'license()' or 'licence()' for distribution details.Natural language support but running in an English localeR is a collaborative project with many contributors.Type 'contributors()' for more information and'citation()' on how to cite R or R packages in publications.Type 'demo()' for some demos, 'help()' for on-line help, or'help.start()' for an HTML browser interface to help.Type 'q()' to quit R.> print(1+1)[1] 2>>[root@dghdp255 ~]#4. Zeppelin UIa) spark2 configSPARK_HOME /usr/hdp/current/sp

Change the livy.spark.master to yarn cluster and add the following environment variable in the zeppelin-env from ambariexport PYSPARK_DRIVER_PYTHON=path_to_python2.7 export PYSPARK_PYTHON=path_to_python2.7 After this restart livy spark interpreter started to work.

PROBLEM: HDP 2.5.3 with Ambari 2.4.2.0 and using Kerberos and Ranger for HBase authorization. We need grant pretty much ALL permissions to the 'default' namespace to every user so they can connect using sqlline.py.1;31mError: org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient permissions for user 'abc@NA.EXAMPLE.COM' (action=create) at org.apache.ranger.authorization.hbase.AuthorizationSession.publishResults(AuthorizationSession.java:261) at org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor.authorizeAccess(RangerAuthorizationCoprocessor.java:595) at org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor.requirePermission(RangerAuthorizationCoprocessor.java:664) at org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor.preCreateTable(RangerAuthorizationCoprocessor.java:769) at org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor.preCreateTable(RangerAuthorizationCoprocessor.java:496) at org.apache.hadoop.hb

PROBLEM: Phoenix ODBC driver strips out "hint" part of "Select" statements. ODBC driver logs looks like below :Mar2016:15:00.601 INFO 6380Statement::SQLSetStmtAttrW:Attribute: SQL_ATTR_MAX_ROWS (1)----Comments: the original query passed in has "hint"Mar2016:15:00.602 INFO 6380StatementState::InternalPrepare:Preparing query:select/*+ INDEX(c.more_xref_cad more_xref_acct_idx) */*from c.more_xref_cad where cad_acct_id =219980018Mar2016:15:00.602 DEBUG 6380RESTAction::HMDebugCallback:Infor type: CURLINFO_TEXT .....Mar2016:15:00.802 DEBUG 6380RESTAction::HMDebugCallback:Info data:Connected to localhost (127.0.0.1) port 8765(#1) Mar2016:15:00.802 DEBUG 6380RESTAction::HMDebugCallback:Infor type: CURLINFO_HEADER_OUT Mar2016:15:00.802 DEBUG 6380RESTAction::HMDebugCallback:Info data: POST / HTTP/1.1Host: localhost:8765Content-Type: application/octet-stream Accept: text/html, image/gif, image/jpeg,*; q=.2,*/*; q=.2 User-Agent: Phoenix ODBC Connection: keep-alive Content-Length: 160 Mar 20 16:15:

PROBLEM : Inner query does not recognize alias for the table and consider it as column family. Following exceptions are seen:0: jdbc:phoenix:localhost:2181> SELECT a.tgb_id FROM "APC:TGB_CURRENT_STATUS" a WHERE a.utc_date_epoch in (SELECT b.utc_date_epoch FROM "APC:TGB_STATUS" b WHERE a.tgb_id = b.tgb_id ORDER BY b.utc_date_epoch DESC limit 2) . . . . . . . . . . . . . . . > ; Error: ERROR 1001 (42I01): Undefined column family. familyName=A.null (state=42I01,code=1001) org.apache.phoenix.schema.ColumnFamilyNotFoundException: ERROR 1001 (42I01): Undefined column family. familyName=A.null at org.apache.phoenix.schema.PTableImpl.getColumnFamily(PTableImpl.java:724) at org.apache.phoenix.compile.FromCompiler$SingleTableColumnResolver.resolveColumn(FromCompiler.java:365)STEPS TO REPRODUCE :1. Create two tables from Phoenix with below schemaCREATE TABLE "${namespace}:TGB_STATUS" ( tgb_id integer not null, utc_date_epoch integer not null, tgb_name varchar(40), city_name varchar(25), tgb

PROBLEM: When doing a select of a relatively large table (a few thousand rows) some rows return partially missing.When increasing the filter to return those specific rows, the values appear as expected.STEPS TO REPRODUCE:1. Create a tableCREATE TABLE IF NOT EXISTS TEST ( BUCKET VARCHAR, TIMESTAMP_DATE TIMESTAMP, TIMESTAMP UNSIGNED_LONG NOT NULL, SRC VARCHAR, DST VARCHAR, ID VARCHAR, ION VARCHAR, IC BOOLEAN NOT NULL, MI UNSIGNED_LONG, AV UNSIGNED_LONG, MA UNSIGNED_LONG, CNT UNSIGNED_LONG, DUMMY VARCHAR CONSTRAINT pk PRIMARY KEY (BUCKET, TIMESTAMP DESC, SRC, DST, ID, ION, IC));2. Use a python script to generate a CSV with 5000 rowsfor i in xrange(5000): print "5SEC,2016-07-21 07:25:35.{i},146908593500{i},WWWWWWWW,AAA,BBBB,CCCCCCCC,false,{i}1181000,1788000{i},2497001{i},{i},aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa{i}".format(i=i)3. Bulk inserting the csv in the tab

PROBLEM : Customers create namespace and assign users "RX" permission , however users are not able to create tables in namespace and require "RWX" permission.CREATE TABLE example_datapipe.table_test1 ( id BIGINT not null primary key); Error: org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient permissions (user=g0123434@ABC.EXAMPLE.COM, scope=SYSTEM:CATALOG, family=0:SALT_BUCKETS, params=[table=SYSTEM:CATALOG,family=0:SALT_BUCKETS],action=WRITE) ROOT CAUSE : Yes this is an expected behavior. To create a Phoenix table, you MUST have the ability to create the HBase table and you MUST have the ability to write to the SYSTEM.CATALOG table.WORKAROUND : If you do not want normal users to have write access on system.catalog, try one of the following alternatives :- 1) Let an admin manually create the Phoenix tables for the unprivileged users.2) Temporarily grant them write access to the SYSTEM.CATALOG table and revoke later.REFERENCE: Internal discussion under EAR-5831

STEPS TO REPRODUCE:1. Setup table and local index:create table mytest (col1 varchar, col2 varchar constraint pk primary key (col1)); upsert into mytest values ('123','123'); upsert into mytest values ('456','456'); create local index myidx on mytest (col2); 2. Obtain current timestamp and prove rows can be seen:select cast(current_time() as bigint) from mytest limit 1; +--------------------------------------------+ | TO_BIGINT(TIME '2017-04-19 09:57:47.867') | +--------------------------------------------+ | 1492595867867 | +--------------------------------------------+ sqlline.py "aw25k-3:2181/hbase-secure;currentSCN=1492595867867" select * from mytest; +-------+-------+ | COL1 | COL2 | +-------+-------+ | 123 | 123 | | 456 | 456 | +-------+-------+ 3. Disconnect sqlline and reconnect without SCN. Drop the local index, insert more rows, recreate the index:sqlline.py drop index myidx on mytest; upsert into mytest values ('678','678'); upsert into mytest values ('456','654'); create loc

The steps below show how to configure authentication between your HDP cluster and WASB using the wasb connector.Note for Cloudbreak users: When you create a cluster with Cloudbreak, you can configure authentication with WASB on the "Add File System" page of the create cluster wizard. If you do this, you do not need to perform the steps below. If you have already created a cluster with Cloudbreak but did not perform WASB configuration on the"Add File System" page of the create cluster wizard, follow the steps below:Prerequisites1. The Azure prerequisite to configuring authentication is having a storage subscription and creating a storage account. For example:2. On the HDP side, you must use a version that supports the WASB integration. I am using HDP 2.6.1, which supports connecting to WASB using the WASB connector.StepsIn order to access data stored in your Azure blob storage account, you must configure your storage account access key in core-site.xml. The configuration property that y

Overview To control access, Azure uses Azure Active Directory (Azure AD), a multi-tenant cloud-based directory and identity management service. To learn more, refer to https://docs.microsoft.com/en-us/azure/active-directory/active-directory-whatis. In short, to configure authentication with ADLS using the client credential, you must register a new application with Active Directory service and then give your application access to your ADL account. After you've performed these steps, you can configure your core-site.xml. Note for Cloudbreak users: When you create a cluster with Cloudbreak, you can configure authentication with ADLS on the "Add File System" page of the create cluster wizard and then you must perform an additional step as described in Cloudbreak documentation. If you do this, you do not need to perform the steps below. If you have already created a cluster with Cloudbreak but did not perform ADLS configuration on the "Add File System" page of the create cluster wizard

Symptoms:Bundle capture will fail with the following error found in the hst-agent.log, during the anonymization process:ERROR anonymize.py:62 - Execution of script /usr/jdk64/jdk1.7.0_67/bin/java -Xmx2048m-Xms1024m -Dlog.file.name=anonymization.log -cp :/etc/hst/conf/:/usr/hdp/share/hst/hst-common/lib/* com.hortonworks.smartsense.anonymization.Main -m /var/lib/smartsense/hst-agent/data/tmp/06-01-17.io-a-00000000-c-00000000_hdp_0_2017-06-01_12-00-00 -c /etc/hst/conf/hst-agent.ini failed ERROR anonymize.py:63 - Execution of command returned 1. Exception in thread "main" java.lang.NullPointerException atcom.hortonworks.smartsense.anonymization.rules.RuleType.from(RuleType.java:47) atcom.hortonworks.smartsense.anonymization.rules.Rules.createRule(Rules.java:161)atcom.hortonworks.smartsense.anonymization.rules.Rules.createRules(Rules.java:148)atcom.hortonworks.smartsense.anonymization.rules.Rules.createRules(Rules.java:132)atcom.hortonworks.smartsense.anonymization.AnonymizerFactory.createA

We are excited to introduce the new Cloud Data Access guide for HDP 2.6.1. The goal of this guide is to provide information and steps required for configuring, using, securing, tuning performance, and troubleshooting access to the cloud storage services using HDP cloud storage connectors available for Amazon Web Services (Amazon S3) and Microsoft Azure (ADLS, WASB). To learn about the architecture of the cloud connectors, refer to Introducing the Cloud Storage Connectors. To get started with your chosen cloud storage service, refer to: Getting Started with Amazon S3Getting Started with ADLSGetting Started with WASBOnce you have configured authentication with the chosen cloud storage service, you can start working with the data. To get started, refer to: Accessing Cloud Data with HiveAccessing Cloud Data with SparkCopying Cloud Data with HadoopIf you have comments or suggestions, corrections or updates regarding our documentation, let us know on HCC. Help us continue to improve our docu

Unlike other services, Knox doesn't expose Java Heap settings via Ambari. Follow these steps to change default Heap settings for Knox:1. On the Knox node, login as root and go to this directory: /usr/hdp/current/knox-server/bin # cd /usr/hdp/current/knox-server/bin 2. Make a copy of the file that we are going to change next. # cp gateway.sh gateway.sh.backup.`date +%m%d%Y-%H-%M-%S`3. Open gateway.sh in text editor and change this line:APP_MEM_OPTS=""to this:APP_MEM_OPTS="-Xmx5g -XX:NewSize=3G -XX:MaxNewSize=3G -verbose:gc -XX:ParallelGCThreads=8 -XX:+UseConcMarkSweepGC -Xloggc:/var/log/knox/knox-gc.log -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps" Shown above are the example values for Knox Heap. This should be updated based on user's environment.General formula for Knox Heap size will be: Recommended KNOX Heap Size = (Webhdfs(ConcNumOfRequest*(replayBufferSize)) + HBase(ConcNumOfRequest*(replayBufferSize)) + Hive(ConcNumOfRequest*(replayBufferSize))) + 20 % for re

Hortonworks Cloud OverviewOverview Hortonworks Cloud (Hortonworks Connected Data cloud) is system for launching and managing a data lake on AWS. You can easily launch HDP clusters for analyzing and processing data. With Apache Hadoop. Use this document as supplement guide to existing official guide by Hortonworks. The doc will guide you in running HDP platform in your AWS environment.Prerequisites Before you launch hortonworks cloud controller make sure You have AWS account. (https://aws.amazon.com/)Amazon key pair to access EC2 instance to run HDP platform. (http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-key-pairs.html#having-ec2-create-your-key-pair)Hortonworks cloud supported in following regions for now.Us-east-1, us-west-2, eu-central-1, eu-west-1 and ap-northeast-1.Architecture OverviewCloud Controller is the primary component of hortonworks cloud. Cloud controller manages one or more HDP clusters on AWS. It runs on EC2 instance and used for launching and managing clus

I was recently involved with, quite possibly, the worst HBase performance debugging issue so far in my lifetime. The issue first arose with a generic problem statement: after X hours of processing, tasks accessing HBase begin to take over 10 times longer than prior. Upon restarting HBase, performance returned to expected levels. There were no obvious errors in HBase logs, HDFS logs, or the host's syslog.This problem would manifest itself on a near-constant period: every X hours after restart. It affected different types of client tasks (those reading and writing), and was not limited to a specific node or set of nodes. Strangely, despite all inspection of HBase logs and profiling information, HBase seemed to be functioning perfectly fine. Just, slower. This lead us to investigate numerous operating system configuration changes and monitoring, none of which completely described the circumstances and symptoms of the problem.After many long days of investigation and some JVM options, we s

With Apache Spark 2.1, SparkR supports virtual environment. With VirtualEnv when you deploy Spark on YARN, each SparkR job will have its own library directory which binds to the executor’s YARN container. All necessary third-party libraries are installed into that directory. After the Spark job finishes, the directory is deleted and there is no interference with other Spark job’s environment.SparkR now supports the following two scenarios:1) Internet enabled cluster allows packages to be installed (say from CRAN) on the executors so that R packages can be used in the tasks.2) For isolated cluster, the driver first downloads the required packages or download packages ahead of time and then add them to the cluster. Spark will distribute the packages to executors and we can install them when that executor runs.Both of these scenarios install R packages in a separate local directory on executor for each user, and will not pollute that executor’s native R environment. This directory will b

In this tutorial, we will set up the Hortonworks Data Cloud for AWS 1.16 (released in June 2017) using the advanced template, which requires you to configure a VPC and an RDS instance and database prior to launching the cloud controller.We will go through the following steps: Subscribing to HDCloud servicesLaunching an RDS instanceLaunching the cloud controller using the advanced templateAccessing the cloud controller UIThis tutorial assumes no prior experience with AWS. If you encounter errors while performing the steps, refer to the Troubleshooting documentation. Let's get started!Prerequisites The prerequisites for this tutorial are:1. You have an AWS account.2. You know in which AWS region you want to launch.3. You have an SSH key pair in the selected region.If you need help with the steps required to meet these prerequisites, refer to the following post:How to set up Hortonworks Data Cloud (HDCloud) for AWSSubscribe to HDCloud ServicesIn order to use the product, you need to subsc

This tutorial has been successfully tried on HDP-2.4.0.0 and Ambari 2.2.1.0.I have my HDP Cluster Kerberized and Ambari has been configured for SSL.Note - Steps are same for Ambari with or without SSL..Please follow below steps for Configuring Hive View on Kerberized HDP Cluster. .Step 1 - Please configure your Ambari Server for Kerberos with the steps mentioned in below article. Please follow steps 1 to 5.https://community.hortonworks.com/articles/40635/configure-tez-view-for-kerberized-hdp-cluster.html.Step 2 - Please add below properties to core-site.xml via Ambari UI and restart required services..Note - If you are running Ambari Server as root user then add below properties hadoop.proxyuser.root.groups=*hadoop.proxyuser.root.hosts=*.If you are running Ambari server as non-root user then please add below properties in core-site.xmlhadoop.proxyuser.<ambari-server-user>.groups=*hadoop.proxyuser.<ambari-server-user>.hosts=*Please replace <ambari-server-user> with use

Issue:On Adding Oozie service to cluster, sometimes its possible user could see an error with simple message Failed to create new war file as below and on top of it Oozie server wouldn't start. INFO: Adding extension: /usr/hdp/current/oozie-server/libext/mysql-jdbc-driver.jar Failed: creating new Oozie WARSolution:Oozie deployment script is trying to unzip the existing Oozie war file and then zip it back up and place it in a sub directory of /tmp/... . This unzip and zip process doubles the storage space required for the Oozie war file. Make sure there is enough /tmp storage for the above process to take place, logs wouldn't provide this info.

Working With S3 Compatible Data Stores (and handling single source failure)With the major outage of S3 in my region, I decided I needed to have an alternative file store. I found a great open source server called Minio that I run on a miniPC running Centos 7. We could also use this solution for connecting to other S3 compatible stores such as RiakCS and Google Cloud Storage. I like to remain cloud and location neutral.In Apache NiFi, it's really easy. You can have two sources and two destinations, instead of just your regular AWS S3, you can have one for AWS S3 and one for another. Or you can use the second as a disaster recovery data backup. Since my Minio box is local, I can store data locally. It's pretty affordable to get a few terabytes connected to a small Linux box to hold some backups. With Apache NiFi, you have queues to buffer a potentially slower ingest/egress. Minio Setupwget https://dl.minio.io/server/minio/release/linux-amd64/minio chmod 755 minionohup ./min

{"authentication":{"class": "org.apache.solr.security.KerberosPlugin", "blockUnknown": true},"authorization":{"class":"solr.RuleBasedAuthorizationPlugin","permissions":[{"name":"security-edit","role":"admin"},{"name":"collection-admin-edit", "role" : "dev"},{"name":"read", "role" : "read_coll_data","path":"/select"}, {"name":"update", "role" : "put_coll_data"}],"user-role":{"a1@EXAMPLE.COM":"admin", "a2@EXAMPLE.COM":"dev", "a3@EXAMPLE.COM":"read_coll_data", "a4@EXAMPLE.COM":"put_coll_data", "a5@EXAMPLE.COM": "read_coll_data","rangeradmin@EXAMPLE.COM":["read_coll_data", "put_coll_data"]}}}This file has two parts, the authentication part and the authorization part. The authentication part stores information about the class being used for authentication. The authorization part is not related to Basic authentication, but is a separate authorization plugin designed to support fine-grained user access control. Each role is comprised of one or more permissions which define what the user is al