Resources

Configure SSL for Ambari Server# On ambari-server host, take a back up of existing filesmkdir /root/ambari-ssl-backup; cp -r /var/lib/ambari-server/keys/* /root/ambari-ssl-backup/# Navigate to the folder where all certificates are stored.cd <dir># Extract certificates from p7b fileopenssl pkcs7 –in ambari-server-host-cert.p7b –print_certs > ambari-server-host-name.crt# Above .crt file contains full certificate chain, so separate server certificate from root chain. Copy the intermediate/root chain to ca.crt and delete intermediate/root chain from ambari-server-host-name.crt# Copy ca.crt, ambari-server-host-name.crt, ambari-server-host-privateKey.key files to /var/lib/ambari-server/keys/cp ca.crt /var/lib/ambari-server/keys/cp ambari-server-host-privateKey.key /var/lib/ambari-server/keys/cp ambari-server-host-name.crt /var/lib/ambari-server/keys/# Update password for the keystore in a file (Assuming it as ‘changeit’)echo "changeit" > pass.txt# Generate the keystore for ambar

This article describes how to enable Knox proxying for Zeppelin Notebook for Wire encrypted environment (i.e SSL is enabled for Knox and Zeppelin) Non Wire encrypted environments Configuring Knox proxying for Zeppelin Notebook in Wire encrypted Environments If you have already configured SSL for zeppelin, please proceed to section 2. If not, please read through section 1. Section 1 : Configuring SSL for Zeppelin Note : The steps mentioned in section 1 are just for example purpose, and the production setup may be different. Also the steps assume no client-side authentication. For client-side authentication, please follow the Zeppelin Component Guide in HDP release documents (HDP 2.6.1 Zeppelin Component Guide : https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.1/bk_zeppelin-component-guide/content/config-ssl-zepp.html ) Create keystore file, truststore file and certificates for each host on the cluster by following these steps Navigate to the directory where you wa

This is a beginner's guide for debugging a running NiFi instance. It covers two important aspects of debugging: adjusting the log levels of what is written to the NiFi application logs and attaching a graphical debugger to a running NiFi instance.RequirementsApache NiFi source code (this tutorial uses NiFi 1.3.0) If you have never build and run NiFi from source, follow the instructions in the NiFi Development Quick Start Guide: https://nifi.apache.org/quickstart.htmlIn this tutorial, {NIFI_ROOT} refers to the location of your NiFi source directory, i.e., your git clone or uncompressed source directory.IntelliJ IDEA (this tutorial was made using IntelliJ IDEA Community 2107.01)Available from: https://www.jetbrains.com/idea/downloadIf you are using other version of Apache NiFi or IntelliJ IDEA, the steps should be very similar though some of the details and example screen shots might vary slightly.Enabling Debug LoggingBy default, the log level for NiFi is set to INFO, meaning log lines

Component AlignmentIf you have ever wanted to line up the components on your canvas, you now have an easy way to do it. Just select the desired components, right-click to see the context menu and select “Align vertically” or “Align horizontally” depending on your desired result."Align vertically" example:“Align horizontally” example: Component LinkingHave you ever wanted to go directly to a process group instead of having to find your way through other process groups to get to it? NiFi 1.2 supports the use of a hyperlink to navigate directly to a component (process group, processor, etc.). For example, the flow shown in the following screenshot is in process group “PG 4” which is embedded within three other process groups: Note that the NiFi URL in the browser now contains component information, specifically the Process Group ID for "PG 4”. The URL can now be shared with other users, and when pasted in their browsers, will take them directly to that process group on their canvas.

NiFi 1.2.0 has a nice, new little feature that will surely please those who may spend a bit of time – for some, perhaps A LOT of time – getting their flow to line up perfectly. The background grid lines can help with this, but the process can still be quite tedious with many components. Now there is a quick, easy way.Select the components on the canvas you want to align. You can do this by either Shift+clicking each component until they are all selected, or while holding Shift, click and (fully) drag around all the components you want to align.Next, simply right-click on any selected component and choose the desired "Align..." option.The next image shows the result after choosing the Align vertically option.If components are different sizes, they will align according to their center points.And that's all there is to it!

Below are the steps to move Namenode from one machine to another via Ambari.These steps have been successfully tested for below version of Ambari and HDPAmbari - 2.2.2.XHDP - 2.2.X/2.3.X/2.4.X.Step 1: Ensure that all the HDFS Components are up and running.Step 2: Select HDFS Services à Click on Service Actions on the top right à Click on Move Namenode ..Step 3: Read the instructions carefully and click on Next .Step 4: Select new target for your Standby NN à Click Next ..Step 5: Review the changes and click on Deploy .Step 6: This wizard will stop required services, setup new NN + ZFKC, Disable HDFS on earlier Namenode and Start services on New Namenode. Once this step is done à Click Next ..Step 7: Follow the Manual steps given in ‘Manual Commands’ section..Step 8 – Confirm that you have performed all the manual steps and click on OK .Step 9 – Once confirmed, Ambari will delete earlier Standby NN and Start all the Services ...Please comment if you have any feedback/questions/suggest

This article will cover debugging MiNiFi C++. This will be the first of a series of debugging, memory leak detection, and profiling MiNiFi C++. In my development of MiNiFi C++ I spent the bulk of my time using common GNU tools for debugging and ensuring we don't have memory leaks. Profiling allows us to baseline. This guide will focus on debugging using the GNU Debugger, GDBDebugging MiNiFi C++ uses CMAKE to generate builds. A precondition for debugging C++ tools is including debug symbols into the executable. To do this you will need to specify a build type of DEBUG or RELWITHDEBINFO, both of which instruct CMAKE to include those symbols.$ cmake -DCMAKE_BUILD_TYPE=RelWithDebInfo ..$ cmake -DCMAKE_BUILD_TYPE=DEBUG .. When you do this, CMAKE will generate compilation code to include debug symbols (-g). When you build the project debug symbols will be included, and allow you to run GDB. GDB is the GNU debugger. When you've installed MiNiFi you can debug minifi in place by running:gdb .

Note that this is a re-post of the original article that was posted at https://blogs.apache.org/nifi/entry/record-oriented-data-with-nifi.INTRO - THE WHATApache NiFi is being used by many companies and organizations to power their data distribution needs. One of NiFi's strengths is that the framework is data agnostic. It doesn't care what type of data you are processing. There are processors for handling JSON, XML, CSV, Avro, images and video, and several other formats. There are also several general-purposes processors, such as RouteText and CompressContent. Data can be hundreds of bytes or many gigabytes. This makes NiFi a powerful tool for pulling data from external sources; routing, transforming, and aggregating it; and finally delivering it to its final destinations.Note that thisWhile this ability to handle any arbitrary data is incredibly powerful, we often see users working with record-oriented data. That is, large volumes of small "records," or "messages," or "events." Everyon

As I was reading over Pig mailing lists, I found a plugin that allows one to turn Eclipse IDE into a Pig editor. There are alternatives like Ambari Pig view and Hue but if you're comfortable using an IDE, then you will feel right at home with this plugin. Setup is fairly straightforward and surprisingly supports Pig from 0.11 to 0.16, which means you can even use it with the latest HDP 2.5 release! The best part, I didn't need to start a Sandbox, download a binary distribution of Pig or mess with any config files, once plugin is installed you can start scripting as if you were working on a cluster. It is still work in progress but feature list is impressive nonetheless. It supports the following features as of 9/13/16: supports Apache Pig Latin 0.11 - 0.16syntax highlightingopen declaration (F3) - for macros, defines, and UDF'sauto complete (ctrl+space) - defines, relations, reserved words, built in functionstoggle comment (ctrl+shift+c) - to comment/uncomment a block using --find refe

ENVIRONMENT HDP-2.6.0.3Ambari 2.5.0.3 SOLUTION1. Install R on each DN$ yum install R-devel libcurl-devel openssl-devel2. Run on each DN$ R> install.packages("knitr")3. Test R from CLI[root@dghdp255 ~]# R -e "print(1+1)"R version 3.3.3 (2017-03-06) -- "Another Canoe"Copyright (C) 2017 The R Foundation for Statistical ComputingPlatform: x86_64-redhat-linux-gnu (64-bit)R is free software and comes with ABSOLUTELY NO WARRANTY.You are welcome to redistribute it under certain conditions.Type 'license()' or 'licence()' for distribution details.Natural language support but running in an English localeR is a collaborative project with many contributors.Type 'contributors()' for more information and'citation()' on how to cite R or R packages in publications.Type 'demo()' for some demos, 'help()' for on-line help, or'help.start()' for an HTML browser interface to help.Type 'q()' to quit R.> print(1+1)[1] 2>>[root@dghdp255 ~]#4. Zeppelin UIa) spark2 configSPARK_HOME /usr/hdp/current/sp

Change the livy.spark.master to yarn cluster and add the following environment variable in the zeppelin-env from ambariexport PYSPARK_DRIVER_PYTHON=path_to_python2.7 export PYSPARK_PYTHON=path_to_python2.7 After this restart livy spark interpreter started to work.

Hi, I've created Support Portal account with my company email and also my client email (2 separate accounts).Is there any possible way to link between this 2 account so that I can open support ticket on behalf of my client? Your help on this is highly appreciated.

PROBLEM: HDP 2.5.3 with Ambari 2.4.2.0 and using Kerberos and Ranger for HBase authorization. We need grant pretty much ALL permissions to the 'default' namespace to every user so they can connect using sqlline.py.1;31mError: org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient permissions for user 'abc@NA.EXAMPLE.COM' (action=create) at org.apache.ranger.authorization.hbase.AuthorizationSession.publishResults(AuthorizationSession.java:261) at org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor.authorizeAccess(RangerAuthorizationCoprocessor.java:595) at org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor.requirePermission(RangerAuthorizationCoprocessor.java:664) at org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor.preCreateTable(RangerAuthorizationCoprocessor.java:769) at org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor.preCreateTable(RangerAuthorizationCoprocessor.java:496) at org.apache.hadoop.hb

PROBLEM: Phoenix ODBC driver strips out "hint" part of "Select" statements. ODBC driver logs looks like below :Mar2016:15:00.601 INFO 6380Statement::SQLSetStmtAttrW:Attribute: SQL_ATTR_MAX_ROWS (1)----Comments: the original query passed in has "hint"Mar2016:15:00.602 INFO 6380StatementState::InternalPrepare:Preparing query:select/*+ INDEX(c.more_xref_cad more_xref_acct_idx) */*from c.more_xref_cad where cad_acct_id =219980018Mar2016:15:00.602 DEBUG 6380RESTAction::HMDebugCallback:Infor type: CURLINFO_TEXT .....Mar2016:15:00.802 DEBUG 6380RESTAction::HMDebugCallback:Info data:Connected to localhost (127.0.0.1) port 8765(#1) Mar2016:15:00.802 DEBUG 6380RESTAction::HMDebugCallback:Infor type: CURLINFO_HEADER_OUT Mar2016:15:00.802 DEBUG 6380RESTAction::HMDebugCallback:Info data: POST / HTTP/1.1Host: localhost:8765Content-Type: application/octet-stream Accept: text/html, image/gif, image/jpeg,*; q=.2,*/*; q=.2 User-Agent: Phoenix ODBC Connection: keep-alive Content-Length: 160 Mar 20 16:15:

PROBLEM : Inner query does not recognize alias for the table and consider it as column family. Following exceptions are seen:0: jdbc:phoenix:localhost:2181> SELECT a.tgb_id FROM "APC:TGB_CURRENT_STATUS" a WHERE a.utc_date_epoch in (SELECT b.utc_date_epoch FROM "APC:TGB_STATUS" b WHERE a.tgb_id = b.tgb_id ORDER BY b.utc_date_epoch DESC limit 2) . . . . . . . . . . . . . . . > ; Error: ERROR 1001 (42I01): Undefined column family. familyName=A.null (state=42I01,code=1001) org.apache.phoenix.schema.ColumnFamilyNotFoundException: ERROR 1001 (42I01): Undefined column family. familyName=A.null at org.apache.phoenix.schema.PTableImpl.getColumnFamily(PTableImpl.java:724) at org.apache.phoenix.compile.FromCompiler$SingleTableColumnResolver.resolveColumn(FromCompiler.java:365)STEPS TO REPRODUCE :1. Create two tables from Phoenix with below schemaCREATE TABLE "${namespace}:TGB_STATUS" ( tgb_id integer not null, utc_date_epoch integer not null, tgb_name varchar(40), city_name varchar(25), tgb

PROBLEM: When doing a select of a relatively large table (a few thousand rows) some rows return partially missing.When increasing the filter to return those specific rows, the values appear as expected.STEPS TO REPRODUCE:1. Create a tableCREATE TABLE IF NOT EXISTS TEST ( BUCKET VARCHAR, TIMESTAMP_DATE TIMESTAMP, TIMESTAMP UNSIGNED_LONG NOT NULL, SRC VARCHAR, DST VARCHAR, ID VARCHAR, ION VARCHAR, IC BOOLEAN NOT NULL, MI UNSIGNED_LONG, AV UNSIGNED_LONG, MA UNSIGNED_LONG, CNT UNSIGNED_LONG, DUMMY VARCHAR CONSTRAINT pk PRIMARY KEY (BUCKET, TIMESTAMP DESC, SRC, DST, ID, ION, IC));2. Use a python script to generate a CSV with 5000 rowsfor i in xrange(5000): print "5SEC,2016-07-21 07:25:35.{i},146908593500{i},WWWWWWWW,AAA,BBBB,CCCCCCCC,false,{i}1181000,1788000{i},2497001{i},{i},aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa{i}".format(i=i)3. Bulk inserting the csv in the tab

PROBLEM : Customers create namespace and assign users "RX" permission , however users are not able to create tables in namespace and require "RWX" permission.CREATE TABLE example_datapipe.table_test1 ( id BIGINT not null primary key); Error: org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient permissions (user=g0123434@ABC.EXAMPLE.COM, scope=SYSTEM:CATALOG, family=0:SALT_BUCKETS, params=[table=SYSTEM:CATALOG,family=0:SALT_BUCKETS],action=WRITE) ROOT CAUSE : Yes this is an expected behavior. To create a Phoenix table, you MUST have the ability to create the HBase table and you MUST have the ability to write to the SYSTEM.CATALOG table.WORKAROUND : If you do not want normal users to have write access on system.catalog, try one of the following alternatives :- 1) Let an admin manually create the Phoenix tables for the unprivileged users.2) Temporarily grant them write access to the SYSTEM.CATALOG table and revoke later.REFERENCE: Internal discussion under EAR-5831

STEPS TO REPRODUCE:1. Setup table and local index:create table mytest (col1 varchar, col2 varchar constraint pk primary key (col1)); upsert into mytest values ('123','123'); upsert into mytest values ('456','456'); create local index myidx on mytest (col2); 2. Obtain current timestamp and prove rows can be seen:select cast(current_time() as bigint) from mytest limit 1; +--------------------------------------------+ | TO_BIGINT(TIME '2017-04-19 09:57:47.867') | +--------------------------------------------+ | 1492595867867 | +--------------------------------------------+ sqlline.py "aw25k-3:2181/hbase-secure;currentSCN=1492595867867" select * from mytest; +-------+-------+ | COL1 | COL2 | +-------+-------+ | 123 | 123 | | 456 | 456 | +-------+-------+ 3. Disconnect sqlline and reconnect without SCN. Drop the local index, insert more rows, recreate the index:sqlline.py drop index myidx on mytest; upsert into mytest values ('678','678'); upsert into mytest values ('456','654'); create loc

The steps below show how to configure authentication between your HDP cluster and WASB using the wasb connector.Note for Cloudbreak users: When you create a cluster with Cloudbreak, you can configure authentication with WASB on the "Add File System" page of the create cluster wizard. If you do this, you do not need to perform the steps below. If you have already created a cluster with Cloudbreak but did not perform WASB configuration on the"Add File System" page of the create cluster wizard, follow the steps below:Prerequisites1. The Azure prerequisite to configuring authentication is having a storage subscription and creating a storage account. For example:2. On the HDP side, you must use a version that supports the WASB integration. I am using HDP 2.6.1, which supports connecting to WASB using the WASB connector.StepsIn order to access data stored in your Azure blob storage account, you must configure your storage account access key in core-site.xml. The configuration property that y

Overview To control access, Azure uses Azure Active Directory (Azure AD), a multi-tenant cloud-based directory and identity management service. To learn more, refer to https://docs.microsoft.com/en-us/azure/active-directory/active-directory-whatis. In short, to configure authentication with ADLS using the client credential, you must register a new application with Active Directory service and then give your application access to your ADL account. After you've performed these steps, you can configure your core-site.xml. Note for Cloudbreak users: When you create a cluster with Cloudbreak, you can configure authentication with ADLS on the "Add File System" page of the create cluster wizard and then you must perform an additional step as described in Cloudbreak documentation. If you do this, you do not need to perform the steps below. If you have already created a cluster with Cloudbreak but did not perform ADLS configuration on the "Add File System" page of the create cluster wizard

Symptoms:Bundle capture will fail with the following error found in the hst-agent.log, during the anonymization process:ERROR anonymize.py:62 - Execution of script /usr/jdk64/jdk1.7.0_67/bin/java -Xmx2048m-Xms1024m -Dlog.file.name=anonymization.log -cp :/etc/hst/conf/:/usr/hdp/share/hst/hst-common/lib/* com.hortonworks.smartsense.anonymization.Main -m /var/lib/smartsense/hst-agent/data/tmp/06-01-17.io-a-00000000-c-00000000_hdp_0_2017-06-01_12-00-00 -c /etc/hst/conf/hst-agent.ini failed ERROR anonymize.py:63 - Execution of command returned 1. Exception in thread "main" java.lang.NullPointerException atcom.hortonworks.smartsense.anonymization.rules.RuleType.from(RuleType.java:47) atcom.hortonworks.smartsense.anonymization.rules.Rules.createRule(Rules.java:161)atcom.hortonworks.smartsense.anonymization.rules.Rules.createRules(Rules.java:148)atcom.hortonworks.smartsense.anonymization.rules.Rules.createRules(Rules.java:132)atcom.hortonworks.smartsense.anonymization.AnonymizerFactory.createA

We are excited to introduce the new Cloud Data Access guide for HDP 2.6.1. The goal of this guide is to provide information and steps required for configuring, using, securing, tuning performance, and troubleshooting access to the cloud storage services using HDP cloud storage connectors available for Amazon Web Services (Amazon S3) and Microsoft Azure (ADLS, WASB). To learn about the architecture of the cloud connectors, refer to Introducing the Cloud Storage Connectors. To get started with your chosen cloud storage service, refer to: Getting Started with Amazon S3Getting Started with ADLSGetting Started with WASBOnce you have configured authentication with the chosen cloud storage service, you can start working with the data. To get started, refer to: Accessing Cloud Data with HiveAccessing Cloud Data with SparkCopying Cloud Data with HadoopIf you have comments or suggestions, corrections or updates regarding our documentation, let us know on HCC. Help us continue to improve our docu

Unlike other services, Knox doesn't expose Java Heap settings via Ambari. Follow these steps to change default Heap settings for Knox:1. On the Knox node, login as root and go to this directory: /usr/hdp/current/knox-server/bin # cd /usr/hdp/current/knox-server/bin 2. Make a copy of the file that we are going to change next. # cp gateway.sh gateway.sh.backup.`date +%m%d%Y-%H-%M-%S`3. Open gateway.sh in text editor and change this line:APP_MEM_OPTS=""to this:APP_MEM_OPTS="-Xmx5g -XX:NewSize=3G -XX:MaxNewSize=3G -verbose:gc -XX:ParallelGCThreads=8 -XX:+UseConcMarkSweepGC -Xloggc:/var/log/knox/knox-gc.log -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps" Shown above are the example values for Knox Heap. This should be updated based on user's environment.General formula for Knox Heap size will be: Recommended KNOX Heap Size = (Webhdfs(ConcNumOfRequest*(replayBufferSize)) + HBase(ConcNumOfRequest*(replayBufferSize)) + Hive(ConcNumOfRequest*(replayBufferSize))) + 20 % for re

Hortonworks Cloud OverviewOverview Hortonworks Cloud (Hortonworks Connected Data cloud) is system for launching and managing a data lake on AWS. You can easily launch HDP clusters for analyzing and processing data. With Apache Hadoop. Use this document as supplement guide to existing official guide by Hortonworks. The doc will guide you in running HDP platform in your AWS environment.Prerequisites Before you launch hortonworks cloud controller make sure You have AWS account. (https://aws.amazon.com/)Amazon key pair to access EC2 instance to run HDP platform. (http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-key-pairs.html#having-ec2-create-your-key-pair)Hortonworks cloud supported in following regions for now.Us-east-1, us-west-2, eu-central-1, eu-west-1 and ap-northeast-1.Architecture OverviewCloud Controller is the primary component of hortonworks cloud. Cloud controller manages one or more HDP clusters on AWS. It runs on EC2 instance and used for launching and managing clus

I was recently involved with, quite possibly, the worst HBase performance debugging issue so far in my lifetime. The issue first arose with a generic problem statement: after X hours of processing, tasks accessing HBase begin to take over 10 times longer than prior. Upon restarting HBase, performance returned to expected levels. There were no obvious errors in HBase logs, HDFS logs, or the host's syslog.This problem would manifest itself on a near-constant period: every X hours after restart. It affected different types of client tasks (those reading and writing), and was not limited to a specific node or set of nodes. Strangely, despite all inspection of HBase logs and profiling information, HBase seemed to be functioning perfectly fine. Just, slower. This lead us to investigate numerous operating system configuration changes and monitoring, none of which completely described the circumstances and symptoms of the problem.After many long days of investigation and some JVM options, we s