Community Articles

In kafka there are three types of communication : 1. Between brokers2. Between client and broker. 3. Broker to zookeeperIn order to communicate in kerberos enabled cluster one needs to authenticate itself. So when broker will try to communicate with other broker in the cluster it will need to authenticate first. Same for the clients communicating to the brokers. In Kafka, JAAS files is used for authentication. Lets first understand what's there in JAAS file. In kafka there are two JAAS files : 1. kafka_jaas.conf 2. kafka_client_jaas.conf. Let's discuss about kafka_jaas.conf. This file will be used for authentication when a broker in a cluster tries to communicate with other brokers in cluster. Take a look at the content and understand its purpose : KafkaServer { com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true keyTab="/etc/security/keytabs/kafka.service.keytab" storeKey=true useTicketCache=false serviceName="kafka" principa

jline-sqlline-mysql-connector.tar.gzStep1:Download the attached tar file and untar ittar -xvf jline_sqlline__mysql_connector.tar Step2:Syntax: [root]# java -Djava.ext.dirs=/path/to/jline_sqlline__mysql_connector/ sqlline.SqlLineExample:[root@mi3 sqlline]# java -Djava.ext.dirs=/root/sqlline/jline-1.0/ sqlline.SqlLinesqlline version 1.0.2 by Marc Prud'hommeaux Step3:Syntax:sqlline> !connect jdbc:mysql://<mysql hostname>:<port>/<db> <username> <password>Example:sqlline> !connect jdbc:mysql://hostabc:3306/hue hue hueConnecting to jdbc:mysql://mi1.openstacklocal:3306/hueConnected to: MySQL (version 5.1.73)Driver: MySQL-AB JDBC Driver (version mysql-connector-java-5.1.17-SNAPSHOT ( Revision: ${bzr.revision-id} ))Autocommit status: trueTransaction isolation: TRANSACTION_REPEATABLE_READ0: jdbc:mysql://mi1.openstacklocal:3306/hue>

We have created a quickstart guide to walk you through step by step going from Google Cloud Platform (GCP) account to provisioning big data workloads on GCP using Hortonworks Cloudbreak. In this overview, we walk through the high level tasks to be performed and point you to the documentation with the step by step instructions.Overview There are 4 main steps involved to get started with GCP.Cloudbreak GCP Prerequisites: Setting up your GCP Environment to provision Cloudbreak and HDP/HDF workloads. Install Cloudbreak on GCP: Setting up Cloudbreak on GCP.Create Cloudbreak Credentials: Setting up your credentials in Cloudbreak to deploy HDP/HDF workloads on GCP.Creating a Cluster on GCP: Step by step walkthrough creating a big data workload on GCP.Cloudbreak GCP PrerequisitesBefore we install Cloudbreak we need to setup your working environment and GCP account for Cloudbreak. In this step, we will setup the following prerequisites.Install Google Cloud SDK: Setting up the Google Cloud SDK a

Introduction & Context There is a reason why I spent my life studying and working in computer science: understanding a computer's psychology is usually fairly straight forward. Indeed, when presented with a specific input, computer programs tend to respond in a very predictable way, as opposed to our fellow human beings. Of course, this observation goes out of the window as our algorithms become increasingly more complex and capable of learning. Regardless, as much as I love computer science, I always had a keen interest in human sciences. Personality psychology is a fascinating subject that has seen its ups and downs as any science topic. At the center of personality psychology reside the big five personality traits: Openness to Experience Conscientiousness Extraversion Agreeableness Neuroticism (or Emotional Stability) This taxonomy was determined by applying statistical models to personality surveys, essentially clustering results of surveys of

Currently Ambari (as of 2.1.2.1) does not support Postgres HA.

What are Ambari Blueprints ?Ambari Blueprints are definition of your HDP cluster in “JSON” format, it contents information about all the hosts in your cluster, their components, mapping of stack components with each hosts or hostgroups and other cool stuff. Using Blueprints we can call Ambari APIs to completely automate HDP installation process. Interesting stuff, isn’t it ?Lets get started with single node cluster installation. Below are the steps to setup single-node HDP cluster with Ambari Blueprints..Note - For Ambari 2.6.X onwards, we will have to register VDF to register internal repository, or else Ambari will pick up latest version of HDP and use the public repos. please see below document for more information. For Ambari version less than 2.6.X, this guide will work without any modifications.Document - https://docs.hortonworks.com/HDPDocuments/Ambari-2.6.0.0/bk_ambari-release-notes/content/ambari_relnotes-2.6.0.0-behavioral-changes.html.Step 1: Install Ambari server using step

The general perception that one needs to thoroughly know the code base for Understanding and Debugging a HDP service. Do appreciate the fact that HDP services including the execution engine (MR, TEZ, SPARK) are all JVM based processes. JVM along with the Operating Systems provides various knobs to know what the process is doing and performing at run-time. The steps mentioned below can be applied for understanding and debugging any JVM process in general.Lets take an example of how HiveServer2 works assuming one is not much acquainted or has a deep understanding of the code base. We are assuming one knows what the service does, but how it works internally the user is not aware of 1. Process Resource Usage Gives a complete overview of the usage pattern of CPU, memory by the process providing a quick insight of the health of the process2. How to figure out which all service the process interacts with ADD JMX parameters to the service allowing you to visualize whats happening with

Heap resigning and Garbage Collection tuning plays a central role in deciding how healthy and efficiently the cluster resources will be utilized. One of the biggest challenges is to fine tune the heap to make sure neither you are underutilized or over utilizing the resources. The following heap sizing has been made after a in depth analysis of the health of individual services. Do remember this is the base line you can add more heap to your resources depending on the kind of work load one executes. Key take away 1. All the services present in HDP are JVM based and all need appropriate heap sizing and GC tuning2. HDP and YARN are the base of all the services, hence make sure NN, DN, RM and NN are given sufficient heap. 3. Rule of thumb, heap sizing till 20 - 30 GB doesnt require any special tuning, anything above the value need fine tuning. <4. 99% of time your cluster is not working efficiently because the services are suffering GC and appear RED in the UI. 5. Most of the Servic

Below script is used to get all the Hive Databases and underlining tables or views or INDEX_TABLES in a cluster into a csv file. This helps in evaluating total counts for metrics or identifying any tables:beeline -u 'jdbc:hive2://zookeeper1:2181,zookeeper2:2181,zookeeper3:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2' --outputformat=csv2 -f table.hql > tableslist.csv In table.hql!tablesHope this helps you.

DB Visualizer is a popular free tool that allows developers to organize development tools for RDBMS development. With Apache Phoenix, that allows SQL like capability for Hbase, we can use DBVisualizer to connect to Apache Phoenix layer for HBase.Verified with following versions.DBVisualizer version 9.2.12hbase-client-1.1.2.2.3.2.0-2950.jarphoenix-4.4.0.2.3.2.0-2950-client.jarFirst Add Phoenix driver to DBVisualizer. In DBVisualizer, go to Tools->Driver Manager and add a new driver. Add both hbase-client and phoenix-client jar. This will add a new Phoenix driver.1. Connecting to Non-Kerberos clusterTo connect to a Non-Kerberos cluster, use jdbc:phoenix:<zookeeper host>:<zookeeper port>:<hbase_z_node> as connection string where hbase_z_node is :/hbase by default.2. Connecting to Kerberos cluster using cached ticketTo connect to a Kerberos cluster, a. add following files to DBVisualizer resources directory.hdfs-site.xmlhbase-site.xmlcore-site.xmlb. Copy krb5.conf file

The article provides a optimized mysql config (Usage of enterprise RDBMS is encouraged ) that can be used, in case mysql is used as the backed DB for Ambari, Hive and Ranger. The configuration is good enough to handle 1000 concurrent users on HS2 and Ambari. The config file my.cnf is generally located at /etc/my.cnfmy.cnf ******************************************************************************************************************************# For advice on how to change settings please see# http://dev.mysql.com/doc/refman/5.6/en/server-configuration-defaults.html[mysqld]## Remove leading # and set to the amount of RAM for the most important data# cache in MySQL. Start at 70% of total RAM for dedicated server, else 10%.# innodb_buffer_pool_size = 128M## Remove leading # to turn on a very important data integrity option: logging# changes to the binary log between backups.# log_bin## Remove leading # to set options mainly useful for reporting servers.# The server defaults are fas

SYMPTOMS: Ambari configures "hdfs - nofile hdfs - nofile 128000" in /etc/security/limits.d/hdfs.conf but when the datanode (or any other) process is started by Ambari, it still runs with a limit of only 8129 open files:sudo grep open /proc/19608/limits Max open files 8192 8192 files (19608 is the pid of the datanode process in this case). As a consequence we get millions of entries in datanode logs resulting in a lot of disk / IO bottleneck.WARN mortbay.log (Slf4jLog.java:warn(89)) - EXCEPTION java.io.IOException: Too many open files ROOT CAUSE: Ambari should check uncommented "session required pam_limits.so" line in /etc/pam.d/su and /etc/pam.d/sudo , otherwise ulimit values from /etc/security/limits.d/hdfs.conf (+ yarn.conf, hive.conf, ams.conf) won't be applied. This is a known issue reported in internal BUG-38892. SOLUTION: Not yet fixed. WORKAROUND: Uncomment the line "session required pam_limits.so " in /etc/pam.d/su on each node and restart services.Log a case with HWX support t

Starting with HDP 2.6.5, it is possible to configure access to Google Cloud Storage from HDP via the gs cloud storage connector. There are two ways to do this:- You can use Cloudbreak to deploy clusters on GCP and help you configure access to GCS from your Cloudbreak-managed HDP cluster. The steps involve (1) Setting up a service account on Google Cloud and (2) Providing the Service Account Email Address to Cloudbreak in the cluster create wizard on the Cloud Storage page. For detailed steps, see https://docs.hortonworks.com/HDPDocuments/Cloudbreak/Cloudbreak-2.7.1/content/gcp-data/index.html - It is also possible to configure access to GCS by setting up a service account, manually placing the service account key on all nodes of the cluster, and setting related properties in Ambari. For detailed steps, see https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.0.0/bk_cloud-data-access/content/authentication-gcp.htmlNote that this capability is currently only available as technical previe

Default location for spark event log history is hdfs:///spark-history (Spark) and hdfs:///spark2-history/ (spark2). This will help to debug spark-history page load issue or if you have huge count of event log files you can archive it by creating the new active location. Following are the steps to change this default location. 1. Create new directory on hdfs for e.g $hdfs dfs -mkdir /spark2-history_new $hdfs dfs -chown spark:hadoop /spark2-history_new 2. Login to Amabri==>Spark==>config. 3. Update following parameters with new path "hdfs:///spark2-history_new/ " as follow. 4. Save the configuration. 5. Restart Spark service to enable new changes. 6. Run spark job, your new event log file will get save in new location. Same you can view using the spark history UI.

Introduction - Here is a small demo how NiFi can help you monitor and alert on YARN Application failure. - Here you can view the screen recording that Demonstrates how it works!Prerequisite - Make sure you have your HDP cluster/Sandbox up and running. - NiFi_0.6.1//HDF_1.2 is available up and running.Steps: 1. Assuming you have NiFi UI Available, lets drop GetHTTP processor to pull data from YARN REST API: Configure Processor with URL given as below, which pulls all Applications in Killed and Failed state. node1 is my Resource Manager: http://node1:8088/ws/v1/cluster/apps?states=KILLED,FAILED Lets schedule the processor to run only every 10sec so that you don’t query too often. 2. As the Rest call outputs the application details in Json format, lets use a SplitJson processor to separate individualapplication details. Provide “JsonPath Expression” value as “$.apps.app” in the configuration. 3. Connect GetHTTP to SplitJson for success relation and auto terminate r

HDF DevOps It's become enough of an ask that I had to post to answer it. The ask is something like this, "What's with all this UI stuff, I want devops, automation, command line et al." So did I in 2002. It's nice to see everything and have a nice diagram on a website without any extra tools or ssh. Okay that didn't convince anyone. So here is a proper DevOps solution for you. Option 1: REST The full documentation for the NiFi REST API is here: https://nifi.apache.org/docs/nifi-docs/rest-api/ The follow is some examples I have accessed via CURL (if you have security, you will need to account for that, see the specifications. curl http://hw13125.local:8080/nifi-api/resources curl http://hw13125.local:8080/nifi-api/tenants/user-groups curl http://hw13125.local:8080/nifi-api/tenants/users curl http://hw13125.local:8080/nifi-api/flow/about curl http://hw13125.local:8080/nifi-api/flow/banners curl http://princeton1.field.hortonworks.com:8080/nifi-api/flow/bulletin-board curl http://hw13125.l

PROBLEM I have a non kerberized cluster. I did apply https://community.hortonworks.com/articles/81910/how-to-enable-user-impersonation-for-jdbc-interpre.html however the jdbc intepreter is still not impersonated. NOTE: From HDP 2.6.2 (Zeppelin 0.7.2) JDBC interpreter on non-kerberised cluster is impersonated by having the following property added into Zeppelin UI -> Interpreter -> JDBC config: hive.proxy.user.property=hive.server2.proxy.user SOLUTION1. Go to Zeppelin UI -> Interpreter The configuration for JDBC hive interpreter should look like 2. Edit JDBC interpreter Remove properties hive.user and hive.password and save the changes. So, now the configuration looks like 3. Go to Zeppelin UI -> Credential Add the credentials for the user like Entity: jdbc.jdbc Username: <username> Password: <password> 4. Run the query Go to your notebook and run the jdbc query for hive. In RM UI this query is now running by YOU

SymptomsWhen attempting to create a Cloudera Altus cluster in AWS, it fails with INTERNAL_CM_CMD_FAIL DiagnosisCM Deployment is failing at the creation / deployment of CM services due to incorrect or unexpected DNS hostnames of the AWS EC2 instances created for Cloudera Altus.Cloudera Altus requires the options DNS hostnames and DNS resolution are enabled (set to yes) within the AWS VPC settings SolutionSet and confirm DNS hostnames and DNS resolution are set to yes in the AWS VPC used for Cloudera Altus.Details for configuring AWS VPCs with DNS is available here:https://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/vpc-dns.htmlOnce the AWS VPC DNS settings of the Cloudera Altus Environment are enabled, ensure to delete the previous cluster creation attempt's EC2 instances, both within a Cloudera Altus console, as well as through the AWS EC2 interfaceRetry creating a new Cloudera Altus cluster with the updated settings

If you have recently upgraded from HDP-2.6.1 to HDP-2.6.2, you may end up facing this issue.I have seen this for a Kerberized Cluster and doAs=true in Hive.You may have carried over same configurations for Zeppelin, Interpreter, Hive from 2.6.1 to 2.6.2 yet a simple query like'show databases' may result in this error stacktrace shown below:org.apache.zeppelin.interpreter.InterpreterException: Error in doAs at org.apache.zeppelin.jdbc.JDBCInterpreter.getConnection(JDBCInterpreter.java:415) at org.apache.zeppelin.jdbc.JDBCInterpreter.executeSql(JDBCInterpreter.java:633) at org.apache.zeppelin.jdbc.JDBCInterpreter.interpret(JDBCInterpreter.java:733) at org.apache.zeppelin.interpreter.LazyOpenInterpreter.interpret(LazyOpenInterpreter.java:101) at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:502) at org.apache.zeppelin.scheduler.Job.run(Job.java:175) at org.apache.zeppelin.scheduler.ParallelScheduler$JobRunner.run(ParallelSc

Objective:Accessing REST API of Kerberized NiFI Cluster using Bearer TokenPrerequisites:CURL or Postman Installed on Your Laptopkinit - Successful for sales1 userFirefox browser with proper config (network.negotiate)Inside Firefox -> Open New Tab & Type -> about:configFilter by networknetwork.negotiate-auth.trusted-uris -> .us-west-2.compute.internalnetwork.negotiate-auth.delegation-uris ->.us-west2.compute.internalSPNEGO NiFi - Successful when you hit NiFi Home Page URL (sales1).Next steps/Plan:On Nifi Home Page, Enable Developer Tools & Monitor Network Logs to get the Bearer Token (under current-user and Authorization part of Request Headers)Lets use the Bearer Token we got to populate the below commands.CURL option:curl ‘

Recently I come across this command line Ldap connection tool, which is very useful while setting Ranger UserSync. This tool collects minimal input from admin about the ldap/AD server and discovers various properties for users and groups in order to successfully pull only targeted Users and Groups from the Ldap/AD server.DetailsLdap Connection check tool is a command line tool and can be run on any machine where Java is installed and Ldap/AD server access is available. This tool can be used to discover not only user sync related properties but also authentication properties if needed. It also generates ambari configuration properties as well as install properties for manual installation. User is also provided an option to discover both the user and group properties together or separately. As part of the tool, a template properties file is provided for the user to update the values specific to the setup.Tool usageIn order to learn details on how to use the tool, the tool also provides

Environment: HDP on AWS EC2 instances.Problem: When trying to execute a hive ddl script, we get the error file does not exist. Upon checking the file exists and also has the read write privileges. Root Cause: The directory where the script exists does not have read write permission to the user.Solution: Open the directory permission to include the necessary read, write permission to the user.Following is a screenshot that shows the problem and solution used to fix it.

Use S3 as storage for Zeppelin Notebooks.Step 1. Use external storage to point to S3 bucket in Cloudbreak advance options. This will use S3access profile and AWS credentials. CB takes care of that set up. In addition to thatStep 2 Change these 3 inzeppelin-env.shexport ZEPPELIN_NOTEBOOK_S3_BUCKET=yourBucketNameexport ZEPPELIN_NOTEBOOK_S3_ENDPOINT="http://s3.amazonaws.com/yourBucketName" export ZEPPELIN_NOTEBOOK_S3_USER=adminStep 3 Change this 1 inzeppelin-site.xmlPoint zeppelin.notebook.storage to org.apache.zeppelin.notebook.repo.S3NotebookRepoDetailed Steps below1.Complete AWS pre-requisites2.Create AWS credentials in Cloudbreak3.Launch HDP cluster on AWS using Cloudbreak.4.Enable Zeppelin Storage on S3.3. Launch HDP Cluster on AWS using Cloudbreak.Not all the screenshots are included, only capturing screenshots that focus on some key Advance features that enable the required Zeppelin Storage. a.Use the Advance tab on Cloudbreak.b.Cloudbreak uses AWS credentials that will provide the

Short Description:This article walks you through how to deploy a custom parser on kerberized cluster in Metron .ArticleWhen adding a net new data source to Metron, the first step is to create kafka topic 'sensor_topic' to push the events from the new telemetry data source into Metron. The second step is to configure Metron to parse the telemetry data source so that downstream processing can be done on it. This wiki doc will walk you through how to perform both of these steps. In this article I have added steps to perform the same operation on a kerberised cluster. On kerberised cluster we need to add the required ACL's to metron user and parser group for the created topic. Also we need to provide the storm authorization while deploying the topology. 1. Switch to the kafka user . su kafka 2. create sensor topic /usr/hdp/current/kafka-broker/bin/kafka-topics.sh --zookeeper $ZOOKEEPER --create --topic sensor_topic_name --partitions 1 --replication-factor 1 3. Add required ACL to metr

Being a secured cluster, the data transfer protocol of DataNode does not use the RPC framework of Hadoop, DataNode must authenticate itself by using privileged ports which are specified by dfs.datanode.address and dfs.datanode.http.address. This authentication is based on the assumption that the attacker won’t be able to get root privileges. When you execute hdfs datanode command as root, server process binds privileged port at first, then drops privilege and runs as the user account specified by HADOOP_SECURE_DN_USER. You can identify that it is a kerberized cluster from the ps -ef output, in the end it says SecureDatanode. "XX:+PrintGCDateStamps -Xms2048m -Xmx2048m -Dhadoop.security.logger=INFO,DRFAS -Dhdfs.audit.logger=INFO,DRFAAUDIT -Dhadoop.security.logger=INFO,RFAS org.apache.hadoop.hdfs.server.datanode.SecureDataNodeStarter" ps -ef|grep datanode ( have cut short the output for space purpose)root 2454 1 0 Jul27 ? 00:00:00 jsvc.exec -Dproc_datanode -outfile /var/log/hadoop/hdfs/j