Resources

The general perception that one needs to thoroughly know the code base for Understanding and Debugging a HDP service. Do appreciate the fact that HDP services including the execution engine (MR, TEZ, SPARK) are all JVM based processes. JVM along with the Operating Systems provides various knobs to know what the process is doing and performing at run-time. The steps mentioned below can be applied for understanding and debugging any JVM process in general.Lets take an example of how HiveServer2 works assuming one is not much acquainted or has a deep understanding of the code base. We are assuming one knows what the service does, but how it works internally the user is not aware of 1. Process Resource Usage Gives a complete overview of the usage pattern of CPU, memory by the process providing a quick insight of the health of the process2. How to figure out which all service the process interacts with ADD JMX parameters to the service allowing you to visualize whats happening with

Heap resigning and Garbage Collection tuning plays a central role in deciding how healthy and efficiently the cluster resources will be utilized. One of the biggest challenges is to fine tune the heap to make sure neither you are underutilized or over utilizing the resources. The following heap sizing has been made after a in depth analysis of the health of individual services. Do remember this is the base line you can add more heap to your resources depending on the kind of work load one executes. Key take away 1. All the services present in HDP are JVM based and all need appropriate heap sizing and GC tuning2. HDP and YARN are the base of all the services, hence make sure NN, DN, RM and NN are given sufficient heap. 3. Rule of thumb, heap sizing till 20 - 30 GB doesnt require any special tuning, anything above the value need fine tuning. <4. 99% of time your cluster is not working efficiently because the services are suffering GC and appear RED in the UI. 5. Most of the Servic

Below script is used to get all the Hive Databases and underlining tables or views or INDEX_TABLES in a cluster into a csv file. This helps in evaluating total counts for metrics or identifying any tables:beeline -u 'jdbc:hive2://zookeeper1:2181,zookeeper2:2181,zookeeper3:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2' --outputformat=csv2 -f table.hql > tableslist.csv In table.hql!tablesHope this helps you.

DB Visualizer is a popular free tool that allows developers to organize development tools for RDBMS development. With Apache Phoenix, that allows SQL like capability for Hbase, we can use DBVisualizer to connect to Apache Phoenix layer for HBase.Verified with following versions.DBVisualizer version 9.2.12hbase-client-1.1.2.2.3.2.0-2950.jarphoenix-4.4.0.2.3.2.0-2950-client.jarFirst Add Phoenix driver to DBVisualizer. In DBVisualizer, go to Tools->Driver Manager and add a new driver. Add both hbase-client and phoenix-client jar. This will add a new Phoenix driver.1. Connecting to Non-Kerberos clusterTo connect to a Non-Kerberos cluster, use jdbc:phoenix:<zookeeper host>:<zookeeper port>:<hbase_z_node> as connection string where hbase_z_node is :/hbase by default.2. Connecting to Kerberos cluster using cached ticketTo connect to a Kerberos cluster, a. add following files to DBVisualizer resources directory.hdfs-site.xmlhbase-site.xmlcore-site.xmlb. Copy krb5.conf file

The article provides a optimized mysql config (Usage of enterprise RDBMS is encouraged ) that can be used, in case mysql is used as the backed DB for Ambari, Hive and Ranger. The configuration is good enough to handle 1000 concurrent users on HS2 and Ambari. The config file my.cnf is generally located at /etc/my.cnfmy.cnf ******************************************************************************************************************************# For advice on how to change settings please see# http://dev.mysql.com/doc/refman/5.6/en/server-configuration-defaults.html[mysqld]## Remove leading # and set to the amount of RAM for the most important data# cache in MySQL. Start at 70% of total RAM for dedicated server, else 10%.# innodb_buffer_pool_size = 128M## Remove leading # to turn on a very important data integrity option: logging# changes to the binary log between backups.# log_bin## Remove leading # to set options mainly useful for reporting servers.# The server defaults are fas

SYMPTOMS: Ambari configures "hdfs - nofile hdfs - nofile 128000" in /etc/security/limits.d/hdfs.conf but when the datanode (or any other) process is started by Ambari, it still runs with a limit of only 8129 open files:sudo grep open /proc/19608/limits Max open files 8192 8192 files (19608 is the pid of the datanode process in this case). As a consequence we get millions of entries in datanode logs resulting in a lot of disk / IO bottleneck.WARN mortbay.log (Slf4jLog.java:warn(89)) - EXCEPTION java.io.IOException: Too many open files ROOT CAUSE: Ambari should check uncommented "session required pam_limits.so" line in /etc/pam.d/su and /etc/pam.d/sudo , otherwise ulimit values from /etc/security/limits.d/hdfs.conf (+ yarn.conf, hive.conf, ams.conf) won't be applied. This is a known issue reported in internal BUG-38892. SOLUTION: Not yet fixed. WORKAROUND: Uncomment the line "session required pam_limits.so " in /etc/pam.d/su on each node and restart services.Log a case with HWX support t

Starting with HDP 2.6.5, it is possible to configure access to Google Cloud Storage from HDP via the gs cloud storage connector. There are two ways to do this:- You can use Cloudbreak to deploy clusters on GCP and help you configure access to GCS from your Cloudbreak-managed HDP cluster. The steps involve (1) Setting up a service account on Google Cloud and (2) Providing the Service Account Email Address to Cloudbreak in the cluster create wizard on the Cloud Storage page. For detailed steps, see https://docs.hortonworks.com/HDPDocuments/Cloudbreak/Cloudbreak-2.7.1/content/gcp-data/index.html - It is also possible to configure access to GCS by setting up a service account, manually placing the service account key on all nodes of the cluster, and setting related properties in Ambari. For detailed steps, see https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.0.0/bk_cloud-data-access/content/authentication-gcp.htmlNote that this capability is currently only available as technical previe

Default location for spark event log history is hdfs:///spark-history (Spark) and hdfs:///spark2-history/ (spark2). This will help to debug spark-history page load issue or if you have huge count of event log files you can archive it by creating the new active location. Following are the steps to change this default location. 1. Create new directory on hdfs for e.g $hdfs dfs -mkdir /spark2-history_new $hdfs dfs -chown spark:hadoop /spark2-history_new 2. Login to Amabri==>Spark==>config. 3. Update following parameters with new path "hdfs:///spark2-history_new/ " as follow. 4. Save the configuration. 5. Restart Spark service to enable new changes. 6. Run spark job, your new event log file will get save in new location. Same you can view using the spark history UI.

Introduction - Here is a small demo how NiFi can help you monitor and alert on YARN Application failure. - Here you can view the screen recording that Demonstrates how it works!Prerequisite - Make sure you have your HDP cluster/Sandbox up and running. - NiFi_0.6.1//HDF_1.2 is available up and running.Steps: 1. Assuming you have NiFi UI Available, lets drop GetHTTP processor to pull data from YARN REST API: Configure Processor with URL given as below, which pulls all Applications in Killed and Failed state. node1 is my Resource Manager: http://node1:8088/ws/v1/cluster/apps?states=KILLED,FAILED Lets schedule the processor to run only every 10sec so that you don’t query too often. 2. As the Rest call outputs the application details in Json format, lets use a SplitJson processor to separate individualapplication details. Provide “JsonPath Expression” value as “$.apps.app” in the configuration. 3. Connect GetHTTP to SplitJson for success relation and auto terminate r

HDF DevOps It's become enough of an ask that I had to post to answer it. The ask is something like this, "What's with all this UI stuff, I want devops, automation, command line et al." So did I in 2002. It's nice to see everything and have a nice diagram on a website without any extra tools or ssh. Okay that didn't convince anyone. So here is a proper DevOps solution for you. Option 1: REST The full documentation for the NiFi REST API is here: https://nifi.apache.org/docs/nifi-docs/rest-api/ The follow is some examples I have accessed via CURL (if you have security, you will need to account for that, see the specifications. curl http://hw13125.local:8080/nifi-api/resources curl http://hw13125.local:8080/nifi-api/tenants/user-groups curl http://hw13125.local:8080/nifi-api/tenants/users curl http://hw13125.local:8080/nifi-api/flow/about curl http://hw13125.local:8080/nifi-api/flow/banners curl http://princeton1.field.hortonworks.com:8080/nifi-api/flow/bulletin-board curl http://hw13125.l

PROBLEM I have a non kerberized cluster. I did apply https://community.hortonworks.com/articles/81910/how-to-enable-user-impersonation-for-jdbc-interpre.html however the jdbc intepreter is still not impersonated. NOTE: From HDP 2.6.2 (Zeppelin 0.7.2) JDBC interpreter on non-kerberised cluster is impersonated by having the following property added into Zeppelin UI -> Interpreter -> JDBC config: hive.proxy.user.property=hive.server2.proxy.user SOLUTION1. Go to Zeppelin UI -> Interpreter The configuration for JDBC hive interpreter should look like 2. Edit JDBC interpreter Remove properties hive.user and hive.password and save the changes. So, now the configuration looks like 3. Go to Zeppelin UI -> Credential Add the credentials for the user like Entity: jdbc.jdbc Username: <username> Password: <password> 4. Run the query Go to your notebook and run the jdbc query for hive. In RM UI this query is now running by YOU

SymptomsWhen attempting to create a Cloudera Altus cluster in AWS, it fails with INTERNAL_CM_CMD_FAIL DiagnosisCM Deployment is failing at the creation / deployment of CM services due to incorrect or unexpected DNS hostnames of the AWS EC2 instances created for Cloudera Altus.Cloudera Altus requires the options DNS hostnames and DNS resolution are enabled (set to yes) within the AWS VPC settings SolutionSet and confirm DNS hostnames and DNS resolution are set to yes in the AWS VPC used for Cloudera Altus.Details for configuring AWS VPCs with DNS is available here:https://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/vpc-dns.htmlOnce the AWS VPC DNS settings of the Cloudera Altus Environment are enabled, ensure to delete the previous cluster creation attempt's EC2 instances, both within a Cloudera Altus console, as well as through the AWS EC2 interfaceRetry creating a new Cloudera Altus cluster with the updated settings

If you have recently upgraded from HDP-2.6.1 to HDP-2.6.2, you may end up facing this issue.I have seen this for a Kerberized Cluster and doAs=true in Hive.You may have carried over same configurations for Zeppelin, Interpreter, Hive from 2.6.1 to 2.6.2 yet a simple query like'show databases' may result in this error stacktrace shown below:org.apache.zeppelin.interpreter.InterpreterException: Error in doAs at org.apache.zeppelin.jdbc.JDBCInterpreter.getConnection(JDBCInterpreter.java:415) at org.apache.zeppelin.jdbc.JDBCInterpreter.executeSql(JDBCInterpreter.java:633) at org.apache.zeppelin.jdbc.JDBCInterpreter.interpret(JDBCInterpreter.java:733) at org.apache.zeppelin.interpreter.LazyOpenInterpreter.interpret(LazyOpenInterpreter.java:101) at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:502) at org.apache.zeppelin.scheduler.Job.run(Job.java:175) at org.apache.zeppelin.scheduler.ParallelScheduler$JobRunner.run(ParallelSc

Objective:Accessing REST API of Kerberized NiFI Cluster using Bearer TokenPrerequisites:CURL or Postman Installed on Your Laptopkinit - Successful for sales1 userFirefox browser with proper config (network.negotiate)Inside Firefox -> Open New Tab & Type -> about:configFilter by networknetwork.negotiate-auth.trusted-uris -> .us-west-2.compute.internalnetwork.negotiate-auth.delegation-uris ->.us-west2.compute.internalSPNEGO NiFi - Successful when you hit NiFi Home Page URL (sales1).Next steps/Plan:On Nifi Home Page, Enable Developer Tools & Monitor Network Logs to get the Bearer Token (under current-user and Authorization part of Request Headers)Lets use the Bearer Token we got to populate the below commands.CURL option:curl ‘

What log shows when a cluster was shutdown?

Recently I come across this command line Ldap connection tool, which is very useful while setting Ranger UserSync. This tool collects minimal input from admin about the ldap/AD server and discovers various properties for users and groups in order to successfully pull only targeted Users and Groups from the Ldap/AD server.DetailsLdap Connection check tool is a command line tool and can be run on any machine where Java is installed and Ldap/AD server access is available. This tool can be used to discover not only user sync related properties but also authentication properties if needed. It also generates ambari configuration properties as well as install properties for manual installation. User is also provided an option to discover both the user and group properties together or separately. As part of the tool, a template properties file is provided for the user to update the values specific to the setup.Tool usageIn order to learn details on how to use the tool, the tool also provides

Environment: HDP on AWS EC2 instances.Problem: When trying to execute a hive ddl script, we get the error file does not exist. Upon checking the file exists and also has the read write privileges. Root Cause: The directory where the script exists does not have read write permission to the user.Solution: Open the directory permission to include the necessary read, write permission to the user.Following is a screenshot that shows the problem and solution used to fix it.

Use S3 as storage for Zeppelin Notebooks.Step 1. Use external storage to point to S3 bucket in Cloudbreak advance options. This will use S3access profile and AWS credentials. CB takes care of that set up. In addition to thatStep 2 Change these 3 inzeppelin-env.shexport ZEPPELIN_NOTEBOOK_S3_BUCKET=yourBucketNameexport ZEPPELIN_NOTEBOOK_S3_ENDPOINT="http://s3.amazonaws.com/yourBucketName" export ZEPPELIN_NOTEBOOK_S3_USER=adminStep 3 Change this 1 inzeppelin-site.xmlPoint zeppelin.notebook.storage to org.apache.zeppelin.notebook.repo.S3NotebookRepoDetailed Steps below1.Complete AWS pre-requisites2.Create AWS credentials in Cloudbreak3.Launch HDP cluster on AWS using Cloudbreak.4.Enable Zeppelin Storage on S3.3. Launch HDP Cluster on AWS using Cloudbreak.Not all the screenshots are included, only capturing screenshots that focus on some key Advance features that enable the required Zeppelin Storage. a.Use the Advance tab on Cloudbreak.b.Cloudbreak uses AWS credentials that will provide the

Short Description:This article walks you through how to deploy a custom parser on kerberized cluster in Metron .ArticleWhen adding a net new data source to Metron, the first step is to create kafka topic 'sensor_topic' to push the events from the new telemetry data source into Metron. The second step is to configure Metron to parse the telemetry data source so that downstream processing can be done on it. This wiki doc will walk you through how to perform both of these steps. In this article I have added steps to perform the same operation on a kerberised cluster. On kerberised cluster we need to add the required ACL's to metron user and parser group for the created topic. Also we need to provide the storm authorization while deploying the topology. 1. Switch to the kafka user . su kafka 2. create sensor topic /usr/hdp/current/kafka-broker/bin/kafka-topics.sh --zookeeper $ZOOKEEPER --create --topic sensor_topic_name --partitions 1 --replication-factor 1 3. Add required ACL to metr

Being a secured cluster, the data transfer protocol of DataNode does not use the RPC framework of Hadoop, DataNode must authenticate itself by using privileged ports which are specified by dfs.datanode.address and dfs.datanode.http.address. This authentication is based on the assumption that the attacker won’t be able to get root privileges. When you execute hdfs datanode command as root, server process binds privileged port at first, then drops privilege and runs as the user account specified by HADOOP_SECURE_DN_USER. You can identify that it is a kerberized cluster from the ps -ef output, in the end it says SecureDatanode. "XX:+PrintGCDateStamps -Xms2048m -Xmx2048m -Dhadoop.security.logger=INFO,DRFAS -Dhdfs.audit.logger=INFO,DRFAAUDIT -Dhadoop.security.logger=INFO,RFAS org.apache.hadoop.hdfs.server.datanode.SecureDataNodeStarter" ps -ef|grep datanode ( have cut short the output for space purpose)root 2454 1 0 Jul27 ? 00:00:00 jsvc.exec -Dproc_datanode -outfile /var/log/hadoop/hdfs/j

Topic: IoT Edge Processing with Apache NiFi and MiniFi and Multiple Deep Learning LibrariesPart 1: Multiple Devices with DataKeywords: Deep Learning On The Edge, GPS Ingestion, Sense-Hat and Rainbow Hat Sensor Ingest, WebCam Image IngestIn preparation for my talk at Strata in NYC, I am updating my IoT demos for more devices, more data types and more actions.I have three streams coming from each device including web camera images. When we are sending data from a MiniFi agent we need to define a port on an Apache NiFi server/cluster to receive it.So I design my MiniFi flow in the Apache NiFi UI (pretty soon there will be a special designer for this). You then highlight everything there and hit Create Template. You can then export it and convert to config.yml. Again, this process will be automated and connected with the NiFi Registry very shortly to make this less clicks.This is an example. When you connect to it in your flow you design in Apache NiFi UI you will connect to

Ingesting Blockchain data from btc.com and blockchain.com(.info).Like everything in Apache NiFi, this is trivial easy to ingest all these different feeds, process them, route them and store them for SQL access.API:blockchain.comhttps://blockchain.info/latestblockhttps://api.blockchain.info/charts/transactions-per-second?timespan=5weeks&rollingAverage=8hours&format=jsonhttps://blockchain.info/blocks/BTC.com?format=jsonhttps://api.blockchain.info/pools?timespan=5dayshttps://api.blockchain.info/statshttps://blockchain.info/tickerhttps://blockchain.info/tobtc?currency=USD&value=10000Blocks For Today

A quick example NiFi flow to demonstrate sending logs to syslog from NiFi.Steps:1. Wait for new compressed files (gz) to land in folder (/tmp/nifi/syslog/src/*.gz) with log lines.2. Decompress the file3. Split the file into individual lines4. Send data to syslogNiFi Template : Download the following NiFi template and import the flow into your NiFi instance. Instructions to import a NiFi template is available here - Importing NiFi templatessend-to-syslog.xmlSend to Syslog Template Walkthrough :The Flow is broken down into 3 sectionsSection 1 : Test Data GenerationSection 2 : Processing (decompress files and send to local syslog)Key HighlightsExtractText Processor :We are using the extract text processor to get the content of the flowfile into the flowfile attribute "msg". We are grabbing the entire content of the flowfile into "msg" by using the regular expression (.*). You may get only specific fields by providing appropriate regular expression here.PutSyslog processor :We are using t

This article mainly focuses mainly on what all supported options available in User sync and how they can be used to target specific use cases. For some common use cases examples and how ranger can be configured in those cases, please refer to Configuring Ranger UserSync with AD/LDAPUser based search without enable group sync: This configuration can be used in cases where customers want to sync users based on a OU or a user search filter and get all the groups the users belong to.This case is supported only when the user object has the group membership attributes like “memberof” or “ismemberof”. This option is available only in Full sync case (i.e., when Incremental sync is disabled in Ranger User sync configuration)User based search with enable group sync: This configuration can be used in cases where - customers want to sync users based on a OU or a user search filter and want to limit the groups that these user belong to based on the OU or a group search filter.Group membership attri

Short Description:In this tutorial we will install the browser connected version of RServer on to the HDP Sandbox.ArticleInstalling RStudio on HDP Sandbox IntroductionRStudio is an Integrated Development Environment (IDE) for the R language which includes a direct code execution console, as well as tools for plotting and debugging, you can find more information about the RStudio features here.RStudio is used as the primary tool in the Predicting Airline Delays using SparkR Hortonworks tutorial in which you will learn to train and analyze Machine Learning models to predict Airline delays.PrerequisitesDownload the latest HDP SandboxComplete the Learning the Ropes of the Hortonworks Sandbox tutorial1. SSH on to the SandboxUse the following command to SSH on to the Sandbox as root user:ssh root@sandbox-hdp.hortonworks.com -p 2222NOTE: If this is your first time signing on the default password is hadoop.2. Begin installationOn CentOS 7, the base OS for the Sandbox, R is available through th