Resources

Mandatory Pre-requisiteUpgrade JDK from 1.7 to 1.8If the step is missed it will result in to below errors. Nodemanager will fail to start if spark shuffle service is enabled."java.lang.UnsupportedClassVersionError: org/apache/spark/network/yarn/YarnShuffleService : Unsupported major.minor version 52.0" Hive Metastore will fail to start."Exception in thread "main" java.lang.UnsupportedClassVersionError: org/apache/hadoop/hive/ql/log/NullAppender : Unsupported major.minor version 52.0" Service checks will fail. YarnHiveFinalize upgrade will fail and cluster have to be downgraded.

Summary Because patients visit many doctors, trends in their ailments and complaints may be difficult to identify. The steps in this article will help you address exactly this problem by creating a TagCloud of the most frequent complaints per patient. Below is a sample: We will generate random HL7 MDM^T02 (v2.3) messages that contain a doctor's note about a fake patient and that patient's fake complaint to their doctor. Apache NiFi will be used to parse these messages and send them to Apache Solr. Finally Banana is used to create the visual dashboard. In the middle of the dashboard is a TagCloud where the more frequently mentioned symptoms for a selected patient appear larger than others. Because this project relies on randomly generated data, some interesting results are possible. In this case, I got lucky and all the symptoms seem related to the patient's most frequent complaint: Morning Alcohol Drinking. The list of all possible symptoms comes from Google searches. Summary of step

This is for HDP 2.5 only. If you are seeing the same error HDP 2.6, there could be something else that has failed before this stage. Please check the full log.   After enabling Hive LLAP, it fails to start with: ERROR impl.LlapZookeeperRegistryImpl: Unable to start curator PathChildrenCache. Exception: {}org.apache.zookeeper.KeeperException$InvalidACLException: KeeperErrorCode = InvalidACL for /llap-sasl/user-hive at org.apache.zookeeper.KeeperException.create(KeeperException.java:121) ~[zookeeper-3.4.6.2.5.0.0-1245.jar:3.4.6-1245--1] at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) ~[zookeeper-3.4.6.2.5.0.0-1245.jar:3.4.6-1245--1] at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783) ~[zookeeper-3.4.6.2.5.0.0-1245.jar:3.4.6-1245--1] at org.apache.curator.utils.ZKPaths.mkdirs(ZKPaths.java:232) ~[curator-client-2.7.1.jar:?] at org.apache.curator.utils.EnsurePath$InitialHelper$1.call(EnsurePath.java:148) ~[curator-client-2.7.1.jar:?] Steps to fix: 1. /

This article provides an overview of various debugging aspects of the system when using LLAP - how are things run, where logs are located, what monitoring options are available, etc. See the parent article for architecture overview and specific issue resolution.HS2 when using LLAP When using LLAP (“Hive interactive”), Ambari starts a separate HS2 (HS2 Interactive). This HS2 is not connected to the other HS2, and has a separate connection string, machine (or ports), configuration and logs . HS2 interactive and LLAP configs in Ambari are split similarly to regular Hive configs, but have “-interactive-“ in their name. HS2 interactive logs are called “hiveServer2Interactive” and are separate from regular HS2 logs.YARN apps when using LLAP; getting query logs Ambari starts LLAP as a YARN app that is shared by everyone using the cluster. It should have N+1 containers, where N is the number of LLAP daemons, and there’s slider AM. LLAP YARN app is called “llap0” by default:Tez sessio

The Apache Ambari project is aimed at making Hadoop management simpler by developing software for provisioning, managing, and monitoring Apache Hadoop clusters. Ambari provides an intuitive, easy-to-use Hadoop management web UI backed by its RESTful APIs.Ambari enables System Administrators to:Provision a Hadoop ClusterAmbari provides a step-by-step wizard for installing Hadoop services across any number of hosts.Ambari handles configuration of Hadoop services for the cluster.Manage a Hadoop ClusterAmbari provides central management for starting, stopping, and reconfiguring Hadoop services across the entire cluster.Monitor a Hadoop ClusterAmbari provides a dashboard for monitoring health and status of the Hadoop cluster.Ambari leverages Ambari Metrics System for metrics collection.Ambari leverages Ambari Alert Framework for system alerting and will notify you when your attention is needed (e.g., a node goes down, remaining disk space is low, etc).Ambari enables Application Developers a

I am an IT Professional with 11 years of exp. I started off my career as a .Net Developer/ SQL database analyst and an administrator. After 3 years, I got side-lined and moved into portfolio management. However my database foundation is strong. I am planning to bounce back into IT development and I see Big Data Analytics is booming currently. Can anyone please suggest me where should I start?  Do you recommend taking training courses with cloud era in order to eventually build an career in this area? if so what would you recommend as I see there are wide variety and I am confused where to begin.

OverviewWe create Apache Hive tables for analytics automagically from DDL generated by Apache NiFi.Hortonworks SAM Run TimeThe results of this produces HBase and Druid analytics.SAM has produced two Druid Datasources that we can slice up and explore in superset.In Apache Zeppelin, we can run our Hive DDL (listed below) and run all of our queries including joining our two patient data tables. We can sort and create some basic graphs.All of this data is available to you in your favorite BI tools.Generated Apache Hive Tables from ORC Files (DDL)CREATE EXTERNAL TABLE IF NOT EXISTS patientdata (PID_SSNNumberPatient INT, email STRING, gender STRING, ip_address STRING, drug_provider STRING, icd9 STRING, icd9_description STRING, icd9P_proc STRING, icd9_proc_description STRING, user_agent STRING, drug_used STRING) STORED AS ORC LOCATION '/patientdata';CREATE EXTERNAL TABLE IF NOT EXISTS ccda (problemSectionact_02observationproblemStatuscodecodeSystemName STRING, vitalSignsSectionorganizerobser

Executor memory size and map join memory for LLAP This article is intended for understanding map join memory sizing and related configurations specific to LLAP. For detailed information about LLAP sizing and setup, please refer to LLAP Sizing and Setup Executor memory size is shared among several components like join hash table loading, map-side aggregation hash tables, PTF operator buffering, shuffle and sort buffers. We recommend 4Gb for a general case (per LLAP Sizing and Setup), configuring less memory can slow down some queries (e.g. hash joins will turn into sort merge joins) and configuring more memory can be wasteful or lead to GC pressure (although more memory can provide more room for hash joins, it can adversely affect query performance when all/many executors load huge hash tables resulting in severe GC pauses). LLAP adds 2 new changes for hash table loaders for map joins Oversubscription of memory (from other executors) to provide more room for in-memory hash tables a

OverviewIn this section of HL7 processing, we look at what to do next with HL7 data now that it has been converted into Big Data friendly AVRO with a schema attached.We can easily combine that data with our other patientdata that came from MQTT. We do a simple JOIN on SSN and send this combined data to Kafka with a new schema and to Druid for analytics.We also Route some of the data to HDFS and HBase for storage and another feed to Druid for analytics.S.A.M. Flow OverviewI have a Simple Hortonworks Streaming Analytics Manager application that ingests two Kafka feeds (one for PatientData and one for HL7 Converted to AVRO by Apache NiFi). I join those two records together on a key and push them to Kafka and to Druid.Our Environment For Running SAM, Just Click what you needI added Druid, Apache HBase, HDFS, Apache Hive, Apache Kafka, Apache Storm and Apache Zookeeper. I am using all except Apache Hive.We can merge together the Kafka feeds sent from Apache NiFi. One feed is of pat

In this article, we’ll discuss tuning repository settings to achieve the highest possible throughput in MiNiFi C++. We’ll begin by benchmarking the SSD that we have. Based on reported numbers, the SSD in this 15” Macbook Pro is rated at 676 MB/s sequential write and 728 MB/s sequential read. Far from the highest, but our goal with this post is to achieve the highest possible throughput with the greatest guarantees. We’ll try different variations with and without volatile stores. Our benchmark shows that we are achieving near optimal speeds across various runs. We’ll create a MiNiFi C++ processor named Throughput Measure [1] that will be placed at the end of our graph to measure bytes output and measure the simple output from our flow. In our case, we’ll generate a flow file and simply measure the output. This won’t give us a completely accurate measure; however, it will be useful for tuning our setup. The first thing we need to be cognizant of is that we need to have enough space in o

Step 1: Collect HL7 Health Records Python to Send JSON Data to MQTT (Data Generated by https://www.mockaroo.com/) import paho.mqtt.client as mqttimport json# MQTTclient = mqtt.Client()client.connect("localhost", 14162, 60)row = [{"PID_SSNNumberPatient":823456789,"email":"ahospital0@census.gov","gender":"Male","ip_address":"138.135.180.206","drug_provider":"OrchidPharma Inc","icd9":"94140","icd9_description":"Deep necrosis of underlying tissues [deep third degree] without mention of loss of a body part, face and head, unspecified site","icd9P_proc":"7942","icd9_proc_description":"Closed reduction of separated epiphysis, radius and ulna","user_agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_5_8) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/14.0.801.0 Safari/535.1","drug_used":"Naratriptan"}]json_string = json.dumps(row)client.publish("patientdata",payload=json_string,qos=1,retain=False)client.disconnect() Step 2: Get to Apache NiFi via FTP, sFTP, File, Apache Kafka, MQTT, REST AP

Finding out the hit rate It is possible to determine cache hit rate per node or per query. Per node, you can see hit rate by looking at LLAP metrics view (<llap node>:15002, see General debugging):Per query, it is possible to see LLAP IO counters (including hit rate) upon running the query by setting hive.tez.exec.print.summary=true, which should produce the counters output at the end of the query, for example - empty cache:Some data in cache:Why is the cache hit rate lowConsider the data size and the cache size across all nodes. E.g. with a 10 Gb cache on each of the 4 LLAP nodes, reading 1 Tb of data cannot achieve more than ~4% cache hit rate even in the perfect case. In practice the rate will be lower due to effects of compression (cache is not snappy/gzip compressed, only encoded), interference from other queries, locality misses, etc. In HDP 2.X/Hive 2.X, cache has coarser granularity to avoid some fragmentation issues that are resolved in HDP 3.0/Hive 3.0. This can cause c

This article is a short summary of LLAP-specific causes of query slowness. Given that a lot of the Hive query execution (compilation, almost all operator logic) stay the same when LLAP is used, queries can be slow for non-LLAP-related reasons; general Hive query performance issues (e.g. bad plan, skew, poor partitioning, slow fs, ...) should also be considered when investigating. These issues are outside of the scope of this article.Queries are slow On HDP 2.5.3/Hive 2.2 and below, there’s a known issue where queries on LLAP can get slower over time on some workloads. Upgrade to HDP 2.6.X/Hive 2.3 or higher.If you are comparing LLAP against containers on the same cluster, check LLAP cluster size compared to the capacity used for containers. If there are 27 nodes for containers, and a 3-node LLAP cluster, it’s possible containers will be faster because they can use many more CPUs/memory.Generally, queries on LLAP run just like in Hive (see Architecture). So, the standard Hive performanc

Introduction: how does LLAP fit into Hive LLAP is a set of persistent daemons that execute fragments of Hive queries. Query execution on LLAP is very similar to Hive without LLAP, except that worker tasks run inside LLAP daemons, and not in containers. High-level lifetime of a JDBC query: Without LLAP With LLAP Query arrives to HS2; it is parsed and compiled into “tasks” Query arrives to HS2; it is parsed and compiled into “tasks” Tasks are handed over to Tez AM (query coordinator) Tasks are handed over to Tez AM (query coordinator) Coordinator (AM) asks YARN for containers Coordinator (AM) locates LLAP instances via ZK Coordinator (AM) pushes task attempts into containers Coordinator (AM) pushes task attempts as fragment into LLAP RecordReader used to read data LLAP IO/cache used to read data or RecordReader used to read data Hive operators are used to pro

Short Description: How to configure KNOX for Hive1 and Hive2(LLAP) in parallel.Article By default KNOX is configured for Hive1, this article will help you to configure KNOX for Hive2(LLAP). Step 1. Before configuring KNOX for Hive2, you have to configure Hive2 for http mode as, Go to Ambari-> Services -> Hive -> configs -> custom hive-interactive-site.xml -> add below properties,hive.server2.thrift.http.path=cliservicehive.server2.transport.mode=http Restart Hive service. Step 2. Now configure KNOX for Hive2(LLAP) as, 1) Go to the below location in your KNOX server machine:-# cd /usr/hdp/<HDP VERSION>/knox/data/services 2) Copy the hive directory present in the location and rename it as llap# cp -rp hive llap3) Edit the services.xml and rewrite.xml as below:-# cd llap/0.13.0/# vim service.xml------------<service role="LLAP" name="llap" version="0.13.0"> <routes> <route path="/llap"/> </routes> <dispatch classname="org.

This is the central page that links to other LLAP debugging articles. You can find articles dealing with specific problems below; to provide some background and to help resolve other issues, there are also some helpful general-purpose articles:A one-page overview of LLAP architecture: https://community.hortonworks.com/articles/149894/llap-a-one-page-architecture-overview.htmlOn the general debugging - where to find logs, UIs, metrics, etc.: https://community.hortonworks.com/articles/149896/llap-debugging-overview-logs-uis-etc.htmlYou might also find this non-debugging LLAP sizing and setup guide interesting: https://community.hortonworks.com/articles/149486/llap-sizing-and-setup.htmlBefore getting to specific issues, there are some limitations with LLAP that you should be aware of; the following significant features are not supported: HA (work in progress).doAs=true (no current plans to support it).Temporary functions (will not be supported; use permanent functions).CLI access (use be

We need to ingest C-CDA files, which often have weird names in them. To Compound the naming issues, the Extract C-CDA processor flattens XML to have periods which are not allowed in Apache Avro names.Apache NiFi DataFlow Ingest C-CDA XML FilesExtract C-CDA AttributesRoute on AttributeAttributeCleaner - my new custom oneAttributeToJSONSetSchemaQueryRecordConvertAvroToORCPutHDFSAttributeCleanerProcessor is my new processor to rename all the attributes. This is a very simple version.These are the converted file names.Above is the JSON fields with values.Route when code field is not null.If code is active then convert to Apache ORC for Hive.SQL TableCREATE EXTERNAL TABLE IF NOT EXISTS ccda (problemSectionact_02observationproblemStatuscodecodeSystemName STRING, vitalSignsSectionorganizerobservations_05idsroot STRING, problemSectionact_02observationproblemStatusstatusCodecode STRING, vitalSignsSectionorganizerobservations_01texttext_01value STRING, vitalSignsSectionorganizerobservations_0

When running a custom Java application that connects via JDBC to Hive, after migration to HDP-2.6.x, the application now fails to start with a NoClassDefFoundError or ClassNotFoundException related to a Hive class, like:Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hive/service/cli/thrift/TCLIService$Ifaceat org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105)at java.sql.DriverManager.getConnection(DriverManager.java:664)at java.sql.DriverManager.getConnection(DriverManager.java:270)Root CausePrior to HDP-2.6.x, the hive-jdbc.jar is a symlink which points to the "standalone" jdbc jar (the one intended to be used for non-hadoop apps, like a generic app that has JDBC driver DB accessibility), for example in HDP 2.5.0:/usr/hdp/current/hive-client/lib/hive-jdbc.jar -> hive-jdbc-1.2.1000.2.5.0.0-1245-standalone.jarBut from newer versions, HDP-2.6.x onwards, the hive-jdbc.jar now points to the "hadoop env" JDBC driver, which has dependencies on many other H

kinit PRINCIPAL -kt /etc/security/keytabs/PRINCIPAL.keytab hive --hiveconf hive.execution.engine=mrSET hive.execution.engine=tez;SET tez.queue.name=QUEUE_NAME;use MON_SCHEMA;select count(*) from TABLE where id =1; By starting Hive CLI with mr, it open the terminal quickly than default, this is because it will not request for AM resource. PS : Hive CLI is not recommended and must be deprecated in your production environment, check here for more info

Complementary article to Hive CLI security to clarify the risk of using Hive CLI.Hive CLI ( or Hive shell ) is not recommended and Apache asked users to move to Beeline even if it's still supported by Hortonworks ( HDP2.6 )Ranger Hive plugin does not enforce permissions for Hive CLI users however it doesn’t bypass systematically “All” Ranger policies, it bypass only hive policies.This risk is therefore for all hive managed tables. ( ones under /apps/hive/warehouse/ ) All external DB/Tables will still be protected by HDFS policies.

When hive.server2.enable.doAs=True , HiveServer2 performs the query processing as the user who submitted the query (usually the user you kinit with, it could be service account or an account assigned to a team ). But if the parameter is set to false, the query will run as the user that the hiveserver2 process runs as, mostly HiveThis will help to : 1-Better control the users via Hive Ranger policies2-Better control the ACLs mappings for Yarn, so you can assign every user on a specific Queue.

Overview This tutorial is intended to walk you through the process of creating a Process Group in NiFi to feed multiple Twitter streams to Elasticsearch. This tutorial is the second part of a two part series. The first part can be found here: HCC Article. In this part of the series, we will create a process group which contains multiple Twitter feeds funneled to a single Elasticsearch instance. This allows you to have multiple feeds of data with different processing needs prior to pushing to Elasticsearch. We will be able to query Elasticsearch to see data from both of our example streams. Admittedly this is a contrived example. However, the concept is fundamentally useful across a variety of NiFi use cases. NOTE: The only required software components are NiFi and Elasticsearch which can be run in just about any Linux environment. However I recommend deploying these as part of your HDP sandbox or test cluster allowing for a broader integration of tools and capabilities such as Pig, Hiv

Ambari kills the process of the namenode when it detects a java error with type : HDFS -> Under hadoop-env template replace-XX:OnOutOfMemoryError=\"/usr/hdp/current/hadoop-hdfs-secondarynamenode/bin/kill-secondary-name-node\"by -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=\"/tmp/heap\"This will only allow you to start the namenode , investigate what is causing the issue.But this is a temporary solution, you need to analyze your heap dump and see what's wrong with it.--SOLR could be one of the causes of this, when creating huge logs that needs to be written to hdfs.You can clear the logs of NN and SNN here : /var/log/hadoop/hdfs/audit/solr/spoolBecareful on deleting only on Standby NN - then do a failover to delete from the other server. do not delete logs while the namenode is active.

There are five ways to connect to HS2 with JDBCDirect - Binary Transport mode (Non-Secure|Secure)Direct - HTTP Transport mode (Non-Secure|Secure)ZooKeeper - Binary Transport mode (Non-Secure|Secure)ZooKeeper - HTTP Transport mode (Non-Secure|Secure)via Knox - HTTP Transport mode Connecting to HS2 via ZooKeeper (3-4) (and knox, if backed by ZooKeeper) provides a level of failover that you can't get directly. When connecting through ZooKeeper, the client is provided server connection information from a list of available servers. This list is managed on the backend and the client isn't aware of them, before the connection. This allows administrators to add additional servers to the list without reconfiguring the clients.NOTE: HS2 in this configuration is considered a Failover and is not automatic once a connection has been established. JDBC connections are stateful. The data and session information kept on HS2 for a connection is LOST when the server goes down. Jobs currently in progre

Part II can be found hereThere are many web UIs--think Ambari, Ranger, etc.--in the HDP platform that require user authentication and traditionally every UI required a separate login. With the federation capabilities provided by Apache Knox, the HDP platform can support federated identity and a single sign-on experience for users.In particular, the flexibility of the Knox authentication and federation providers allows KnoxSSO to support generalized authentication events, via exchange of a common JWT-based token. Without the token exchange capabilities offered by KnoxSSO, each UI would need to integrate with each desired IAM solution on its own.KnoxSSO comes with its own Form-based IdP. This allows for easily integrating a form-based login with the enterprise AD/LDAP server. We will cover that in a future article, as we'll focus at present on the integration with Okta using SAML 2.0. You can find more information regarding SAML on Okta's website: https://developer.okta.com/standards/SAM