Resources

In this episode of Big Data Roaring News, Dave laments another announcement of Hadoop’s demise and exposes A.I. imposters. Jhon has articles comparing Ranger with Sentry and Apache Nifi reaching the ripe age of 1.7 with a Minifi charged practical demo to prove the point.https://roaringelephant.org/2018/07/24/episode-98-roaring-news/ Play in new window | Download (Duration: 22:16 — 15.5MB)Hadoop’s star dims in the era of cloud object data storage and stream computing https://siliconangle.com/blog/2018/07/09/hadoops-star-dims-era-cloud-object-data-storage-stream-computing/The rise of “pseudo-ai” how tech firms quietly use humans to do bots work https://www.theguardian.com/technology/2018/jul/06/artificial-intelligence-ai-humans-bots-tech-companiesApache Ranger Vs Sentry https://www.linkedin.com/pulse/apache-ranger-vs-sentry-mythily-rajavelu/How to build an IIoT system using Apache NiFi, MiNiFi, C2 Server, MQTT and Raspberry Pi https://medium.freecodecamp.org/building-an-iiot-sys

Episode 97 – ODPi: A new world for data governancehttps://roaringelephant.org/2018/07/17/episode-97-odpi-a-new-world-for-data-governance/In this episode, we welcome back John Mertic one more time. It was quite obvious that John had lots more to talk about at the end of our last interview with him. ODPi has recently reinvented itself, moving away from a strict distribution standards body towards data governance and reference specifications. Play in new window | Download (Duration: 1:07:57 — 46.9MB)John MerticDirector of Program Management for ODPi, R Consortium, and Open Mainframe Projecthttps://www.linkedin.com/in/jmertic/ODPi website links: https://www.odpi.org/https://www.odpi.org/blog/2018/04/04/the-state-of-open-source-and-big-data-three-years-laterhttps://www.odpi.org/projects/data-governance-pmchttps://www.odpi.org/events Please use the Contact Form on this blog or our twitter feed to send us your questions, or to suggest future episode topics you would like us to cover.

Hadoop Distributed File System HDFS is a Java-based file system that provides scalable and reliable data storage, and it was designed to span large clusters of commodity servers. HDFS has demonstrated production scalability of up to 200 PB of storage and a single cluster of 4500 servers, supporting close to a billion files and blocks. When that quantity and quality of enterprise data is available in HDFS, and YARN enables multiple data access applications to process it, Hadoop users can confidently answer questions that eluded previous data platforms.HDFS is a scalable, fault-tolerant, distributed storage system that works closely with a wide variety of concurrent data access applications, coordinated by YARN. HDFS will “just work” under a variety of physical and systemic circumstances. By distributing storage and computation across many servers, the combined storage resource can grow linearly with demand while remaining economical at every amount of storage.Take Away 1. HDFS is based

JSTACK and JMAP CollectionJstack CollectionStep 1: Switch as the service user that started the process.#su - <service-user-who-started-the-process>Step 2: Capture the process ID#ps -ef | grep <process-name>#ps -ef | grep hivehive 21887 1 0 Aug01 ? 00:58:04 /usr/jdk64/jdk1.8.0_112/bin/java -Xmx1024m -Dhdp.version=2.6.5.0-292 -Djava.net.preferIPv4Stack=true -Dhdp.version=2.6.5.0-292 -Dhadoop.log.dir=/var/log/hadoop/hive -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/usr/hdp/2.6.5.0-292/hadoop -Dhadoop.id.str=hive -Dhadoop.root.logger=INFO,console -Djava.library.path=:/usr/hdp/2.6.5.0-292/hadoop/lib/native/Linux-amd64-64:/usr/hdp/2.6.5.0-292/hadoop/lib/native/Linux-amd64-64:/usr/hdp/2.6.5.0-292/hadoop/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Xmx1024m -Xmx2048m -Djava.util.logging.config.file=/usr/hdp/current/hive-server2/conf/conf.server/parquet-logging.properties -Dhadoop.security.logger=INFO,NullAppender org.apache.h

How to use make a simple flow using SNMPSET/SNMPGET Append A - Troubleshooting common mistakesPre-requisites: - Nifi Cluster Installed (I'm using HDF 3.1.2.0 hence with Nifi 1.5) - Centos7 With your NIFI cluster installed, you're up to initiate this step-by-step. First of all, we'll need to install a snmp server to retrieve/set values of/to MIB's. In my case I'm using net-snmp tool, and it comes with some MIB samples to play. Here we're going to use the SNMPv2-MIB.txt under the /usr/share/snmp/mibs path 1)Install net-snmpyum install -y net-snmp net-snmp-utils net-snmp-libs 2)Give full access of read-write to anyone at public community (DON'T DO IT IN PROD, please)echo > /etc/snmp/snmpd.conf printf "agentAddress udp:161\nrwcommunity public\nrwuser public\nrocommunity public default system\nrouser public" >> /etc/snmp/snmpd.conf 3)Start the SNMP SERVER DAEMON - port 161service snmpd start 4)Test a simple snmpwalk to see if those changes at step 2 are workingsnmpwalk -v2c -mALL -

IoT Edge Processing with Apache NiFi and MiniFi and Multiple Deep Learning Libraries SeriesFor: https://conferences.oreilly.com/strata/strata-ny/public/schedule/detail/68140In preparation for my talk on utilizing edge devices for deep learning, IoT sensor reading and big data processing I have updated my environment to the latest and greatest tools available.With the upgrade of HDF to 3.2, I can now use Apache NiFi 1.7 and MiniFi 0.5 for IoT data ingestion, simple event processing, conversion, data processing, data flow and storage.The architecture diagram above shows the basic flow we are utilizing.IoT Step by Step Raspberry Pi with latest patches, Python, GPS software, USB Camera, Sensor libraries, Java 8, MiniFi 0.5, TensorFlow and Apache MXNet installed.minifi flow pushes JSON and JPEGs over HTTP(s) / Site-to-Site to an Apache NiFi gateway server.Option: NiFi can push to a central NiFi cloud cluster and/or Kafka cluster both of which running on HDF 3.2 environments.Apache NiFi c

Short Description: Using JOLT transformation to modify a JSON object.Article: In NIFI, JOLT Transformation can be used to help modifying a JSON object, the below code snippets are an example of how JOLT can be used to shift elements inside a JSON Object, in this case, the intention is to move and repeat "productionID" and "processID" inside of the array. Original JSON Object: { "productionID": "BHS0928MDB8391", "productionLine": [ { "model": "Cooper", "year": 2018, "type": "Hatchback", "motorization": "Electric", "colour": "Midnight Black", "stageID": "MGOP94810482042" }, { "model": "Cooper", "year": 2018, "type": "Hatchback", "motorization": "Diesel", "colour": "Silver", "stageID": "MGOP9183740194" } ], "processID": "884910421743POR4183UK"} Target JSON Object: { "productionLine" : [ { "model" : "Cooper", "year" : 2018, "type" : "Hatchback", "motorization" : "Electric", "colour" : "Midni

This tutorial is part two of a two-part series. In this tutorial, we'll verify Spark 2.1 functionality using Zeppelin on an HDP 2.6 cluster deployed using Cloudbreak. The first tutorial covers using Cloudbreak to deploy the cluster. You can find the first tutorial here: HCC Article Prerequisites You should already have completed part one of this tutorial series and already have an Cloudbreak HDP 2.6 with Spark 2.1 cluster running. Scope This tutorial was tested in the following environment: Cloudbreak 1.14.4 AWS EC2 HDP 2.6 Spark 2.1 Zeppelin 0.7 Steps Login into Ambari As mentioned in the prerequisites, you should already have a cluster built using Cloudbreak. Click on the cluster summary box in the Cloudbreak UI to display the cluster details. Now click on the link to your Ambari cluster. You may see something similar to this: Your screen may vary depending on your browser of choice. I'm using Chrome. This warning is because we are using self-signed certificates which are not

This article has steps to identify where most of the small file are located in a large HDFS cluster. Below are some articles regarding the small file issues and how to analyze. https://community.hortonworks.com/articles/15104/small-files-in-hadoop.htmlhttps://community.hortonworks.com/repos/105768/identifying-small-file-offenders.htmlhttps://community.hortonworks.com/articles/142016/analyze-fsimage-from-a-non-smartsense-cluster.htmlSmartsense Activity analyzer currently doesn't show the report based on hdfs locations but it shows other useful information like Top N user with Small files, Trends, etc.In a large HDFS cluster with heavy workload env, it is often hard to locate where the most # of small files are located by using 'fsck' or 'hdfs dfs -ls -R' outputs as they can take a long time to retrieve the data and will have to repeat cmds several times to get the desired output.I have taken below approach to spot the HDFS locations where most of the small files exist in a large HDFS cl

Q1.Does Hive LLAP supports stored procedures? UDF’s https://community.hortonworks.com/articles/117833/creating-custom-udf-and-adding-udf-jar-to-hive-lla.htmlQuestion on handling small files problemIf ACID tables are not used then how to handle small files problem in Hive?. Is there any archival process to follow like creating HAR files?Alter Table/Partition Concatenate Version information In Hive release 0.8.0 RCFile added support for fast block level merging of small RCFiles using concatenate command. In Hive release 0.14.0ORC files added support fast stripe level merging of small ORC files using concatenate command.ALTER TABLE table_name [PARTITION (partition_key = 'partition_value' [, ...])] CONCATENATE;If the table or partition contains many small RCFiles or ORC files, then the above command will merge them into larger files. In case of RCFile the merge happens at block level whereas for ORC files the merge happens at stripe level thereby avoiding the overhead of decompressing and

IntroductionMotivation In many production clusters, the servers (or VMs) have multiple physical Network Interface Cards (NICs) and/or multiple Virtual Network Interfaces (VIFs) and are configured to participate in multiple IP address spaces (“networks”). This may be done for a variety of reasons, including but not limited to the following: to improve availability or bandwidth through redundancy to partition traffic for security, bandwidth, or other management reasons etc. In many of these “multi-homed” clusters it is desirable to configure the various Hadoop-related services to “listen” on all the network interfaces rather than on just one. Clients, on the other hand, must be given a specific target to which they should initiate connections. Note that NIC Bonding (also known as NIC Teaming or Link Aggregation), where two or more physical NICs are made to look like a single network interface, is not included in this topic. The settings discussed below are usually not applicab

HIVE LLAP - a one-page architecture overviewhttps://community.hortonworks.com/articles/149894/llap-a-one-page-architecture-overview.htmlHive - Understanding concurrent sessions + queue allocation + preemptionhttps://community.hortonworks.com/articles/56636/hive-understanding-concurrent-sessions-queue-alloc.htmlHive LLAP Dashboardshttps://docs.hortonworks.com/HDPDocuments/Ambari-2.6.0.0/bk_ambari-operations/content/grafana_hive_llap_dashboards.htmlHive LLAP Logs infohttps://community.hortonworks.com/articles/149896/llap-debugging-overview-logs-uis-etc.htmlMonitoring LLAP metricshttp://www.kartikramalingam.com/hive-llap/Debugging Hive LLAP Queryhttps://community.hortonworks.com/articles/149896/llap-debugging-overview-logs-uis-etc.htmlQuestion on Hive LLAP benchmarksPlease share if any Hive LLAP benchmarks?https://hortonworks.com/blog/3x-faster-interactive-query-hive-llap/LLAP TuningHere is an excellent article on LLAP tuning.https://community.hortonworks.com/articles/149486/llap-sizing-a

SYMPTOM: hiveserver2 is failing with OutOfMemoryError very frequently, user increased HiveServer2 heap size to 16GB but the still faces the same issue.we enabled heap dump using -XX:+HeapDumpOnOutOfMemoryError to see what object are causing the hiveserver2 heap to grow such big.ROOT CAUSE:After initial analysis we notice that some of the connection objects are set with the fetchSize of 50M,we enquire user about that which reveals that there are some connection strings where the fetchSize was set as 50M.with the fetchSize setting in place HiveServer2 is taking a lot of heap space while fetching the query result and going into out of memory.WORKAROUND:NARESOLUTION:Ask user to remove the fetchSize setting from connection string. there is an improvement in place in the hive community tracked as https://issues.apache.org/jira/browse/HIVE-14901 to put a 'guardrail' against someone using too high a value for fetchSize.

Background:In NiFI, the PutSQL Processor is unable to perform the batch inserts although a batch size had been configured in the processor. There is a commit for each insert because the content of each FlowFile is a unique SQL statement.Lets take an example workflow,CSV file -> 50 lines (name, jobtitle) -> 50 flow filesFor each line, PutSQL processor (batchsize-10) is invoked. Because each flow file is unique in its content, 50 commits are performed.Line1 -> Flowfile#1Insert into Employee (name, job title) VALUES ('Bryan B','Director')Line2 -> Flowfile#2Insert into Employee (name, job title) VALUES ('Joe W','CTO')The PutSQL processor batches FlowFiles by matching SQL statements. To match SQL statements with unique inserts, you must configure the dataflow so that the SQL statements make use of the "?" to escape parameters. In this case, the parameters to use must exist as FlowFile attributes with the naming convention sql.args.N.type and sql.args.N.value, where N is a positi

In the following video, I demonstrate how to create a Cloudbreak credential on Google Cloud Video link: https://youtu.be/uVYpgz9m4eE The Cloudbreak version used in this video is Cloudbreak 2.7.1. To obtain the roles that need to be assigned to the service account, refer to Service account for GCP credential documentation. If you are using a corporate Google Cloud account you may be unable to perform some of the steps (such as service account creation and role assignment) by yourself and you may have to contact your Google Cloud admin to perform these steps for you. If you are using a Cloudbreak version different than 2.7.1, refer to the equivalent documentation for that version. You can access Cloudbreak documentation from the Hortonworks docs page at https://docs.hortonworks.com.

In the following video, I demonstrate how to create a key-based Cloudbreak credential on AWS Video link: https://youtu.be/SIYhzlMQf8Y The Cloudbreak version used in this video is Cloudbreak 2.7.1. To obtain the policy that need to be assigned to the user, refer to Create CredentialRole documentation. If you are using a corporate AWS account you may be unable to perform some of the steps (such as user creation and policy assignment) by yourself and you may have to contact your AWS admin to perform these steps for you. If you are using a Cloudbreak version different than 2.7.1, refer to the equivalent documentation for that version. You can access Cloudbreak documentation from the Hortonworks docs page at https://docs.hortonworks.com.

In the following video, I demonstrate how to create an app-based Cloudbreak credential on Azure Video link: https://youtu.be/NYQZd2AmM6g The Cloudbreak version used in this video is Cloudbreak 2.7.1. If you are using a corporate Azure account you may be unable to perform some of the steps (such as role creation and policy assignment) by yourself and you may have to contact your Azure admin to perform these steps for you. If you are using a Cloudbreak version different than 2.7.1, refer to the equivalent documentation for that version. You can access Cloudbreak documentation from the Hortonworks docs page at https://docs.hortonworks.com.

In the following video, I demonstrate how to create an interactive Cloudbreak credential on AzureVideo link: https://youtu.be/Q9fqmOZqnnw The Cloudbreak version used in this video is Cloudbreak 2.7.1. If you are using a corporate Azure account, you may be unable to use this method due to the fact that the Owner role is required. In this case, you should use the app-based credential method. See Create an app-based credential documentation.If you are using a Cloudbreak version different than 2.7.1, refer to the equivalent documentation for that version. You can access Cloudbreak documentation from the Hortonworks docs page at https://docs.hortonworks.com.

In the following video, I demonstrate how to create a role-based Cloudbreak credential on AWS Video link: https://youtu.be/XlvUahqwThI The Cloudbreak version used in this video is Cloudbreak 2.7.1. To obtain the policy that need to be assigned to the user, refer to Create CredentialRole documentation.If you are using a corporate AWS account you may be unable to perform some of the steps (such as role creation and policy assignment) by yourself and you may have to contact your AWS admin to perform these steps for you.If you are using a Cloudbreak version different than 2.7.1, refer to the equivalent documentation for that version. You can access Cloudbreak documentation from the Hortonworks docs page at https://docs.hortonworks.com.

There are at times we would need to move kerberos database to different nodes or upgrade the OS of KDC node ( for e.x CentOS6 to CentOS7 ). Obviously you would not want to lose you the kdc users especially if your HDP cluster is configured to use this kdc.Follow below steps to backup and restore kerberos database.prerequisite* Backup the keytab from the HDP cluster under /etc/security/keytabs from all nodes.* Note down your kdc admin principal and password* Backup /etc/krb5.conf* Backup /var/kerberos directoryBackup* Take the kerberos database dump using below command ( to be executed on node running kerberos )kdb5_util dump kdb5_dump.txt* Safely backup the kdb5_dump.txt.Restore* Restore the kerberos database execute below commandkdb5_util load kdb5_dump.txt * Restore the /etc/krb5.conf from backup* Restore /var/kerberos/krb5kdc/kdc.conf from backup* Restore /var/kerberos/krb5kdc/kadm5.acl from backup* Run below command to store master principal in stash file ( kdc admin password is r

For HDFS analysis, an Activity Analyzer needs to be deployed to each NameNode in the cluster. These instances will automatically begin processing the fsimage on startup and will reprocess the latest fsimage data once every 24 hours. By default, when deployed on a NameNode, these Activity Analyzers do not process YARN, or MapReduce & Tez utilization data; This is to reduce the amount of processing done on servers hosting critical services like the NameNode.Unless we install analyzer on namenode, it will not be able to find fsimage in directory and hence fail with below error2018-08-23 14:28:04,401 ERROR [pool-3-thread-1] ActivityManager:78 - Failed to process activity id null of type HDFSjava.lang.NullPointerException at java.io.File.<init>(File.java:277) at com.hortonworks.smartsense.activity.hdfs.HDFSImageProcessor.processActivity(HDFSImageProcessor.java:71) at com.hortonworks.smartsense.activity.ActivityManager$1.call(ActivityManager.java:69) at

Overview As mentioned in the article parent to this article, the objective of this part 1 is to ingest data from the New York Times API, run HTML scraping, personality detection and expose it to a Kafka topic, as depicted by the architecture below: The end goal is thus to implement a NiFi flow like the following: To implement it, this article will be divided in 4 section: Section 1: Create a NYT API consumer Section 2: Create a custom BoilerPipe Extractor Processor Section 3: Create a custom Mairesse Personality Processor Section 4: Create and configure a Kafka topic Section 1: Create a NYT API consumer Disclaimer: I basically copy-pasted this from my previous article, Determining the big 5 traits of Personality Psychology of news articles using NiFi, Hive & Zeppelin. Step 1: Obtaining an API Key This step is very straight forward. Go to https://developer.nytimes.com/signup and sign-up for a key. Step 2: Configuring InvokeHTTP The InvokeHTTP is

IntroductionA few weeks ago, I published an article called Determining the big 5 traits of Personality Psychology of news articles using NiFi, Hive & Zeppelin. Since then, I have worked diligently to improve on this first iteration, with the objective to mock up at the heart of every company today: create an end-to-end platform that uses machine learning to not only generate insights, but keeps improving and feeding consumer applications.While doing this work was a way for me to get into the needy-greedy of the latest Hortonworks tools, I decided to share it with the world in the form of a series of tutorial articles, because I believe it is a great way to get familiar with the stack. Architecture overviewLuckily, the Hortonworks platform has all the elements needed to create this end-to-end platform. The figure below gives an overview of this series of articles architecture:As you can see, the goal of this platform is to: Ingest data from news articles (directly from the NYT API a

Env : HDP 2.52 node kafka cluster having topic name 'testtopic' with partition set as 2 and replication set as 2.I am running two consumer with consumer id 'test'.1. what happen when Consumer start freshConsumer NetworkClient will request metadata <- return cluster information2016-12-17 23:21:05 DEBUG clients.NetworkClient:619 - Sending metadata request ClientRequest(expectResponse=true, callback=null, request=RequestSend(header={api_key=3,api_version=0,correlation_id=1,client_id=consumer-1}, body={topics=[testtopic]}), isInitiatedByNetworkClient, createdTimeMs=1481997065894, sendTimeMs=0) to node -2 2016-12-17 23:21:06 DEBUG clients.Metadata:172 - Updated cluster metadata version 2 to Cluster(nodes = [Node(1002, rkk2.hdp.local, 6667), Node(1001, rkk1.hdp.local, 6667)], partitions = [Partition(topic = testtopic, partition = 1, leader = 1002, replicas = [1002,1001,], isr = [1002,1001,], Partition(topic = testtopic, partition = 0, leader = 1001, replicas = [1002,1001,], isr = [1001,10