Community Articles

Matteo and Sijie from Streamlio reached out to us and let us know they had an update on Apache Pulsar. It turned out they had a lot to talk about so we cut the interview in two parts. the first of which was published in episode 101. Here is the second part with information on version 2.0 and the future of the Apache Pulsar project.https://roaringelephant.org/2018/08/28/episode-103-apache-pulsar-version-2-0-with-matteo-and-sijie-from-streamlio/ Play in new window | Download (Duration: 43:31 — 30.1MB)The first subject taken on by Sijie is Pulsar Functions, followed by Matteo talking about the new schema registry and Topic Compaction. With a new major version being released, users will probably want to upgrade so we asked the guys about the upgrade path. The rest of the episode, Matteo and Sijie share what they can regarding the future Pulsar Roadmap.Matteo Merli (https://www.linkedin.com/in/matteomerli/) Co-Founder – Software EngineerSijie Guo (https://www.linkedin.com/in/sa

Big Data News at the end of the summer is not easy to find, but we did end up with three topics to discuss: from isolating GPUs in Hadoop 3.x to replicating big data (to the cloud) and quick tips from Adam’s blog.https://roaringelephant.org/2018/08/21/episode-102-roaring-news/ Play in new window | Download (Duration: 22:07 — 15.4MB)First Class GPUs support in Apache Hadoop 3.1, YARN & HDP 3.0 https://hortonworks.com/blog/gpus-support-in-apache-hadoop-3-1-yarn-hdp-3/Replicating big datasets in the cloud https://medium.com/hotels-com-technology/replicating-big-datasets-in-the-cloud-c0db388f6ba2https://dataworkssummit.com/berlin-2018/session/tools-and-approaches-for-migrating-big-datasets-to-the-cloud/https://www.slideshare.net/Hadoop_Summit/tools-and-approaches-for-migrating-big-datasets-to-the-cloudQuick Tip: The easiest way to grab data out of a web page in Python https://medium.com/@ageitgey/quick-tip-the-easiest-way-to-grab-data-out-of-a-web-page-in-python-7153cecfca58Plea

Matteo and Sijie from Streamlio reached out to us and let us know they had an update on Apache Pulsar. It turned out they had a lot to talk about so we cut the interview in two parts and here is the first part where they introduce Apache Pulsar, go in depth on the correct deployment scaling of a stable Pulsar cluster and clarify Pulsars “at least once vs exactly once” strategy. Part two will go in more depth on what’s new. Stay tuned!https://roaringelephant.org/2018/08/14/episode-101-apache-pulsar-update-with-matteo-and-sijie-from-streamlio/ Play in new window | Download (Duration: 1:05:48 — 45.4MB)Matteo Merli (https://www.linkedin.com/in/matteomerli/) Co-Founder – Software EngineerSijie Guo (https://www.linkedin.com/in/samuelguo/) Co-FounderApache Pulsar (incubating) https://pulsar.apache.org/Please use the Contact Form on this blog or our twitter feed to send us your questions, or to suggest future episode topics you would like us to cover.

https://roaringelephant.org/2018/08/07/episode-100-celebrating-our-centennial/100 Big Data episodes! We made it, in no small part thanks to our audience: you are who keeps us going! In this episode we celebrate our centennial by going over the history of Hadoop releases, highlighting the most noteworthy events along the way. Join us down the twisty paths of our memory lanes!Play in new window | Download (Duration: 1:07:19 — 46.5MB)The blockchain related Linkedin post Jhon likedThe sources for this episode: http://hadoop.apache.org/releases.htmlhttps://en.wikipedia.org/wiki/Apache_HadoopDebate over which company had contributed more to Hadoop: http://hortonworks.com/blog/reality-check-contributions-to-apache-hadoop/Thank you for being part of the ride and now on to episode 200! Please use the Contact Form on this blog or our twitter feed to send us your questions, or to suggest future episode topics you would like us to cover.

Apache Spark is a fast, in-memory data processing engine with elegant and expressive development APIs to allow data workers to efficiently execute streaming, machine learning or SQL workloads that require fast iterative access to datasets. With Spark running on Apache Hadoop YARN, developers everywhere can now create applications to exploit Spark’s power, derive insights, and enrich their data science workloads within a single, shared dataset in Hadoop.The Hadoop YARN-based architecture provides the foundation that enables Spark and other applications to share a common cluster and dataset while ensuring consistent levels of service and response. Spark is now one of many data access engines that work with YARN in HDP.https://hortonworks.com/apache/spark/Spark code samples: http://bytepadding.com/spark/Take Away1. Spark is a library and not a service.2. Spark interacts with multiple services like HDFS and YARN to process data. 3. Spark client also has YARN client wrapped within it. 4. Sp

The Roaring Elephant podcast was a guest at the Codemotion conference in Amsterdam a little while ago. This episode contains the audio of the talk we did on the State of Big Data.https://roaringelephant.org/2018/07/31/episode-99-the-state-of-big-data/ Play in new window | Download (Duration: 45:28 — 31.5MB)Our talk was dfinitely light on slideware, but if you want to see the video cast of our presentation, you can find it on the Codemotion youtube channel:Codemotion Amsterdam 2018: The State of Big Data by Roaring Elephant podcast Please use the Contact Form on this blog or our twitter feed to send us your questions, or to suggest future episode topics you would like us to cover.

In this episode of Big Data Roaring News, Dave laments another announcement of Hadoop’s demise and exposes A.I. imposters. Jhon has articles comparing Ranger with Sentry and Apache Nifi reaching the ripe age of 1.7 with a Minifi charged practical demo to prove the point.https://roaringelephant.org/2018/07/24/episode-98-roaring-news/ Play in new window | Download (Duration: 22:16 — 15.5MB)Hadoop’s star dims in the era of cloud object data storage and stream computing https://siliconangle.com/blog/2018/07/09/hadoops-star-dims-era-cloud-object-data-storage-stream-computing/The rise of “pseudo-ai” how tech firms quietly use humans to do bots work https://www.theguardian.com/technology/2018/jul/06/artificial-intelligence-ai-humans-bots-tech-companiesApache Ranger Vs Sentry https://www.linkedin.com/pulse/apache-ranger-vs-sentry-mythily-rajavelu/How to build an IIoT system using Apache NiFi, MiNiFi, C2 Server, MQTT and Raspberry Pi https://medium.freecodecamp.org/building-an-iiot-sys

Episode 97 – ODPi: A new world for data governancehttps://roaringelephant.org/2018/07/17/episode-97-odpi-a-new-world-for-data-governance/In this episode, we welcome back John Mertic one more time. It was quite obvious that John had lots more to talk about at the end of our last interview with him. ODPi has recently reinvented itself, moving away from a strict distribution standards body towards data governance and reference specifications. Play in new window | Download (Duration: 1:07:57 — 46.9MB)John MerticDirector of Program Management for ODPi, R Consortium, and Open Mainframe Projecthttps://www.linkedin.com/in/jmertic/ODPi website links: https://www.odpi.org/https://www.odpi.org/blog/2018/04/04/the-state-of-open-source-and-big-data-three-years-laterhttps://www.odpi.org/projects/data-governance-pmchttps://www.odpi.org/events Please use the Contact Form on this blog or our twitter feed to send us your questions, or to suggest future episode topics you would like us to cover.

Hadoop Distributed File System HDFS is a Java-based file system that provides scalable and reliable data storage, and it was designed to span large clusters of commodity servers. HDFS has demonstrated production scalability of up to 200 PB of storage and a single cluster of 4500 servers, supporting close to a billion files and blocks. When that quantity and quality of enterprise data is available in HDFS, and YARN enables multiple data access applications to process it, Hadoop users can confidently answer questions that eluded previous data platforms.HDFS is a scalable, fault-tolerant, distributed storage system that works closely with a wide variety of concurrent data access applications, coordinated by YARN. HDFS will “just work” under a variety of physical and systemic circumstances. By distributing storage and computation across many servers, the combined storage resource can grow linearly with demand while remaining economical at every amount of storage.Take Away 1. HDFS is based

JSTACK and JMAP CollectionJstack CollectionStep 1: Switch as the service user that started the process.#su - <service-user-who-started-the-process>Step 2: Capture the process ID#ps -ef | grep <process-name>#ps -ef | grep hivehive 21887 1 0 Aug01 ? 00:58:04 /usr/jdk64/jdk1.8.0_112/bin/java -Xmx1024m -Dhdp.version=2.6.5.0-292 -Djava.net.preferIPv4Stack=true -Dhdp.version=2.6.5.0-292 -Dhadoop.log.dir=/var/log/hadoop/hive -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/usr/hdp/2.6.5.0-292/hadoop -Dhadoop.id.str=hive -Dhadoop.root.logger=INFO,console -Djava.library.path=:/usr/hdp/2.6.5.0-292/hadoop/lib/native/Linux-amd64-64:/usr/hdp/2.6.5.0-292/hadoop/lib/native/Linux-amd64-64:/usr/hdp/2.6.5.0-292/hadoop/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Xmx1024m -Xmx2048m -Djava.util.logging.config.file=/usr/hdp/current/hive-server2/conf/conf.server/parquet-logging.properties -Dhadoop.security.logger=INFO,NullAppender org.apache.h

How to use make a simple flow using SNMPSET/SNMPGET Append A - Troubleshooting common mistakesPre-requisites: - Nifi Cluster Installed (I'm using HDF 3.1.2.0 hence with Nifi 1.5) - Centos7 With your NIFI cluster installed, you're up to initiate this step-by-step. First of all, we'll need to install a snmp server to retrieve/set values of/to MIB's. In my case I'm using net-snmp tool, and it comes with some MIB samples to play. Here we're going to use the SNMPv2-MIB.txt under the /usr/share/snmp/mibs path 1)Install net-snmpyum install -y net-snmp net-snmp-utils net-snmp-libs 2)Give full access of read-write to anyone at public community (DON'T DO IT IN PROD, please)echo > /etc/snmp/snmpd.conf printf "agentAddress udp:161\nrwcommunity public\nrwuser public\nrocommunity public default system\nrouser public" >> /etc/snmp/snmpd.conf 3)Start the SNMP SERVER DAEMON - port 161service snmpd start 4)Test a simple snmpwalk to see if those changes at step 2 are workingsnmpwalk -v2c -mALL -

IoT Edge Processing with Apache NiFi and MiniFi and Multiple Deep Learning Libraries SeriesFor: https://conferences.oreilly.com/strata/strata-ny/public/schedule/detail/68140In preparation for my talk on utilizing edge devices for deep learning, IoT sensor reading and big data processing I have updated my environment to the latest and greatest tools available.With the upgrade of HDF to 3.2, I can now use Apache NiFi 1.7 and MiniFi 0.5 for IoT data ingestion, simple event processing, conversion, data processing, data flow and storage.The architecture diagram above shows the basic flow we are utilizing.IoT Step by Step Raspberry Pi with latest patches, Python, GPS software, USB Camera, Sensor libraries, Java 8, MiniFi 0.5, TensorFlow and Apache MXNet installed.minifi flow pushes JSON and JPEGs over HTTP(s) / Site-to-Site to an Apache NiFi gateway server.Option: NiFi can push to a central NiFi cloud cluster and/or Kafka cluster both of which running on HDF 3.2 environments.Apache NiFi c

Short Description: Using JOLT transformation to modify a JSON object.Article: In NIFI, JOLT Transformation can be used to help modifying a JSON object, the below code snippets are an example of how JOLT can be used to shift elements inside a JSON Object, in this case, the intention is to move and repeat "productionID" and "processID" inside of the array. Original JSON Object: { "productionID": "BHS0928MDB8391", "productionLine": [ { "model": "Cooper", "year": 2018, "type": "Hatchback", "motorization": "Electric", "colour": "Midnight Black", "stageID": "MGOP94810482042" }, { "model": "Cooper", "year": 2018, "type": "Hatchback", "motorization": "Diesel", "colour": "Silver", "stageID": "MGOP9183740194" } ], "processID": "884910421743POR4183UK"} Target JSON Object: { "productionLine" : [ { "model" : "Cooper", "year" : 2018, "type" : "Hatchback", "motorization" : "Electric", "colour" : "Midni

This tutorial is part two of a two-part series. In this tutorial, we'll verify Spark 2.1 functionality using Zeppelin on an HDP 2.6 cluster deployed using Cloudbreak. The first tutorial covers using Cloudbreak to deploy the cluster. You can find the first tutorial here: HCC Article Prerequisites You should already have completed part one of this tutorial series and already have an Cloudbreak HDP 2.6 with Spark 2.1 cluster running. Scope This tutorial was tested in the following environment: Cloudbreak 1.14.4 AWS EC2 HDP 2.6 Spark 2.1 Zeppelin 0.7 Steps Login into Ambari As mentioned in the prerequisites, you should already have a cluster built using Cloudbreak. Click on the cluster summary box in the Cloudbreak UI to display the cluster details. Now click on the link to your Ambari cluster. You may see something similar to this: Your screen may vary depending on your browser of choice. I'm using Chrome. This warning is because we are using self-signed certificates which are not

This article has steps to identify where most of the small file are located in a large HDFS cluster. Below are some articles regarding the small file issues and how to analyze. https://community.hortonworks.com/articles/15104/small-files-in-hadoop.htmlhttps://community.hortonworks.com/repos/105768/identifying-small-file-offenders.htmlhttps://community.hortonworks.com/articles/142016/analyze-fsimage-from-a-non-smartsense-cluster.htmlSmartsense Activity analyzer currently doesn't show the report based on hdfs locations but it shows other useful information like Top N user with Small files, Trends, etc.In a large HDFS cluster with heavy workload env, it is often hard to locate where the most # of small files are located by using 'fsck' or 'hdfs dfs -ls -R' outputs as they can take a long time to retrieve the data and will have to repeat cmds several times to get the desired output.I have taken below approach to spot the HDFS locations where most of the small files exist in a large HDFS cl

Q1.Does Hive LLAP supports stored procedures? UDF’s https://community.hortonworks.com/articles/117833/creating-custom-udf-and-adding-udf-jar-to-hive-lla.htmlQuestion on handling small files problemIf ACID tables are not used then how to handle small files problem in Hive?. Is there any archival process to follow like creating HAR files?Alter Table/Partition Concatenate Version information In Hive release 0.8.0 RCFile added support for fast block level merging of small RCFiles using concatenate command. In Hive release 0.14.0ORC files added support fast stripe level merging of small ORC files using concatenate command.ALTER TABLE table_name [PARTITION (partition_key = 'partition_value' [, ...])] CONCATENATE;If the table or partition contains many small RCFiles or ORC files, then the above command will merge them into larger files. In case of RCFile the merge happens at block level whereas for ORC files the merge happens at stripe level thereby avoiding the overhead of decompressing and

IntroductionMotivation In many production clusters, the servers (or VMs) have multiple physical Network Interface Cards (NICs) and/or multiple Virtual Network Interfaces (VIFs) and are configured to participate in multiple IP address spaces (“networks”). This may be done for a variety of reasons, including but not limited to the following: to improve availability or bandwidth through redundancy to partition traffic for security, bandwidth, or other management reasons etc. In many of these “multi-homed” clusters it is desirable to configure the various Hadoop-related services to “listen” on all the network interfaces rather than on just one. Clients, on the other hand, must be given a specific target to which they should initiate connections. Note that NIC Bonding (also known as NIC Teaming or Link Aggregation), where two or more physical NICs are made to look like a single network interface, is not included in this topic. The settings discussed below are usually not applicab

HIVE LLAP - a one-page architecture overviewhttps://community.hortonworks.com/articles/149894/llap-a-one-page-architecture-overview.htmlHive - Understanding concurrent sessions + queue allocation + preemptionhttps://community.hortonworks.com/articles/56636/hive-understanding-concurrent-sessions-queue-alloc.htmlHive LLAP Dashboardshttps://docs.hortonworks.com/HDPDocuments/Ambari-2.6.0.0/bk_ambari-operations/content/grafana_hive_llap_dashboards.htmlHive LLAP Logs infohttps://community.hortonworks.com/articles/149896/llap-debugging-overview-logs-uis-etc.htmlMonitoring LLAP metricshttp://www.kartikramalingam.com/hive-llap/Debugging Hive LLAP Queryhttps://community.hortonworks.com/articles/149896/llap-debugging-overview-logs-uis-etc.htmlQuestion on Hive LLAP benchmarksPlease share if any Hive LLAP benchmarks?https://hortonworks.com/blog/3x-faster-interactive-query-hive-llap/LLAP TuningHere is an excellent article on LLAP tuning.https://community.hortonworks.com/articles/149486/llap-sizing-a

SYMPTOM: hiveserver2 is failing with OutOfMemoryError very frequently, user increased HiveServer2 heap size to 16GB but the still faces the same issue.we enabled heap dump using -XX:+HeapDumpOnOutOfMemoryError to see what object are causing the hiveserver2 heap to grow such big.ROOT CAUSE:After initial analysis we notice that some of the connection objects are set with the fetchSize of 50M,we enquire user about that which reveals that there are some connection strings where the fetchSize was set as 50M.with the fetchSize setting in place HiveServer2 is taking a lot of heap space while fetching the query result and going into out of memory.WORKAROUND:NARESOLUTION:Ask user to remove the fetchSize setting from connection string. there is an improvement in place in the hive community tracked as https://issues.apache.org/jira/browse/HIVE-14901 to put a 'guardrail' against someone using too high a value for fetchSize.

Background:In NiFI, the PutSQL Processor is unable to perform the batch inserts although a batch size had been configured in the processor. There is a commit for each insert because the content of each FlowFile is a unique SQL statement.Lets take an example workflow,CSV file -> 50 lines (name, jobtitle) -> 50 flow filesFor each line, PutSQL processor (batchsize-10) is invoked. Because each flow file is unique in its content, 50 commits are performed.Line1 -> Flowfile#1Insert into Employee (name, job title) VALUES ('Bryan B','Director')Line2 -> Flowfile#2Insert into Employee (name, job title) VALUES ('Joe W','CTO')The PutSQL processor batches FlowFiles by matching SQL statements. To match SQL statements with unique inserts, you must configure the dataflow so that the SQL statements make use of the "?" to escape parameters. In this case, the parameters to use must exist as FlowFile attributes with the naming convention sql.args.N.type and sql.args.N.value, where N is a positi

In the following video, I demonstrate how to create a Cloudbreak credential on Google Cloud Video link: https://youtu.be/uVYpgz9m4eE The Cloudbreak version used in this video is Cloudbreak 2.7.1. To obtain the roles that need to be assigned to the service account, refer to Service account for GCP credential documentation. If you are using a corporate Google Cloud account you may be unable to perform some of the steps (such as service account creation and role assignment) by yourself and you may have to contact your Google Cloud admin to perform these steps for you. If you are using a Cloudbreak version different than 2.7.1, refer to the equivalent documentation for that version. You can access Cloudbreak documentation from the Hortonworks docs page at https://docs.hortonworks.com.

In the following video, I demonstrate how to create a key-based Cloudbreak credential on AWS Video link: https://youtu.be/SIYhzlMQf8Y The Cloudbreak version used in this video is Cloudbreak 2.7.1. To obtain the policy that need to be assigned to the user, refer to Create CredentialRole documentation. If you are using a corporate AWS account you may be unable to perform some of the steps (such as user creation and policy assignment) by yourself and you may have to contact your AWS admin to perform these steps for you. If you are using a Cloudbreak version different than 2.7.1, refer to the equivalent documentation for that version. You can access Cloudbreak documentation from the Hortonworks docs page at https://docs.hortonworks.com.

In the following video, I demonstrate how to create an app-based Cloudbreak credential on Azure Video link: https://youtu.be/NYQZd2AmM6g The Cloudbreak version used in this video is Cloudbreak 2.7.1. If you are using a corporate Azure account you may be unable to perform some of the steps (such as role creation and policy assignment) by yourself and you may have to contact your Azure admin to perform these steps for you. If you are using a Cloudbreak version different than 2.7.1, refer to the equivalent documentation for that version. You can access Cloudbreak documentation from the Hortonworks docs page at https://docs.hortonworks.com.

In the following video, I demonstrate how to create an interactive Cloudbreak credential on AzureVideo link: https://youtu.be/Q9fqmOZqnnw The Cloudbreak version used in this video is Cloudbreak 2.7.1. If you are using a corporate Azure account, you may be unable to use this method due to the fact that the Owner role is required. In this case, you should use the app-based credential method. See Create an app-based credential documentation.If you are using a Cloudbreak version different than 2.7.1, refer to the equivalent documentation for that version. You can access Cloudbreak documentation from the Hortonworks docs page at https://docs.hortonworks.com.