Resources

Summary:The release of HDF 3.3 brings about a significant number of improvements in HDF. This article shows how you can use ambari-bootstrap project to easily generate a blueprint and deploy either HDF only clusters or combined HDP/HDF clusters in 5 easy steps. To quickly setup a single node setup, prebuilt AMIs are available for AWS as well as a script that automates these steps, so you can deploy the cluster in a few commands.Steps for each of the below option are described in this article:A. Single-node prebuilt AMIs on AWSB. Single-node fresh installsC. Multi-node fresh installsA. Single-node prebuilt AMI on AWS:Steps to launch the AMI1. Launch Amazon AWS console page in your browser by clicking here and sign in with your credentials. Once signed in, you can close this browser tab.2. Select the AMI from ‘N. California’ region by clicking one of the below options To spin up HDP 3.1/HDF 3.3, click hereTo spin up HDF 3.3 only cluster, click hereNow choose instance type: select ‘m4.2xl

Issue Description : Unable to Register new Version in Ambari-2.6.2.x versions using Redhat Satellite/spacewalk, The Save button is disabled or grayed out the Javascript logic of validating the URL fails if the Redhat satellite/spacewalk is ticked and save button remains disabled Root cause : https://issues.apache.org/jira/browse/AMBARI-22952 Solution : 1) Use javascript Workaround Go to Ambari-Server node, go to folder(for ambari-2.6.2.2 folder name will be ADMIN_VIEW\{2.6.2.2\} : cd /var/lib/ambari-server/resources/views/work/ADMIN_VIEW{2.6.2.0}/scripts Back up the current main.js cp main.js /tmp/main.js vi main.js find the line : $scope.isSaveButtonDisabled = function() { Change this functions return type from return !(enabled && $scope.validBaseUrlsExist()); to return !($scope.useRedhatSatellite || (enabled && $scope.validBaseUrlsExist())); Save it and Try again after Hard Reloading the Browser. There is an another option of adding the same via CU

This is a short how-to leveraging Zeppelin and Solr's native SQL capabilities to query the Ranger audit logs in real time. The capability to query Ranger audits has been in existence for quite a while and there are multiple articles available demonstrating how to apply a Hive external table on top of Ranger audits stored in HDFS. This article demonstrates how to leverage Zeppelin and Solr SQL to query Solr in real time without additional step of creating an external table on top of the HDFS audit.First thing you need is access to your Solr instance. I'm using the default instance packaged with Ambari infra. The Solr admin UI is available at the following address: http://{ambari-infra-ip}:8886. In the UI, you can issue arbitrary queries using standard Solr syntax. I am new to Solr and found the query syntax cumbersome. Instead, I decided to leverage Solr SQL, available as of version 6. HDP 3.0 ships with Solr 7.3.The next step is to set up Zeppelin interpreter for Solr via JDBC. Steps f

Setting up High Availability for Oozie server. Prerequisite: - To have multiple Oozie instances we need to have either MYSql/Postgres/Oracle DB instead of the Dubey Database We need to have HA Proxy /F5 Setup for Load Balancing. For Production I would suggest to get the F5 Load Balancer which should be managed by the Network Team. Steps to setup Multiple oozie servers followed by HAProxy setup. Please find the attached Steps to add Multiple Oozie Instances https://docs.hortonworks.com/HDPDocuments/Ambari-2.6.2.2/bk_ambari-operations/content/adding_an_oozie_server_component.html). Setting up the ZK Quorum and oozie_base_url Inorder to enable the HA feature of oozie we need to use zookeeper quorum and expose the Url of the Proxy server. Oozie-site.xml ozie.zookeeper.connection.string provide the Zookeepr quorum in my lab e.g. "hwc3206-node2.hogwarts-labs.com:2181,hwc3206-node3.hogwarts-labs.com:2181,hwc3206-node4.hogwarts-labs.com:2181" oozie.services.ext add th

Short Description:Often times there will be a need to ingest binary files to Hadoop (like PDF, JPG, PNG) where you will want to store them in HBase directly and not on HDFS itself. This article describes an example how this could be achieved. ArticleThe maximum number of files in HDFS depends on the amount of memory available for the NameNode.Each file object and each block object takes about 150 bytes of the memory. For example, if you have 10 million files and each file has 1 one block each, then you would need about 3GB of memory for the NameNode. This could pose a problem if you would like to store trillions of files, where you will run out of RAM on the NameNode trying to track all of these files. One option to resolve this issue is to store your blobs in an Object store. There are many to choose from, especially if you use your favorite cloud provider. Another alternative would be to look at Ozone: https://hortonworks.com/blog/ozone-object-store-hdfs/Below is an example how you c

During launch of HDP or HDF on azure via cloudbreak, if the following provisioning error is thrown (Check cloudbreak logs):log:55 INFO  c.m.a.m.r.Deployments checkExistence - [owner:xxxxx] [type:STACK] [id:2] [name:sparky] [flow:xxx] [tracking:] <-- 404 Not Found https://management.azure.com/subscriptions/xxxxxx/resourcegroups/spark. (104 ms, 92-byte body)/cbreak_cloudbreak_1 | 2018-09-05 14:25:22,882 [reactorDispatcher-24] launch:136 ERROR c.s.c.c.a.AzureResourceConnector - [owner:xxxxxx] [type:STACK] [id:2] [name:sparky] [flow:xxxxxx] [tracking:] Provisioning error:This means the instance type selected is not available within the region. Please change region where instance is available or change to instance type which is available within region.

Exploring Apache Flink with HDPApache Flink is an open source platform for distributed stream and batch data processing. More details on Flink and how it is being used in the industry today available here: http://flink-forward.org/?post_type=session. There are a few ways you can explore Flink on HDP 2.3:1. Compilation on HDP 2.3.2To compile Flink from source on HDP 2.3 you can use these commands:curl -o /etc/yum.repos.d/epel-apache-maven.repo https://repos.fedorapeople.org/repos/dchen/apache-maven/epel-apache-maven.repoyum -y install apache-maven-3.2*git clone https://github.com/apache/flink.gitcd flinkmvn clean install -DskipTests -Dhadoop.version=2.7.1.2.3.2.0-2950 -Pvendor-reposNote that with this option I ran into a classpath bug and raised it here: https://issues.apache.org/jira/browse/FLINK-30322. Run using precompiledtarballwget http://www.gtlib.gatech.edu/pub/apache/flink/flink-0.9.1/flink-0.9.1-bin-hadoop27.tgztar xvzf flink-0.9.1-bin-hadoop27.tgzcd flink-0.9.1export HADOOP_

If you have started using Hive LLAP, you would have noticed that by default its configured to use log4j2.Default configuration makes use of advanced features from log4j2 like Rolling Over logs based on time interval and size.With time, a lot of old log files would have accumulated and typically you would compress those files manually or add additional jars and change configuration when using log4j1 to achieve the sameWith log4j2, a simple change in configuration can ensure that every time a log file is rolled over, it gets compressed for optimal use of storage space.Default configuration:To automatically compress the rolled over log files, update the highlighted line to:appender.DRFA.filePattern = ${sys:hive.log.dir}/${sys:hive.log.file}.%d{yyyy-MM-dd}-%i.gz-%i will ensure that in a rare scenario when there has been increased logging and the threshold size can be been reached more than once in the specified interval, the previously rolled over file won't get over written..gz will ensur

Test if I can publish an article. This errors out for me, and I want to know if this works now.

Overview Following my series on news author personality detection, let's dive into the streaming analytics side of things, and continue building our data science and engineering platform. This tutorial does assume that you completed part 1 of the series. After completing this part, your architecture will look like this: This tutorial will be divided in sections: Section 1: Create a custom UDF for SAM Section 2: Create a SAM application to sink to Druid Section 3: Monitor your streaming application Section 4: Create a simple superset visualization Note: Creating a custom UDF is a purely academical exercise, as is the use of SAM for sinking into Druid, considering I could have connected Kafka to Druid directly. However, one of the goal of this series is to learn as much as possible about the Hortonworks stack. Therefore, custom UDF and SAM is on the menu. Section 1: Create a custom UDF for SAM Step 1: Download the structure for your custom UDF Using the example giv

We recently had a customer who was doing stress testing on a very small cluster with just four Datanodes. Their tests included simulated crashes and sudden power-offs. They were getting recurring client errors of hdfs.DFSClient: Exception in createBlockOutputStream with various log messages including:Client:EOFException: Premature EOF: no length prefix availableIOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to tryDatanode:ReplicaAlreadyExistsException: Block <number> already exists in state FINALIZED and thus cannot be createdReplicaNotFoundException: Cannot append to a non-existent replicaFsDatasetImpl (FsDatasetImpl.java:recoverAppend(1053)) - Recover failed append to BP-<number>ReplicaNotFoundException: Cannot append to a replica with unexpeted generation stampNamenode:BlockPlacementPolicy - Failed to place enough replicas, still in need of 1 to reach 3The reason for these errors has to do with: HDFS

This Article is intended to Ambari Administrators who will be managing ambari .And for the steps mentioned below, you need access to ambari Server and ambari database.For Ambari versions below ambari-2.6.2.2 refer to comments here : https://community.hortonworks.com/questions/449/how-to-reset-ambari-admin-password.htmlThis article is for Ambari-2.7.0.0 and above.Problem Statement: you have changed the password of the admin user in UI and now you forgot the password. you want to reset the admin password.Steps1) Stop ambari serverambari-server stop2) login to your database (default password will be bigdata) psql -U ambari ambarior ( for mysql)mysql -U ambari -D ambari3) execute the command : update user_authentication set authentication_key='538916f8943ec225d97a9a86a2c6ec0818c1cd400e09e03b660fdaaec4af29ddbb6f2b1033b81b00' where user_id in (select user_id from users where user_name='admin');4) exit the database and restart ambari-serverambari-server startNow password is reset to default

Ozone is an Object store for Hadoop. It is a redundant, distributed object store built by leveraging primitives present in HDFS. Below are some key features of ozone: A Hadoop compatible file system called Ozone File system that allows programs like Hive or Spark to run against Ozone without any modifications.Ozone supports RPC and REST API for accessing the store.Built to support billions of keys in distributed environment.Ozone can run concurrently with HDFS.Like many other object stores, Ozone has a notion of volume. Only Administrators can create Volumes. Users create buckets in the volumes. To store data inside a bucket, users create keys.An ozone file system allows other Hadoop ecosystem applications like Hive and Spark to use ozone. Once a bucket is created, it is trivial to create an ozone file system.A 10-thousand foot view of Ozone OzoneManager (Om) acts as namespace manager. All ozone entities like volumes, buckets and keys are managed by Om. Om talks to an independent

IoT Edge Processing with Apache NiFi and MiniFi and Multiple Deep Learning Libraries SeriesFor: https://conferences.oreilly.com/strata/strata-ny/public/schedule/detail/68140See Part 1: https://community.hortonworks.com/articles/215079/iot-edge-processing-with-deep-learning-on-hdf-32-a.htmlSee Part 2: https://community.hortonworks.com/articles/215258/iot-edge-processing-with-deep-learning-on-hdf-32-a-1.htmlHive - SQL - IoT Data StorageIn this section, we will focus on converting JSON to AVRO to Apache ORC and storage options in Apache Hive 3. I am doing two styles of storage for one of the tables, rainbow. I am storing ORC files with an external table as well as using the Streaming API to store into an ACID table.NiFi - SQL - On Streams - CalciteSELECT *FROM FLOWFILEWHERE CAST(memory AS FLOAT) > 0 SELECT *FROM FLOWFILEWHERE CAST(tempf AS FLOAT) > 65I check the flows as they are ingested real-time and filter based on conditions such as memory or temperature. This makes for so

IoT Edge Processing with Deep Learning on HDF 3.2 and HDP 3.0 - Part 2For: https://conferences.oreilly.com/strata/strata-ny/public/schedule/detail/68140See Pre-Work: https://community.hortonworks.com/articles/203638/ingesting-multiple-iot-devices-with-apache-nifi-17.html See Part 1: https://community.hortonworks.com/articles/215079/iot-edge-processing-with-deep-learning-on-hdf-32-a.htmlStep By Step ProcessingStep 1: Install Apache NiFi (One or More Nodes or clusters) Choose: https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.2.0/installing-hdf/content/install-ambari.html or docker pull hortonworks/nifi Apache NiFi Configuration for IoT https://community.hortonworks.com/articles/67756/ingesting-log-data-using-minifi-nifi.html You will need to set: nifi.remote.input.host and nifi.remote.input.socket.port in the conf/nifi.properties or Ambari settings. Step 2: Install Apache NiFi - MiniFi on Your Device(s) Download MiniFi (https://nifi.apache.org/minifi/download.html) Yo

When you apply aggregate function on a column whose data type is "void" you get the above error.How you get "void" as datatype?When you create a table from another table(CTAS) & hardcode NULL as column values in the base table, you get the datatype as "void" for those columns. Consider the following example. This is a bug https://issues.apache.org/jira/browse/HIVE-11217hive> create table test1 as select null as col1, null as col2;Query ID = hive_20161227141826_d746f7f8-f867-4376-8b58-2e03e63c9763Total jobs = 1Launching Job 1 out of 1Status: Running (Executing on YARN cluster with App id application_1481936603759_0026)-------------------------------------------------------------------------------- VERTICES STATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED--------------------------------------------------------------------------------Map 1 .......... SUCCEEDED 1 1 0 0 0 0------------------------------------------------

Apache Ambari is a tool for provisioning, managing, and monitoring Apache Hadoop clusters. Ambari provides central management for starting, stopping, and reconfiguring Hadoop services across the entire cluster.Ambari-server Sends Commands to Ambari-agent for such operations in JSON format which is saved in /var/lib/ambari-agent/data folder. these json (command-*.json, status_command.json) Files may contain usernames and passwords which is required for few services at startupCurrently the command JSON files are stored on disk in /var/lib/ambari-agent/data. These files may be cleared out over time, but there is a need to have them removed as soon as they are no longer needed.Ambari-2.7.0 Provides a provision for deleting the command JSON files once Operation is successful to reduce the risk of leaking sensitive data stored at plaintext in the command JSON files.More Details about this can be found at the Apache JIRA: https://issues.apache.org/jira/browse/AMBARI-24228How To Enable this fe

Objective This tutorial is intended to walk you through the process of using the GetTwitter, UpdateAttribute, ReplaceText and PutElasticsearch processors in Apache NiFi to modify Twitter JSON data before sending it to Elasticsearch. This tutorial is the first part of a two part series. In this part of the series, we will create a single data flow that adds an additional field to the JSON data called twitterFilterAttribute using the ReplaceText processor. This will allow us to query Elasticsearch using a fielded query like q=twitterFilterAttribute:elasticsearch. The second part of the series will build on this example to create a process group with two GetTwitter feeds: one with elasticsearch term filter and the other with a solr term filter. Admittedly this is a contrived example. However, the concept is fundamentally useful across a variety of NiFi use cases. NOTE: The only required software components are NiFi and Elasticsearch which can be run in just about any linux environment

Why to use spark-submit if spark-shell is there ?1. Spark shell spawns executors on random nodes and hence chances of data Locality will be very less. 2. Spark-submit based on the Nodes where the data is saved spawns the excutors hence spark-submit will be more performant as compared to spark-shell.3. Spark shell is good in situations when data exploration needs to be done as it provides a interactive CLI to run your code.

Matteo and Sijie from Streamlio reached out to us and let us know they had an update on Apache Pulsar. It turned out they had a lot to talk about so we cut the interview in two parts. the first of which was published in episode 101. Here is the second part with information on version 2.0 and the future of the Apache Pulsar project.https://roaringelephant.org/2018/08/28/episode-103-apache-pulsar-version-2-0-with-matteo-and-sijie-from-streamlio/ Play in new window | Download (Duration: 43:31 — 30.1MB)The first subject taken on by Sijie is Pulsar Functions, followed by Matteo talking about the new schema registry and Topic Compaction. With a new major version being released, users will probably want to upgrade so we asked the guys about the upgrade path. The rest of the episode, Matteo and Sijie share what they can regarding the future Pulsar Roadmap.Matteo Merli (https://www.linkedin.com/in/matteomerli/) Co-Founder – Software EngineerSijie Guo (https://www.linkedin.com/in/sa

Big Data News at the end of the summer is not easy to find, but we did end up with three topics to discuss: from isolating GPUs in Hadoop 3.x to replicating big data (to the cloud) and quick tips from Adam’s blog.https://roaringelephant.org/2018/08/21/episode-102-roaring-news/ Play in new window | Download (Duration: 22:07 — 15.4MB)First Class GPUs support in Apache Hadoop 3.1, YARN & HDP 3.0 https://hortonworks.com/blog/gpus-support-in-apache-hadoop-3-1-yarn-hdp-3/Replicating big datasets in the cloud https://medium.com/hotels-com-technology/replicating-big-datasets-in-the-cloud-c0db388f6ba2https://dataworkssummit.com/berlin-2018/session/tools-and-approaches-for-migrating-big-datasets-to-the-cloud/https://www.slideshare.net/Hadoop_Summit/tools-and-approaches-for-migrating-big-datasets-to-the-cloudQuick Tip: The easiest way to grab data out of a web page in Python https://medium.com/@ageitgey/quick-tip-the-easiest-way-to-grab-data-out-of-a-web-page-in-python-7153cecfca58Plea

Matteo and Sijie from Streamlio reached out to us and let us know they had an update on Apache Pulsar. It turned out they had a lot to talk about so we cut the interview in two parts and here is the first part where they introduce Apache Pulsar, go in depth on the correct deployment scaling of a stable Pulsar cluster and clarify Pulsars “at least once vs exactly once” strategy. Part two will go in more depth on what’s new. Stay tuned!https://roaringelephant.org/2018/08/14/episode-101-apache-pulsar-update-with-matteo-and-sijie-from-streamlio/ Play in new window | Download (Duration: 1:05:48 — 45.4MB)Matteo Merli (https://www.linkedin.com/in/matteomerli/) Co-Founder – Software EngineerSijie Guo (https://www.linkedin.com/in/samuelguo/) Co-FounderApache Pulsar (incubating) https://pulsar.apache.org/Please use the Contact Form on this blog or our twitter feed to send us your questions, or to suggest future episode topics you would like us to cover.

https://roaringelephant.org/2018/08/07/episode-100-celebrating-our-centennial/100 Big Data episodes! We made it, in no small part thanks to our audience: you are who keeps us going! In this episode we celebrate our centennial by going over the history of Hadoop releases, highlighting the most noteworthy events along the way. Join us down the twisty paths of our memory lanes!Play in new window | Download (Duration: 1:07:19 — 46.5MB)The blockchain related Linkedin post Jhon likedThe sources for this episode: http://hadoop.apache.org/releases.htmlhttps://en.wikipedia.org/wiki/Apache_HadoopDebate over which company had contributed more to Hadoop: http://hortonworks.com/blog/reality-check-contributions-to-apache-hadoop/Thank you for being part of the ride and now on to episode 200! Please use the Contact Form on this blog or our twitter feed to send us your questions, or to suggest future episode topics you would like us to cover.

Apache Spark is a fast, in-memory data processing engine with elegant and expressive development APIs to allow data workers to efficiently execute streaming, machine learning or SQL workloads that require fast iterative access to datasets. With Spark running on Apache Hadoop YARN, developers everywhere can now create applications to exploit Spark’s power, derive insights, and enrich their data science workloads within a single, shared dataset in Hadoop.The Hadoop YARN-based architecture provides the foundation that enables Spark and other applications to share a common cluster and dataset while ensuring consistent levels of service and response. Spark is now one of many data access engines that work with YARN in HDP.https://hortonworks.com/apache/spark/Spark code samples: http://bytepadding.com/spark/Take Away1. Spark is a library and not a service.2. Spark interacts with multiple services like HDFS and YARN to process data. 3. Spark client also has YARN client wrapped within it. 4. Sp

The Roaring Elephant podcast was a guest at the Codemotion conference in Amsterdam a little while ago. This episode contains the audio of the talk we did on the State of Big Data.https://roaringelephant.org/2018/07/31/episode-99-the-state-of-big-data/ Play in new window | Download (Duration: 45:28 — 31.5MB)Our talk was dfinitely light on slideware, but if you want to see the video cast of our presentation, you can find it on the Codemotion youtube channel:Codemotion Amsterdam 2018: The State of Big Data by Roaring Elephant podcast Please use the Contact Form on this blog or our twitter feed to send us your questions, or to suggest future episode topics you would like us to cover.