Community Articles

This video describes the steps performed to write custom UDFs in Hive.     Open the video on YouTube here  

This video focuses on how to retrieve the cURL commands for various tasks performed in Ambari UI.     Open YouTube video here  

This video explains how to move the master service from one node to another using Ambari Web UI. The video also talks about the need for this option.     Open the video on YouTube here

This video describes why Beeline must be used over Hive CLI when Ranger is in place. It is recommended to use Beeline when Ranger is enabled in the clusters.     Open the video on YouTube here

This video describes the configuration for Ranger SSL.  This video reviews the architecture and steps needed to secure Ranger Admin UI. Some basic troubleshooting tips are also covered.     Open YouTube video here

This is a how to video on Configuring Hive View 2.0 to view Ranger Policies associated with any table, which can be used by an Ambari Admin to configure the views. This will ease the task of Hive users to directly view the policies from the views table rather than checking them from the Ranger Admin UI. Additionally, this video covers the policies for table level which can be helpful in debugging the security issues with Hive and Ranger.     Open YouTube video here

Hey folks, I was recently doing some basic benchmarks and then I tried to use our classical DFSIO suite but I found myself stuck with the following error : main : run as user is hdfsmain : requested yarn user is hdfsRequested user hdfs is bannedAs TestDFSIO default output dir is /benchmark/TestDFSIO and it is automatically created with the following permission :inode="/benchmarks/TestDFSIO/io_control/in_file_test_io_0":hdfs:hdfs:drwxr-xr-xThe use of hdfs user is mandatory. At least that's what i thought...Why HDFS user is banned ?After looking around some configuration files I found the banned.user property in : /etc/hadoop/cong/container-executor.cfg#/*# * Licensed to the Apache Software Foundation (ASF) under one# * or more contributor license agreements. See the NOTICE file# * distributed with this work for additional information# * regarding copyright ownership. The ASF licenses this file# * to you under the Apache License, Version 2.0 (the# * "License"); you may not use this

Hey everyone, Today's video gives some high level detail about how the Cloudera Data Platform is secure by design. We designed CDP to safeguard your organization's data within the virtual network of your chosen public cloud vendor. Also, we took all of the hard work associated with setting up security tools, such as kerberos, and automated it. This way the platform is secure and ready to start using right away. Here's a short quick video explaining these concepts. Let us know by adding a comment below what topics you care about and keep checking back as we will be posting new videos about CDP and more at least once a week.   More information: What is Cloudera's Shared Data Experience (SDX)? https://community.cloudera.com/t5/Community-Articles/What-is-Cloudera-s-Shared-Data-Experience-SDX/ta-p/284102  

Motivation: As zeppelin is used for Data Ingestion/Discovery/Analytics/Visualization, the muti-purpose notebook's usage is increasing tremendously pushing users to find a way to scale out. Intention of the article is to give a fair idea on how zeppelin can be scaled out with the current available options in 2.6.xDisclaimer: This article is based on my personal experience and knowledge. Don't take it as a standard guidelines, understand the concept and modify it for your environmental best practices and use case.Beefing up the Zeppelin/interpreter Memory, tweaking the performance affecting factors will help to a certain extend but when the usage of notebook increase it is wise to scale out.In this article we are going discuss on various considerations and best practices while scaling out. Note: Please refer Key factors that affects Zeppelin's Performance before deciding to scaling out, It will be also having the best practices to maintain zeppelin EnvironmentBenefits of multiple in

Introduction Knowing the performance of Kafka in general or on your hardware is an important part of capacity planning. Sizing can be hard to calculate, with different message sizes, retention periods, partitions, replica factor, network speeds and even synchronous vs asynchronous to pick from. So many decisions to make, but what type of performance can you realistically expect ?   Benchmarking for Kafka has been done before, however this is a couple of years out of date. There are newer versions of Kafka, newer hardware, faster networking all which have improved performance by many factors. But how ? This article will run the same tests as before with the latest and greatest to see what the improvements are.   Hardware specification: Cloud Provider AWS using m5.4xlarge * 6 machines OS Centos 7 CPU 16 cores RAM 64GB Disk 50GB OS, 6 * 20 GB SSD IO Optimised for Kafka log dirs Networking 10GB/s Kafka Version CDH 6.3 with Kafka 2.3 Message size 100 bytes per me

Hey everyone,  Check out our first video in the Data Hub Collection where we cover the basics of the Data Hub experience. Cloudera Data Hub lets users easily provision clusters with the services you care about with the security and governance your organization requires.    Let us know by adding a comment below what topics you care about and keep checking back as we will be posting new videos about CDP and more at least once a week.    

The purpose of this article is to provide the steps needed to create your own certificates for securing your NiFi instance(s). The article will also cover creating your own Certificate Authority (CA) that you can use to sign all the certificates you create. This article is not intended to be a best practices guide to creating secure keys. While we will provides tips, users should carefully research the various security options available when creating keys. This procedure assumes you have Java Keytool and OpenSSL installed on your system. HDF 1.x or Apache NiFi 0.x Secured UI:HDF 2.x or Apache NiFi 1.x Secured UI:Creating your Certificate Authority: You only need to create one CA, which you will use to sign the keys for every one of your servers/VMs and users (You only need to create keys for users if your NiFi has not been configured to use LDAP authentication). What is a CA? The CA acts as a trusted entity for validating the authenticity of certificates. The CA is used to certify t

Apache Metron vs. OpenSocApache Metron inherits the advantages of OpenSoc which enables fast processing of events from variety sources. One of its intent is to overcome the shortcomings of OpenSoc. The main challenges of OpenSoc architecture are: Does not take advantage of full parallelism. For each topic, the enrichment topology contains a set of bolts which run in serially. This architecture does not take advantage of the parallelism that storm could provide. It can be observed in OpenSoc architecture in Figure 1.Hard to extend. To add a new data source, it has to create a new topology to fulfill that requirement. Hard to maintain. With the number of data sources increases, there are lots of “redundant” topologies. Many logics of bolts in enrichment topology are similar, however, due to the architecture of OpenSoc, each topic has to maintain its own line of code. If any change is needed for a certain shared logic for topics, it has to be modified across multiple lines of code, which

Traditional Prometheus Architecture(Image Courtesy: https://prometheus.io/docs/introduction/overview/)Prometheus is great. It has a huge number of integrations and provides a great metric monitoring platform, especially when working with Kubernetes. However, it does have a few shortcomings. The aim of this alternative architecture is to preserve the best parts of Prometheus while augmenting its weaker points with more powerful technologies. The service discovery and metric scraping framework in Prometheus is its greatest strength, but it is greatly limited by its tight coupling to the TSDB system inside Prometheus. While it is possible to replace the TSDB inside Prometheus with an external database, the data retrieval process only supports writing into this one database.Maui ArchitectureThe greatest strength of Prometheus, the service discovery and metric scraping framework, can now be used within Apache NiFi with the introduction of the GetPrometheusMetrics processor. This proces

Ever wondered what Cloudera's Shared Data Experience (SDX) is? Want to know how SDX can help you or your organization? Well here's a video that will hopefully clear up some confusion and tell you about some of the powerful things SDX enables.    Let us know what topics you care about and keep checking back as we will be posting new videos about CDP and more at least once a week.  

Since getting started with Hadoop & ELK I have taken the original Hortonworks 5.x Elasticsearch and Kibana Management Pack for HDP 2.6.5 and upgraded it to ELK 6.3.2.  During this project I also added Logstash, FileBeat, and MetricBeat.  Next, I upgraded the Management Pack to work with HDP 3.x and HDF 3.x.     In this article I am going to share with you all of the steps required to upgrade the ELK 6.3.2 Management Pack to latest, greatest versions of ELK 7.4.2.  You can find all of the files in my GitHub repo: DFHZ ELK Mpack   First, I create a known working test cluster making sure the 3.4-0 management pack is operational and my test environment is suitable. Commands for a single node HDF test cluster: # history 1 hostnamectl set-hostname elk.cloudera.com 2 yum install nano wget -y 3 nano /etc/sysconfig/selinux 4 nano /etc/cloud/cloud.cfg 5 nano /etc/hosts 6 reboot 9 wget -O /etc/yum.repos.d/ambari.repo http://public-re

New in Cloudera AI 1.2 we’re excited to announce support for hosting persistent web based applications and dashboards using frameworks like Flask, Dash and Shiny to share analytics results and insights with business stakeholders.   Example Use case   Here’s a quick example of how it works based on a use-case involving New York City transportation data: Part 1 of the analysis compares workday trip data to facilitate the appropriate choice of transportation (bike vs. car), given a time of the day and route (Analysis based on this blog post by “Todd  W. Schneider”.) Part 2 of the analysis demonstrates outlier detection for trip durations, which might be interesting for internal audit purposes. This example uses the most recent data from Citibike and NYC TLC for-hire vehicle (FHV) data.   Publishing the app   Two pre-trained machine learning models (simple sklearn logistic regression models) are served by CML to be accessed by the application via API. As our fo

Apache NiFi flow to recursively call the HDFS Namenode WebHDFS API to track disk consumption per user across the cluster for graphing, monitoring, capacity planning and alerting.   It can often times be hard to track the disk consumption per user on your HDFS filesystem. There isn't really a single command to tell you how much disk space a specific user is using across the filesystem (over many directories), nor can you easily track this over time to predict future consumption.   This article attempts to solve both problems. Using a NiFi flow to recursively pull the status of each file stored on HDFS, and storing that in a database for visualising through Superset and/or Grafana.   You can run REST queries against the HDFS active Namenode, and get a listing of all the files and directories at a specific location. Here is an example of query against the Namenode for '/user/willie':       curl -s 'http://localhost:50070/webhdfs/v1/user/willie/?op=liststatu

I have written articles in the past benchmarking Hadoop cloud environments such at BigStep and AWS. What I didn't dive into those article is how I ran the script. I built scripts to rapidly launch TeraGen, TeraSort, and TeraValidate. Why? I found myself running the same script over and over and over again. Why not make it easier by simply executing a shell script.All scripts I mentioned are located here.Grab the following filesteragen.shterasort.shvalidate.shTo run TeraGen, TeraSort, and TeraValidate a determination of the volume of data and number of records is required. For example you can generate 500GB of data with 5000000000 rows. The script comes with the following predefined sets#SIZE=500G#ROWS=5000000000#SIZE=100G#ROWS=1000000000#This will be used as it only value uncommented outSIZE=1T ROWS=10000000000# SIZE=10G#ROWS=100000000# SIZE=1G# ROWS=10000000Above 1T (for terabyte) and rows 10000000000 are uncommented out. Meaning this script will generate 1TB of data with 10000000

/hbase , /hbase-unsecure , /hbase-secureThis is the root znode for HBase. Older versions used to have just /hbase while newer ones distinguish whether cluster is secured or unsecured. /hbase-unsecure/hbaseid UUID of the cluster. Also stored in /hbase/hbase.id file in HDFS. /hbase-unsecure/masterContains the active master server hostname.Written during master server startup. /hbase-unsecure/backup-mastersAll standby master servers are registered here /hbase-unsecure/meta-region-serverRegisters the hostname of region server which holds Meta table. /hbase-unsecure/rsActs as the root node for all region servers to list themselves when they start. It is used to track server failures. Each znode inside is ephemeral and its name is the server name of the region server. /hbase-unsecure/splitWAL The parent znode for all log-splitting-related coordination /hbase-unsecure/balancerStatus of load balancer enabled / disabled on cluster. /hbase-unsecure/region-in-transitionList of regions in tra

Apache Ranger is a framework to enable, monitor and manage comprehensive data security across the Big Data platform. Ranger allows you to centralize security administration to manage all security related tasks in a central UI or using REST APIs. You can setup fine grained authorization to do a specific action and/or operation with the platform's components/tools through role based and/or attribute based access control. Ranger also supports central auditing of all events across the cluster on access/edit/update/delete operations. Ranger has support for the following components through plugins: HDFS HBase Hive YARN KNOX Storm SOLR Kafka NiFi NiFi-Registry Atlas This article will provide an example of setting up Ranger database policies as well as column masking rules. The example will also show what happens when you have two tables, with one table having column masking rules applied for a specific user, and trying to join the two tables. Scenario: Two users: user_sales, user_hr Two t

INTRODUCTION With IoT growing at an exponential rate it is imperative that we continuously adapt our streaming architectures to stay ahead of the game.Apache Flink provides many great capabilities for stream processing that can be applied to a modern data architecture to enable advanced stream processing like windowing of streaming data, anomaly detection, and process monitoring to name a few. Something more important that any one tool; however, is the overall architecture of your solution and how it enables your full data lifecycle.In order to provide a robust solution you must take into account and make decisions that select the best tools for each part of your data's journey. Let's look at an example that will be growing rapidly over the next few years. Cellular IoT devices! In particular the IoT devices mad by particle.io.   I will walk you through the end to end solution designed to: Deploy a cellular IoT device at the edge programmed to collect Temperature re

Objective of this article is to demonstrate how to rapidly deploy a demo Druid & LLAP cluster preloaded with 20 years (nearly 113 million records) of airline data ready for analytics using CloudBreak on any IaaS. Entire deployment is UI driven without the need for a large overhead of administration. All artifacts mentioned in this article are publicly available for reuse to try on your ownProlegomenonTime series is an incredible capability highly leveraged within the IoT space. Current solution sets offer non scalable & expensive or distributed processing engines lacking low latency OLAP speeds. Druid is an OLAP time series engine backed by a Lambda architecture. Druid out of the box SQL capabilities are severely limited and without join support. Layering HiveQL over Druid brings the best of both worlds. Hive 3 also offers HiveQL over Kafka essentially making Hive a true SQL federation engine. With Druid’s native integration with Kafka, streaming data from Kafka directly i

Exploring Apache NiFi 1.10:   Parameters and Stateless Engine Apache NiFi is now available in 1.10!   https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12344993   You can now use JDK 8 or JDK 11!   I am running in JDK 11, seems a bit faster.   A huge feature is the addition of Parameters!   And you can use these to pass parameters to Apache NiFi Stateless!   A few lesser Processors have been moved from the main download, see here for migration hints: https://cwiki.apache.org/confluence/display/NIFI/Migration+Guidance   Release Notes:   https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.10.0   Example Source Code:   https://github.com/tspannhw/stateless-examples   More New Features:   ParquetReader/Writer (See:  https://www.datainmotion.dev/2019/10/migrating-apache-flume-flows-to-apache_7.html) Prometheus Reporting Tas

CDP has recently been introduced to the market, and we all want to learn about it, so I decided to contribute a bit.   In this series of tutorials, I want to explain the basics of CDP Data Hub and provide some ease of automation. Indeed, the CDP control plane user interface provides a thorough workflow that automates the creation of main elements that ensures security, governance and scalability.     As depicted above, these main elements are: CDP Environment: Cloud hosted (in your cloud) resources for CDP deployment CDP Data Lake: Host of CDP SDX, the shared service layer providing all security and governance CDP Data Hub cluster(s): where users run their workloads Note: CDP offers a lot more than these basic elements, and a lot more ways to configure them; this is a 101 tutorial and will not address all that CDP has to offer. For more information, visit the CDP documentation or product page.   Instead, this tutorial series will teach you: How to create a CDP