Resources

Overview There are many useful articles as well official Cloudbreak documentation covers everything in great depth. This short article walks you through how to deploy the Cloudbreak instance within the existing VPC and Subnet using the AWS Quickstart deployment. Cloudbreak deployment options The cloudbreak deployment options are explained in detail here. If you notice the AWS specific Networking options, the Quickstart by default creates the new VPC and for the custom VPC the production install is recommended. In case you are doing poc and quickly want to try the Quickstart option but wanted to use the existing VPC, you can do that by enhancing the CloudFormation template which is described in next session. CloudFormation template changes When you launch the CloudFormation template for AWS Quickstart by default it selects the existing CloudFormation template https://s3.amazonaws.com/cbd-quickstart/cbd-quickstart-2.7.0.template. Instead of using the default template use t

I need to parse Kerberos KDC Log files (including the currently filling file) to find users with their host that are connecting.It seems using Grok in NiFi we can parse out a lot of different parts of these files and use them for filtering and alerting with ease.This is what many of the lines in the log file look like:Jan 01 03:31:01 somenewserver-310 krb5kdc[28593](info): AS_REQ (4 etypes {18 17 16 23}) 192.168.237.220: ISSUE: authtime 1546278185, etypes {rep=18 tkt=16 ses=18}, nn/somenewserver-310.field.hortonworks.com@HWX.COM for nn/somenewserver-310.field.hortonworks.com@HWX.COMState of the Tail ProcessorTail a FileWe also have the option of using the GrokReader listed in an article included to immediately convert matching records to output formats like JSON or Avro and then partition into groups. We'll do that in a later article.In this one, we can get a line from the file via Tail, read a list of files and fetch one at a time or generate a flow file for testing. Once we had

The following parameters decide if ACID is enabled or disabled in your cluster. ACID Enable:hive.support.concurrency=true;hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;hive.enforce.bucketing=true;hive.exec.dynamic.partition.mode=nostrict;*** The following parameters are required for standalone hive metastore ***hive.compactor.initiator.on=true;hive.compactor.worker.threads=1NOTE: Even though HiveServer2 runs with an embedded metastore, a standalone Hive metastore is required for ACID support to function properly. If you are not using ACID support with HiveServer2, you do not need a standalone metastore.ACID Disable:If not using ACID, then make sure the value of the parameters are set as below to avoid any hive locking issueshive.support.concurrency=false;hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager;hive.enforce.bucketing=false;hive.exec.dynamic.partition.mode=strict;hive.compactor.initiator.on=false;hive.compactor.worker.threads=0;

Problem:some of the rest call you might not face this issue , but some cases where A service to service call is made , for ex. hbase makes call to ranger admin to download the policy using following rest call.https://localhost:6182/service/plugins/policies/download/in this case ranger admin trust store should have certificate of Client who is trying to download the policy. so for example if hbase tries to download the policy then since we had already setup the ssl for ranger and plugins and ranger admin is having certificate for hbase plugin in ranger admin trust store so this call will work finebut if you try to make this call using curl from your own rest client then it will failcurl -k -u admin:admin 'https://localhost:6182/service/plugins/policies/download/' it will throw following kind of error:ERROR org.apache.ranger.common.ServiceUtil (ServiceUtil.java:1376) - Unauthorized access. Unable to get client certificate. serviceName=cl1_hadoop2016-07-06 05:51:46,264 [http-bio-6182-exec

Here is a self explanatory tutorial. The official Cloudbreak documentation explains how to retrieve your Cloudbreak token via curl or python (see here).I have been playing with Node.JS on my side, so here is how to do it in Node.JS using request:process.env["NODE_TLS_REJECT_UNAUTHORIZED"] = 0;var request = require("request");var options = { method: 'POST', url: 'https://[YOUR_CB_URL]/identity/oauth/authorize', qs: { response_type: 'token', client_id: 'cloudbreak_shell', 'scope.0': 'openid', source: 'login', redirect_uri: 'http://cloudbreak.shell', accept: 'application/x-www-form-urlencoded' }, headers: { 'Content-Type': 'application/x-www-form-urlencoded', accept: 'application/x-www-form-urlencoded' }, body: 'credentials={"username":"[YOUR_USER]","password":"[YOUR_PASSWORD]"' };request(options, function (error, response, body) { if (error) throw new Error(error); const querystring = require('querystring'); console.log(querystring.parse(response.h

Note: this article only deals with the disk space of each format, not the performance comparison. When using Hive as your engine for SQL queries, you might want to consider using ORC or Parquet file formats for your data. There are numerous advantages to consider when choosing ORC or Parquet. Firstly, both will give you columnar compression of the data within, whereas a plain text file will have no compression at all. Secondly, indexes within ORC or Parquet will help with query speed as some basic statistics are stored inside the files, such as min,max value, number of rows etc. Both formats can also express complex data structures (such as hierarchies) which a plain CSV file cannot do. The combination of these can boost your query many times over. Have a look at the article here to read more about the underlying mechanics of both formats: https://www.datanami.com/2018/05/16/big-data-file-formats-demystified/ More information on the ORC file format: https://cwiki.apache.org/confluenc

The number can be found in the database which is used to store hive meta data.If MySql is the backend db, select count(*) from TBLS;

In Data Science, R is commonly used for analytics and data exploration. When moving to a Hadoop architecture and a connected data platform a big question is what happens to my already existing R scripts?You can transition nicely to Hadoop using the rHadoop package for R which allows you to read from hdfs and get data back into a dataframe in R. To enable this you first need to get the R package:install.packages("rhadoop")You can also wget the packagewget https://cran.r-project.org/src/contrib/Archive/rHadoopClient/rHadoopClient_0.2.tar.gzand then install.packages("/path/to/package")now you can read a file in using the rHadoopClient:rHadoopClient::read.hdfs("/path/to/data.csv")That's all you need to get started. This allows you to change your file read steps in your R scripts to point to HDFS and still run your R scripts as you are used to doing.

1. Introduction1.1 HDP 2.6.3 provides Apache Spark 2.2 with Apache ORC 1.4 Since Apache Spark 1.4.1, Spark supports ORC as one of its FileFormat. This article introduces how to use another faster ORC file format with Apache Spark 2.2 in HDP 2.6.3. First, in order to show how to choose a FileFormat, Section 1.2 will show an example which writes and reads with ORCFileFormat. Section 2 shows a brief performance comparison, and Section 3 explains more use cases and ORC configurations. Section 4 summarizes ORC-related Apache Spark fixes included in HDP 2.6.3.1.2 Usage Example: Write and Read with ORCFileFormat %spark2.spark// Save 5 rows into an ORC file.spark.range(5).write.format("orc").mode("overwrite").save("/tmp/orc")// Read a DataFrame from the ORC file with the existing ORCFileFormat.spark.read.format("orc").load("/tmp/orc").count// Read a DataFrame from the ORC file with a new ORCFileFormat.spark.read.format("org.apache.spark.sql.execution.datasources.orc").load("/tmp/orc").countres

Abstract With native SQL Commands and the publically available Hivemall components we are able to build an advanced analytics and machine learning process for a recommendation system.Setup the EnvironmentWe will setup this environment with the latest version of the Hortonworks Sandbox and the Hivemall Repo as shown below to prepare our infrastructure for eg some relatively complicated Machine Learning concepts with a relatively simple environment.Step 1- Download and install Hortonworks Sandbox HDP 3.0.1 Step 2 - Download the Hivemall artefacts Hivemall Repo and copy into your HDFS directory (so all nodes in the cluster have access to the same)login as user: hive with the su -hive commandwget https://search.maven.org/remotecontent?filepath=org/apache/hivemall/hivemall-all/0.5.2-incubating/hivemall-all-0.5.2-incubating.jar -O hivemall-all-0.5.2-incubating.jarhdfs dfs -mkdir /apps/hive/warehouse/libhdfs dfs -put hivemall-all-0.5.2-incubating.jar /apps/hive/warehouse/lib Step 3A - Dow

Disclaimer: This article is based on my personal experience and knowledge. Don't take it as a standard guidelines, understand the concept and modify it for your environmental best practices and use case.Its always suggest contact Hortonworks Support if you have trouble in your production cluster Problem Statement: My Cluster is upgraded from ambari-2.7.1 to ambari-2.7.3 and HDP-3.0.1 to HDP-3.1. Whenever I am trying to change some configs and save it. the Below Error is shown : Error message: </strong>Stack Advisor reported an error . Exit Code: 2. Error: KeyError: 'beeline_jdbc_url_default' StdOut file: /var/run/ambari-server/stack-recommendations/12/stackadvisor.outStdErr file: /var/run/ambari-server/stack-recommendations/12/stackadvisor.err and in Ambari-server.log I can see Caused by: org.apache.ambari.server.api.services.stackadvisor.StackAdvisorException: Stack Advisor reported an error. Exit Code: 2. Error: KeyError: 'beeline_jdbc_url_default'StdOut file: /var/run/ambar

AbstractMachine learning and supervised learning at scale is a common task that can be addressed with Apache Hive by leveraging Hivemall features and user-defined functions (UDFs) for HiveQL, which is strongly optimized for machine learning (ML) and data science. For example, logistic regression models, one of the basic tenants of ML, can also be easily built with Hivemall. The following example should explain the basic usage of Hivemall with examples of supervised learning of a simple regressor.Setup the environment-> HDP 3.0.1 Sandbox - Hivemall 0.5.2 see Machine Learning with Apache Hive SQL using HivemallExplore the Training Data Database: foodmart - Table: sales_fact _dec_1998 and customer Create Table customer_training asselect s1.customer_id, c1.country, c1.gender, c1.member_card,c1.num_cars_owned, count(s1.product_id) as purchase_cntfrom sales_fact_dec_1998 s1 join customer c1WHERE s1.customer_id = c1.customer_idgroup by s1.customer_id, c1.country, c1.gender, c1.member_c

Kafka's mirroring feature makes it possible to maintain a replica of an existing Kafka cluster. This tool uses Kafka consumer to consume messages from the source cluster, and re-publishes those messages to the target cluster using an embedded Kafka producer. Running Mirror Maker To set up a mirror, run kafka.tools.MirrorMaker. The following table lists configuration options. At a minimum, MirrorMaker requires one or more consumer configuration files, a producer configuration file, and either a whitelist or a blacklist of topics. In the consumer and producer configuration files, point the consumer to the the source cluster, and point the producer to the destination (mirror) cluster, respectively.bin/kafka-run-class.sh kafka.tools.MirrorMaker --consumer.config sourceCluster1Consumer.config --consumer.config sourceCluster2Consumer.config --num.streams 2 --producer.config targetClusterProducer.config --whitelist=".*"Pa

Docker on YARN is relatively easy to set up on an existing cluster, but full clusters are not always available.My Ember Project was created to simplify dev/test for technologies managed by Ambari and Cloudera Manager. It provides utilities for creating dockerized clusters that use far fewer resources than a full bare metal or VM-based cluster. Additionally, by using pre-built images, the time it takes to get a cluster up and running can be reduced to less than 10 minutes.The following four commands are all that is necessary to download and run a ~5GB image that comes preinstalled with Ambari, Zookeeper, HDFS, and YARN with Docker on YARN pre-configured. Docker containers spawned by YARN will be created on the host machine as peers to the container with YARN inside. All containers are launched into the "ember" docker network by default. Once the container is downloaded, it takes less than 5 minutes to start all services.curl -L https://github.com/SamHjelmfelt/Ember/archive/v1.1.zip -o E

Ingesting Drone Data From DJII Ryze Tello Drones Part 1 - Setup and PracticeIn Part 1, we will setup our drone, our communication environment, capture the data and do initial analysis. We will eventually grab live video stream for object detection, real-time flight control and real-time data ingest of photos, videos and sensor readings. We will have Apache NiFi react to live situations facing the drone and have it issue flight commands via UDP.In this initial section, we will control the drone with Python which can be triggered by NiFi. Apache NiFi will ingest log data that is stored as CSV files on a NiFi node connected to the drone's WiFi. This will eventually move to a dedicated embedded device running MiniFi.This is a small personal drone with less than 13 minutes of flight time per battery. This is not a commercial drone, but gives you an idea of the what you can do with drones.Drone Live Communications for Sensor Readings and Drone ControlYou must connect to the drone's WiFi, whi

Okay, so in preparation for the DataWorks Summit :: San Jose I was going over the Spark 2 cluster we give our students, you know - testing the important labs, etc. and lo and behold I found a problem: Testing my old code for Spark 2 Structured Streaming with Apache Kafka was suddenly broken with an odd Exception "MicroBatch Execution: Query [id = blah blah blah ... java.lang.AbstractMethodError." But I have no abstract methods in my code, and this has always worked before ... so what the heck?:18/06/1219:14:59 ERROR MicroBatchExecution: Query [id = bf5215b6-6dec-4ed3-be11-991da4025bc8, runId = 78680de5-be6f-4b60-a635-1cc96003b054] terminated with errorjava.lang.AbstractMethodErrorat org.apache.spark.internal.Logging$class.initializeLogIfNecessary(Logging.scala:99)at org.apache.spark.sql.kafka010.KafkaSourceProvider.initializeLogIfNecessary(KafkaSourceProvider.scala:40)at org.apache.spark.internal.Logging$class.log(Logging.scala:46)at org.apache.spark.sql.kafka010.KafkaSourceProvider.lo

Problem statement: How to upgrade Apache Ambari 2.6.2 to Apache Ambari 2.7.3 Abstract:Apache Ambari 2.7.3 is the latest among Ambari 2.7.x releases. Ambari 2.7.0, which was the first release in the 2.7.x series introduced significant improvements from its predecessor - Ambari 2.6.2. This tutorial will help users upgrade from Ambari 2.6.2.2 to Ambari 2.7.3 Procedure: I. Prerequisites 1. Backup Ambari configuration file # mkdir /root/backups# cp /etc/ambari-server/conf/ambari.properties /root/backups 2. Turn off Service Auto RestartFrom Ambari UI: Admin > Service Auto Start. Set Auto Start Services to Disabled. Click Save. 3. Run Service Checks on all Ambari servicesOn each of the Ambari services installed on your cluster, Run Service Checks as shown below:From Ambari UI: <Service_Name> > Service Actions > Run Service CheckFor example: HDFS > Service Actions > Run Service Check4. Start and Stop all of the Ambari services from Ambari UI.II. Stop Services1.

I'd like to thank @Jean-Philippe Player, @bpreachuk, @ghagleitner, @gopal, @ndembla and @Prasanth Jayachandran for providing input and content for this article.Introduction This document describes LLAP setup for reasonable performance with a typical workload. It is intended as a starting point, not as the definitive answer to all tuning questions. However, recommendations in this document have been applied both to our own instances as well as instances we help other teams to run.The context for the examples Each step below will be accompanied by an example based on allocating ~50% of a cluster to LLAP, out of 12 nodes with 16 cores and 96Gb dedicated to YARN NodeManager each.Basic cluster setupPick the fraction of the cluster to be used by LLAP This is determined by the workload (amount of data, concurrency, types of queries, desired latencies); typical values can be in 15-50% range on an existing cluster, or it is possible to give entire cluster to LLAP, depending on customer us

DISCLAIMER: This is a method that is not recommended for any production use, only for development purposes. Only Cloudbreak official versions are supported, and upgrading to a non-supported version could cause unforeseen issues, and loss of official Hortonworks support. Moreover, this method only works for upgrades, not downgrades; downgrades would result in data loss. Recently, I have been playing with Cloudbreak and have been needing to upgrade my local version. While it follows the principles detailed in Hortonworks documentation, I thought I'd share a quick step by step guide on how to upgrade to any version of Cloudbreak. 1. Stop Cloudbreak on your machine: cbd kill 1. Go to https://mvnrepository.com/artifact/com.sequenceiq/cloudbreak and find the version you need (e.g. 2.8.1-rc.48) 2. On the VM where Cloudbreak is running, navigate to the directory where your Profile file is located. For example: cd /var/lib/cloudbreak-deployment/ 2. Run the following commands to downlo

Image Data Flow for Industrial ImagingOBJECTIVE:Ingest and store manufacturing quality assurance images, measurements, and metadata in a cost-effective and simple-to-retrieve-from platform that can provide analytic capability in the future.OVERVIEW:In high-speed manufacturing, imaging systems may be used to identify material imperfections, monitor thermal state, or identify when tolerances are exceeded. Many commercially-available systems automate measurement and reporting of specific tests, but combining results from multiple instrumentation vendors, longer-term storage, process analytics, and comprehensive auditability require different technology.Using HDF’s NiFi and HDP’s HDFS, Hive or Hbase, and Zeppelin, one can build a cost-effective and performant solution to store and retrieve these images, as well as provide a platform for machine learning based on that data.Sample files and code, including the Zeppelin notebook, can be found on this github repository: https://github.com/wcbd

This article series is an expansion into the technical details behind the Big Data Processing Engines blog post: https://hortonworks.com/blog/big-data-processing-engines-which-one-do-i-use-part-1/ Intro to performance analysis Here, we will be deep diving into the performance side of the three Big Data Processing Engines discussed in the above blog post: Druid, HBase, and Hive. I ran a number of query types to represent the various workloads generally executed on these processing engines, and measured their performance in a consistent manner. I ran a few tests on each of the three engines to showcase their different strengths, and show where some are less effective (but could still fill the gap in a pinch). We will be executing the following queries: Simple count of all records in a table, highlighting aggregation capabilities A select with a where clause, highlighting drill-down and "needle in the haystack" OLTP queries A join, showcasing ad-hoc analysis across the dataset Updates, s

Pre-requisites Basic knowledge of NiFi, processors, custom properties, and configuring a NiFi flowSalesforce developer account with Bulk API (SOAP API) accessWorking HDF clusterGeneral WorkflowYou will be performing the following steps for each API interaction after authenticating into Salesforce and getting your SessionID. The following is for an enterprise account. If you have a partner account, these instructions are still mostly valid but you may need to tweak your endpoints slightly when communicating with SFDC. Retrieve an XML request body from disk or another accessible locationModify the XML if necessary based on requirementsSend your request, receive a response from the SFDC APIParse the XML result and pull out any data that is necessary to continue the interactionAuthenticate into SalesforceYou will first need to authenticate into Salesforce using your username and password. Create a file named login.txt that has the following information in it:<?xml version="1.0" encoding

This is the second article in the 2-part series (see Part 1) where we look at yet another architectural variation of the distributed pricing engine discussed here, this time leveraging a P2P cluster design through the Akka framework deployed on Kubernetes with a native API Gateway using Ambassador that in turn leverages Envoy, an L4/L7 proxy, and monitoring infrastructure powered by Prometheus.The Cloud Native JourneyIn the previous article, we just scratched the surface of one of the top level CNCF projects – Kubernetes - the core container management, orchestration and scheduling platform. In this article, we will take our cloud native journey to the next level by exploring more nuanced aspects of Kubernetes (support for stateful applications through StatefulSets, Service, Ingress) including the use of package manager for Kubernetes – Helm.We will also look at the two other top level projects of CNCF – Prometheus and Envoy and how these form the fundamental building blocks of any pro

This article is the first in a series of articles discussing various topics regarding frontend UI/UX development on the Apache NiFi project (versions 1.1.2+). This series will explore some of the architecture and bootstrapping of the client side JavaScript as well as various "recipes", case studies, and examples on how to accomplish certain tasks.Introduction Keeping up with modern web development best practices, frameworks, and anti-patterns can be intimidating. Frameworks like AngularJS, Ember.js, Knockout, Backbone.js turned the JavaScript world inside out just 3-5 years ago with their declarative styles and TDD approach. Couple that with all the ever changing ECMAscript standard, differing support of said standard in various browsers, and the fact that as memory and storage capabilities have continued to increase and at lower price points more and more functionality is expected from, and placed on, the client and you start to understand that web development is still maturing and

Problem Statement : The hdfs encryption guide for HDP 3.0.1 states "In Ambari, replace the current value of dfs.permissions.superusergroup with the group name “operator”. however, this option is not available from Ambari. RootCause : As per fix of Apache bug : https://issues.apache.org/jira/browse/AMBARI-22086 ,ambari considers the dfs.permissions.superusergroup as a group property and by default the group property is not meant to be edited in ambari-ui. So the ui disabled the editing of this property by default.Workaround :Navigate to ambari-server and change this config via configs.py/var/lib/ambari-server/resources/scripts/configs.py -l <AMBARI_HOSTNAME>-t 8080-u <USER_NAME>-p <PASSWORD>-a <GET/SET/DELETE>-n <CLUSTER_NAME>-c <CONFIG_TYPE>-k <KEY>-v <VALUE>For ex :[root@asnaik-asnaik1 ~]# /var/lib/ambari-server/resources/scripts/configs.py --helpUsage: configs.py [options]Options: -h, --help show this help message and exit