Community Articles

Ingesting Drone Data From DJII Ryze Tello Drones Part 1 - Setup and PracticeIn Part 1, we will setup our drone, our communication environment, capture the data and do initial analysis. We will eventually grab live video stream for object detection, real-time flight control and real-time data ingest of photos, videos and sensor readings. We will have Apache NiFi react to live situations facing the drone and have it issue flight commands via UDP.In this initial section, we will control the drone with Python which can be triggered by NiFi. Apache NiFi will ingest log data that is stored as CSV files on a NiFi node connected to the drone's WiFi. This will eventually move to a dedicated embedded device running MiniFi.This is a small personal drone with less than 13 minutes of flight time per battery. This is not a commercial drone, but gives you an idea of the what you can do with drones.Drone Live Communications for Sensor Readings and Drone ControlYou must connect to the drone's WiFi, whi

Okay, so in preparation for the DataWorks Summit :: San Jose I was going over the Spark 2 cluster we give our students, you know - testing the important labs, etc. and lo and behold I found a problem: Testing my old code for Spark 2 Structured Streaming with Apache Kafka was suddenly broken with an odd Exception "MicroBatch Execution: Query [id = blah blah blah ... java.lang.AbstractMethodError." But I have no abstract methods in my code, and this has always worked before ... so what the heck?:18/06/1219:14:59 ERROR MicroBatchExecution: Query [id = bf5215b6-6dec-4ed3-be11-991da4025bc8, runId = 78680de5-be6f-4b60-a635-1cc96003b054] terminated with errorjava.lang.AbstractMethodErrorat org.apache.spark.internal.Logging$class.initializeLogIfNecessary(Logging.scala:99)at org.apache.spark.sql.kafka010.KafkaSourceProvider.initializeLogIfNecessary(KafkaSourceProvider.scala:40)at org.apache.spark.internal.Logging$class.log(Logging.scala:46)at org.apache.spark.sql.kafka010.KafkaSourceProvider.lo

Problem statement: How to upgrade Apache Ambari 2.6.2 to Apache Ambari 2.7.3 Abstract:Apache Ambari 2.7.3 is the latest among Ambari 2.7.x releases. Ambari 2.7.0, which was the first release in the 2.7.x series introduced significant improvements from its predecessor - Ambari 2.6.2. This tutorial will help users upgrade from Ambari 2.6.2.2 to Ambari 2.7.3 Procedure: I. Prerequisites 1. Backup Ambari configuration file # mkdir /root/backups# cp /etc/ambari-server/conf/ambari.properties /root/backups 2. Turn off Service Auto RestartFrom Ambari UI: Admin > Service Auto Start. Set Auto Start Services to Disabled. Click Save. 3. Run Service Checks on all Ambari servicesOn each of the Ambari services installed on your cluster, Run Service Checks as shown below:From Ambari UI: <Service_Name> > Service Actions > Run Service CheckFor example: HDFS > Service Actions > Run Service Check4. Start and Stop all of the Ambari services from Ambari UI.II. Stop Services1.

I'd like to thank @Jean-Philippe Player, @bpreachuk, @ghagleitner, @gopal, @ndembla and @Prasanth Jayachandran for providing input and content for this article.Introduction This document describes LLAP setup for reasonable performance with a typical workload. It is intended as a starting point, not as the definitive answer to all tuning questions. However, recommendations in this document have been applied both to our own instances as well as instances we help other teams to run.The context for the examples Each step below will be accompanied by an example based on allocating ~50% of a cluster to LLAP, out of 12 nodes with 16 cores and 96Gb dedicated to YARN NodeManager each.Basic cluster setupPick the fraction of the cluster to be used by LLAP This is determined by the workload (amount of data, concurrency, types of queries, desired latencies); typical values can be in 15-50% range on an existing cluster, or it is possible to give entire cluster to LLAP, depending on customer us

DISCLAIMER: This is a method that is not recommended for any production use, only for development purposes. Only Cloudbreak official versions are supported, and upgrading to a non-supported version could cause unforeseen issues, and loss of official Hortonworks support. Moreover, this method only works for upgrades, not downgrades; downgrades would result in data loss. Recently, I have been playing with Cloudbreak and have been needing to upgrade my local version. While it follows the principles detailed in Hortonworks documentation, I thought I'd share a quick step by step guide on how to upgrade to any version of Cloudbreak. 1. Stop Cloudbreak on your machine: cbd kill 1. Go to https://mvnrepository.com/artifact/com.sequenceiq/cloudbreak and find the version you need (e.g. 2.8.1-rc.48) 2. On the VM where Cloudbreak is running, navigate to the directory where your Profile file is located. For example: cd /var/lib/cloudbreak-deployment/ 2. Run the following commands to downlo

Image Data Flow for Industrial ImagingOBJECTIVE:Ingest and store manufacturing quality assurance images, measurements, and metadata in a cost-effective and simple-to-retrieve-from platform that can provide analytic capability in the future.OVERVIEW:In high-speed manufacturing, imaging systems may be used to identify material imperfections, monitor thermal state, or identify when tolerances are exceeded. Many commercially-available systems automate measurement and reporting of specific tests, but combining results from multiple instrumentation vendors, longer-term storage, process analytics, and comprehensive auditability require different technology.Using HDF’s NiFi and HDP’s HDFS, Hive or Hbase, and Zeppelin, one can build a cost-effective and performant solution to store and retrieve these images, as well as provide a platform for machine learning based on that data.Sample files and code, including the Zeppelin notebook, can be found on this github repository: https://github.com/wcbd

This article series is an expansion into the technical details behind the Big Data Processing Engines blog post: https://hortonworks.com/blog/big-data-processing-engines-which-one-do-i-use-part-1/ Intro to performance analysis Here, we will be deep diving into the performance side of the three Big Data Processing Engines discussed in the above blog post: Druid, HBase, and Hive. I ran a number of query types to represent the various workloads generally executed on these processing engines, and measured their performance in a consistent manner. I ran a few tests on each of the three engines to showcase their different strengths, and show where some are less effective (but could still fill the gap in a pinch). We will be executing the following queries: Simple count of all records in a table, highlighting aggregation capabilities A select with a where clause, highlighting drill-down and "needle in the haystack" OLTP queries A join, showcasing ad-hoc analysis across the dataset Updates, s

Pre-requisites Basic knowledge of NiFi, processors, custom properties, and configuring a NiFi flowSalesforce developer account with Bulk API (SOAP API) accessWorking HDF clusterGeneral WorkflowYou will be performing the following steps for each API interaction after authenticating into Salesforce and getting your SessionID. The following is for an enterprise account. If you have a partner account, these instructions are still mostly valid but you may need to tweak your endpoints slightly when communicating with SFDC. Retrieve an XML request body from disk or another accessible locationModify the XML if necessary based on requirementsSend your request, receive a response from the SFDC APIParse the XML result and pull out any data that is necessary to continue the interactionAuthenticate into SalesforceYou will first need to authenticate into Salesforce using your username and password. Create a file named login.txt that has the following information in it:<?xml version="1.0" encoding

This is the second article in the 2-part series (see Part 1) where we look at yet another architectural variation of the distributed pricing engine discussed here, this time leveraging a P2P cluster design through the Akka framework deployed on Kubernetes with a native API Gateway using Ambassador that in turn leverages Envoy, an L4/L7 proxy, and monitoring infrastructure powered by Prometheus.The Cloud Native JourneyIn the previous article, we just scratched the surface of one of the top level CNCF projects – Kubernetes - the core container management, orchestration and scheduling platform. In this article, we will take our cloud native journey to the next level by exploring more nuanced aspects of Kubernetes (support for stateful applications through StatefulSets, Service, Ingress) including the use of package manager for Kubernetes – Helm.We will also look at the two other top level projects of CNCF – Prometheus and Envoy and how these form the fundamental building blocks of any pro

This article is the first in a series of articles discussing various topics regarding frontend UI/UX development on the Apache NiFi project (versions 1.1.2+). This series will explore some of the architecture and bootstrapping of the client side JavaScript as well as various "recipes", case studies, and examples on how to accomplish certain tasks.Introduction Keeping up with modern web development best practices, frameworks, and anti-patterns can be intimidating. Frameworks like AngularJS, Ember.js, Knockout, Backbone.js turned the JavaScript world inside out just 3-5 years ago with their declarative styles and TDD approach. Couple that with all the ever changing ECMAscript standard, differing support of said standard in various browsers, and the fact that as memory and storage capabilities have continued to increase and at lower price points more and more functionality is expected from, and placed on, the client and you start to understand that web development is still maturing and

Problem Statement : The hdfs encryption guide for HDP 3.0.1 states "In Ambari, replace the current value of dfs.permissions.superusergroup with the group name “operator”. however, this option is not available from Ambari. RootCause : As per fix of Apache bug : https://issues.apache.org/jira/browse/AMBARI-22086 ,ambari considers the dfs.permissions.superusergroup as a group property and by default the group property is not meant to be edited in ambari-ui. So the ui disabled the editing of this property by default.Workaround :Navigate to ambari-server and change this config via configs.py/var/lib/ambari-server/resources/scripts/configs.py -l <AMBARI_HOSTNAME>-t 8080-u <USER_NAME>-p <PASSWORD>-a <GET/SET/DELETE>-n <CLUSTER_NAME>-c <CONFIG_TYPE>-k <KEY>-v <VALUE>For ex :[root@asnaik-asnaik1 ~]# /var/lib/ambari-server/resources/scripts/configs.py --helpUsage: configs.py [options]Options: -h, --help show this help message and exit

Manufacturing IoT/Process Monitoring DemoSummaryYou will be provided with background on what type of usage cases Manufacturing Shop Floors are implementing to maintain a competitive advantage in their respective marketplaces. And then we will walk through, step by step, instructions on how to build your own Manufacturing IoT - Process Monitoring Dashboard environment. This dashboard will be isolated in this demo to two fans on a single Manufacturing Shop Floor and a single cluster for demo purposes.What’s IncludedInstructions and pointers on how to:Obtain and deploy HDP, HDF, and KEPServerEXSetup KEPServerEX to access an OPC TCP UA ServerSetup KEPServerEX - IoT Gateway as a REST API EndpointA pre-build Nifi Flow (fans_demo_v7.xml)Use either the Nifi “InvokeHTTP” or “GetFile” processorInbound Source can be an IoT Gateway REST API or Fan Event File Dump dataA Fan Events Data Tar file containing a dump of individual fan events (fans.tar.gz)A Hive Create Table query (create_table_fans_demo

IoT Series: Sensors: Utilizing Breakout Garden Hat: Part 2 - Integrating MQTT, TensorFlow and Kafka StreamsSee Part 1: https://community.hortonworks.com/content/kbentry/229522/iot-series-sensors-utilizing-breakout-garden-hat-p.htmlIn this second part, I have incremented the functionality in the Python capture, MiniFi, NiFi and post-NiFi processing. I have added a Kafka Streams Java application.With this NiFi flow we are consuming the MQTT and Kafka messages send by the Kafka Streams application.In one flow, we received MQTT messages, pull out the entire flow as a message and send to a Slack channel.In another flow we ingest two types of Kafka messages and store the JSON ones that have a schema in an HBase table via the record processor. In this flow we receive from the local NiFi router that was called by MiniFi over S2S/HTTP(s). We build two types of messages and send them to Kafka 2.0 brokers. One is the full JSON message with a schema, the other is just the temperature. We

Implementing Streaming Machine Learning and Deep Learning In Production Part 1After we have done our data exploration with Apache Zeppelin, Hortonworks Data Analytics Studio and other Data Science Notebooks and Tools, we will start building iterations of ever improving models that need to be used in live environments. These will need to run at scale and score millions of records in real-time streams.These can be in various frameworks, versions,types and many options of data required.There are a number of things we need to think about when doing this.Model Deployment OptionsApache SparkApache Storm (Hortonworks Streaming Analytics Manager - SAM)Apache Kafka StreamsApache NiFiYARN 3.1YARN SubmarineTensorFlow Serving on YARNCloudera Data Science WorkbenchRequirementsClassificationREST APISecurityAutomationData LineageSchema Versioning, REST API and ManagementData ProvenanceScriptingIntegration with KafkaContainerized Services Support Docker Containers running on YARN Support Dockerized

Problem statement : i am upgrading my ambari-2.6.2.2 to ambari-2.7.3 and the ambari-server upgrade commands fails with exception : java.sql.SQLSyntaxErrorException: Unknown table 'ambari_configuration' in information_schema[root@slambe-1 java]# ambari-server upgradeUsing python /usr/bin/pythonUpgrading ambari-serverINFO: Upgrade Ambari ServerINFO: Updating Ambari Server properties in ambari.properties ...INFO: Updating Ambari Server properties in ambari-env.sh ...INFO: Original file ambari-env.sh keptINFO: Fixing database objects ownerAmbari Server configured for MySQL. Confirm you have made a backup of the Ambari Server database [y/n] (n)? yINFO: Upgrading database schemaERROR: Unexpected ValueError: No JSON object could be decodedFor more info run ambari-server with -v or --verbose optionwhen i see ambari-server logs i see the following exception:2018-12-28 13:59:07,062 INFO [main] DBAccessorImpl:869 - Executing query: CREATE TABLE ambari_configuration (category_name VARCHAR(100)

Beeline is a JDBC client tool which is used to connect to HiveServer2 or HiveServer2Interactive(LLAP). Beeline, which connects to HiveServer2 and requires access to only one .jar file: hive-jdbc-<version>-standalone.jar Hortonworks recommends using HiveServer2 and a JDBC client (such as Beeline) as the primary way to access Hive. This approach uses SQL standard-based authorization or Ranger-based authorization. However, some users may wish to access Hive data from other applications, such as Pig. For these use cases, use the Hive CLI and storage-based authorization. Connecting to Hive with Beeline The following examples demonstrate how to use Beeline to connect to Hive for all possible variations of these modes. Embedded Client Use the following syntax to connect to Hive from Beeline in embedded mode: !connect jdbc:hive2:// Remote Client with HiveServer2 TCP Transport Mode and SASL Authentication Use the following syntax to connect to HiveServer2 in TCP mode from a remot

IntroductionIn Customizing Atlas (Part1): Model governance, traceability and registry I provided a brief overview of Atlas types and entities and showed how to customize them to fit your needs. I showed the specific example of a Model type used to govern your deployed data science models and complex Spark code.In Customizing Atlas (Part2): Deep source metadata and embedded entities I showed how to customize Atlas to hold knowledge of ingested data that goes deep beyond the data itself, e.g. detailes of the device that generated the data. I also showed how to implement the pattern of embedding an entity (not string) as an attribute value in your custom type. The result is a clickable hyperlink in the UI that opens that entity and its metadata.In this post I will:show how to extend your Atlas lineage to include processing and outputs on non-Hadoop systemsrepresent the above as a single lineage that connects data in Hadoop to a reporting system which generates a report and sends an ema

Here is the steps to access HBase UI through Knox. This is tried and tested with Ambari 2.4 and HDP2.51. Make sure Knox is configured properly and it works fine.2. ssh to Knox gateway host and go to /var/lib/knox/data-2.5.3.0-37/services3. download the configurations from https://github.com/apache/knox/tree/v0.11.0/gateway-service-definitions/src/main/resources/services/hbaseui/1.1.0/ URL.4. make sure your folder structure should look alike /var/lib/knox/data-2.5.3.0-37/services/hbaseui/1.1.0 and should have rewrite.xml and service.xml files5. change the owner/Group permissions to Knox for /var/lib/knox/data-2.5.3.0-37/services/hbaseui/ and subdirectory6. Go to Knox configurations Modify "Advanced topology" with below service tag <service> <role>HBASEUI</role> <url>http://HBASEMANTERHOST:16010</url> </service> <service>7. Restart Knox service.8. You should be able to access HBase UI from

IntroductionIn the previous post Customizing Atlas (Part1): Model governance, traceability and registry we: provided a brief overview of Atlas types and entitiesshowed how to customize Atlas types and entities to fit your own needs and appear in Atlas search and lineagecustomized a special type called model, which inherited from Process and empowered Atlas to govern the deployment of data science modelscommented on operationalizing custom entitiesIn this post we will: customize an Atlas type to represent deep source metadata (metadata beyond source data itself)customize an Atlas type to represent devices (metadata about the actual device that generates data)embed the device entity in the deep source entity (make the device entity an attribute value in the deep source metadata)show how device as an attribute value is a clickable link in the Atlas UI that opens to the full device entityConcepts and ExampleMain ConceptsThe main concepts are summarized in the table below.Example: Gene sequ

Disclaimer:This article is based on my personal experience and knowledge.Don't take it as a standard guidelines, understand the concept and modify it for your environmental best practices and use case. Always contact Hortonworks support if its production clusterProblem Description: I am currently using ambari-2.6.2 version. I am trying to update some configs via configs.py and my python version is python-2.7.9 and my configs.py is failing with below error :[root@asnaik1 certs]# /var/lib/ambari-server/resources/scripts/configs.py --port=8443 --action=set --host=asnaik1.openstacklocal --cluster=asnaik --config-type=kafka-env --user=admin --password=admin --key=kafka_log_dir --value=/tmp --protocol=https2018-12-05 10:24:57,615 INFO ### Performing "set":2018-12-05 10:24:57,615 INFO ### new property - "kafka_log_dir":"/tmp"Traceback (most recent call last):File "/var/lib/ambari-server/resources/scripts/configs.py", line 364, in <module>sys.exit(main())File "/var/lib/ambari-server/reso

Introduction Continuing my series on Beast Mode Quotient, let's automate the creation and termination of data science ready clusters. As always, since this is step 2 of a series of article, this tutorial depends on my previous article. Pre-Requisites In order to run this tutorial you will need to have a CB instance available. There are plenty of good tutorial out there, I recommend this one Agenda This tutorial is divided in the following sections: Section 1: Create a blueprints & recipes to run a minimal data science ephemeral cluster Section 2: Add blueprint and recipes via Cloudbreak interface Section 3: Automate cluster launch and terminate clusters Section 1: Create a blueprints & recipes to run a minimal data science ephemeral cluster A Cloudbreak blueprint has 3 parts: Part 1: Blueprint details Part 2: Services Configuration Part 3: Host Components configuration Here are the details of each part for our blueprint. Blueprint details This part i

With Stochastic Streaming Algorithms it's possible to analyze highly efficient and performant massive amounts of data. The main characteristics of sketches are very small size (sub-lineare space consumption), single-pass, mergeable and deliver approximate results in proven error bounds. If an approximate result is acceptable, then it is possible that data sketches are orders-of-magnitude faster compared to classic indexing or aggregation techniques. (I don't go into the mathematical discussion, more details here DataSketches)The following example shows how to use Data Sketches within Hive (also possible with Druid):Download and install Hortonworks Sandbox (it comes with demo database: foodmart)Download Java artifact sketches-hive-0.11.0-with-shaded-core.jar (provided by Yahoo) here jar repo and copy the jar file into HDFS /apps/hive/warehouse/lib directory#Login as hive user hive@sandbox-hdpwget https://search.maven.org/remotecontent?filepath=com/yahoo/datasketches/sketches-hive/0.11.0

Install AWS CLI from the steps provided in the below linkhttps://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.htmlVerify AWS CLI installation$aws --versionConfigure AWS credentials$aws configure Download Ozone 0.3.0-alpha tarball from here, untar it.Go to the $PWD/ozone-0.3.0-aplha/compose/ozones3 directory, and start the server:docker-compose up -dCreate alias commandalias ozones3api='aws s3api --endpoint http://localhost:9878'Create bucket:$ ozones3api create-bucket --bucket documentsPut objects to bucket:$ ozones3api put-object --bucket documents --key S3Doc --body ./S3.md$ ozones3api put-object --bucket documents --key hddsDoc --body ./Hdds.md$ ozones3api put-object --bucket documents --key javaDoc --body ./JavaApi.mdList objects in a bucket:$ ozones3api list-objects --bucket documents{"Contents": [{"LastModified": "2018-11-02T21:57:40.875Z","ETag": "1541195860875","StorageClass": "STANDARD","Key": "hddsDoc","Size": 2845},{"LastModified": "2018-11-02T22:36:23.358Z","ET

This article is designed to the steps need to convert an existing Spark pipeline model into a MLeap bundle. To follow the steps required to build the pipeline model used in the tutorial, please download the complete Apache Zeppelin notebook here on my GitHub site. Tools Used In Tutorial Spark 2.3 MLeap Spark 0.13 Step 1 - Load MLeap Spark DependenciesThis dependency is required to convert Spark models into MLeap bundles. in the following section of code, decency files are added to my Zeppelin notebook. Keep in mind this document needs to be present on Maven. %dep//In Zeppelin//Run this before initializing Spark Contextz.load("ml.combust.mleap:mleap-spark_2.11:0.13.0")Step 2 - Build Pipeline Model The Spark pipeline model needs to be built before it can be converted. In this section of code, a Spark pipeline model is built from previously defined stages. To view the complete code list, please download the notebook from my Github site. A pipeline model will be built once it

Post HDP upgrade you will observe all previous version HDP directories will be exist under /usr/hdp/ directory (Including current active). Once upgrade is finalized you can clear those older version directories using Ambari. use following "curl" command. Ssh to ambari server and run following command. (You can use any linux box in network which has curl installed and able to access ambari console URL).curl 'http://<$AMBARI_SERVER>:8080/api/v1/clusters/<$MyClusterName>/requests' -u admin:<Password> -H "X-Requested-By: ambari" -X POST -d'{"RequestInfo":{"context":"remove_previous_stacks", "action" : "remove_previous_stacks", "parameters" : {"version":"<$Current_Active_version>"}}, "Requests/resource_filters": [{"hosts":"<$ClusterNode1>, <$ClusterNode2>"}]}' $MBARI_SERVER => Ambari server name and port. If you are using SSL use the SSL port no. $MyClusterName => Enter cluster name you want to remove directories.$ Current_Active_version =&gt