Part 4: Create an end-to-end application automating BMQ calculation and prediction

[#1]

Introduction

Finally, we get to put all this together! This is less of a tutorial, and more of a show case of the capabilities of this solution. Please refere to previous articles to build this on your own.

The meat of this article is showcasing the React/NodeJS application you can build to communicate with Nifi, Cloudbreak and your ephemeral cluster running Zeppelin and Spark.

Architecture

Below is a high-level architecture of the solution:

You can find all the code for this here: https://github.com/paulvid/bmq-app

The code includes:

• Latest Zeppelin Prediction Notebook
• All Nifi templates
• All Blueprints/Recipes for Cloudbreak
• The source code of the app

Agenda

This article is divided in the following sections:

• Section 1: Monitoring your current data with Nifi
• Section 2: Launching an ephemeral cluster with Cloudbreak
• Section 3: Running a prediction model with Spark Zeppelin

Section 1: Monitoring your current data with Nifi

Section 2: Launching an ephemeral cluster with Cloudbreak

Section 3: Running a prediction model with Spark Zeppelin