Community Articles

Find and share helpful community-sourced technical articles.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Advanced Search

News Authors Personality Detection - Part 4: Create Nifi Web Services and a consumer application

pvidal
Cloudera Employee

Created on ‎10-15-2018 03:14 PM - edited ‎09-16-2022 01:44 AM

Overview

Last article of the series, and a fun one. As usual, this article is the continuation of part 1, part 2 and part 3, so make sure you read those before starting. The completion of this tutorial will finalize your end to end architecture, as depicted here:

92854-screen-shot-2018-10-12-at-111244-am.png

This tutorial will be divided in 2 sections:

  • Section 1: Create a Web Service with Nifi
  • Section 2: Create a consumer application with code Igniter

Note: Section 2 will not be detailed. First, because web application development is not the main goal of Hortonworks Community Connection. Second, because I am pretty sure I did a terrible job at it considering it is not my domain of expertise, and I shouldn't spread mediocrity to the world!

Section 1: Create a Web Service with Nifi

For your entire project to work, you will have to create more than one web service although this section only goes through the creation of one (see the flow overview below). To get all the flows required for your app to work, check github here.

92855-screen-shot-2018-10-12-at-112122-am.png

Step 1: Create a request handler

StandardHttpContextMap

This can't be more straight forward. Create a StandardHttpContextMap controller service with default options and enable it.

92856-screen-shot-2018-10-12-at-112839-am.png

HandleHttpRequest: Receive request and data

Create a processor simple HandleHttpRequest processor.

92858-screen-shot-2018-10-12-at-113314-am.png

Here the properties you need to change (all other are standard):

HandleHttpRequest 

8012

HTTP Context Map 

StandardHttpContextMap (the controller you just created)

Allowed Paths 

/dashboard

Step 2: Retrieve data from Hbase

Execute SQL

Create 4 execute SQLs that will run queries retrieving all the data you need. They all rely on a DBCPConnectionPool controller for Phoenix. To understand how to create that, please refer to part 3.

Each and every Execute SQL will have a select count in them. I'm not going to list them here, but I included a article for all web services to this article.

92859-screen-shot-2018-10-12-at-113500-am.png

Convert Avro to JSON

Yet again a very straight forward configuration. For each ExecuteSQL you created, create a ConvertAvroToJSON processor with standard configuration:

92861-screen-shot-2018-10-15-at-112649-am.png

Step 3: Merge and prepare Response

Merge Content

Merge all outputs from the Convert Avro to JSON processors using a standard MergeContent processor:

92863-screen-shot-2018-10-15-at-112649-am.png

Replace Text

Use a ReplaceText to add brackets before and after the merge content, so that your web app can read it:

92864-screen-shot-2018-10-15-at-120936-pm.png

Step 4: Send response back

Handle Http Response

Finally, use a standard configuration for a HandleHttpResponse processor to return the web response:

92865-screen-shot-2018-10-15-at-121244-pm.png

Start your flow, then ou should be able to call your service by going to this URL: 

http://[your_nifi_address]:8012/dashboard

The response should be something along these lines: 

[{"BYLINES": 781},{"TOTAL": 6153},{"TOTAL_VIEWS": 4147},{"NEWS_DESKS": 63}]

Section 2: Create a consumer application with code Igniter

Once you've created all the web services, you will need to create a web app to consume these services. I created one using code igniter, and you can see all the code on my github. As I mentioned before, I'm not going to go deep in this configuration, but to make this app run, you will need to follow these steps:

  • Step 1: Instantiate a Linux web server with Apache, Maria DB, and PHP 7
  • Step 2: Create a adminlte database on your server with the appropriate data by running the SQL script below
  • Step 3: Clone my github to the root of your apache server
  • Step 4: Change the config.php base_url to match your server, and the database.php to match your credentials
-- MySQL dump 10.14  Distrib 5.5.60-MariaDB, for Linux (x86_64)
--
-- Host: localhost    Database: adminlte
-- ------------------------------------------------------
-- Server version 5.5.60-MariaDB
/*!40101 SET @OLD_CHARACTER_SET_CLIENT=@@CHARACTER_SET_CLIENT */;
/*!40101 SET @OLD_CHARACTER_SET_RESULTS=@@CHARACTER_SET_RESULTS */;
/*!40101 SET @OLD_COLLATION_CONNECTION=@@COLLATION_CONNECTION */;
/*!40101 SET NAMES utf8 */;
/*!40103 SET @OLD_TIME_ZONE=@@TIME_ZONE */;
/*!40103 SET TIME_ZONE='+00:00' */;
/*!40014 SET @OLD_UNIQUE_CHECKS=@@UNIQUE_CHECKS, UNIQUE_CHECKS=0 */;
/*!40014 SET @OLD_FOREIGN_KEY_CHECKS=@@FOREIGN_KEY_CHECKS, FOREIGN_KEY_CHECKS=0 */;
/*!40101 SET @OLD_SQL_MODE=@@SQL_MODE, SQL_MODE='NO_AUTO_VALUE_ON_ZERO' */;
/*!40111 SET @OLD_SQL_NOTES=@@SQL_NOTES, SQL_NOTES=0 */;
--
-- Database creation
--
CREATE DATABASE adminlte;
use adminlte;
--
-- Table structure for table `activity_logs`
--
DROP TABLE IF EXISTS `activity_logs`;
/*!40101 SET @saved_cs_client     = @@character_set_client */;
/*!40101 SET character_set_client = utf8 */;
CREATE TABLE `activity_logs` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `title` text NOT NULL,
  `user` text NOT NULL,
  `ip_address` text NOT NULL,
  `created_at` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
  `updated_at` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
  PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=18 DEFAULT CHARSET=latin1;
/*!40101 SET character_set_client = @saved_cs_client */;
--
-- Dumping data for table `activity_logs`
--
LOCK TABLES `activity_logs` WRITE;
/*!40000 ALTER TABLE `activity_logs` DISABLE KEYS */;
INSERT INTO `activity_logs` VALUES (1,'Administrator Logged in','1','10.42.80.145','2018-10-11 18:34:37','0000-00-00 00:00:00'),(2,'User: Administrator Logged Out','1','10.42.80.145','2018-10-11 19:44:48','0000-00-00 00:00:00'),(3,'Administrator Logged in','1','10.42.80.145','2018-10-11 19:44:54','0000-00-00 00:00:00'),(4,'User: Administrator Logged Out','1','10.42.80.145','2018-10-11 20:52:02','0000-00-00 00:00:00'),(5,'Administrator Logged in','1','10.42.80.145','2018-10-11 20:52:06','0000-00-00 00:00:00'),(6,'User: Administrator Logged Out','1','10.42.80.145','2018-10-11 20:56:16','0000-00-00 00:00:00'),(7,'Administrator Logged in','1','10.42.80.145','2018-10-11 20:56:24','0000-00-00 00:00:00'),(8,'User: Administrator Logged Out','1','10.42.80.145','2018-10-11 20:59:01','0000-00-00 00:00:00'),(9,'Administrator Logged in','1','10.42.80.145','2018-10-11 21:02:36','0000-00-00 00:00:00'),(10,'User: Administrator Logged Out','1','10.42.80.145','2018-10-11 21:06:35','0000-00-00 00:00:00'),(11,'Administrator Logged in','1','10.42.80.145','2018-10-11 21:12:24','0000-00-00 00:00:00'),(12,'Administrator Logged in','1','10.42.80.172','2018-10-12 15:52:55','0000-00-00 00:00:00'),(13,'Administrator Logged in','1','10.42.80.85','2018-10-13 11:45:37','0000-00-00 00:00:00'),(14,'Administrator Logged in','1','10.42.80.153','2018-10-13 12:27:08','0000-00-00 00:00:00'),(15,'User: Administrator Logged Out','1','10.42.80.153','2018-10-13 12:27:14','0000-00-00 00:00:00'),(16,'Administrator Logged in','1','10.42.80.153','2018-10-13 12:27:40','0000-00-00 00:00:00'),(17,'Administrator Logged in','1','10.42.80.204','2018-10-15 16:21:06','0000-00-00 00:00:00');
/*!40000 ALTER TABLE `activity_logs` ENABLE KEYS */;
UNLOCK TABLES;
--
-- Table structure for table `permissions`
--
DROP TABLE IF EXISTS `permissions`;
/*!40101 SET @saved_cs_client     = @@character_set_client */;
/*!40101 SET character_set_client = utf8 */;
CREATE TABLE `permissions` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `title` text NOT NULL,
  `code` text,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=16 DEFAULT CHARSET=latin1;
/*!40101 SET character_set_client = @saved_cs_client */;
--
-- Dumping data for table `permissions`
--
LOCK TABLES `permissions` WRITE;
/*!40000 ALTER TABLE `permissions` DISABLE KEYS */;
INSERT INTO `permissions` VALUES (1,'Users List','users_list'),(2,'Add Users','users_add'),(3,'Edit Users','users_edit'),(4,'Delete Users','users_delete'),(5,'Users View','users_view'),(6,'Activity Logs List','activity_log_list'),(7,'Acivity Log View','activity_log_view'),(8,'Roles List','roles_list'),(9,'Add Roles','roles_add'),(10,'Edit Roles','roles_edit'),(11,'Permissions List','permissions_list'),(12,'Add Permissions','permissions_add'),(13,'Permissions Edit','permissions_edit'),(14,'Delete Permissions','permissions_delete'),(15,'Company Settings','company_settings');
/*!40000 ALTER TABLE `permissions` ENABLE KEYS */;
UNLOCK TABLES;
--
-- Table structure for table `role_permissions`
--
DROP TABLE IF EXISTS `role_permissions`;
/*!40101 SET @saved_cs_client     = @@character_set_client */;
/*!40101 SET character_set_client = utf8 */;
CREATE TABLE `role_permissions` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `role` int(11) NOT NULL,
  `permission` text NOT NULL,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=16 DEFAULT CHARSET=latin1;
/*!40101 SET character_set_client = @saved_cs_client */;
--
-- Dumping data for table `role_permissions`
--
LOCK TABLES `role_permissions` WRITE;
/*!40000 ALTER TABLE `role_permissions` DISABLE KEYS */;
INSERT INTO `role_permissions` VALUES (1,1,'users_list'),(2,1,'users_add'),(3,1,'users_edit'),(4,1,'users_delete'),(5,1,'users_view'),(6,1,'activity_log_list'),(7,1,'activity_log_view'),(8,1,'roles_list'),(9,1,'roles_add'),(10,1,'roles_edit'),(11,1,'permissions_list'),(12,1,'permissions_add'),(13,1,'permissions_edit'),(14,1,'permissions_delete'),(15,1,'company_settings');
/*!40000 ALTER TABLE `role_permissions` ENABLE KEYS */;
UNLOCK TABLES;
--
-- Table structure for table `roles`
--
DROP TABLE IF EXISTS `roles`;
/*!40101 SET @saved_cs_client     = @@character_set_client */;
/*!40101 SET character_set_client = utf8 */;
CREATE TABLE `roles` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `title` text NOT NULL,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=2 DEFAULT CHARSET=latin1;
/*!40101 SET character_set_client = @saved_cs_client */;
--
-- Dumping data for table `roles`
--
LOCK TABLES `roles` WRITE;
/*!40000 ALTER TABLE `roles` DISABLE KEYS */;
INSERT INTO `roles` VALUES (1,'Admin');
/*!40000 ALTER TABLE `roles` ENABLE KEYS */;
UNLOCK TABLES;
--
-- Table structure for table `settings`
--
DROP TABLE IF EXISTS `settings`;
/*!40101 SET @saved_cs_client     = @@character_set_client */;
/*!40101 SET character_set_client = utf8 */;
CREATE TABLE `settings` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `key` text NOT NULL,
  `value` text NOT NULL,
  `created_at` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=4 DEFAULT CHARSET=latin1;
/*!40101 SET character_set_client = @saved_cs_client */;
--
-- Dumping data for table `settings`
--
LOCK TABLES `settings` WRITE;
/*!40000 ALTER TABLE `settings` DISABLE KEYS */;
INSERT INTO `settings` VALUES (1,'company_name','Company Name','2018-06-21 17:37:59'),(2,'company_email','testcompany@gmail.com','2018-07-11 11:09:58'),(3,'timezone','Asia/Kolkata','2018-07-15 19:54:17');
/*!40000 ALTER TABLE `settings` ENABLE KEYS */;
UNLOCK TABLES;
--
-- Table structure for table `users`
--
DROP TABLE IF EXISTS `users`;
/*!40101 SET @saved_cs_client     = @@character_set_client */;
/*!40101 SET character_set_client = utf8 */;
CREATE TABLE `users` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `name` text NOT NULL,
  `username` text NOT NULL,
  `email` text NOT NULL,
  `password` text NOT NULL,
  `phone` text NOT NULL,
  `address` longtext NOT NULL,
  `last_login` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00' ON UPDATE CURRENT_TIMESTAMP,
  `role` int(11) NOT NULL,
  `reset_token` text NOT NULL,
  `created_at` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
  `updated_at` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
  PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=2 DEFAULT CHARSET=latin1;
/*!40101 SET character_set_client = @saved_cs_client */;
--
-- Dumping data for table `users`
--
LOCK TABLES `users` WRITE;
/*!40000 ALTER TABLE `users` DISABLE KEYS */;
INSERT INTO `users` VALUES (1,'Administrator','admin','admin@gmail.com','21232f297a57a5a743894a0e4a801fc3','','','2018-10-15 21:10:51',1,'','2018-06-27 18:30:16','0000-00-00 00:00:00');
/*!40000 ALTER TABLE `users` ENABLE KEYS */;
UNLOCK TABLES;
/*!40103 SET TIME_ZONE=@OLD_TIME_ZONE */;
/*!40101 SET SQL_MODE=@OLD_SQL_MODE */;
/*!40014 SET FOREIGN_KEY_CHECKS=@OLD_FOREIGN_KEY_CHECKS */;
/*!40014 SET UNIQUE_CHECKS=@OLD_UNIQUE_CHECKS */;
/*!40101 SET CHARACTER_SET_CLIENT=@OLD_CHARACTER_SET_CLIENT */;
/*!40101 SET CHARACTER_SET_RESULTS=@OLD_CHARACTER_SET_RESULTS */;
/*!40101 SET COLLATION_CONNECTION=@OLD_COLLATION_CONNECTION */;
/*!40111 SET SQL_NOTES=@OLD_SQL_NOTES */;

After you're done with that and have restarted your web server, go to: 

http://[YOUR_SERVER]/index.php

You can now connect using admin/admin. You should see screens like these ones:

92866-screen-shot-2018-10-15-at-122057-pm.png

92867-screen-shot-2018-10-15-at-122110-pm.png

92868-screen-shot-2018-10-15-at-122123-pm.png

92869-screen-shot-2018-10-15-at-122132-pm.png

Preview file
142 KB
1,155 Views
Take a Tour of the Community
Don't have an account?
Register
Your experience may be limited. Sign in to explore more.
Announcements
Community Announcements
March 2023 Community Highlights
What's New @ Cloudera
Cloudera DataFlow Designer for self-service data flow development is now generally available to all CDP Public Cloud customers
What's New @ Cloudera
Cloudera Operational Database (COD) UI provides the JWT configuration details to connect to your HBase client
What's New @ Cloudera
Cloudera Operational Database (COD) UI allows storage type selection when creating an operational database
What's New @ Cloudera
Release of Cloudera Streaming Analytics (CSA) 1.9.0 for CDP 7.1.7 SP2 & CDP 7.1.8
View More Announcements
Version history
Last update:
‎09-16-2022 01:44 AM
Updated by:
Cloudera Employee
Contributors
Top Kudoed Authors
View Full Leaderboard