Introduction Let's jump into tutorial 2 from my AI to Edge series! This tutorial details the creation of a Nifi flow executing the ONNX model we trained in my last article. More precisely we will try to feed these 3 handwritten digits and predict their value: Note: as always, all code/files referenced in this tutorial can be found on my github, here. Agenda Below is an overview of the flow: As you can see, the flow is divided in the following sections: Section 1: Listening to a folder for new png files Section 2: Resizing these images to 28x28 (size used to train our model) Section 3: Converting these images to CSV (format used to train our model) Section 4: Running our predictive model Section 1: Listening to a folder for new png files Step 1: Setup a variable for the root folder This will be useful when we deploy the flow to a minifi flow. Go to your variables and create the following: Name: root_folder Value: location of your download of my github Step 2: List files in folder Create a ListFiles processor and modify the following properties: Input Directory: ${root_folder}NIFI/png/original/ File Filter: [^\.].*.png Step 3: Fetch files in folder Create a FetchFiles processor with default parameters. Note: The List/Fetch paradigm is very powerful because it will allow us to continuously look for new images without reprocessing all of them. ListFiles is a stateful processor. If you're unfamiliar with the concept I encourage you to read about it on this community. Section 2: Resizing these images to 28x28 Step 1: Resize Image Create a ResizeImage processor and modify the following properties: Image Width (in pixels): 28 Image Height (in pixels): 28 Step 2: Enter out attributes for resized images Create an UpdateAttribute processor, aimed at defining the folder and filename of the resized images, by adding the following properties to the processor: filedirectory: ${root_folder}NIFI/png/resized/ filename: resized_${filename} Section 3: Converting these images to CSV Step 1: Saving modified image Create a PutFile processor and modify the following properties to store the converted image in the resized folder: Directory: ${filedirectory} Step 2: Execute a python script to convert images to CSV In this step we will create an ExecuteStreamCommand processor that will run the convertImg.sh python script. The script takes the resized image file, converts it to grayscale, and converts it into an inverted CSV to match the input of our model. Below is the script itself: #!/usr/bin/env python3 import os,png,array import pandas as pd import time import sys from PIL import Image from PIL import ImageOps columnNames = list() for i in range(784): pixel = 'pixel' pixel += str(i) columnNames.append(pixel) train_data = pd.DataFrame(columns = columnNames) start_time = time.time() img_name = sys.argv[1] img = Image.open(img_name) img = img.convert('LA') rawData = img.load() data = [] for y in range(28): for x in range(28): data.append(rawData[x,y][0]) print(i) k = 0 #print data train_data.loc[i] = [255-data[k] for k in range(784)] csvFile = sys.argv[2] print(csvFile) train_data.to_csv(csvFile,index = False) As you can see it expects two arguments: Location of the resized image ( img_name = sys.argv[1] ) Location of the target CSV ( csvFile = sys.argv[2] ) Thus, you will modify the following properties in the ExecuteStreamCommand processor: Command Arguments: ${root_folder}NIFI/png/resized/${filename};${root_folder}NIFI/csv/${filename}.csv Command Path: ${root_folder}NIFI/convertImg.sh Section 4: Running our predictive model Step 1: Enter input attributes for model execution Create an UpdateAttribute processor, aimed at defining the locations of the CSV file and the ONNX model, by adding the following properties to the processor: filename: ${root_folder}NIFI/csv/${filename}.csv onnxModel: ${root_folder}NOTEBOOKS/model.onnx Step 2: Use python to run the model with onnxruntime In this step we will create an ExecuteStreamCommand processor that will run the runModel.sh python script. The script takes the CSV version of the image and run the ONNX model created in the last tutorial with this CSV as an input. Below is the script itself: #!/usr/bin/env python3 import onnxruntime as rt import onnx as ox import numpy import pandas as pd import shutil import sys test=pd.read_csv(sys.argv[1]) X_test = test.values.astype('float32') X_test = X_test.reshape(X_test.shape[0], 28, 28,1) session = rt.InferenceSession(sys.argv[2]) input_name = session.get_inputs()[0].name label_name = session.get_outputs()[0].name prediction = session.run([label_name], {input_name: X_test.astype(numpy.float32)})[0] number = 0 for i in range(0, 9): if (prediction[0][i] == 1.0): number = i print(number) As you can see it expects two arguments: Location of the CSV ( test=pd.read_csv(sys.argv[1]) ) Location of the ONNX model ( session = rt.InferenceSession(sys.argv[2]) ) Thus, you will modify the following properties in the ExecuteStreamCommand processor: Command Arguments: ${filename};${onnxModel} Command Path: ${root_folder}NIFI/runModel.sh Results If you run the flow against the image in my github, you will see 3 output flowfiles, predicting the value of the handwritten digit, like shown below: ... View more

Introduction Time for the tutorial 1 of a series detailing how to go from AI to Edge! Note: all code/files referenced in this tutorial can be found on my github, here. Agenda This tutorial is divided in the following sections: Section 1: Create a custom Docker container running Jupyter for CDSW Section 2: Automate Jupyter launch in a CDSW project Section 3: Train and save a model reading MNSIT database Section 1: Create a custom Docker container running Jupyter for CDSW This is fairly straight forward to implement, as it is detailed in the official documentation. Note: make sure that dock is signed in with your Dockerhub username/password (not email) otherwise the docker push will not work. Step 1: Create a repository in docker hub Go to docker hub and sign in with your account. Create a new repository as follows: You should see something like this: Step 2: Creating a custom docker file Go to a folder on your computer can create this docker file (saving it as Dockerfile 😞 FROM docker.repository.cloudera.com/cdsw/engine:7 RUN pip3 install --upgrade pip RUN pip3 install keras RUN pip3 install tensorflow RUN pip3 install sklearn RUN pip3 install jupyter RUN pip3 install 'prompt-toolkit==1.0.15' RUN pip3 install onnxruntime RUN pip3 install keras2onnx Step 3: Build the container Run the following command in the folder where the file has been saved: docker build -t YOUR_USER/YOUR_REPO:YOUR_TAG . -f Dockerfile Step 4: Publish it to docker hub Run the following command on your computer: docker push YOUR_USER/YOUR_REPO:YOUR_TAG Section 2: Automate Jupyter launch in a CDSW project Step 1: Create a shell script to run Jupyter In CDSW 1.5, you can't add a CMD or an ENTRYPOINT to your docker file. Therefore, you will need to add a .bashrc file to your CDSW project, with the following code: processes=`ps -ef | grep jupyter | wc -l` if (( $processes == 2 )) ; then echo "Jupyter is already running!" elif (( $processes == 1 )) ; then jupyter notebook --no-browser --ip=0.0.0.0 --port=8080 --NotebookApp.token= else echo "Invalid number of processes, relaunch your session!" fi Save this file to a github repository. Step 2: Add the custom engine to CDSW In CDSW config, use the docker hub image you created as your default engine: Step 3: Create a project in CDSW with .bashrc In CDSW, create a new project using the github repository you just created: Note: You can create a blank project and add the .bashrc file to it, but this automates it. Step 4: Launch a CDSW session with Jupyter In your project, open workbench and launch a session with your custom engine. Run terminal access and Jupyter will launch. You will then see the following on your 9 dots, allowing you to run Jupyter: Section 3: Train and save a model reading MNSIT database The model training is very well explained in the original Kaggle article that can be found here. A reviewed version of this notebook can be found on my github. The main thing that was added to the notebook is the publishing of the model: # Convert into ONNX format with onnxmltools import keras2onnx onnx_model = keras2onnx.convert_keras(model, model.name) import onnx temp_model_file = 'model.onnx' onnx.save_model(onnx_model, temp_model_file) After the notebook runs, you should see the model.onnx file created. ... View more

Introduction Machine Learning and Artificial Intelligence frameworks are numerous and their impact on the future of computer science need no emphasis. However, deploying these models can be complex and fairly manual without the appropriate multi-function ecosystem, especially when deploying these models to the edge. Luckily, Cloudera comprehensive data management suite make this endeavor very easy.During this series, I will present how to create a Deep Learning model trained to read digits from the MNIST database and deploy it to the edge. This article is an introduction to the architecture and pre-requisites necessary for this tutorial. It will refer to sub articles that will be tutorials that anyone can follow to implement understand how to take an AI model and operationalize it to the edge Architecture overview The figure below gives a highlight of my hybrid cloud platform: As you can see, it is comprised of the three main functions: Cloudera Data Science Workbench: data science hub used to train and save the model, leveraging like Tensorflow, Jupyter and ONNX. Cloudera Flow Management: leverages Nifi to develop a flow reading from an image and running the ONNX model. Cloudera Edge Management: allows for the deployment of Minifi flows to edge deployment Pre-Requisites To run this tutorial I used the following main elements of the Cloudera Stack: CDSW 1.5 HDF 3.3 HDP 3.1 CEM 1.0 Implementation Tutorials The implementation will be detailed in the upcoming following tutorial articles: Part 1: CDSW model training using a custom docker image with Jupyter and save it using ONNX Part 2: Nifi flow creation to parse new images and run the model Part 3: Flow deployment to Minifi using CEM ... View more

Introduction CFM is out for CDH and that's awesome. While the install documentation is very thorough, I thought I'd share a screenshot based tutorial, avoiding some of the gotchas of the installation Pre-Requisites Healthy CDH 6.2 cluster Admin privileges on the cluster and its nodes wget installed Open JDK 1.8 installed (see doc) Zookeeper service installed (see doc) Step 1: Get the CFM CSD Files Connect to your Cloudera Manager node as root , and enter the following: cd /opt/cloudera/csd wget http://archive.cloudera.com/CFM/csd/1.0.0.0/NIFI-1.9.0.1.0.0.0-90.jar wget http://archive.cloudera.com/CFM/csd/1.0.0.0/NIFICA-1.9.0.1.0.0.0-90.jar wget http://archive.cloudera.com/CFM/csd/1.0.0.0/NIFIREGISTRY-0.3.0.1.0.0.0-90.jar chown cloudera-scm:cloudera-scm NIFI*.jar chmod 644 NIFI*.jar service cloudera-scm-server restart After restart, log to CM and restart the Cloudera Management Service: Wait for the restart to finish: Step 2: Get the CFM Parcel Go to the parcels configuration (parcel icon top right): Go to parcel configuration and add http://archive.cloudera.com/CFM/parcels/1.0.0.0/ to the parcel repository list by clicking on the "+" icon: Click on the "Check for New Parcels" button: You should see a CFM parcel, click on "Download": After the download completes, distribute and activate the parcel. Step 4: Install services Go to your CM main page, and click on "Add Service": Select the Nifi service: Select the host onto which you want to install Nifi: Click on next and wait for the service to install. Step 5: Profit You can now go to your Nifi URL: Bonus: Changing election timeout By default, the nifi.cluster.flow.election.max.wait.time property is set to 5 minutes, which can be confusing by giving rise to a log message like Cluster is still in the process of voting on the appropriate Data Flow . You can modify the property by adding it to the nifi.properties.xml in CM, as follows: Restart your Nifi instance via CM, and voilà! ... View more

My blood recently turned from green to blue (after the Hortonworks-Cloudera merger) and I couldn't be more excited to play with new toys. What I am particularly excited about is Cloudera Data Science Workbench. But, like in everything I do, I am very lazy. So here is a quick tutorial to install Altus Director, and use it to deploy a CDH 5.15 + CDSW cluster. Step 1: Install Altus Director Many ways to do that, but the one I chose was the AWS install, detailed here: https://www.cloudera.com/documentation/director/latest/topics/director_aws_setup_client.html The installation documentation is very well done, but here are the important excerpts Create a VPC for your Altus instance Follow the documentation. Few important points: In the name of laziness I also recommend to add a 0-65535 rule from your personal IP. Your VPC should have an internet gateway associated with it (you could do it without, but would require you manually pulling the CM/CDH software down and make internal repositories within your subnet) Do not forget to open all traffic to your security group as described here. Your deployment will not work otherwise. Launch a Redhat 7.3 instance You can either search communities AMIs, or use this one: ami-6871a115 Install Altus Connect to your ec2 instance: ssh -i your_file.pem ec2-user@your_instance_ip Install JDK and wget sudo yum install java-1.8.0-openjdk sudo yum install wget Install/Start Altus server and client: cd /etc/yum.repos.d/ sudo wget "http://archive.cloudera.com/director6/6.1/redhat7/cloudera-director.repo" sudo yum install cloudera-director-server cloudera-director-client sudo service cloudera-director-server start sudo systemctl disable firewalld sudo systemctl stop firewalld Connect to Altus Director Go to http://your_instance_ip:7189/ and connect with admin/admin Step 2: Modify the Director configuration file CDSW cluster configuration can be found here https://github.com/cloudera/director-scripts/blob/master/configs/aws.cdsw.conf Modify the configuration file to use: Your AWS accessKeyId/secretAccessKey Your AWS region Your AWS subnetId (same as the one you created for your Director instance) Your AWS securityGroupsIds (same as the one you created for your Director instance) Your private key path (e.g. /home/ec2-user/field.pem) Your AWS image (e.g. ami-6871a115) Step 3: Launch the cluster via director client Go to your EC2 instance where Director is installed, and load your modified configuration file as well as the appropriate key. Finally, run the following: cloudera-director bootstrap-remote your_configuration_file.conf \ --lp.remote.username=admin \ --lp.remote.password=admin Step 4: Access Cloudera Manager You can follow the bootstrapping of the cluster both on command line or in the Director interface; once done, you can connect to Cloudera Manager using: http://your_manager_instance_ip:7180/ Step 5: Configure CDSW domain with your IP Cloudera Data Science Workbench uses DNS. The correct approach is to setup a wildcard DNS record is required, as described here. However, for testing purposes I used nip.io. The only parameter to change is the Cloudera Data Science Workbench Domain, from cdsw.my-domain.com as the conf file sets it up to, to cdsw.[YOUR_AWS_PUBLIC_IP].nip.io , as depicted below: Restart the CDSW service, then you should be able to access CDSW by clicking on the CDSW Web UI link. Register for a new account and you will have access to CDSW: ... View more

Introduction Overview Cloudera/Hortonworks offers today one of the most comprehensive data management platform, with components allowing you data flow management to governed and distributed Data Science workloads. With so many toys to play with, I thought I'd share an easy way to setup a simple cluster that will, using Cloudbreak, setup the following main components on Azure cloud: Hortonworks Data Platform 3.1 Hortonworks Data Flow 3.3 Data Platform Search 4.0 Cloudera Data Science Workbench 1.5 Note: This is not a production-ready setup, but merely a first step to customizing your deployment using the Cloudera toolkit. Pre-Requisites Account on Azure with permission to assign roles Cloudbreak 2.9 (you can always set one up on your machine using: https://community.hortonworks.com/articles/194076/using-vagrant-and-virtualbox-to-create-a-local-ins.html) Tutorial steps Step 1: Setup Azure Credentials Step 2: Setup blueprint and cluster extensions Step 3: Create cluster Step 1: Setup Azure Credentials Find your Azure subscription and tenant ID To find your subscription ID, go to the search box and look for subscription; you should find it as such: For the tenant ID, use the Azure AD Directory ID: Setup your credentials in Cloudbreak This part is extremely well documented in Cloudbreak's documentation portal: https://docs.hortonworks.com/HDPDocuments/Cloudbreak/Cloudbreak-2.9.0/create-credential-azure/content/cb_create-app-based-credential.html. Note: Because of IT restrictions on my side, I chose to use an app based credential setup, but if you have enough privileges, Cloudbreak creates the app and assigns roles automagically for you. Step 2: Setup blueprint and cluster extensions Blueprint First upload the blueprint below: { "Blueprints": { "blueprint_name": "edge-to-ai-3.1", "stack_name": "HDP", "stack_version": "3.1" }, "configurations": [ { "yarn-site": { "properties": { "yarn.nodemanager.resource.cpu-vcores": "6", "yarn.nodemanager.resource.memory-mb": "60000", "yarn.scheduler.maximum-allocation-mb": "14" } } }, { "hdfs-site": { "properties": { "dfs.cluster.administrators": "hdfs" } } }, { "capacity-scheduler": { "properties": { "yarn.scheduler.capacity.maximum-am-resource-percent": "0.4", "yarn.scheduler.capacity.root.capacity": "67", "yarn.scheduler.capacity.root.default.capacity": "67", "yarn.scheduler.capacity.root.default.maximum-capacity": "67", "yarn.scheduler.capacity.root.llap.capacity": "33", "yarn.scheduler.capacity.root.llap.maximum-capacity": "33", "yarn.scheduler.capacity.root.queues": "default,llap" } } }, { "ranger-hive-audit": { "properties": { "xasecure.audit.destination.hdfs.file.rollover.sec": "300" }, "properties_attributes": {} } }, { "hive-site": { "hive.exec.compress.output": "true", "hive.merge.mapfiles": "true", "hive.metastore.dlm.events": "true", "hive.metastore.transactional.event.listeners": "org.apache.hive.hcatalog.listener.DbNotificationListener", "hive.repl.cm.enabled": "true", "hive.repl.cmrootdir": "/apps/hive/cmroot", "hive.repl.rootdir": "/apps/hive/repl", "hive.server2.tez.initialize.default.sessions": "true", "hive.server2.transport.mode": "http" } }, { "hive-interactive-env": { "enable_hive_interactive": "true", "hive_security_authorization": "Ranger", "num_llap_nodes": "1", "num_llap_nodes_for_llap_daemons": "1", "num_retries_for_checking_llap_status": "50" } }, { "hive-interactive-site": { "hive.exec.orc.split.strategy": "HYBRID", "hive.llap.daemon.num.executors": "5", "hive.metastore.rawstore.impl": "org.apache.hadoop.hive.metastore.cache.CachedStore", "hive.stats.fetch.bitvector": "true" } }, { "spark2-defaults": { "properties": { "spark.datasource.hive.warehouse.load.staging.dir": "/tmp", "spark.datasource.hive.warehouse.metastoreUri": "thrift://%HOSTGROUP::master1%:9083", "spark.hadoop.hive.zookeeper.quorum": "{{zookeeper_quorum_hosts}}", "spark.sql.hive.hiveserver2.jdbc.url": "jdbc:hive2://{{zookeeper_quorum_hosts}}:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2-interactive", "spark.sql.hive.hiveserver2.jdbc.url.principal": "hive/_HOST@EC2.INTERNAL" }, "properties_attributes": {} } }, { "gateway-site": { "properties": { "gateway.path": "{{cluster_name}}" }, "properties_attributes": {} } }, { "admin-topology": { "properties": { "content": "\n \n\n \n\n \n authentication\n ShiroProvider\n true\n \n sessionTimeout\n 30\n \n \n main.ldapRealm\n org.apache.hadoop.gateway.shirorealm.KnoxLdapRealm\n \n \n main.ldapRealm.userDnTemplate\n uid={0},ou=people,dc=hadoop,dc=apache,dc=org\n \n \n main.ldapRealm.contextFactory.url\n ldap://54.219.163.9:33389\n \n \n main.ldapRealm.contextFactory.authenticationMechanism\n simple\n \n \n urls./**\n authcBasic\n \n \n\n \n authorization\n AclsAuthz\n true\n \n \n\n \n KNOX\n \n\n " }, "properties_attributes": {} } }, { "ranger-admin-site": { "properties": { "ranger.jpa.jdbc.url": "jdbc:postgresql://localhost:5432/ranger" }, "properties_attributes": {} } }, { "ranger-env": { "properties": { "is_solrCloud_enabled": "true", "keyadmin_user_password": "{{{ general.password }}}", "ranger-atlas-plugin-enabled": "Yes", "ranger-hdfs-plugin-enabled": "Yes", "ranger-hive-plugin-enabled": "Yes", "ranger-knox-plugin-enabled": "Yes", "ranger_admin_password": "{{{ general.password }}}", "rangertagsync_user_password": "{{{ general.password }}}", "rangerusersync_user_password": "{{{ general.password }}}" }, "properties_attributes": {} } }, { "ams-hbase-site": { "properties": { "hbase.cluster.distributed": "true", "hbase.rootdir": "file:///hadoopfs/fs1/metrics/hbase/data" } } }, { "atlas-env": { "properties": { "atlas.admin.password": "admin", "atlas_solr_shards": "2", "content": "\n # The java implementation to use. If JAVA_HOME is not found we expect java and jar to be in path\n export JAVA_HOME={{java64_home}}\n\n # any additional java opts you want to set. This will apply to both client and server operations\n {% if security_enabled %}\n export ATLAS_OPTS=\"{{metadata_opts}} -Djava.security.auth.login.config={{atlas_jaas_file}}\"\n {% else %}\n export ATLAS_OPTS=\"{{metadata_opts}}\"\n {% endif %}\n\n # metadata configuration directory\n export ATLAS_CONF={{conf_dir}}\n\n # Where log files are stored. Defatult is logs directory under the base install location\n export ATLAS_LOG_DIR={{log_dir}}\n\n # additional classpath entries\n export ATLASCPPATH={{metadata_classpath}}\n\n # data dir\n export ATLAS_DATA_DIR={{data_dir}}\n\n # pid dir\n export ATLAS_PID_DIR={{pid_dir}}\n\n # hbase conf dir\n export HBASE_CONF_DIR=\"/etc/ams-hbase/conf\"\n\n # Where do you want to expand the war file. By Default it is in /server/webapp dir under the base install dir.\n export ATLAS_EXPANDED_WEBAPP_DIR={{expanded_war_dir}}\n export ATLAS_SERVER_OPTS=\"-server -XX:SoftRefLRUPolicyMSPerMB=0 -XX:+CMSClassUnloadingEnabled -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:+PrintTenuringDistribution -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=$ATLAS_LOG_DIR/atlas_server.hprof -Xloggc:$ATLAS_LOG_DIRgc-worker.log -verbose:gc -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=1m -XX:+PrintGCDetails -XX:+PrintHeapAtGC -XX:+PrintGCTimeStamps\"\n {% if java_version == 8 %}\n export ATLAS_SERVER_HEAP=\"-Xms{{atlas_server_xmx}}m -Xmx{{atlas_server_xmx}}m -XX:MaxNewSize={{atlas_server_max_new_size}}m -XX:MetaspaceSize=100m -XX:MaxMetaspaceSize=512m\"\n {% else %}\n export ATLAS_SERVER_HEAP=\"-Xms{{atlas_server_xmx}}m -Xmx{{atlas_server_xmx}}m -XX:MaxNewSize={{atlas_server_max_new_size}}m -XX:MaxPermSize=512m\"\n {% endif %}\n", "hbase_conf_dir": "/etc/ams-hbase/conf" } } }, { "kafka-broker": { "properties": { "default.replication.factor": "1", "offsets.topic.replication.factor": "1" }, "properties_attributes": {} } }, { "hbase-env": { "properties": { "phoenix_sql_enabled": "true" }, "properties_attributes": {} } }, { "druid-common": { "properties": { "druid.extensions.loadList": "[\"postgresql-metadata-storage\", \"druid-datasketches\", \"druid-hdfs-storage\", \"druid-kafka-indexing-service\", \"ambari-metrics-emitter\"]", "druid.indexer.logs.directory": "/user/druid/logs", "druid.indexer.logs.type": "hdfs", "druid.metadata.storage.connector.connectURI": "jdbc:postgresql://%HOSTGROUP::master1%:5432/druid", "druid.metadata.storage.connector.password": "druid", "druid.metadata.storage.connector.user": "druid", "druid.metadata.storage.type": "postgresql", "druid.selectors.indexing.serviceName": "druid/overlord", "druid.storage.storageDirectory": "/user/druid/data", "druid.storage.type": "hdfs" }, "properties_attributes": {} } }, { "druid-overlord": { "properties": { "druid.indexer.runner.type": "remote", "druid.indexer.storage.type": "metadata", "druid.port": "8090", "druid.service": "druid/overlord" }, "properties_attributes": {} } }, { "druid-middlemanager": { "properties": { "druid.indexer.runner.javaOpts": "-server -Xmx2g -Duser.timezone=UTC -Dfile.encoding=UTF-8 -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager -Dhdp.version={{stack_version}} -Dhadoop.mapreduce.job.classloader=true", "druid.port": "8091", "druid.processing.numThreads": "2", "druid.server.http.numThreads": "50", "druid.service": "druid/middlemanager", "druid.worker.capacity": "3" }, "properties_attributes": {} } }, { "druid-coordinator": { "properties": { "druid.coordinator.merge.on": "false", "druid.port": "8081" }, "properties_attributes": {} } }, { "druid-historical": { "properties": { "druid.port": "8083", "druid.processing.numThreads": "2", "druid.server.http.numThreads": "50", "druid.server.maxSize": "300000000000", "druid.service": "druid/historical" }, "properties_attributes": {} } }, { "druid-broker": { "properties": { "druid.broker.http.numConnections": "5", "druid.cache.type": "local", "druid.port": "8082", "druid.processing.numThreads": "2", "druid.server.http.numThreads": "50", "druid.service": "druid/broker" }, "properties_attributes": {} } }, { "druid-router": { "properties": {}, "properties_attributes": {} } }, { "superset": { "properties": { "SECRET_KEY": "{{{ general.password }}}", "SUPERSET_DATABASE_TYPE": "sqlite" }, "properties_attributes": {} } }, { "nifi-ambari-config": { "nifi.max_mem": "4g", "nifi.security.encrypt.configuration.password": "{{{ general.password }}}", "nifi.sensitive.props.key": "{{{ general.password }}}" } }, { "nifi-properties": { "nifi.security.user.login.identity.provider": "", "nifi.sensitive.props.key": "{{{ general.password }}}" } }, { "nifi-registry-ambari-config": { "nifi.registry.security.encrypt.configuration.password": "{{{ general.password }}}" } }, { "nifi-registry-properties": { "nifi.registry.db.password": "{{{ general.password }}}", "nifi.registry.sensitive.props.key": "{{{ general.password }}}" } }, { "registry-common": { "properties": { "adminPort": "7789", "database_name": "registry", "jar.storage": "/hdf/registry", "jar.storage.hdfs.url": "hdfs://localhost:9090", "jar.storage.type": "local", "port": "7788", "registry.schema.cache.expiry.interval": "3600", "registry.schema.cache.size": "10000", "registry.storage.connector.connectURI": "jdbc:mysql://localhost:3306/registry", "registry.storage.connector.password": "registry", "registry.storage.connector.user": "registry", "registry.storage.query.timeout": "30", "registry.storage.type": "mysql" }, "properties_attributes": {} } }, { "hbase-site": { "properties": { "hbase.bucketcache.combinedcache.enabled": "true", "hbase.bucketcache.ioengine": "file:/hbase/cache", "hbase.bucketcache.size": "24000", "hbase.defaults.for.version.skip": "true", "hbase.hregion.max.filesize": "21474836480", "hbase.hregion.memstore.flush.size": "536870912", "hbase.region.server.rpc.scheduler.factory.class": "org.apache.hadoop.hbase.ipc.PhoenixRpcSchedulerFactory", "hbase.regionserver.global.memstore.size": "0.4", "hbase.regionserver.handler.count": "60", "hbase.regionserver.wal.codec": "org.apache.hadoop.hbase.regionserver.wal.IndexedWALEditCodec", "hbase.rootdir": "/apps/hbase", "hbase.rpc.controllerfactory.class": "org.apache.hadoop.hbase.ipc.controller.ServerRpcControllerFactory", "hbase.rs.cacheblocksonwrite": "true", "hfile.block.bloom.cacheonwrite": "true", "hfile.block.cache.size": "0.4", "hfile.block.index.cacheonwrite": "true", "phoenix.functions.allowUserDefinedFunctions": "true", "phoenix.query.timeoutMs": "60000" }, "properties_attributes": {} } }, { "hbase-env": { "properties": { "hbase_java_io_tmpdir": "/tmp", "hbase_log_dir": "/var/log/hbase", "hbase_master_heapsize": "1024m", "hbase_pid_dir": "/var/run/hbase", "hbase_regionserver_heapsize": "16384m", "hbase_regionserver_shutdown_timeout": "30", "hbase_regionserver_xmn_max": "16384", "hbase_regionserver_xmn_ratio": "0.2", "hbase_user": "hbase", "hbase_user_nofile_limit": "32000", "hbase_user_nproc_limit": "16000", "phoenix_sql_enabled": "true" }, "properties_attributes": {} } } ], "host_groups": [ { "cardinality": "1", "components": [ { "name": "RANGER_TAGSYNC" }, { "name": "RANGER_USERSYNC" }, { "name": "RANGER_ADMIN" }, { "name": "KNOX_GATEWAY" }, { "name": "HIVE_SERVER" }, { "name": "HIVE_METASTORE" }, { "name": "DRUID_OVERLORD" }, { "name": "DRUID_COORDINATOR" }, { "name": "DRUID_ROUTER" }, { "name": "DRUID_BROKER" }, { "name": "SECONDARY_NAMENODE" }, { "name": "HISTORYSERVER" }, { "name": "APP_TIMELINE_SERVER" }, { "name": "REGISTRY_SERVER" }, { "name": "NIFI_REGISTRY_MASTER" }, { "name": "DATANODE" }, { "name": "YARN_CLIENT" }, { "name": "HDFS_CLIENT" }, { "name": "TEZ_CLIENT" }, { "name": "INFRA_SOLR_CLIENT" }, { "name": "ZOOKEEPER_CLIENT" }, { "name": "MAPREDUCE2_CLIENT" }, { "name": "ATLAS_CLIENT" }, { "name": "HBASE_CLIENT" }, { "name": "HIVE_CLIENT" }, { "name": "LOGSEARCH_LOGFEEDER" }, { "name": "SPARK2_CLIENT" } ], "name": "master1" }, { "cardinality": "1+", "components": [ { "name": "NODEMANAGER" }, { "name": "DATANODE" }, { "name": "HBASE_REGIONSERVER" }, { "name": "MAPREDUCE2_CLIENT" }, { "name": "YARN_CLIENT" }, { "name": "HDFS_CLIENT" }, { "name": "TEZ_CLIENT" }, { "name": "ZOOKEEPER_CLIENT" }, { "name": "ATLAS_CLIENT" }, { "name": "HBASE_CLIENT" }, { "name": "HIVE_CLIENT" }, { "name": "METRICS_MONITOR" }, { "name": "LOGSEARCH_LOGFEEDER" }, { "name": "KAFKA_BROKER" }, { "name": "NIFI_MASTER" }, { "name": "ZOOKEEPER_SERVER" }, { "name": "HBASE_CLIENT" }, { "name": "SPARK2_CLIENT" } ], "name": "worker" }, { "cardinality": "1", "components": [ { "name": "ATLAS_SERVER" }, { "name": "HBASE_MASTER" }, { "name": "METRICS_COLLECTOR" }, { "name": "RESOURCEMANAGER" }, { "name": "DRUID_HISTORICAL" }, { "name": "DRUID_MIDDLEMANAGER" }, { "name": "LIVY2_SERVER" }, { "name": "SPARK2_JOBHISTORYSERVER" }, { "name": "DATANODE" }, { "name": "HIVE_CLIENT" }, { "name": "ZOOKEEPER_CLIENT" }, { "name": "ATLAS_CLIENT" }, { "name": "MAPREDUCE2_CLIENT" }, { "name": "TEZ_CLIENT" }, { "name": "HBASE_CLIENT" }, { "name": "METRICS_MONITOR" }, { "name": "LOGSEARCH_LOGFEEDER" }, { "name": "LOGSEARCH_SERVER" }, { "name": "NAMENODE" }, { "name": "SUPERSET" }, { "name": "NIFI_CA" }, { "name": "INFRA_SOLR" }, { "name": "METRICS_GRAFANA" }, { "name": "METRICS_MONITOR" }, { "name": "HBASE_MASTER" }, { "name": "HBASE_CLIENT" }, { "name": "SPARK2_CLIENT" } ], "name": "master2" }, { "name": "cdsw_worker", "cardinality": "1+", "components": [ { "name": "SPARK2_CLIENT" }, { "name": "ZOOKEEPER_CLIENT" }, { "name": "YARN_CLIENT" }, { "name": "HDFS_CLIENT" }, { "name": "MAPREDUCE2_CLIENT" }, { "name": "HIVE_CLIENT" }, { "name": "NODEMANAGER" }, { "name": "DATANODE" }, { "name": "KAFKA_BROKER" }, { "name": "NIFI_MASTER" }, { "name": "ZOOKEEPER_SERVER" }, { "name": "HBASE_REGIONSERVER" }, { "name": "HBASE_CLIENT" }, { "name": "METRICS_MONITOR" }, { "name": "TEZ_CLIENT" } ] } ], "settings": [ { "recovery_settings": [ { "recovery_enabled": "false" } ] } ] } Recipes Pre Ambari start recipe to setup metastores #!/usr/bin/env bash # Intialize MetaStores yum install -y https://download.postgresql.org/pub/repos/yum/9.6/redhat/rhel-7-x86_64/pgdg-redhat96-9.6-3.noarch.rpm yum install -y postgresql96-server yum install -y postgresql96-contrib /usr/pgsql-9.6/bin/postgresql96-setup initdb sed -i 's,#port = 5432,port = 5433,g' /var/lib/pgsql/9.6/data/postgresql.conf echo '' > /var/lib/pgsql/9.6/data/pg_hba.conf echo 'local all das,streamsmsgmgr,cloudbreak,registry,ambari,postgres,hive,ranger,rangerdba,rangeradmin,rangerlogger,druid trust ' >> /var/lib/pgsql/9.6/data/pg_hba.conf echo 'host all das,streamsmsgmgr,cloudbreak,registry,ambari,postgres,hive,ranger,rangerdba,rangeradmin,rangerlogger,druid 0.0.0.0/0 trust ' >> /var/lib/pgsql/9.6/data/pg_hba.conf echo 'host all das,streamsmsgmgr,cloudbreak,registry,ambari,postgres,hive,ranger,rangerdba,rangeradmin,rangerlogger,druid ::/0 trust ' >> /var/lib/pgsql/9.6/data/pg_hba.conf echo 'local all all peer ' >> /var/lib/pgsql/9.6/data/pg_hba.conf echo 'host all all 127.0.0.1/32 ident ' >> /var/lib/pgsql/9.6/data/pg_hba.conf echo 'host all all ::1/128 ident ' >> /var/lib/pgsql/9.6/data/pg_hba.conf systemctl enable postgresql-9.6.service systemctl start postgresql-9.6.service echo "CREATE DATABASE streamsmsgmgr;" | sudo -u postgres psql -U postgres -h localhost -p 5433 echo "CREATE USER streamsmsgmgr WITH PASSWORD 'streamsmsgmgr';" | sudo -u postgres psql -U postgres -h localhost -p 5433 echo "GRANT ALL PRIVILEGES ON DATABASE streamsmsgmgr TO streamsmsgmgr;" | sudo -u postgres psql -U postgres -h localhost -p 5433 echo "CREATE DATABASE druid;" | sudo -u postgres psql -U postgres echo "CREATE DATABASE ranger;" | sudo -u postgres psql -U postgres echo "CREATE DATABASE registry;" | sudo -u postgres psql -U postgres echo "CREATE USER druid WITH PASSWORD 'druid';" | sudo -u postgres psql -U postgres echo "CREATE USER registry WITH PASSWORD 'registry';" | sudo -u postgres psql -U postgres echo "CREATE USER rangerdba WITH PASSWORD 'rangerdba';" | sudo -u postgres psql -U postgres echo "CREATE USER rangeradmin WITH PASSWORD 'ranger';" | sudo -u postgres psql -U postgres echo "GRANT ALL PRIVILEGES ON DATABASE druid TO druid;" | sudo -u postgres psql -U postgres echo "GRANT ALL PRIVILEGES ON DATABASE registry TO registry;" | sudo -u postgres psql -U postgres echo "GRANT ALL PRIVILEGES ON DATABASE ranger TO rangerdba;" | sudo -u postgres psql -U postgres echo "GRANT ALL PRIVILEGES ON DATABASE ranger TO rangeradmin;" | sudo -u postgres psql -U postgres #ambari-server setup --jdbc-db=postgres --jdbc-driver=/usr/share/java/postgresql-jdbc.jar if [[ $(cat /etc/system-release|grep -Po Amazon) == "Amazon" ]]; then echo '' > /var/lib/pgsql/9.5/data/pg_hba.conf echo 'local all cloudbreak,ambari,postgres,hive,ranger,rangerdba,rangeradmin,rangerlogger,druid,registry trust ' >> /var/lib/pgsql/9.5/data/pg_hba.conf echo 'host all cloudbreak,ambari,postgres,hive,ranger,rangerdba,rangeradmin,rangerlogger,druid,registry 0.0.0.0/0 trust ' >> /var/lib/pgsql/9.5/data/pg_hba.conf echo 'host all cloudbreak,ambari,postgres,hive,ranger,rangerdba,rangeradmin,rangerlogger,druid,registry ::/0 trust ' >> /var/lib/pgsql/9.5/data/pg_hba.conf echo 'local all all peer ' >> /var/lib/pgsql/9.5/data/pg_hba.conf echo 'host all all 127.0.0.1/32 ident ' >> /var/lib/pgsql/9.5/data/pg_hba.conf echo 'host all all ::1/128 ident ' >> /var/lib/pgsql/9.5/data/pg_hba.conf sudo -u postgres /usr/pgsql-9.5/bin/pg_ctl -D /var/lib/pgsql/9.5/data/ reload else echo '' > /var/lib/pgsql/data/pg_hba.conf echo 'local all cloudbreak,ambari,postgres,hive,ranger,rangerdba,rangeradmin,rangerlogger,druid,registry trust ' >> /var/lib/pgsql/data/pg_hba.conf echo 'host all cloudbreak,ambari,postgres,hive,ranger,rangerdba,rangeradmin,rangerlogger,druid,registry 0.0.0.0/0 trust ' >> /var/lib/pgsql/data/pg_hba.conf echo 'host all cloudbreak,ambari,postgres,hive,ranger,rangerdba,rangeradmin,rangerlogger,druid,registry ::/0 trust ' >> /var/lib/pgsql/data/pg_hba.conf echo 'local all all peer ' >> /var/lib/pgsql/data/pg_hba.conf echo 'host all all 127.0.0.1/32 ident ' >> /var/lib/pgsql/data/pg_hba.conf echo 'host all all ::1/128 ident ' >> /var/lib/pgsql/data/pg_hba.conf sudo -u postgres pg_ctl -D /var/lib/pgsql/data/ reload fi yum remove -y mysql57-community* yum remove -y mysql56-server* yum remove -y mysql-community* rm -Rvf /var/lib/mysql yum install -y epel-release yum install -y libffi-devel.x86_64 ln -s /usr/lib64/libffi.so.6 /usr/lib64/libffi.so.5 yum install -y mysql-connector-java* ambari-server setup --jdbc-db=mysql --jdbc-driver=/usr/share/java/mysql-connector-java.jar if [ $(cat /etc/system-release|grep -Po Amazon) == Amazon ]; then yum install -y mysql56-server service mysqld start else yum localinstall -y https://dev.mysql.com/get/mysql-community-release-el7-5.noarch.rpm yum install -y mysql-community-server systemctl start mysqld.service fi chkconfig --add mysqld chkconfig mysqld on ln -s /usr/share/java/mysql-connector-java.jar /usr/hdp/current/hive-client/lib/mysql-connector-java.jar ln -s /usr/share/java/mysql-connector-java.jar /usr/hdp/current/hive-server2-hive2/lib/mysql-connector-java.jar mysql --execute="CREATE DATABASE druid DEFAULT CHARACTER SET utf8" mysql --execute="CREATE DATABASE registry DEFAULT CHARACTER SET utf8" mysql --execute="CREATE DATABASE streamline DEFAULT CHARACTER SET utf8" mysql --execute="CREATE DATABASE streamsmsgmgr DEFAULT CHARACTER SET utf8" mysql --execute="CREATE USER 'das'@'localhost' IDENTIFIED BY 'dasuser'" mysql --execute="CREATE USER 'das'@'%' IDENTIFIED BY 'dasuser'" mysql --execute="CREATE USER 'ranger'@'localhost' IDENTIFIED BY 'ranger'" mysql --execute="CREATE USER 'ranger'@'%' IDENTIFIED BY 'ranger'" mysql --execute="CREATE USER 'rangerdba'@'localhost' IDENTIFIED BY 'rangerdba'" mysql --execute="CREATE USER 'rangerdba'@'%' IDENTIFIED BY 'rangerdba'" mysql --execute="CREATE USER 'registry'@'localhost' IDENTIFIED BY 'registry'" mysql --execute="CREATE USER 'registry'@'%' IDENTIFIED BY 'registry'" mysql --execute="CREATE USER 'streamsmsgmgr'@'localhost' IDENTIFIED BY 'streamsmsgmgr'" mysql --execute="CREATE USER 'streamsmsgmgr'@'%' IDENTIFIED BY 'streamsmsgmgr'" mysql --execute="CREATE USER 'druid'@'%' IDENTIFIED BY 'druid'" mysql --execute="CREATE USER 'streamline'@'%' IDENTIFIED BY 'streamline'" mysql --execute="CREATE USER 'streamline'@'localhost' IDENTIFIED BY 'streamline'" mysql --execute="GRANT ALL PRIVILEGES ON *.* TO 'das'@'localhost'" mysql --execute="GRANT ALL PRIVILEGES ON *.* TO 'das'@'%'" mysql --execute="GRANT ALL PRIVILEGES ON *.* TO 'das'@'localhost' WITH GRANT OPTION" mysql --execute="GRANT ALL PRIVILEGES ON *.* TO 'das'@'%' WITH GRANT OPTION" mysql --execute="GRANT ALL PRIVILEGES ON *.* TO 'ranger'@'localhost'" mysql --execute="GRANT ALL PRIVILEGES ON *.* TO 'ranger'@'%'" mysql --execute="GRANT ALL PRIVILEGES ON *.* TO 'ranger'@'localhost' WITH GRANT OPTION" mysql --execute="GRANT ALL PRIVILEGES ON *.* TO 'ranger'@'%' WITH GRANT OPTION" mysql --execute="GRANT ALL PRIVILEGES ON *.* TO 'rangerdba'@'localhost'" mysql --execute="GRANT ALL PRIVILEGES ON *.* TO 'rangerdba'@'%'" mysql --execute="GRANT ALL PRIVILEGES ON *.* TO 'rangerdba'@'localhost' WITH GRANT OPTION" mysql --execute="GRANT ALL PRIVILEGES ON *.* TO 'rangerdba'@'%' WITH GRANT OPTION" mysql --execute="GRANT ALL PRIVILEGES ON druid.* TO 'druid'@'%' WITH GRANT OPTION" mysql --execute="GRANT ALL PRIVILEGES ON *.* TO 'registry'@'localhost'" mysql --execute="GRANT ALL PRIVILEGES ON *.* TO 'registry'@'%'" mysql --execute="GRANT ALL PRIVILEGES ON *.* TO 'registry'@'localhost' WITH GRANT OPTION" mysql --execute="GRANT ALL PRIVILEGES ON *.* TO 'registry'@'%' WITH GRANT OPTION" mysql --execute="GRANT ALL PRIVILEGES ON *.* TO 'streamsmsgmgr'@'localhost'" mysql --execute="GRANT ALL PRIVILEGES ON *.* TO 'streamsmsgmgr'@'%'" mysql --execute="GRANT ALL PRIVILEGES ON *.* TO 'streamsmsgmgr'@'localhost' WITH GRANT OPTION" mysql --execute="GRANT ALL PRIVILEGES ON *.* TO 'streamsmsgmgr'@'%' WITH GRANT OPTION" mysql --execute="GRANT ALL PRIVILEGES ON streamline.* TO 'streamline'@'%' WITH GRANT OPTION" mysql --execute="CREATE DATABASE beast_mode_db DEFAULT CHARACTER SET utf8" mysql --execute="CREATE USER 'bmq_user'@'localhost' IDENTIFIED BY 'Be@stM0de'" mysql --execute="CREATE USER 'bmq_user'@'%' IDENTIFIED BY 'Be@stM0de'" mysql --execute="GRANT ALL PRIVILEGES ON beast_mode_db.* TO 'bmq_user'@'localhost'" mysql --execute="GRANT ALL PRIVILEGES ON beast_mode_db.* TO 'bmq_user'@'%'" mysql --execute="GRANT ALL PRIVILEGES ON beast_mode_db.* TO 'bmq_user'@'localhost' WITH GRANT OPTION" mysql --execute="GRANT ALL PRIVILEGES ON beast_mode_db.* TO 'bmq_user'@'%' WITH GRANT OPTION" mysql --execute="FLUSH PRIVILEGES" mysql --execute="COMMIT" #remount tmpfs to ensure NOEXEC is disabled if grep -Eq '^[^ ]+ /tmp [^ ]+ ([^ ]*,)?noexec[, ]' /proc/mounts; then echo "/tmp found as noexec, remounting..." mount -o remount,size=10G /tmp mount -o remount,exec /tmp else echo "/tmp not found as noexec, skipping..." fi Pre Ambari start recipe to grow the root volume for the CDSW worker #!/usr/bin/env bash # WARNING: This script is only for RHEL7 on Azure # growing the /dev/sda2 partition sed -e 's/\s*\([\+0-9a-zA-Z]*\).*/\1/' << EOF | fdisk /dev/sda d # delete 2 # delete partition 2 n # new p # partition 2 # partition 2 # default # default w # write the partition table q # and we're done EOF reboot Post cluster install recipe to setup CDSW #!/usr/bin/env bash # WARNING: This script is only for RHEL7 on Azure # growing the /dev/sda2 partition xfs_growfs /dev/sda2 # Some of these installs may be unecessary but are included for completeness against documentation yum -y install nfs-utils libseccomp lvm2 bridge-utils libtool-ltdl ebtables rsync policycoreutils-python ntp bind-utils nmap-ncat openssl e2fsprogs redhat-lsb-core socat selinux-policy-base selinux-policy-targeted # CDSW wants a pristine IPTables setup iptables -P INPUT ACCEPT iptables -P FORWARD ACCEPT iptables -P OUTPUT ACCEPT iptables -t nat -F iptables -t mangle -F iptables -F iptables -X # set java_home on centos7 #echo 'export JAVA_HOME=$(readlink -f /usr/bin/javac | sed "s:/bin/javac::")' >> /etc/profile #export JAVA_HOME=$(readlink -f /usr/bin/javac | sed "s:/bin/javac::") echo 'export JAVA_HOME=/usr/lib/jvm/java' >> /etc/profile export JAVA_HOME='/usr/lib/jvm/java' # Fetch public IP export MASTER_IP=$(hostname --ip-address) # Fetch public FQDN for Domain export DOMAIN=$(curl https://ipv4.icanhazip.com) cd /hadoopfs/ mkdir cdsw # Install CDSW #wget -q --no-check-certificate https://s3.eu-west-2.amazonaws.com/whoville/v2/temp.blob #mv temp.blob cloudera-data-science-workbench-1.5.0.818361-1.el7.centos.x86_64.rpm wget -q https://archive.cloudera.com/cdsw1/1.5.0/redhat7/yum/RPMS/x86_64/cloudera-data-science-workbench-1.5.0.849870-1.el7.centos.x86_64.rpm yum install -y cloudera-data-science-workbench-1.5.0.849870-1.el7.centos.x86_64.rpm # Install Anaconda curl -Ok https://repo.anaconda.com/archive/Anaconda2-5.2.0-Linux-x86_64.sh chmod +x ./Anaconda2-5.2.0-Linux-x86_64.sh ./Anaconda2-5.2.0-Linux-x86_64.sh -b -p /anaconda # create unix user useradd tutorial echo "tutorial-password" | passwd --stdin tutorial su - hdfs -c 'hdfs dfs -mkdir /user/tutorial' su - hdfs -c 'hdfs dfs -chown tutorial:hdfs /user/tutorial' # CDSW Setup sed -i "s@MASTER_IP=\"\"@MASTER_IP=\"${MASTER_IP}\"@g" /etc/cdsw/config/cdsw.conf sed -i "s@JAVA_HOME=\"/usr/java/default\"@JAVA_HOME=\"$(echo ${JAVA_HOME})\"@g" /etc/cdsw/config/cdsw.conf sed -i "s@DOMAIN=\"cdsw.company.com\"@DOMAIN=\"${DOMAIN}.xip.io\"@g" /etc/cdsw/config/cdsw.conf sed -i "s@DOCKER_BLOCK_DEVICES=\"\"@DOCKER_BLOCK_DEVICES=\"${DOCKER_BLOCK}\"@g" /etc/cdsw/config/cdsw.conf sed -i "s@APPLICATION_BLOCK_DEVICE=\"\"@APPLICATION_BLOCK_DEVICE=\"${APP_BLOCK}\"@g" /etc/cdsw/config/cdsw.conf sed -i "s@DISTRO=\"\"@DISTRO=\"HDP\"@g" /etc/cdsw/config/cdsw.conf sed -i "s@ANACONDA_DIR=\"\"@ANACONDA_DIR=\"/anaconda/bin\"@g" /etc/cdsw/config/cdsw.conf # CDSW will break default Amazon DNS on 127.0.0.1:53, so we use a different IP sed -i "s@nameserver 127.0.0.1@nameserver 169.254.169.253@g" /etc/dhcp/dhclient-enter-hooks cdsw init echo "CDSW will shortly be available on ${DOMAIN}" # after the init, we wait until we are able to create the tutorial user export respCode=404 while (( $respCode != 201 )) do sleep 10 export respCode=$(curl -iX POST http://${DOMAIN}.xip.io/api/v1/users/ -H 'Content-Type: application/json' -d '{"email":"tutorial@tutorial.com","name":"tutorial","username":"tutorial","password":"tutorial-password","type":"user","admin":true}' | grep HTTP | awk '{print $2}') done exit 0 Note: this script is using xip.io and hacks into unix to create user and hadoop folders, not a recommendation in production! Management packs IMAGE You will need two management packs for this setup, using the URL detailed below: HDF mpack: http://s3.amazonaws.com/dev.hortonworks.com/HDF/centos7/3.x/BUILDS/3.3.1.0-10/tars/hdf_ambari_mp/hdf-ambari-mpack-3.3.1.0-10.tar.gz Search mpack: http://public-repo-1.hortonworks.com/HDP-SOLR/hdp-solr-ambari-mp/solr-service-mpack-4.0.0.tar.gz Step 3: Create cluster This step uses Cloudbreak's Create Cluster wizard, and is pretty self-explanatory following screenshots, but I will add specific parameters in text form for convenience Note: Do not forget to toggle the advanced mode when running the wizard (top of the screen) General Configuration Image Settings Hardware and Storage Note: Make sure to use 100 GB as the root volume size for CDSW. Network and Availability Cloud Storage Cluster Extensions External Sources Gateway Configuration Network Security Groups Security Result After the cluster created, you should have access to the following screen in Cloudbreak: You can now access Ambari via the link provided, and CDSW using http://[CDSW_WORKER_PUBLIC_IP].xip.io ... View more

Introduction Before the official release of Cloudera Data Flow (ex. Hortonworks Data Flow), you may want to play with Nifi and Hive on CDH. However, because CDH 5 is using a fork of Hive 1.1, the HiveQL processors and controller services included on the official Apache release will not work, so you need to have your own, as explained in this article: Connecting NiFi to CDH Hive. This article is awesome, but does not focus on Kerberos/SSL; since I had to do the configuration myself, I thought I would share the knowledge. Note: You could use a DBCP connection to connect to Cloudera Hive but it will not allow you to use the proper authentication. Pre-Requisites To connect to Hive with SSL and Kerberos, you will need the following: A running version of Nifi (I used Apache 1.9 in this example) A Kerberized CDH cluster with Hive on SSL (I used CDH 5.15 in this example) The certificate to add to your keystore for SSL connection A keytab for a specific user authorized in the cluster The krb5 configuration file from the cluster hive-site.xml, core-site.xml and hdfs-site.xml from your cluster Nifi processors and services compiled for Hive 1.1 on CDH (can be compiled like described in the article I linked to) Step 1: Add certificate to Java truststore The goal of this step is to add your certificate to the Java cacerts that is used to run Nifi. In order to import your certificate, run the following command: keytool –importcert –alias HS2server -keystore [LOCATION_OF_CACERTS] –file [LOCATION_OF_YOUR_CERTIFICATE] I'm running on MacOS, so my cacerts is under /Library/Java/JavaVirtualMachines/jdk1.8.0_181.jdk/Contents/Home/jre/lib/security/cacerts , so I ran: keytool –importcert –alias HS2server -keystore /Library/Java/JavaVirtualMachines/jdk1.8.0_181.jdk/Contents/Home/jre/lib/security/cacerts –file /Users/pvidal/Documents/customers/quest/config/tls/rootCA.pem Step 2: Prepare Nifi Note: This step will require a Nifi restart, so I suggest to stop Nifi before following these instructions and then start it afterwards Add the krb5 conf file to Nifi Properties Go to your Nifi conf folder and modify the nifi.properties file to add the following: nifi.kerberos.krb5.file=[LOCATION_OF_YOUR_KRB5.CONF] Load the processors and services to Nifi Go to your Nifi lib folder and add the necessary NARs; I added the following: -rwxr-xr-x@ 1 pvidal admin 14800 Mar 5 16:39 nifi-hive-services-api-nar-1.9.0.1.0.0.0-49.nar -rwxr-xr-x@ 1 pvidal admin 164674666 Mar 5 16:39 nifi-hive_1_1-nar-1.9.0.1.0.0.0-49.nar Step 3: Configure Nifi Note: Remember to restart Nifi before this step. Configure a KeytabCredentialsService Go to your controller services, and add a new KeytabCredentialsService. Configure the service as such: Kerberos Keytab: [LOCATION_OF_YOUR_KEYTAB] Kerberos Principal: [NAME_OF_YOUR_PRINCIPAL] Enable the service. Configure a Hive_1_1ConnectionPool Go to your controller services, and add a new Hive_1_1ConnectionPool (from the NAR you imported). Configure the service as such: Database Connection URL: jdbc:hive2://[YOUR_HIVE_HOST]:10000/default;principal=hive/_HOST@[YOUR_DOMAIN,SAME AS PRINICPAL];ssl=true Hive Configuration Resources: [LOCATION_OF_HIVE_SITE.XML],[LOCATION_OF_CORE_SITE.XML],[LOCATION_OF_HDFS_SITE.XML] Kerberos Credentials Service: [YOUR_KEYTABCREDENTIALSSERVICE] Enable the service. Configure a simple flow I configured a simple flow that only contains: A SelectHive_1_1QL (from the NAR I imported) A convert Avro to JSON (to make it readable) A log message The only bit of configuration I had to do was referencing the Hive_1_1ConnectionPool I created earlier, as depicted below: Note: With the official release of CDF, all of this will be MUCH simpler, with no need for NAR import. If you're not excited about it, I am! ... View more

I have been playing quite a bit with CDSW lately. Here is a quick article on how to setup a CDSW project in scala connecting to an external RDBMS Step 1: Create a new CDSW Project Using the CDSW UI, create a new Scala Project: Step 2: Reference the external Jar in your spark-defaults.conf Open your project, and edit your spark-defaults.conf to add an external jar: spark.jars=http://central.maven.org/maven2/mysql/mysql-connector-java/5.1.6/mysql-connector-java-5.1.6.jar Step 3: Create a simple Scala file to connect to the DB Create a new file and add this code in it: val sqlContext = new org.apache.spark.sql.SQLContext(sc) val df = sqlContext.read.format("jdbc").option("url", "jdbc:mysql://[YOUR_SERVER_IP]:3306/[YOUR_DB]").option("driver", "com.mysql.jdbc.Driver").option("dbtable", "[YOUR_TABLE]").option("user", "[YOUR_USER]").option("password", "[YOUR_PWD]").load() df.show() Step 4: Run your application Launch a session and run your code: ... View more

Introduction Finally, we get to put all this together! This is less of a tutorial, and more of a show case of the capabilities of this solution. Please refere to previous articles to build this on your own. The meat of this article is showcasing the React/NodeJS application you can build to communicate with Nifi, Cloudbreak and your ephemeral cluster running Zeppelin and Spark. Architecture Below is a high-level architecture of the solution: You can find all the code for this here: https://github.com/paulvid/bmq-app The code includes: Latest Zeppelin Prediction Notebook All Nifi templates All Blueprints/Recipes for Cloudbreak The source code of the app Agenda This article is divided in the following sections: Section 1: Monitoring your current data with Nifi Section 2: Launching an ephemeral cluster with Cloudbreak Section 3: Running a prediction model with Spark Zeppelin Section 1: Monitoring your current data with Nifi Section 2: Launching an ephemeral cluster with Cloudbreak Section 3: Running a prediction model with Spark Zeppelin ... View more

Introduction It's been a while, but I'm finally finishing this series on Best Mode Quotient. This article is fairly straight forward as we look how to build the infrastructure of model training using Zeppelin and Spark. Pre-Requisites In order to run this install, you will have to have deployed a BMQ ephemeral cluster as detailed in this article and this repository. Moreover you will have to have gathered data on your fitness as explained in part 1 of these series. Based on this data, I created 3 indexes: Intensity Index: How intense was the workout I did that day (based on distance, pace and elevation) Fatigue Index: How much rest I had on that day (based on hours of sleep and rest heart rate) BMQ Index: How my BMQ fairs compare to the max BMQ (5) You can then agglomerate these 3 indexes, having BMQ and Fatigue on the same day correlating to the Intensity Index of the previous day. All data will be shared in the last article of the series. You will also notice that the model uses parameterized sleep and rest HR. The whole flow is to be revealed in part 4 🙂 Agenda This tutorial is divided in the following sections: Section 1: Create a mysql interpreter for JDBC in Zeppelin Section 2: Create training set for BMQ prediction Section 3: Create a prediction model Section 4: Save the results in a table Section 1: Create a mysql interpreter for JDBC in Zeppelin Login to Zeppelin, then go to top right corner > Admin > Interpreter and edit the jdbc interpreter. Add the following parameters: mysql.driver: com.mysql.jdbc.Driver mysql.url: jdbc:mysql://localhost:3306/beast_mode_db mysql.user: bmq_user mysql.password: Be@stM0de Add artifact: mysql:mysql-connector-java:5.1.38 Restart the interpreter and you should be good to go. Section 2: Create training set for BMQ prediction Create a new note called BMQ Predictions and add the following code, using the jdbc interpreter you just built. Delete existing training tables if any %jdbc(mysql) drop table if exists training_set Create training set based on fatigue and intensity indexes %jdbc(mysql) create table training_set as ( select @rowid:=@rowid+1 as rowid, bmq_index.date, bmq_index, fatigue_index, intensity_index from bmq_index, fatigue_index, intensity_index, (select @rowid:=0) as init where bmq_index.date = fatigue_index.date and date_sub(bmq_index.date, INTERVAL 1 DAY) = intensity_index.date order by bmq_index.date asc) View Data %jdbc(mysql) select * from training_set Delete existing prediction tables if any %jdbc(mysql) drop table if exists prediction Create a table we want to apply the algo against %jdbc(mysql) create table prediction as ( select date(training_date) as date, estimated_intensity_index, round(( (1-((select (sleep_hours*60) from PREDICTION_PARAMETERS)/(select max(TOTAL_MINUTES_ASLEEP) from SLEEP_HISTORY)))*0.6 + (1-((select min(REST_HR) from HEALTH_HISTORY)/(select rest_hr from PREDICTION_PARAMETERS)))*0.4 ) *100,2) as estimated_fatigue_index, 0.0 as predicted_bmq from training_plan) View Data %jdbc(mysql) select * from prediction Section 3: Create a prediction model DISCLAIMER: This model needs to be worked on; the purpose of this article is to establish the principal architecture, not give the final most tuned model as I plan on improving on it. This part uses the spark interpreter of Zeppelin to vectorize, normalize and train a model Create dataframe from MySQL tables: %spark2 import org.apache.spark.sql.SQLContext import org.apache.spark.ml.regression.LinearRegression import org.apache.spark.ml.feature.Interaction import org.apache.spark.ml.feature.VectorAssembler import org.apache.spark.ml.feature.Normalizer val sqlcontext = new org.apache.spark.sql.SQLContext(sc) val df = sqlcontext.read.format("jdbc").option("url", "jdbc:mysql://localhost:3306/beast_mode_db").option("driver", "com.mysql.jdbc.Driver").option("dbtable", "training_set").option("user", "bmq_user").option("password", "Be@stM0de").load() df.show() %spark2 val target_df = sqlcontext.read.format("jdbc").option("url", "jdbc:mysql://localhost:3306/beast_mode_db").option("driver", "com.mysql.jdbc.Driver").option("dbtable", "prediction").option("user", "bmq_user").option("password", "Be@stM0de").load() target_df.show() Vectorize Dataframes val assembler1 = new VectorAssembler(). setInputCols(Array( "fatigue_index","intensity_index")). setOutputCol("features"). transform(df) assembler1.show() %spark2 val assembler2 = new VectorAssembler(). setInputCols(Array( "estimated_fatigue_index","estimated_intensity_index")). setOutputCol("features"). transform(target_df) assembler2.show() Normalize Dataframes %spark2 val normalizer = new Normalizer() .setInputCol("features") .setOutputCol("normFeatures") .setP(2.0) .transform(assembler1) normalizer.show() %spark2 val targetNormalizer = new Normalizer() .setInputCol("features") .setOutputCol("normFeatures") .setP(2.0) .transform(assembler2) targetNormalizer.show() Train and evaluate Model %spark2 val Array(trainingData, testData) = normalizer.randomSplit(Array(0.7, 0.3)) %spark2 val lr = new LinearRegression() .setLabelCol("bmq_index") .setFeaturesCol("normFeatures") .setMaxIter(10) .setRegParam(1.0) .setElasticNetParam(1.0) val lrModel = lr.fit(trainingData) lrModel.transform(testData).select("features","normFeatures", "bmq_index", "prediction").show() %spark2 println(s"Coefficients: ${lrModel.coefficients} Intercept: ${lrModel.intercept}") %spark2 val trainingSummary = lrModel.summary println(s"numIterations: ${trainingSummary.totalIterations}") println(s"objectiveHistory: [${trainingSummary.objectiveHistory.mkString(",")}]") trainingSummary.residuals.show() println(s"RMSE: ${trainingSummary.rootMeanSquaredError}") println(s"r2: ${trainingSummary.r2}") %spark2 val targetTable = lrModel.transform(targetNormalizer).select("date", "estimated_intensity_index", "estimated_fatigue_index", "prediction") targetTable.show() Section 4: Save the results in a table Finally, we will take the results of the prediction and put them in a table called BMQ_PREDICTIONS for later use: Rename the dataframe to match target columns name %spark2 val newNames = Seq("date", "estimated_intensity_index", "estimated_fatigue_index", "predicted_bmq") val targetTableRenamed = targetTable.toDF(newNames: _*) Delete target table if exists %jdbc(mysql) drop table if exists BMQ_PREDICTIONS Write data %spark2 val prop = new java.util.Properties prop.setProperty("driver", "com.mysql.jdbc.Driver") prop.setProperty("user", "bmq_user") prop.setProperty("password", "Be@stM0de") targetTableRenamed.write.mode("append").jdbc("jdbc:mysql://localhost:3306/beast_mode_db", "BMQ_PREDICTIONS", prop) View data %jdbc(mysql) select * from BMQ_PREDICTIONS ... View more

Here is a self explanatory tutorial. The official Cloudbreak documentation explains how to retrieve your Cloudbreak token via curl or python (see here). I have been playing with Node.JS on my side, so here is how to do it in Node.JS using request: process.env["NODE_TLS_REJECT_UNAUTHORIZED"] = 0; var request = require("request"); var options = { method: 'POST', url: 'https://[YOUR_CB_URL]/identity/oauth/authorize', qs: { response_type: 'token', client_id: 'cloudbreak_shell', 'scope.0': 'openid', source: 'login', redirect_uri: 'http://cloudbreak.shell', accept: 'application/x-www-form-urlencoded' }, headers: { 'Content-Type': 'application/x-www-form-urlencoded', accept: 'application/x-www-form-urlencoded' }, body: 'credentials={"username":"[YOUR_USER]","password":"[YOUR_PASSWORD]"' }; request(options, function (error, response, body) { if (error) throw new Error(error); const querystring = require('querystring'); console.log(querystring.parse(response.headers['location'])['access_token']); }); ... View more

DISCLAIMER: This is a method that is not recommended for any production use, only for development purposes. Only Cloudbreak official versions are supported, and upgrading to a non-supported version could cause unforeseen issues, and loss of official Hortonworks support. Moreover, this method only works for upgrades, not downgrades; downgrades would result in data loss. Recently, I have been playing with Cloudbreak and have been needing to upgrade my local version. While it follows the principles detailed in Hortonworks documentation, I thought I'd share a quick step by step guide on how to upgrade to any version of Cloudbreak. 1. Stop Cloudbreak on your machine: cbd kill 1. Go to https://mvnrepository.com/artifact/com.sequenceiq/cloudbreak and find the version you need (e.g. 2.8.1-rc.48) 2. On the VM where Cloudbreak is running, navigate to the directory where your Profile file is located. For example: cd /var/lib/cloudbreak-deployment/ 2. Run the following commands to download the binary: export CBD_VERSION=2.8.1-rc.48 curl -Ls public-repo-1.hortonworks.com/HDP/cloudbreak/cloudbreak-deployer_${CBD_VERSION}_$(uname)_x86_64.tgz | tar -xz -C /bin cbd 3. Verify the version: cbd version 4. Regenerate assets: cbd regenerate 5. Restart Cloudbreak: cbd start ... View more

Introduction Continuing my series on Beast Mode Quotient, let's automate the creation and termination of data science ready clusters. As always, since this is step 2 of a series of article, this tutorial depends on my previous article. Pre-Requisites In order to run this tutorial you will need to have a CB instance available. There are plenty of good tutorial out there, I recommend this one Agenda This tutorial is divided in the following sections: Section 1: Create a blueprints & recipes to run a minimal data science ephemeral cluster Section 2: Add blueprint and recipes via Cloudbreak interface Section 3: Automate cluster launch and terminate clusters Section 1: Create a blueprints & recipes to run a minimal data science ephemeral cluster A Cloudbreak blueprint has 3 parts: Part 1: Blueprint details Part 2: Services Configuration Part 3: Host Components configuration Here are the details of each part for our blueprint. Blueprint details This part is fairly simple; we want to run an HDP 3.1 cluster, and I'm naming it bmq-data-science. "Blueprints": { "blueprint_name": "bmq-data-science", "stack_name": "HDP", "stack_version": "3.1" } Host Components configuration Similarly, I configured one Host with all the components needed for my services "host_groups": [ { "name": "master", "cardinality": "1", "components": [ { "name": "ZOOKEEPER_SERVER" }, { "name": "NAMENODE" }, { "name": "SECONDARY_NAMENODE" }, { "name": "RESOURCEMANAGER" }, { "name": "HISTORYSERVER" }, { "name": "APP_TIMELINE_SERVER" }, { "name": "LIVY2_SERVER" }, { "name": "SPARK2_CLIENT" }, { "name": "SPARK2_JOBHISTORYSERVER" }, { "name": "ZEPPELIN_MASTER" }, { "name": "METRICS_GRAFANA" }, { "name": "METRICS_MONITOR" }, { "name": "DATANODE" }, { "name": "HIVE_SERVER" }, { "name": "HIVE_METASTORE" }, { "name": "HIVE_CLIENT" }, { "name": "YARN_CLIENT" }, { "name": "HDFS_CLIENT" }, { "name": "ZOOKEEPER_CLIENT" }, { "name": "TEZ_CLIENT" }, { "name": "NODEMANAGER" }, { "name": "MAPREDUCE2_CLIENT" } ] } ] Services Configuration For our purposes, I want to create a minimum cluster that will run YARN, HDFS, HIVE, SPARK and ZEPPELIN (plus all the necessary compute engines behind it). I therefore configured these services according to the Cloudbreak examples that are available in the default Cloudbreak blueprints: "configurations": [ { "yarn-site": { "properties": { "yarn.nodemanager.resource.cpu-vcores": "6", "yarn.nodemanager.resource.memory-mb": "23296", "yarn.scheduler.maximum-allocation-mb": "23296" } } }, { "core-site": { "properties_attributes": {}, "properties": { "fs.s3a.threads.max": "1000", "fs.s3a.threads.core": "500", "fs.s3a.max.total.tasks": "1000", "fs.s3a.connection.maximum": "1500" } } }, { "capacity-scheduler": { "properties": { "yarn.scheduler.capacity.root.queues": "default", "yarn.scheduler.capacity.root.capacity": "100", "yarn.scheduler.capacity.root.maximum-capacity": "100", "yarn.scheduler.capacity.root.default.capacity": "100", "yarn.scheduler.capacity.root.default.maximum-capacity": "100" } } }, { "spark2-defaults": { "properties_attributes": {}, "properties": { "spark.sql.hive.hiveserver2.jdbc.url": "jdbc:hive2://%HOSTGROUP::master%:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2", "spark.sql.hive.hiveserver2.jdbc.url.principal": "hive/_HOST@EC2.INTERNAL", "spark.datasource.hive.warehouse.metastoreUri": "thrift://%HOSTGROUP::master%:9083", "spark.datasource.hive.warehouse.load.staging.dir": "/tmp", "spark.hadoop.hive.zookeeper.quorum": "%HOSTGROUP::master%:2181" } } }, { "hive-site": { "hive.metastore.warehouse.dir": "/apps/hive/warehouse", "hive.exec.compress.output": "true", "hive.merge.mapfiles": "true", "hive.server2.tez.initialize.default.sessions": "true", "hive.server2.transport.mode": "http", "hive.metastore.dlm.events": "true", "hive.metastore.transactional.event.listeners": "org.apache.hive.hcatalog.listener.DbNotificationListener", "hive.repl.cm.enabled": "true", "hive.repl.cmrootdir": "/apps/hive/cmroot", "hive.repl.rootdir": "/apps/hive/repl" } }, { "hdfs-site": { "properties_attributes": {}, "properties": { } } } ] You can find the complete blueprint and other recipes on my github, here Creating a recipe Recipes in Cloubreak are very useful and allow you to run scripts before a cluster is launched or after. In this example, I created a PRE-AMBARI-START recipe that creates the appropriate Postgres and MySQL services on my master box, as well as recreating the DB for my BMQ analysis: #!/bin/bash # Cloudbreak-2.7.2 / Ambari-2.7.0 - something is install pgsq95 yum remove -y postgresql95* # Install pgsql96 yum install -y https://download.postgresql.org/pub/repos/yum/9.6/redhat/rhel-7-x86_64/pgdg-redhat96-9.6-3.noarch.rpm yum install -y postgresql96-server yum install -y postgresql96-contrib /usr/pgsql-9.6/bin/postgresql96-setup initdb sed -i 's,#port = 5432,port = 5433,g' /var/lib/pgsql/9.6/data/postgresql.conf systemctl enable postgresql-9.6.service systemctl start postgresql-9.6.service yum remove -y mysql57-community* yum remove -y mysql56-server* yum remove -y mysql-community* rm -Rvf /var/lib/mysql yum install -y epel-release yum install -y libffi-devel.x86_64 ln -s /usr/lib64/libffi.so.6 /usr/lib64/libffi.so.5 yum install -y mysql-connector-java* ambari-server setup --jdbc-db=mysql --jdbc-driver=/usr/share/java/mysql-connector-java.jar if [ $(cat /etc/system-release|grep -Po Amazon) == Amazon ]; then yum install -y mysql56-server service mysqld start else yum localinstall -y https://dev.mysql.com/get/mysql-community-release-el7-5.noarch.rpm yum install -y mysql-community-server systemctl start mysqld.service fi chkconfig --add mysqld chkconfig mysqld on ln -s /usr/share/java/mysql-connector-java.jar /usr/hdp/current/hive-client/lib/mysql-connector-java.jar ln -s /usr/share/java/mysql-connector-java.jar /usr/hdp/current/hive-server2-hive2/lib/mysql-connector-java.jar mysql --execute="CREATE DATABASE beast_mode_db DEFAULT CHARACTER SET utf8" mysql --execute="CREATE USER 'bmq_user'@'localhost' IDENTIFIED BY 'Be@stM0de'" mysql --execute="CREATE USER 'bmq_user'@'%' IDENTIFIED BY 'Be@stM0de'" mysql --execute="GRANT ALL PRIVILEGES ON beast_mode_db.* TO 'bmq_user'@'localhost'" mysql --execute="GRANT ALL PRIVILEGES ON beast_mode_db.* TO 'bmq_user'@'%'" mysql --execute="GRANT ALL PRIVILEGES ON beast_mode_db.* TO 'bmq_user'@'localhost' WITH GRANT OPTION" mysql --execute="GRANT ALL PRIVILEGES ON beast_mode_db.* TO 'bmq_user'@'%' WITH GRANT OPTION" mysql --execute="FLUSH PRIVILEGES" mysql --execute="COMMIT" Section 2: Add blueprint and recipes via Cloudbreak interface This part is super simple. Follow the User interface to load the files you just created, as depicted below Adding a blueprint Adding a recipe Section 3: Automate cluster launch and terminate clusters This is where the fun begins. For this part I created two scripts for launching and terminating the ephemeral cluster (that will then be called by the BMQ app). Both scripts rely on the cb cli that you can download from your CB instance: Launch Cluster Script As you will see below, the script is divided in the following part: Part 1: Reference the location of the cb-cli for ease of use Part 2: Dumps the content of my long lasting cluster to a recipe that will load them Part 3: Use CB api to add the recipe to my CB instance Part 4: Launch the cluster via cb-cli #!/bin/bash ############################### # 0. Initializing environment # ############################### export PATH=$PATH:/Users/pvidal/Documents/Playground/cb-cli/ ################################################### # 1. Dumping current data and adding it to recipe # ################################################### rm -rf poci-bmq-data-science.sh >/dev/null 2>&1 echo "mysql -u bmq_user -pBe@stM0de beast_mode_db --execute=\"""$(mysqldump -u bm_user -pHWseftw33# beast_mode_db 2> /dev/null)""\"" >> poci-bmq-data-science.sh ################################## # 2. Adding recipe to cloudbreak # ################################## TOKEN=$(curl -k -iX POST -H "accept: application/x-www-form-urlencoded" -d 'credentials={"username":"pvidal@hortonworks.com","password":"HWseftw33#"}' "https://192.168.56.100/identity/oauth/authorize?response_type=token&client_id=cloudbreak_shell&scope.0=openid&source=login&redirect_uri=http://cloudbreak.shell" | grep location | cut -d'=' -f 3 | cut -d'&' -f 1) echo $TOKEN ENCODED_RECIPE=$(base64 poci-bmq-data-science.sh) curl -X DELETE https://192.168.56.100/cb/api/v1/recipes/user/poci-bmq-data-science -H "Authorization: Bearer $TOKEN" -k curl -X POST https://192.168.56.100/cb/api/v1/recipes/user -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' -H 'cache-control: no-cache' -d " { \"name\": \"poci-bmq-data-science\", \"description\": \"Recipe loading BMQ data post BMQ cluster launch\", \"recipeType\": \"POST_CLUSTER_INSTALL\", \"content\": \"$POST_CLUSTER_INSTALL\" }" -k ######################## # 3. Launching cluster # ######################## cb cluster create --cli-input-json tp-bmq-data-science.json --name bmq-data-science-$(date +%s) Terminate Cluster Script This script is much simpler; it uses cb cli to list the clusters running and terminate them: #!/bin/bash ############################### # 0. Initializing environment # ############################### export PATH=$PATH:/Users/pvidal/Documents/Playground/cb-cli/ ################################################ # 1. Get a list of clusters and terminate them # ################################################ cb cluster list | grep Name | awk -F \" '{print $4}' | while read cluster; do echo "Terminating ""$cluster""..." cb cluster delete --name $cluster done Conclusion With this framework, I'm able to launch and terminate clusters in a matter of minutes, as depicted below. Next step will be to model a way to calculate accurate prediction for BMQ! ... View more

Introduction Cloudbreak documentation is fairly thorough and can be found on Hortonworks' documentation portal (e.g. here). However, if you are as lazy as I am, you'll appreciate the few quick examples I'm listing in this article. First and foremost, you can access the documentation of available APIs on your CB instance by using the following URL https://[YOUR_CB_URL]/cb/apidocs ; you should see something like this: Authenticating Authentication in cloudbreak is OAuth 2, and getting a detailed in the Cloudbreak API. Here is simple curl call to get a token: curl -k -iX POST -H "accept: application/x-www-form-urlencoded" -d 'credentials={"username":"[YOUR_USER]","password":"[YOUR_PWD]"}' "https://[YOUR_CB_URL]/identity/oauth/authorize?response_type=token&client_id=cloudbreak_shell&scope.0=openid&source=login&redirect_uri=http://cloudbreak.shell" | grep location | cut -d'=' -f 3 | cut -d'&' -f 1 GET Example: Listing all blueprints The biggest trick to the CB API is to know the URL to use and to remember to disable SSL Certificate Verification (using the -k in curl). Here is an example call that lists all blueprints: TOKEN=$(curl -k -iX POST -H "accept: application/x-www-form-urlencoded" -d 'credentials={"username":"[YOUR_USER]","password":"[YOUR_PWD]"}' "https://[YOUR_CB_URL]/identity/oauth/authorize?response_type=token&client_id=cloudbreak_shell&scope.0=openid&source=login&redirect_uri=http://cloudbreak.shell" | grep location | cut -d'=' -f 3 | cut -d'&' -f 1) curl -X GET https://YOUR_CB_URL]/cb/api/v1/blueprints/account -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' -H 'cache-control: no-cache' -k POST Example: Adding an mpack TOKEN=$(curl -k -iX POST -H "accept: application/x-www-form-urlencoded" -d 'credentials={"username":"[YOUR_USER]","password":"[YOUR_PWD]"}' "https://[YOUR_CB_URL]/identity/oauth/authorize?response_type=token&client_id=cloudbreak_shell&scope.0=openid&source=login&redirect_uri=http://cloudbreak.shell" | grep location | cut -d'=' -f 3 | cut -d'&' -f 1) curl -X POST https://[YOUR_CB_URL]/cb/api/v1/mpacks/account -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' -H 'cache-control: no-cache' -d '{ "name": "hdf-3-2-aws", "description": "HDF 3.2 Mangement Pack for AWS", "mpackUrl": "http://public-repo-1.hortonworks.com/HDF/amazonlinux2/3.x/updates/3.2.0.0/tars/hdf_ambari_mp/hdf-ambari-mpack-3.2.0.0-520.tar.gz", "purge": false, "force": false }' -k POST Example: Uploading a user blueprint To upload your own blueprints, remember to encode your blueprint in base64 TOKEN=$(curl -k -iX POST -H "accept: application/x-www-form-urlencoded" -d 'credentials={"username":"[YOUR_USER]","password":"[YOUR_PWD]"}' "https://[YOUR_CB_URL]/identity/oauth/authorize?response_type=token&client_id=cloudbreak_shell&scope.0=openid&source=login&redirect_uri=http://cloudbreak.shell" | grep location | cut -d'=' -f 3 | cut -d'&' -f 1) ENCODED_BLUEPRINT=$(base64 [YOUR_BP_JSON_LOCATION]) curl -X POST https://[YOUR_CB_URL]/cb/api/v1/blueprints/user -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' -H 'cache-control: no-cache' -d " { \"ambariBlueprint\": \"$ENCODED_BLUEPRINT\", \"description\": \"Blueprint for HWX Data Science Uploaded\", \"inputs\": [], \"tags\": {}, \"name\": \"data-science-workshop-upload\", \"hostGroupCount\": 1, \"status\": \"USER_MANAGED\", \"public\": true }" -k ... View more

Introduction Before being able to predict one's Beast Mode Quotient, we must find an easy way to ingest data from multiple different sources, in this case flat files and APIs. Luckily for everyone in the world, HDF, specifically Nifi is designed to handle multiple data inputs and outputs extremely efficiently, as I will detail in this tutorial. A few notes about the choices of technologies selected for this tutorial and series of article: Target - MySQL: I selected MySQL because this is what my BMQ app is going to use, but it could be any target depending on your use-case. Sources - Fitbit, Strava, MyFitnessPal: There are a bunch of options out there to track health. Fitbit and Strava are the only ones offering a comprehensive and public API (looking at you Garmin, and Apple Health; both have no public APIs, and make it not easy to extract data via file, either having to parse .fit files or only exporting the full set of data); as for MyFitnessPal, I could have used Fitbit API (after sync with MFP) to get Calories data, but I figured it would be nice to demonstrate file ingestion. Notions you will learn in this tutorial Here are some high level concepts highlighted in this article, that you can re-apply to any implementation: Running MySQL queries including create, upsert and selection of date series Using Nifi processors maintaining state Configuring schema registry for file format conversion in Nifi (e.g CSV > JSON) Using Nifi to call OAuth 2 APIs (using refresh token, bearer tokens, etc.) Implementing and using a distributed map cache in Nifi Parsing and splitting JSONs in Nifi Implementing simple math operations using Nifi language Agenda This tutorial is divided in the following sections: Section 1: Setting up a MySQL DB to host health and fitness data Section 2: Creating a Nifi flow to consume MyFitnessPal CSV exports Section 3: Creating a Nifi flow to consume Fitbit health & sleep data Section 4: Creating a Nifi flow to consume Strava activity data Section 1: Setting up a MySQL DB to host health and fitness data Step 1: Create the MySQL DB and users Before being able to do anything, you will have to create a MySQL instance. There are a bunch of tutorials out there that explain you how to do it, but here is one for centos 7 for instance. Once you have this data setup connect as root and run the following database creation script: DROP DATABASE IF EXISTS `BEAST_MODE_DB`; CREATE DATABASE `BEAST_MODE_DB`; CREATE USER 'bm_user'@'%' IDENTIFIED BY '[YOUR_PASSWORD]'; GRANT ALL ON BEAST_MODE_DB.* TO 'bm_user'@'%' IDENTIFIED BY '[YOUR_PASSWORD]'; Step 2: Create the DB tables Connect as bm_user and run this script to create the appropriate tables: -- MySQL dump 10.14 Distrib 5.5.60-MariaDB, for Linux (x86_64) -- -- Host: localhost Database: BEAST_MODE_DB -- ------------------------------------------------------ -- Server version 5.5.60-MariaDB /*!40101 SET @OLD_CHARACTER_SET_CLIENT=@@CHARACTER_SET_CLIENT */; /*!40101 SET @OLD_CHARACTER_SET_RESULTS=@@CHARACTER_SET_RESULTS */; /*!40101 SET @OLD_COLLATION_CONNECTION=@@COLLATION_CONNECTION */; /*!40101 SET NAMES utf8 */; /*!40103 SET @OLD_TIME_ZONE=@@TIME_ZONE */; /*!40103 SET TIME_ZONE='+00:00' */; /*!40014 SET @OLD_UNIQUE_CHECKS=@@UNIQUE_CHECKS, UNIQUE_CHECKS=0 */; /*!40014 SET @OLD_FOREIGN_KEY_CHECKS=@@FOREIGN_KEY_CHECKS, FOREIGN_KEY_CHECKS=0 */; /*!40101 SET @OLD_SQL_MODE=@@SQL_MODE, SQL_MODE='NO_AUTO_VALUE_ON_ZERO' */; /*!40111 SET @OLD_SQL_NOTES=@@SQL_NOTES, SQL_NOTES=0 */; USE BEAST_MODE_DB; -- -- Table structure for table `ACTIVITY_HISTORY` -- DROP TABLE IF EXISTS `ACTIVITY_HISTORY`; /*!40101 SET @saved_cs_client = @@character_set_client */; /*!40101 SET character_set_client = utf8 */; CREATE TABLE `ACTIVITY_HISTORY` ( `ID` bigint(20) NOT NULL, `START_TIME` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, `DURATION` double DEFAULT NULL, `DISTANCE` double DEFAULT NULL, `ELEVATION_GAIN` double DEFAULT NULL, `AVG_HR` double DEFAULT NULL, `AVG_PACE` double DEFAULT NULL, PRIMARY KEY (`ID`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8; /*!40101 SET character_set_client = @saved_cs_client */; -- -- Table structure for table `BMQ_HISTORY` -- DROP TABLE IF EXISTS `BMQ_HISTORY`; /*!40101 SET @saved_cs_client = @@character_set_client */; /*!40101 SET character_set_client = utf8 */; CREATE TABLE `BMQ_HISTORY` ( `ID` mediumint(9) NOT NULL AUTO_INCREMENT, `BMQ` int(11) NOT NULL, `TYPE` char(30) DEFAULT NULL, `TIME_ENTERED` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, PRIMARY KEY (`ID`) ) ENGINE=InnoDB AUTO_INCREMENT=30 DEFAULT CHARSET=utf8; /*!40101 SET character_set_client = @saved_cs_client */; -- -- Table structure for table `GENERATOR` -- DROP TABLE IF EXISTS `GENERATOR`; /*!40101 SET @saved_cs_client = @@character_set_client */; /*!40101 SET character_set_client = utf8 */; CREATE TABLE `GENERATOR` ( `ID` mediumint(9) NOT NULL AUTO_INCREMENT, `VALUE` varchar(50) DEFAULT NULL, PRIMARY KEY (`ID`) ) ENGINE=InnoDB AUTO_INCREMENT=161 DEFAULT CHARSET=latin1; /*!40101 SET character_set_client = @saved_cs_client */; -- -- Dumping data for table `GENERATOR` -- LOCK TABLES `GENERATOR` WRITE; /*!40000 ALTER TABLE `GENERATOR` DISABLE KEYS */; INSERT INTO `GENERATOR` VALUES (1,'dummy'),(2,'dummy'),(3,'dummy'),(4,'dummy'),(5,'dummy'),(6,'dummy'),(7,'dummy'),(8,'dummy'),(9,'dummy'),(10,'dummy'),(11,'dummy'),(12,'dummy'),(13,'dummy'),(14,'dummy'),(15,'dummy'),(16,'dummy'),(17,'dummy'),(18,'dummy'),(19,'dummy'),(20,'dummy'),(21,'dummy'),(22,'dummy'),(23,'dummy'),(24,'dummy'),(25,'dummy'),(26,'dummy'),(27,'dummy'),(28,'dummy'),(29,'dummy'),(30,'dummy'),(31,'dummy'),(32,'dummy'),(33,'dummy'),(34,'dummy'),(35,'dummy'),(36,'dummy'),(37,'dummy'),(38,'dummy'),(39,'dummy'),(40,'dummy'),(41,'dummy'),(42,'dummy'),(43,'dummy'),(44,'dummy'),(45,'dummy'),(46,'dummy'),(47,'dummy'),(48,'dummy'),(49,'dummy'),(50,'dummy'),(51,'dummy'),(52,'dummy'),(53,'dummy'),(54,'dummy'),(55,'dummy'),(56,'dummy'),(57,'dummy'),(58,'dummy'),(59,'dummy'),(60,'dummy'),(61,'dummy'),(62,'dummy'),(63,'dummy'),(64,'dummy'),(65,'dummy'),(66,'dummy'),(67,'dummy'),(68,'dummy'),(69,'dummy'),(70,'dummy'),(71,'dummy'),(72,'dummy'),(73,'dummy'),(74,'dummy'),(75,'dummy'),(76,'dummy'),(77,'dummy'),(78,'dummy'),(79,'dummy'),(80,'dummy'),(81,'dummy'),(82,'dummy'),(83,'dummy'),(84,'dummy'),(85,'dummy'),(86,'dummy'),(87,'dummy'),(88,'dummy'),(89,'dummy'),(90,'dummy'),(91,'dummy'),(92,'dummy'),(93,'dummy'),(94,'dummy'),(95,'dummy'),(96,'dummy'),(97,'dummy'),(98,'dummy'),(99,'dummy'),(100,'dummy'),(101,'dummy'),(102,'dummy'),(103,'dummy'),(104,'dummy'),(105,'dummy'),(106,'dummy'),(107,'dummy'),(108,'dummy'),(109,'dummy'),(110,'dummy'),(111,'dummy'),(112,'dummy'),(113,'dummy'),(114,'dummy'),(115,'dummy'),(116,'dummy'),(117,'dummy'),(118,'dummy'),(119,'dummy'),(120,'dummy'),(121,'dummy'),(122,'dummy'),(123,'dummy'),(124,'dummy'),(125,'dummy'),(126,'dummy'),(127,'dummy'),(128,'dummy'),(129,'dummy'),(130,'dummy'),(131,'dummy'),(132,'dummy'),(133,'dummy'),(134,'dummy'),(135,'dummy'),(136,'dummy'),(137,'dummy'),(138,'dummy'),(139,'dummy'),(140,'dummy'),(141,'dummy'),(142,'dummy'),(143,'dummy'),(144,'dummy'),(145,'dummy'),(146,'dummy'),(147,'dummy'),(148,'dummy'),(149,'dummy'),(150,'dummy'),(151,'dummy'),(152,'dummy'),(153,'dummy'),(154,'dummy'),(155,'dummy'),(156,'dummy'),(157,'dummy'),(158,'dummy'),(159,'dummy'),(160,'dummy'); /*!40000 ALTER TABLE `GENERATOR` ENABLE KEYS */; UNLOCK TABLES; -- -- Table structure for table `HEALTH_HISTORY` -- DROP TABLE IF EXISTS `HEALTH_HISTORY`; /*!40101 SET @saved_cs_client = @@character_set_client */; /*!40101 SET character_set_client = utf8 */; CREATE TABLE `HEALTH_HISTORY` ( `DIARY_DAY` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, `TOTAL_CALORIES_OUT` double NOT NULL, `TOTAL_MINUTES_RECORDED` double NOT NULL, `STEPS` double DEFAULT NULL, `ELEVATION` double DEFAULT NULL, `AVG_REST_HR` double DEFAULT NULL, `REST_MINUTES` double DEFAULT NULL, `REST_CAL_OUT` double DEFAULT NULL, `AVG_FAT_BURN_HR` double DEFAULT NULL, `FAT_BURN_MINUTES` double DEFAULT NULL, `FAT_BURN_CAL_OUT` double DEFAULT NULL, `AVG_CARDIO_HR` double DEFAULT NULL, `CARDIO_MINUTES` double DEFAULT NULL, `CARDIO_CAL_OUT` double DEFAULT NULL, `AVG_PEAK_HR` double DEFAULT NULL, `PEAK_MINUTES` double DEFAULT NULL, `PEAK_CAL_OUT` double DEFAULT NULL, PRIMARY KEY (`DIARY_DAY`) ) ENGINE=InnoDB DEFAULT CHARSET=latin1; /*!40101 SET character_set_client = @saved_cs_client */; -- -- Table structure for table `NUTRITION_HISTORY` -- DROP TABLE IF EXISTS `NUTRITION_HISTORY`; /*!40101 SET @saved_cs_client = @@character_set_client */; /*!40101 SET character_set_client = utf8 */; CREATE TABLE `NUTRITION_HISTORY` ( `DIARY_DAY` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, `MEAL` varchar(50) NOT NULL, `TOTAL_CALORIES_IN` double NOT NULL, `CARB` double NOT NULL, `PROT` double NOT NULL, `FAT` double NOT NULL, PRIMARY KEY (`DIARY_DAY`,`MEAL`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8; /*!40101 SET character_set_client = @saved_cs_client */; -- -- Table structure for table `SLEEP_HISTORY` -- DROP TABLE IF EXISTS `SLEEP_HISTORY`; /*!40101 SET @saved_cs_client = @@character_set_client */; /*!40101 SET character_set_client = utf8 */; CREATE TABLE `SLEEP_HISTORY` ( `DIARY_DAY` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, `TOTAL_MINUTES_ASLEEP` double NOT NULL, `TOTAL_MINUTES_IN_BED` double NOT NULL, `REM_MINUTES` double NOT NULL, `LIGHT_MINUTES` double NOT NULL, `DEEP_MINUTES` double NOT NULL, `WAKE_MINUTES` double NOT NULL, PRIMARY KEY (`DIARY_DAY`) ) ENGINE=InnoDB DEFAULT CHARSET=latin1; /*!40101 SET character_set_client = @saved_cs_client */; /*!40103 SET TIME_ZONE=@OLD_TIME_ZONE */; /*!40101 SET SQL_MODE=@OLD_SQL_MODE */; /*!40014 SET FOREIGN_KEY_CHECKS=@OLD_FOREIGN_KEY_CHECKS */; /*!40014 SET UNIQUE_CHECKS=@OLD_UNIQUE_CHECKS */; /*!40101 SET CHARACTER_SET_CLIENT=@OLD_CHARACTER_SET_CLIENT */; /*!40101 SET CHARACTER_SET_RESULTS=@OLD_CHARACTER_SET_RESULTS */; /*!40101 SET COLLATION_CONNECTION=@OLD_COLLATION_CONNECTION */; /*!40111 SET SQL_NOTES=@OLD_SQL_NOTES */; -- Dump completed on 2018-11-07 0:14:54 You will note a GENERATOR table has been created. This is necessary to create date series queries, as detailed later. Step 3: Update your BMQ every day To get your perceived BMQ, you will have to enter it manually. Before we develop the BMQ app to do that, you can enter it using the following example queries every day: insert into BMQ_HISTORY (BMQ, TYPE, TIME_ENTERED) values ('3', 'morning', '2018-11-06 7:00:00'); insert into BMQ_HISTORY (BMQ, TYPE, TIME_ENTERED) values ('3', 'pre-workout', '2018-11-06 7:50:00'); insert into BMQ_HISTORY (BMQ, TYPE, TIME_ENTERED) values ('4', 'post-workout', '2018-11-06 8:50:00'); insert into BMQ_HISTORY (BMQ, TYPE, TIME_ENTERED) values ('3', 'evening', now()); Section 2: Creating a Nifi flow to consume MyFitnessPal CSV exports The goal is to setup the following flow: Step 1: Survey folder for new CSV Files List Files in MFP Folder: ListFile Processor This processor will monitor a folder for new files, based on timestamp, then maintain the last file read in its state, as seen below after right-clicking on the processor and selecting view state: Here is a list of properties to configure that are not default: Input directory: [YOUR_LOCAL_PATH] File filter: .* Run Schedule: 12 h Get Latest File: FetchFile Processor This processor gets the file from the list generated previously and sends it in a flow file. It is configured using default properties. Step 2: Convert CSV to JSON Update Schema Name: UpdateAttribute Processor This processor updates the schema name attribute (later used by next processors and controller services). HortonworksSchemaRegistry Controller Service This controller service is pointing to your HDF Schema Registry. The only non-default property to configure in the controller service before enabling is: Schema Registry URL: http://[YOUR_SCHEMA_REGISTRY_HOST]:7788/api/v1/ CSVReader Controller Service This controller service is going to read your CSV and use your Hortonworks Schema Registry to parse its schema. The following non-default properties should be configured in the controller service before enabling is: Schema Access Strategy: Use 'Schema Name' Property Schema Registry: HortonworksSchemaRegistry Treat First Line as Header: true Ignore CSV Header Column Names: true JsonRecordSetWriter Controller Service This controller service is going to write your JSON output based on your Hortonworks Schema Registry. The following non-default properties should be configured in the controller service before enabling is: Schema Write Strategy: Set 'schema.name' Attribute Schema Access Strategy: Use 'Schema Name' Property Schema Registry: HortonworksSchemaRegistry Convert CSV to JSON: ConvertRecord Processor This processor does the record conversion. Configure it with your enabled controllers as such: Record Reader: CSVReader Record Writer: JsonRecordSetWriter Step 3: Get fields needed for insert SplitJson Processor This processor splits each meal in your export in a flow file, by using the following non-default property: JsonPath Expression: $.[*] Extract Relevant Attributes: EvaluateJsonPath Processor This processor extracts the relevant elements of your split JSON into attributes. It is configured using the following: Destination: flowfile-attribute carb: $.carbohydrates_g diary_day: $.date fat: $.fat_g meal: $.meal prot: $.protein_g total_calories_in: $.calories Step 4: Insert Data to MySQL Create Upsert Query: ReplaceText Processor This processor creates the query to be executed on your MySQL server to upsert the different lin. Here is the ReplacementValue you should use: INSERT INTO NUTRITION_HISTORY (DIARY_DAY, MEAL, TOTAL_CALORIES_IN, CARB, PROT, FAT) VALUES ('${diary_day}', '${meal}', ${total_calories_in}, ${carb}, ${prot}, ${fat}) ON DUPLICATE KEY UPDATE TOTAL_CALORIES_IN = ${total_calories_in}, CARB = ${carb}, PROT = ${prot}, FAT = ${fat} DBCPConnectionPool Controller Service This controller service is enabling connection to your MySQL server, configured by the following non-default properties: Database Connection URL: jdbc:mysql://[YOUR_MYSQL_SERVER_ADDRESS]:3306/BEAST_MODE_DB?useLegacyDatetimeCode=false&serverTimezone=America/New_York Database Driver Class Name: com.mysql.jdbc.Driver Database Driver Location(s): file:///home/nifi/mysql_jdbc_drivers/mysql-connector-java-8.0.11.jar (or wherever you put your mysql jdbc driver) Upsert into MySQL: PutSQL Processor This processor executes the query generated in the ReplaceText processor, and relies on your DBCPConnectionPool controller service, configured as such: JDBC Connection Pool: DBCPConnectionPool Section 3: Creating a Nifi flow to consume Fitbit health & sleep data The goal is to setup the following flow (separated in two screenshot, so you know it's going to be fun). It assumes that you registered an application with the Fitbit API and have a refresh/bearer token ready with the right permissions (see details over at Fitbit documentation). Step 1: Get Delta parameters Get Latest Date: ExecuteSQL Processor This processor runs a query to get a series of dates not yet in the BMQ DB in order to run the Fitbit API. It is relying on the configuration of DBCPConnectionPool detailed above (configured in the property Database Connection Pooling Service, and a 12 h Run Schedule). Here is the query it should run: select list_date from (select date_format( adddate((select IFNULL(max(hist.diary_day),'2018-11-07') from HEALTH_HISTORY hist), @num:=@num+1), '%Y-%m-%d' ) list_date from GENERATOR, (select @num:=-1) num limit 150) dates where list_date < CURRENT_DATE() I realize this is not the only method to generate such a list (using GenerateTableFetch, etc.). This makes sure that I only run what the DB needs if something else changes the DB. ConvertAvroToJSON Processor This processor converts the Avro response of the previous processor. Make sure you configure the following property to avoid splitting error when only one record is returned. Wrap Single Record: true SplitJson Processor This processor splits each record in the query in a flow file, by using the following non-default property: JsonPath Expression: $.* Extract Relevant Attributes: EvaluateJsonPath Processor This processor extracts the relevant elements of your split JSON into attributes. It is configured using the following: Destination: flowfile-attribute DIARY_DAY: $.list_date Step 2: Handle Token Refresh DistributedMapCacheServer Controller Service This controller service is the server that will run the distributed cache. I should be configured using all default properties before being enabled. DistributedMapCacheClientService Controller Service This controller service is connecting to the DistributedMapCacheServer configured previously. Before enabling it, just configure the following: Server Hostname: [YOUR_HOSTNAME] Get Token From Cache: FetchDistributedMapCache Processor This processor will try and fetch a refresh token from the DistributedMapCacheServer. If not found, we will send it to an UpdateAttribute to set a default value (e.g. for the first run), otherwise we will send it to the InvokeHTTP to get a new token. Here are the non-default properties to be configured: Cache Entry Identifier: refresh-token-fitbit Distributed Cache Service: DistributedMapCacheClientService Put Cache Value In Attribute: CurrentRefreshToken A few notes: This configuration requests a refresh token every run, which works with my 12 h run schedule. Ideally, you would store the expiration in distributed cache as well and only request a refresh token then, but I didn't want to overcomplicate things. Moreover, in a production environment, I would probably recommend to persist these token on disk and not only in memory in case of failure. Use Default Value if Not Found: UpdateAttribute Processor This processor updates the CurrentRefreshToken if FetchDistributedMapCache returns not-found. Configure it using this property: CurrentRefreshToken: [A_VALID_REFRESH_TOKEN] Refresh Fitbit Token: InvokeHTTP Processor This calls the Fitbit API to get a new valid token and a refresh token. Configure the following non-default properties to run it: HTTP Method: POST Remote URL: https://api.fitbit.com/oauth2/token?grant_type=refresh_token&refresh_token=${CurrentRefreshToken} Content-Type: application/x-www-form-urlencoded Authorization: Basic [YOUR_AUTHORIZATION_CODE] (see Fibit Documentation) Extract Token: EvaluateJsonPath Processor This processor extracts the bearer and refresh tokens from the HTTP response. It is configured using the following: Destination: flowfile-attribute carb: $.access_token diary_day: $.refresh_token Extract Token: EvaluateJsonPath Processor This processor extracts the bearer and refresh tokens from the HTTP response, then continues to two routes: one storing back the refresh token in cache, the other one using the bearer token to call the Fitbit API. It is configured using the following: Destination: flowfile-attribute carb: $.access_token diary_day: $.refresh_token Put Refresh Token in Flowfile: ReplaceText Processor This processor puts the refresh token in flowfile in order to be consumed: ${REFRESH_TOKEN} PutDistributedMapCache Processor This processor takes the refresh token in the flowfile and stores it under refresh-token-fitbit for next time the flow is executed: Cache Entry Identifier: refresh-token-fitbit Distributed Cache Service: DistributedMapCacheClientService Step 3: Call Fitbit APIs (Health and Sleep) For this step, I will only detail one processor. First, because, as you can see on the screenshot above, both Get FitBit Daily Summary are very similar (they are just calling diffrent endpoints of the fitbit API). Secondly, because the flow Extract Relevant Attributes > Create SQL Query > Upsert to MySQL has been described above. Finally, because I will keep the Calculate Averages and Sums for the next flow I run in Strava. All detailed processors can be found here: feed-fitbit-anonymized.xml Get FitBit Daily Summary: InvokeHTTP Processor This calls the Fitbit API to get a daily health summary: HTTP Method: GET Remote URL: https://api.fitbit.com/1/user/-/activities/date/${DIARY_DAY}.json Authorization: Bearer ${BEARER_TOKEN} Section 4: Creating a Nifi flow to consume Strava activity data As for the previous section, the goal is to setup a flow calling Strava's activity API. It assumes that you registered an application with the Strava API and have a refresh/bearer token ready with the right permissions (see details over at Strava documentation). As opposed with Fitbit the refresh token remains the same, so no need for distributed cache. I'm only going to detail the Convert Units processor to talk about Nifi language. The rest of the flow can be found here: feed-strava-anonymized.xml Convert Units: UpdateAttribute Processor This processor uses some of the data extracted to convert units before putting them into attributes later on used in your upsert query. Here are the calculation executed: AVG_HR: ${AVG_HR:isEmpty():ifElse(0.0, ${AVG_HR})} (makes sure that we have a value if the API does not return HR, in the case of HR not present in activity) AVG_PACE: ${MILE_IN_METERS:divide(${AVG_PACE:toDecimal():multiply(60.0)})} (converts speed from m/s to min/mile) DISTANCE: ${DISTANCE:divide(${MILE_IN_METERS})} (converts distance from meters to miles) DURATION: ${DURATION:toDecimal():divide(60.0)} (converts duration from seconds to mins) START_TIME: ${START_TIME:replaceAll("T"," "):replaceAll("Z","")} (converts timestamp format to MySQL format) ... View more

Introduction Knowing how and when to leverage cloud infrastructure and when to use on premises equipment is at the heart of today's mature enterprise data strategies. I decided to dive deeper into the subject, and, like I did with the Personality Detection data science and engineering platform I built, I wanted to use an example to illustrate how to approach how to architect a hybrid cloud infrastructure. These types of endeavors can be quite theoretical (read hard to follow and potentially boring), so I decided to attach this hybrid cloud platform demonstration to something very concrete: my own health. I signed up for a marathon that will take place March 29, and I am starting to train November 5th. I plan to use my training, sleep, nutrition and general health data to evaluate and eventually predict my perceived level of fatigue that I coined under the term: "Beast Mode Quotient" or BMQ. This article is an introduction to the architecture and data flows I will put in place. It will refer to sub articles that will be tutorials that anyone can follow to implement their hybrid cloud strategies. Architecture overview The figure below gives a highlight of my hybrid cloud platform: As you can see, it is comprised of the following elements: A Data Fabric Layer (Data Plane Services) hosting the BMQ dashboard app, as well as Cloubreak in order to deploy ephemeral clusters One permanent cluster, hosting data ingestion from various health sources (e.g. Strava, MyFitnessPal, Fitbit). This ingestion will also execute the ML pipeline to predict level of fatigue Any number of ephemeral clusters executing the training and generation of the ML pipeline Enable Analytics & model training on the data stored in MySQL using Zeppelin notebooks & Spark, that would then feed back the BMQ application Enable custom application to consumer the data extracted and analyzed Note: I realize that this architecture is showcasing a near-future state (Cloudbreak is not technically part of Data Plane Service yet). But this series of article is a long term project, so the architecture is set a few weeks into the future. Data Flows The platform will interact in two major ways, detailed in the flows below. Data Flow 1: Refreshing Data Data Flow 2: Generating ML Pipeline using ephemeral clusters Metrics analyzed and predicted From a high vantage point, the platform will collect, analyze and predict the following pieces of data. My goal is to log all these pieces of data trough various health and fitness services, create a BMQ predictive model that I will then apply to my upcoming training plan. Activity Data Distance Duration Avg. Heart Rate Elevation Avg. Pace Nutrition Data Calories in % Carbs % Proteins % Fats Sleep Data Total sleep % REM % Light sleep % Deep sleep % Awake Health Data Total Cal. Burned Avg. Heart Rate Steps Elevation Weight Perceived Fatigue BMQ: Morning BMQ: Pre-Workout BMQ: Post-Workout BMQ: Evening Implementation Tutorials The implementation of this platform will be detailed in the upcoming following tutorial articles: Part 1: Create Nifi Flows to ingest API and flat files and load them onto MySQL Part 2: Create Cloudbreak blueprints to deploy data science ready ephemeral clusters Part 3: Use Spark and Zeppelin to train fatigue prediction models Part 4: Create an end-to-end application automating BMQ calculation and prediction ... View more

Overview Last article of the series, and a fun one. As usual, this article is the continuation of part 1, part 2 and part 3, so make sure you read those before starting. The completion of this tutorial will finalize your end to end architecture, as depicted here: [ADD IMAGE] This tutorial will be divided in 2 sections: Section 1: Create a Web Service with Nifi Section 2: Create a consumer application with code Igniter Note: Section 2 will not be detailed. First, because web application development is not the main goal of Hortonworks Community Connection. Second, because I am pretty sure I did a terrible job at it considering it is not my domain of expertise, and I shouldn't spread mediocrity to the world! Section 1: Create a Web Service with Nifi For your entire project to work, you will have to create more than one web service although this section only goes through the creation of one (see the flow overview below). To get all the flows required for your app to work, check github here. [ADD FLOW IMAGE] Step 1: Create a request handler StandardHttpContextMap This can't be more straight forward. Create a StandardHttpContextMap controller service with default options and enable it. [ADD CONTROLLER IMAGE] HandleHttpRequest: Receive request and data Create a processor simple HandleHttpRequest processor. [ADD PROCESSOR IMAGE] Here the properties you need to change (all other are standard): HandleHttpRequest 8012 HTTP Context Map StandardHttpContextMap (the controller you just created) Allowed Paths /dashboard Step 2: Retrieve data from Hbase Execute SQL Create 4 execute SQLs that will run queries retrieving all the data you need. They all rely on a DBCPConnectionPool controller for Phoenix. To understand how to create that, please refer to part 3. Each and every Execute SQL will have a select count in them. I'm not going to list them here, but I included a template for all web services to this article ... View more

Overview Last article of the series, and a fun one. As usual, this article is the continuation of part 1, part 2 and part 3, so make sure you read those before starting. The completion of this tutorial will finalize your end to end architecture, as depicted here: This tutorial will be divided in 2 sections: Section 1: Create a Web Service with Nifi Section 2: Create a consumer application with code Igniter Note: Section 2 will not be detailed. First, because web application development is not the main goal of Hortonworks Community Connection. Second, because I am pretty sure I did a terrible job at it considering it is not my domain of expertise, and I shouldn't spread mediocrity to the world! Section 1: Create a Web Service with Nifi For your entire project to work, you will have to create more than one web service although this section only goes through the creation of one (see the flow overview below). To get all the flows required for your app to work, check github here. Step 1: Create a request handler StandardHttpContextMap This can't be more straight forward. Create a StandardHttpContextMap controller service with default options and enable it. HandleHttpRequest: Receive request and data Create a processor simple HandleHttpRequest processor. Here the properties you need to change (all other are standard): HandleHttpRequest 8012 HTTP Context Map StandardHttpContextMap (the controller you just created) Allowed Paths /dashboard Step 2: Retrieve data from Hbase Execute SQL Create 4 execute SQLs that will run queries retrieving all the data you need. They all rely on a DBCPConnectionPool controller for Phoenix. To understand how to create that, please refer to part 3. Each and every Execute SQL will have a select count in them. I'm not going to list them here, but I included a article for all web services to this article. Convert Avro to JSON Yet again a very straight forward configuration. For each ExecuteSQL you created, create a ConvertAvroToJSON processor with standard configuration: Step 3: Merge and prepare Response Merge Content Merge all outputs from the Convert Avro to JSON processors using a standard MergeContent processor: Replace Text Use a ReplaceText to add brackets before and after the merge content, so that your web app can read it: Step 4: Send response back Handle Http Response Finally, use a standard configuration for a HandleHttpResponse processor to return the web response: Start your flow, then ou should be able to call your service by going to this URL: http://[your_nifi_address]:8012/dashboard The response should be something along these lines: [{"BYLINES": 781},{"TOTAL": 6153},{"TOTAL_VIEWS": 4147},{"NEWS_DESKS": 63}] Section 2: Create a consumer application with code Igniter Once you've created all the web services, you will need to create a web app to consume these services. I created one using code igniter, and you can see all the code on my github. As I mentioned before, I'm not going to go deep in this configuration, but to make this app run, you will need to follow these steps: Step 1: Instantiate a Linux web server with Apache, Maria DB, and PHP 7 Step 2: Create a adminlte database on your server with the appropriate data by running the SQL script below Step 3: Clone my github to the root of your apache server Step 4: Change the config.php base_url to match your server, and the database.php to match your credentials -- MySQL dump 10.14 Distrib 5.5.60-MariaDB, for Linux (x86_64) -- -- Host: localhost Database: adminlte -- ------------------------------------------------------ -- Server version 5.5.60-MariaDB /*!40101 SET @OLD_CHARACTER_SET_CLIENT=@@CHARACTER_SET_CLIENT */; /*!40101 SET @OLD_CHARACTER_SET_RESULTS=@@CHARACTER_SET_RESULTS */; /*!40101 SET @OLD_COLLATION_CONNECTION=@@COLLATION_CONNECTION */; /*!40101 SET NAMES utf8 */; /*!40103 SET @OLD_TIME_ZONE=@@TIME_ZONE */; /*!40103 SET TIME_ZONE='+00:00' */; /*!40014 SET @OLD_UNIQUE_CHECKS=@@UNIQUE_CHECKS, UNIQUE_CHECKS=0 */; /*!40014 SET @OLD_FOREIGN_KEY_CHECKS=@@FOREIGN_KEY_CHECKS, FOREIGN_KEY_CHECKS=0 */; /*!40101 SET @OLD_SQL_MODE=@@SQL_MODE, SQL_MODE='NO_AUTO_VALUE_ON_ZERO' */; /*!40111 SET @OLD_SQL_NOTES=@@SQL_NOTES, SQL_NOTES=0 */; -- -- Database creation -- CREATE DATABASE adminlte; use adminlte; -- -- Table structure for table `activity_logs` -- DROP TABLE IF EXISTS `activity_logs`; /*!40101 SET @saved_cs_client = @@character_set_client */; /*!40101 SET character_set_client = utf8 */; CREATE TABLE `activity_logs` ( `id` int(11) NOT NULL AUTO_INCREMENT, `title` text NOT NULL, `user` text NOT NULL, `ip_address` text NOT NULL, `created_at` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP, `updated_at` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00', PRIMARY KEY (`id`) ) ENGINE=InnoDB AUTO_INCREMENT=18 DEFAULT CHARSET=latin1; /*!40101 SET character_set_client = @saved_cs_client */; -- -- Dumping data for table `activity_logs` -- LOCK TABLES `activity_logs` WRITE; /*!40000 ALTER TABLE `activity_logs` DISABLE KEYS */; INSERT INTO `activity_logs` VALUES (1,'Administrator Logged in','1','10.42.80.145','2018-10-11 18:34:37','0000-00-00 00:00:00'),(2,'User: Administrator Logged Out','1','10.42.80.145','2018-10-11 19:44:48','0000-00-00 00:00:00'),(3,'Administrator Logged in','1','10.42.80.145','2018-10-11 19:44:54','0000-00-00 00:00:00'),(4,'User: Administrator Logged Out','1','10.42.80.145','2018-10-11 20:52:02','0000-00-00 00:00:00'),(5,'Administrator Logged in','1','10.42.80.145','2018-10-11 20:52:06','0000-00-00 00:00:00'),(6,'User: Administrator Logged Out','1','10.42.80.145','2018-10-11 20:56:16','0000-00-00 00:00:00'),(7,'Administrator Logged in','1','10.42.80.145','2018-10-11 20:56:24','0000-00-00 00:00:00'),(8,'User: Administrator Logged Out','1','10.42.80.145','2018-10-11 20:59:01','0000-00-00 00:00:00'),(9,'Administrator Logged in','1','10.42.80.145','2018-10-11 21:02:36','0000-00-00 00:00:00'),(10,'User: Administrator Logged Out','1','10.42.80.145','2018-10-11 21:06:35','0000-00-00 00:00:00'),(11,'Administrator Logged in','1','10.42.80.145','2018-10-11 21:12:24','0000-00-00 00:00:00'),(12,'Administrator Logged in','1','10.42.80.172','2018-10-12 15:52:55','0000-00-00 00:00:00'),(13,'Administrator Logged in','1','10.42.80.85','2018-10-13 11:45:37','0000-00-00 00:00:00'),(14,'Administrator Logged in','1','10.42.80.153','2018-10-13 12:27:08','0000-00-00 00:00:00'),(15,'User: Administrator Logged Out','1','10.42.80.153','2018-10-13 12:27:14','0000-00-00 00:00:00'),(16,'Administrator Logged in','1','10.42.80.153','2018-10-13 12:27:40','0000-00-00 00:00:00'),(17,'Administrator Logged in','1','10.42.80.204','2018-10-15 16:21:06','0000-00-00 00:00:00'); /*!40000 ALTER TABLE `activity_logs` ENABLE KEYS */; UNLOCK TABLES; -- -- Table structure for table `permissions` -- DROP TABLE IF EXISTS `permissions`; /*!40101 SET @saved_cs_client = @@character_set_client */; /*!40101 SET character_set_client = utf8 */; CREATE TABLE `permissions` ( `id` int(11) NOT NULL AUTO_INCREMENT, `title` text NOT NULL, `code` text, PRIMARY KEY (`id`) ) ENGINE=InnoDB AUTO_INCREMENT=16 DEFAULT CHARSET=latin1; /*!40101 SET character_set_client = @saved_cs_client */; -- -- Dumping data for table `permissions` -- LOCK TABLES `permissions` WRITE; /*!40000 ALTER TABLE `permissions` DISABLE KEYS */; INSERT INTO `permissions` VALUES (1,'Users List','users_list'),(2,'Add Users','users_add'),(3,'Edit Users','users_edit'),(4,'Delete Users','users_delete'),(5,'Users View','users_view'),(6,'Activity Logs List','activity_log_list'),(7,'Acivity Log View','activity_log_view'),(8,'Roles List','roles_list'),(9,'Add Roles','roles_add'),(10,'Edit Roles','roles_edit'),(11,'Permissions List','permissions_list'),(12,'Add Permissions','permissions_add'),(13,'Permissions Edit','permissions_edit'),(14,'Delete Permissions','permissions_delete'),(15,'Company Settings','company_settings'); /*!40000 ALTER TABLE `permissions` ENABLE KEYS */; UNLOCK TABLES; -- -- Table structure for table `role_permissions` -- DROP TABLE IF EXISTS `role_permissions`; /*!40101 SET @saved_cs_client = @@character_set_client */; /*!40101 SET character_set_client = utf8 */; CREATE TABLE `role_permissions` ( `id` int(11) NOT NULL AUTO_INCREMENT, `role` int(11) NOT NULL, `permission` text NOT NULL, PRIMARY KEY (`id`) ) ENGINE=InnoDB AUTO_INCREMENT=16 DEFAULT CHARSET=latin1; /*!40101 SET character_set_client = @saved_cs_client */; -- -- Dumping data for table `role_permissions` -- LOCK TABLES `role_permissions` WRITE; /*!40000 ALTER TABLE `role_permissions` DISABLE KEYS */; INSERT INTO `role_permissions` VALUES (1,1,'users_list'),(2,1,'users_add'),(3,1,'users_edit'),(4,1,'users_delete'),(5,1,'users_view'),(6,1,'activity_log_list'),(7,1,'activity_log_view'),(8,1,'roles_list'),(9,1,'roles_add'),(10,1,'roles_edit'),(11,1,'permissions_list'),(12,1,'permissions_add'),(13,1,'permissions_edit'),(14,1,'permissions_delete'),(15,1,'company_settings'); /*!40000 ALTER TABLE `role_permissions` ENABLE KEYS */; UNLOCK TABLES; -- -- Table structure for table `roles` -- DROP TABLE IF EXISTS `roles`; /*!40101 SET @saved_cs_client = @@character_set_client */; /*!40101 SET character_set_client = utf8 */; CREATE TABLE `roles` ( `id` int(11) NOT NULL AUTO_INCREMENT, `title` text NOT NULL, PRIMARY KEY (`id`) ) ENGINE=InnoDB AUTO_INCREMENT=2 DEFAULT CHARSET=latin1; /*!40101 SET character_set_client = @saved_cs_client */; -- -- Dumping data for table `roles` -- LOCK TABLES `roles` WRITE; /*!40000 ALTER TABLE `roles` DISABLE KEYS */; INSERT INTO `roles` VALUES (1,'Admin'); /*!40000 ALTER TABLE `roles` ENABLE KEYS */; UNLOCK TABLES; -- -- Table structure for table `settings` -- DROP TABLE IF EXISTS `settings`; /*!40101 SET @saved_cs_client = @@character_set_client */; /*!40101 SET character_set_client = utf8 */; CREATE TABLE `settings` ( `id` int(11) NOT NULL AUTO_INCREMENT, `key` text NOT NULL, `value` text NOT NULL, `created_at` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP, PRIMARY KEY (`id`) ) ENGINE=InnoDB AUTO_INCREMENT=4 DEFAULT CHARSET=latin1; /*!40101 SET character_set_client = @saved_cs_client */; -- -- Dumping data for table `settings` -- LOCK TABLES `settings` WRITE; /*!40000 ALTER TABLE `settings` DISABLE KEYS */; INSERT INTO `settings` VALUES (1,'company_name','Company Name','2018-06-21 17:37:59'),(2,'company_email','testcompany@gmail.com','2018-07-11 11:09:58'),(3,'timezone','Asia/Kolkata','2018-07-15 19:54:17'); /*!40000 ALTER TABLE `settings` ENABLE KEYS */; UNLOCK TABLES; -- -- Table structure for table `users` -- DROP TABLE IF EXISTS `users`; /*!40101 SET @saved_cs_client = @@character_set_client */; /*!40101 SET character_set_client = utf8 */; CREATE TABLE `users` ( `id` int(11) NOT NULL AUTO_INCREMENT, `name` text NOT NULL, `username` text NOT NULL, `email` text NOT NULL, `password` text NOT NULL, `phone` text NOT NULL, `address` longtext NOT NULL, `last_login` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00' ON UPDATE CURRENT_TIMESTAMP, `role` int(11) NOT NULL, `reset_token` text NOT NULL, `created_at` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00', `updated_at` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00', PRIMARY KEY (`id`) ) ENGINE=InnoDB AUTO_INCREMENT=2 DEFAULT CHARSET=latin1; /*!40101 SET character_set_client = @saved_cs_client */; -- -- Dumping data for table `users` -- LOCK TABLES `users` WRITE; /*!40000 ALTER TABLE `users` DISABLE KEYS */; INSERT INTO `users` VALUES (1,'Administrator','admin','admin@gmail.com','21232f297a57a5a743894a0e4a801fc3','','','2018-10-15 21:10:51',1,'','2018-06-27 18:30:16','0000-00-00 00:00:00'); /*!40000 ALTER TABLE `users` ENABLE KEYS */; UNLOCK TABLES; /*!40103 SET TIME_ZONE=@OLD_TIME_ZONE */; /*!40101 SET SQL_MODE=@OLD_SQL_MODE */; /*!40014 SET FOREIGN_KEY_CHECKS=@OLD_FOREIGN_KEY_CHECKS */; /*!40014 SET UNIQUE_CHECKS=@OLD_UNIQUE_CHECKS */; /*!40101 SET CHARACTER_SET_CLIENT=@OLD_CHARACTER_SET_CLIENT */; /*!40101 SET CHARACTER_SET_RESULTS=@OLD_CHARACTER_SET_RESULTS */; /*!40101 SET COLLATION_CONNECTION=@OLD_COLLATION_CONNECTION */; /*!40111 SET SQL_NOTES=@OLD_SQL_NOTES */; After you're done with that and have restarted your web server, go to: http://[YOUR_SERVER]/index.php You can now connect using admin/admin. You should see screens like these ones: ... View more

Overview Time to continue my series on news author personality detection! This tutorial will look at data science, Zeppelin notebooks with the idea of predicting the popularity of an article based on the personality of an author. This tutorial does assume that you completed part 1 and part 2 of the series. After completing this part, your architecture will look like this: This tutorial will be divided in sections: Section 1: Update Nifi to send article data to Phoenix Section 2: Read Phoenix data with Spark in Zeppelin Note Section 3: Train Linear Regression Model Section 4: Make and store article popularity predictions Note: This exercise is a very simplified way to apply machine learning. I spent very little time selecting the appropriate algorithm, accumulating enough data or even challenging whether or not the personality of an author is a viable predictor for personality. This is a full time job, it's called being a data scientist! But hopefully this will enable the real data scientist to please stand up and build useful stuff using the Hortonworks stack. Section 1: Update Nifi to send article data to Phoenix This section's goal is to first enhance our previous Nifi flow to upsert data into HBase using Phoenix, as well as add a new flow collecting data from the New York Times "most popular" API. You can find all these flows under my github here. I used Nifi registry to push directly to github; if you want to learn how to do that, check out this article: How to configure git for Nifi Registry in HDF 3.2. Step 1: Create the appropriate tables using Phoenix Connect to your server and run phoenix-sqlline (with the appropriate user): $ ssh [YOUR_MACHINE] $ sudo su hbase $ phoenix-sqlline Create three tables: One to store author personality evaluation per article One to store article popularity One to store popularity prediction based on author personality Here is the SQL create code: CREATE TABLE article_evaluation ( web_url VARCHAR NOT NULL PRIMARY KEY, snippet VARCHAR, byline VARCHAR, pub_date TIMESTAMP, headline VARCHAR, document_type VARCHAR, news_desk VARCHAR, filename VARCHAR, last_updated VARCHAR, extraversion DOUBLE, emotional_stability DOUBLE, agreeableness DOUBLE, conscientiousness DOUBLE, openness_to_experience DOUBLE ); CREATE TABLE article_stats ( web_url VARCHAR NOT NULL PRIMARY KEY, views TINYINT, last_updated VARCHAR ); CREATE TABLE article_popularity_prediction ( web_url VARCHAR NOT NULL PRIMARY KEY, extraversion DOUBLE, emotional_stability DOUBLE, agreeableness DOUBLE, conscientiousness DOUBLE, openness_to_experience DOUBLE, prediction DOUBLE ); Step 2: Update your existing flow to upsert to Phoenix Our goal is to add the following processors to our previous flow (established in part 1 of this series): Update Attribute: Escape characters Create a processor and configure it to create/update 3 attributes: filename: filename used to store the article text into a hdfs file (optional and not part of this tutorial, but the field is needed for the upsert) headline: article headline snippet: article snippet Here is the code for each attribute: filename ${web_url:substring(${web_url:lastIndexOf('/'):plus(1)},${web_url:length()})} headline ${headline:replaceAll('\'',''):replaceAll('\"',''):replaceAll(';','')} snippet ${snippet:replaceAll('\'',''):replaceAll('\"',''):replaceAll(';','')} Replace Text: Create Upsert query This one is straight forward, as depicted below: Your replacement value should be: upsert into article_evaluation values('${web_url}','${snippet}','${byline}','${pub_date}','${headline}','${document_type}','${news_desk}','${filename}','${now()}',${mairessepersonalityrecognition.extraversion},${mairessepersonalityrecognition.emotional_stability},${mairessepersonalityrecognition.agreeableness},${mairessepersonalityrecognition.conscientiousness},${mairessepersonalityrecognition.openness_to_experience}) PutSQL: Run Query This processor is straight forward too, but will require to first create a DBCPConnectionPool controller service: Assuming the nifi node has a phoenix client, here is the parameters you will need to setup: Database Connection URL jdbc:phoenix:localhost:2181:/hbase-unsecure Database Driver Class Name org.apache.phoenix.jdbc.PhoenixDriver Database Driver Location(s) file:///usr/hdp/current/phoenix-client/phoenix-client.jar Finally, configure a PutSQL as depicted below. You will note that I turned off the support for fragmented transactions. This is not important here but will in the next flow where we split the output of the API. Step 3: Create a new flow for article popularity The flow itself is pretty self explanatory, as shown below: Invoke HTTP: Call Most Popular NYT API Remote URL should be (to get an API key, see part 1): http://api.nytimes.com/svc/mostpopular/v2/mostviewed/all-sections/1.json?api-key=[YOUR_API_KEY] SplitJson: Split Response into 10 flow files JsonPath Expression should be: $.results[*] EvaluateJsonPath: Extract Relevant Attributes Extract url from JSON: $.url Extract views from JSON: $.views ReplaceText: Create Phoenix Query Replacement Value: upsert into article_stats values('${url}',${views},'${now()}') PutSQL: Run Phoenix Upsert SQL This is the exact same processor as in your other flow :) Section 2: Read Phoenix data with Spark in Zeppelin Note Step 1: Download the latest phoenix Spark 2 Jar Go to the maven repository and get the latest jar for phoenix and spark2 (see repository here). Once downloaded add it to the spark2 libraries, under the following folder: /usr/hdp/current/spark2-client/jars. In this directory copy the phoenix client if it's not present yet from /usr/hdp/current/phoenix-client/. $ cd /usr/hdp/current/spark2-client/jars $ wget http://repo.hortonworks.com/content/repositories/releases/org/apache/phoenix/phoenix-spark2/4.7.0.2.6.5.3000-28/phoenix-spark2-4.7.0.2.6.5.3000-28.jar $ cp /usr/hdp/current/phoenix-client/phoenix-5.0.0.3.0.1.0-187-client.jar . $ ls -ltra phoenix* -rw-r--r--. 1 root root 222910655 Oct 2 17:35 phoenix-5.0.0.3.0.1.0-187-client.jar -rwxr-xr-x. 1 root root 83908 Oct 2 17:35 phoenix-spark2-4.7.0.2.6.5.3000-28.jar Restart spark from Ambari. Step 2: Create a new note to retrieve data from Phoenix Open Zeppelin and create a new note (I named mine Linear Regression). Using pyspark, retrieve data from the article_evaluation and article_stats tables into a dataframe, and join them based on the web_url: %pyspark from pyspark.sql.functions import lit article_evaluation_table = sqlContext.read .format("org.apache.phoenix.spark") .option("table", "article_evaluation") .option("zkUrl", "localhost:2181") .load() article_stats_table = sqlContext.read .format("org.apache.phoenix.spark") .option("table", "article_stats") .option("zkUrl", "localhost:2181") .load() aet = article_evaluation_table.alias('aet') ast = article_stats_table.alias('ast') inner_join = aet.join(ast, aet.WEB_URL == ast.WEB_URL) article_df = inner_join.select('EXTRAVERSION','OPENNESS_TO_EXPERIENCE','CONSCIENTIOUSNESS','AGREEABLENESS','EMOTIONAL_STABILITY','VIEWS') no_views_df = article_evaluation_table.select('WEB_URL','EXTRAVERSION','OPENNESS_TO_EXPERIENCE','CONSCIENTIOUSNESS','AGREEABLENESS','EMOTIONAL_STABILITY').withColumn('VIEWS',lit(0)) Section 3: Train Linear Regression Model In the same note, create a paragraph that creates a vectorized dataframe for your model to use: %pyspark from pyspark.ml.feature import VectorAssembler vectorAssembler = VectorAssembler(inputCols = ['EXTRAVERSION', 'OPENNESS_TO_EXPERIENCE', 'CONSCIENTIOUSNESS', 'AGREEABLENESS', 'EMOTIONAL_STABILITY'], outputCol = 'features') varticle_df = vectorAssembler.transform(article_df) varticle_df = vtrainingDF.select(['features', 'VIEWS']) vno_views_df = vectorAssembler.transform(no_views_df) train_df = varticle_df Use the dataframe created to train a linear regression model: %pyspark from pyspark.ml.regression import LinearRegression lr = LinearRegression(featuresCol = 'features', labelCol='VIEWS', maxIter=10, regParam=0.3, elasticNetParam=0.8) lr_model = lr.fit(train_df) print("Coefficients: " + str(lr_model.coefficients)) print("Intercept: " + str(lr_model.intercept)) Check the summary of your model: %pyspark trainingSummary = lr_model.summary print("RMSE: %f" % trainingSummary.rootMeanSquaredError) print("r2: %f" % trainingSummary.r2) train_df.describe().show() Section 4: Make and store article popularity predictions Run Your model on all articles in the article evaluation table: %pyspark predictions = lr_model.transform(vno_views_df) df_to_write = predictions.select("WEB_URL","EXTRAVERSION","OPENNESS_TO_EXPERIENCE","CONSCIENTIOUSNESS","AGREEABLENESS","EMOTIONAL_STABILITY","prediction") Upload these results to the article_popularity_prediction table using phoenix %pyspark df_to_write.write .format("org.apache.phoenix.spark") .mode("overwrite") .option("table", "article_popularity_prediction") .option("zkUrl", "localhost:2181") .save() Finally, if you want to check your predictions versus the actual view, create a jdbc(phoenix) paragraph, as follows: %jdbc(phoenix) select article_popularity_prediction.web_url as URL, article_popularity_prediction. prediction as predicted_views, article_stats.views from article_popularity_prediction, article_stats where article_popularity_prediction.web_url = article_stats.web_url As you can see the model is far from perfect. I would also argue that creating a PMML from the model would have been a better way to go for our architecture (calling the PMML from Nifi as data flows), instead of loading results offline like we do here. Again, this is a full time job 🙂 See you next article, where we will create a dashboard to leverage this data science and data engineering we just created! Note: for reference, I included my Zeppelin note here: linear-regression-zeppelin-note.json ... View more

Overview As you may or may not know, Nifi-Registry 0.2.0 (included in HDF 3.2) allows the storage of versioned flows in a git repository. @Andrew Lim wrote a great article on how to configure this storage for nifi-registry as you can see here: https://community.hortonworks.com/articles/210286/storing-apache-nifi-versioned-flows-in-a-git-repos.html I thought I'd give a few pointers on how to do it in HDF 3.2, following these four simple steps: Step 1: Setup your git repository Step 2: Clone repository on HDF server Step 3: Configure nifi-registry via Ambari Step 4: Restart nifi-registry Step 1: Setup your git repository This step is essentially exactly what was highlighted by Andrew. Create a new repository Create a token Go to to GitHub’s “Developer settings” and create a new “Personal access token”, as follows: Step 2: Clone repository on HDF server Connect to the server running nifi-registry on your hdf cluster. Make sure that git is installed; if it isn't run the following and follow instructions: $ sudo yum install git Go to the nifi_registry_internal_dir configured for your registry (/var/lib/nifi-registry by default) and clone your repo as the nifiregistry user: $ sudo su nifiregistry $ cd /var/lib/nifi-registry $ mkdir git $ cd git $ git clone [address_of_your_repo] Step 3: Configure nifi-registry via Ambari The configuration for git is already in your providers.xml commented out. You will have to swap it for the default one that uses a local folder. In your nifi-registry configs tab, go to Advanced nifi-registry-providers-env and change it as follows: The key changes should look like this: <!-- <flowPersistenceProvider> <class>org.apache.nifi.registry.provider.flow.FileSystemFlowPersistenceProvider</class> <property name="Flow Storage Directory">{{nifi_registry_internal_dir}}/flow_storage</property> </flowPersistenceProvider> --> <flowPersistenceProvider> <class>org.apache.nifi.registry.provider.flow.git.GitFlowPersistenceProvider</class> <property name="Flow Storage Directory">{{nifi_registry_internal_dir}}/git/perso-recognition-nifi-flows</property> <property name="Remote To Push">origin</property> <property name="Remote Access User">paulvid</property> <property name="Remote Access Password">[your_key]</property> </flowPersistenceProvider> Step 4: Restart nifi-registry Using Ambari, restart nifi-registry: Existing buckets will be automatically uploaded when you commit changes, and your repository will look like this: ... View more

Overview Following my series on news author personality detection, let's dive into the streaming analytics side of things, and continue building our data science and engineering platform. This tutorial does assume that you completed part 1 of the series. After completing this part, your architecture will look like this: This tutorial will be divided in sections: Section 1: Create a custom UDF for SAM Section 2: Create a SAM application to sink to Druid Section 3: Monitor your streaming application Section 4: Create a simple superset visualization Note: Creating a custom UDF is a purely academical exercise, as is the use of SAM for sinking into Druid, considering I could have connected Kafka to Druid directly. However, one of the goal of this series is to learn as much as possible about the Hortonworks stack. Therefore, custom UDF and SAM is on the menu. Section 1: Create a custom UDF for SAM Step 1: Download the structure for your custom UDF Using the example give by the SAM documentation, fork/clone this repository: link. Using your favorite IDE, open the cloned folder and navigate to the sam-custom-udf folder: Step 2: Create a new class Under [YOUR_PATH]/sam-custom-extensions/sam-custom-udf/src/main/java/hortonworks/hdf/sam/custom/udf/math add a new class called NormalizeByline (I realize this is not a math function, but again, this is an exercise). This class removes the "By " at the beginning of a by line and capitalizes it. It should contain the following code: package hortonworks.hdf.sam.custom.udf.math; import com.hortonworks.streamline.streams.rule.UDF; import org.apache.commons.lang.WordUtils; public class NormalizeByline implements UDF<String, String> { public String evaluate(String value) { String result = WordUtils.capitalize(value.replaceAll("By ", "")); return result; } } Step 3: Compile your code Go to [YOUR_PATH]/sam-custom-extensions/sam-custom-udf/ and run the following command: $ mvn install You should see something like this: [INFO] Scanning for projects... [WARNING] [WARNING] Some problems were encountered while building the effective model for hortonworks.hdf.sam.custom:sam-custom-udf:jar:3.1.0.0-564 [WARNING] 'version' contains an expression but should be a constant. @ hortonworks.hdf.sam.custom:sam-custom-extensions:${sam.extensions.version}, /Users/pvidal/Documents/sam-custom/sam-custom-extensions/pom.xml, line 6, column 13 [WARNING] 'build.plugins.plugin.version' for org.apache.maven.plugins:maven-compiler-plugin is missing. @ hortonworks.hdf.sam.custom:sam-custom-extensions:${sam.extensions.version}, /Users/pvidal/Documents/sam-custom/sam-custom-extensions/pom.xml, line 80, column 12 [WARNING] 'build.plugins.plugin.version' for org.apache.maven.plugins:maven-jar-plugin is missing. @ hortonworks.hdf.sam.custom:sam-custom-extensions:${sam.extensions.version}, /Users/pvidal/Documents/sam-custom/sam-custom-extensions/pom.xml, line 94, column 21 [WARNING] 'build.plugins.plugin.version' for org.apache.maven.plugins:maven-shade-plugin is missing. @ hortonworks.hdf.sam.custom:sam-custom-extensions:${sam.extensions.version}, /Users/pvidal/Documents/sam-custom/sam-custom-extensions/pom.xml, line 89, column 21 [WARNING] [WARNING] It is highly recommended to fix these problems because they threaten the stability of your build. [WARNING] [WARNING] For this reason, future Maven versions might no longer support building such malformed projects. [WARNING] [INFO] [INFO] -------------< hortonworks.hdf.sam.custom:sam-custom-udf >-------------- [INFO] Building sam-custom-extensions-udfs 3.1.0.0-564 [INFO] --------------------------------[ jar ]--------------------------------- [INFO] [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ sam-custom-udf --- [WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent! [INFO] Copying 3 resources [INFO] [INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ sam-custom-udf --- [INFO] Changes detected - recompiling the module! [WARNING] File encoding has not been set, using platform encoding UTF-8, i.e. build is platform dependent! [INFO] Compiling 4 source files to /Users/pvidal/Documents/sam-custom/sam-custom-extensions/sam-custom-udf/target/classes [INFO] [INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ sam-custom-udf --- [WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent! [INFO] skip non existing resourceDirectory /Users/pvidal/Documents/sam-custom/sam-custom-extensions/sam-custom-udf/src/test/resources [INFO] [INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ sam-custom-udf --- [INFO] No sources to compile [INFO] [INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ sam-custom-udf --- [INFO] No tests to run. [INFO] [INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ sam-custom-udf --- [INFO] Building jar: /Users/pvidal/Documents/sam-custom/sam-custom-extensions/sam-custom-udf/target/sam-custom-udf-3.1.0.0-564.jar [INFO] [INFO] --- maven-install-plugin:2.4:install (default-install) @ sam-custom-udf --- [INFO] Installing /Users/pvidal/Documents/sam-custom/sam-custom-extensions/sam-custom-udf/target/sam-custom-udf-3.1.0.0-564.jar to /Users/pvidal/.m2/repository/hortonworks/hdf/sam/custom/sam-custom-udf/3.1.0.0-564/sam-custom-udf-3.1.0.0-564.jar [INFO] Installing /Users/pvidal/Documents/sam-custom/sam-custom-extensions/sam-custom-udf/pom.xml to /Users/pvidal/.m2/repository/hortonworks/hdf/sam/custom/sam-custom-udf/3.1.0.0-564/sam-custom-udf-3.1.0.0-564.pom [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 1.981 s [INFO] Finished at: 2018-09-04T16:46:04-04:00 [INFO] ------------------------------------------------------------------------ Once compiled, a new jar (sam-custom-udf-3.1.0.0-564.jar) should be created under [YOUR_PATH]/sam-custom-extensions/sam-custom-udf/target Step 4: Add UDF to SAM In your SAM application, go to Configuration > Application Resources UDF. Click on the + sign on the right side of the screen, and enter the following parameters: Don't forget to upload your recently created jar (sam-custom-udf-3.1.0.0-564.jar). Once done, click OK. Section 2: Create a SAM application to sink to Druid Step 1: Configure Service Pool & Environment In your SAM application, go to Configuration > Service Pool. Use the Auto Add feature by specifying your Ambari's API: http://ambari_host:port/api/v1/clusters/CLUSTER_NAME Once Auto Add is complete, you should see something like this: Then, go to Configuration > Environments and create a new environment selecting Kafka, Druid and Storm, as follows: Step 2: Create application Go to My Applications. Use the + sign in the top right of the screen to create a new application using the environment we just configured, as follows: Step 3: Add Kafka Source In the edit screen of your new application, add a Kafka source: Most of the parameters are auto detected from your environment (including kafka topic and schema registry created in previous tutorial), as detailed below for mine: Cluster Name: pvidal_personality_recognition Security Protocol: PLAINTEXT Bootstrap Servers: pvidal-personality-recognition0.field.hortonworks.com:6667 Kafka Topic: personality-recognition Reader Schema Branch: MASTER Reader Schema Version: 1 Consumer Group Id: persorecognition_group (you can put whatever you want here) For this exercise, no need to concern yourself with optional parameters. Step 4: Add Projection In the edit screen of your new application, add a projection to use your custom UDF, and link it to your kafka source. The configuration is very straightforward, as depicted below: Step 5: Add Druid sink In the edit screen of your new application, add a Druid sink, and link it to your projection. You will need to configure two screens for this sink: the required part for Druid dimensions, and the optional part for aggregations, to be used later on in superset. The required part should have the following configuration elements: Cluster Name: pvidal_personality_recognition Name of the indexing service: druid/overlord Service discovery path: /druid/discovery Datasource Name: personality-detection-datasource (you can put whatever you want here) Zookeeper connection string: pvidal-personality-recognition0.field.hortonworks.com:2181,pvidal-personality-recognition1.field.hortonworks.com:2181,pvidal-personality-recognition2.field.hortonworks.com:2181 Dimensions: Everything but the 5 traits of personality for which we will create aggregations Timestamp field name: processingTime Window Period: PT10M Index Retry Period: PT10M Segment Granularity: Minute Query Granularity: Second In the optional section of the configuration, you will leave every parameters as default, but you will have to create, for each personality trait (agreeableness, extraversion, openness_to_experience, conscientiousness, emotional_stability), 3 aggregators: Double Sum Double Max Double Min The figure below details the configuration of the agreeableness Double Sum aggregator: Section 3: Monitor your streaming application After finalizing your application, launch it by clicking on the paper plane icon of SAM. This will deploy the application. Once launched the application can be monitored via the Storm UI. Go to the My Applications screen, click on your application, then click on the storm UI icon on the top right of the screen: The icon opens a new tab to the Storm UI, where you can monitor your app for error, lag, etc. I also recommend to get familiar with navigating indexing and segmentation from the Druid Overlord or Druid console: Section 4: Create a simple superset visualization Step 1: Refresh / Verify Datasource In Superset, click on Refresh Druid Metadata: You should see your datasource listed in the Druid datasources: Click on the edit icon of your datasource and verify your druid columns. It should look something like this: Verify your metrics as well. It should look something like this: Step 2: Create visualization Go back to the druid datasources screen and click on your datasource. You can then create visualizations based on the data collected. Below is a simple example of personality trait, per byline, using the min of each aggregate: Superset of course gives you the opportunity to create a lot better visualizations, including timeseries, if you are not creatively impaired like I am. See you soon for part 3, where we will dive deeper into machine learning and optimize our platform. ... View more

Overview As mentioned in the article parent to this article, the objective of this part 1 is to ingest data from the New York Times API, run HTML scraping, personality detection and expose it to a Kafka topic, as depicted by the architecture below: The end goal is thus to implement a NiFi flow like the following: To implement it, this article will be divided in 4 section: Section 1: Create a NYT API consumer Section 2: Create a custom BoilerPipe Extractor Processor Section 3: Create a custom Mairesse Personality Processor Section 4: Create and configure a Kafka topic Section 1: Create a NYT API consumer Disclaimer: I basically copy-pasted this from my previous article, Determining the big 5 traits of Personality Psychology of news articles using NiFi, Hive & Zeppelin. Step 1: Obtaining an API Key This step is very straight forward. Go to https://developer.nytimes.com/signup and sign-up for a key. Step 2: Configuring InvokeHTTP The InvokeHTTP is used here with all default parameters, except for the URL. Here are some key configuration items and a screenshot of the Processor configuration: HTTP Method: GET Remote URL: http://api.nytimes.com/svc/search/v2/articlesearch.json?fq=source:("The New York Times")&page=0&sort=newest&fl=web_url,snippet,headline,pub_date,document_type,news_desk,byline≈i-key=[YOUR_KEY] (This URL selects article from the New York Times as a source, and only selects some of the fields I am interested in: web_url,snippet,headline,pub_date,document_type,news_desk,byline). Content-Type: ${mime.type} Run Schedule: 5 mins (could be set to a little more, I'm not sure the frequency at which new articles are published) Step 3: Extracting results from Invoke HTTP response The API call parameter page=0 returns results 0-9; for this exercise, I'm only interested in the latest article, so I setup an evaluateJSONpath processor take care of that, as you can see below: Section 2: Create a custom BoilerPipe Extractor Processor The goal of this section is to create a simple processor that extracts article text based on a URL, using the BoilerPipe library. Step 1: Install Maven Having Maven installed is a pre-requisite (so is having Java installed, a computer and the knowledge & will to run this tutorial). There are many ways to install maven, depending on your machine, but for MacOS I recommend using the homebrew package manager and simply run $ brew install maven Step 2: Generate a NiFi processor archetype Once installed, create a new folder, wherever you'd like and create a nifi processor archetype as such: $ mkdir tutorial $ cd tutorial/ $ mvn archetype:generate [INFO] Scanning for projects... [INFO] [INFO] ------------------< org.apache.maven:standalone-pom >------------------- [INFO] Building Maven Stub Project (No POM) 1 [INFO] --------------------------------[ pom ]--------------------------------- [INFO] [INFO] >>> maven-archetype-plugin:3.0.1:generate (default-cli) > generate-sources @ standalone-pom >>> [INFO] [INFO] <<< maven-archetype-plugin:3.0.1:generate (default-cli) < generate-sources @ standalone-pom <<< [INFO] [INFO] [INFO] --- maven-archetype-plugin:3.0.1:generate (default-cli) @ standalone-pom --- [INFO] Generating project in Interactive mode [INFO] No archetype defined. Using maven-archetype-quickstart (org.apache.maven.archetypes:maven-archetype-quickstart:1.0) Choose archetype: 1: remote -> am.ik.archetype:maven-reactjs-blank-archetype (Blank Project for React.js) 2: remote -> am.ik.archetype:msgpack-rpc-jersey-blank-archetype (Blank Project for Spring Boot + Jersey) 3: remote -> am.ik.archetype:mvc-1.0-blank-archetype (MVC 1.0 Blank Project) 4: remote -> am.ik.archetype:spring-boot-blank-archetype (Blank Project for Spring Boot) [...] 2214: remote -> xyz.luan.generator:xyz-generator (-) Choose a number or apply filter (format: [groupId:]artifactId, case sensitive contains): 1224: nifi Choose archetype: 1: remote -> org.apache.nifi:nifi-processor-bundle-archetype (-) 2: remote -> org.apache.nifi:nifi-service-bundle-archetype (-) Choose a number or apply filter (format: [groupId:]artifactId, case sensitive contains): : 1 Choose org.apache.nifi:nifi-processor-bundle-archetype version: 1: 0.0.2-incubating 2: 0.1.0-incubating 3: 0.2.0-incubating 4: 0.2.1 5: 0.3.0 6: 0.4.0 7: 0.4.1 8: 0.5.0 9: 0.5.1 10: 0.6.0 11: 0.6.1 12: 0.7.0 13: 0.7.1 14: 0.7.2 15: 0.7.3 16: 0.7.4 17: 1.0.0-BETA 18: 1.0.0 19: 1.0.1 20: 1.1.0 21: 1.1.1 22: 1.1.2 23: 1.2.0 24: 1.3.0 25: 1.4.0 26: 1.5.0 27: 1.6.0 28: 1.7.0 29: 1.7.1 Choose a number: 29: 29 Define value for property 'groupId': com.github.paulvid Define value for property 'artifactId': boilerpipe-article-extractor Define value for property 'version' 1.0-SNAPSHOT: : 1.7.0.3.2.0.0-520 Define value for property 'artifactBaseName': BoilerpipeArticleExtractor Define value for property 'package' com.github.paulvid.processors.BoilerpipeArticleExtractor: : [INFO] Using property: nifiVersion = 1.7.1 Confirm properties configuration: groupId: com.github.paulvid artifactId: boilerpipe-article-extractor version: 1.7.0.3.2.0.0-520 artifactBaseName: BoilerpipeArticleExtractor package: com.github.paulvid.processors.BoilerpipeArticleExtractor nifiVersion: 1.7.1 Y: : Y [INFO] ---------------------------------------------------------------------------- [INFO] Using following parameters for creating project from Archetype: nifi-processor-bundle-archetype:1.7.1 [INFO] ---------------------------------------------------------------------------- [INFO] Parameter: groupId, Value: com.github.paulvid [INFO] Parameter: artifactId, Value: boilerpipe-article-extractor [INFO] Parameter: version, Value: 1.7.0.3.2.0.0-520 [INFO] Parameter: package, Value: com.github.paulvid.processors.BoilerpipeArticleExtractor [INFO] Parameter: packageInPathFormat, Value: com/github/paulvid/processors/BoilerpipeArticleExtractor [INFO] Parameter: package, Value: com.github.paulvid.processors.BoilerpipeArticleExtractor [INFO] Parameter: artifactBaseName, Value: BoilerpipeArticleExtractor [INFO] Parameter: version, Value: 1.7.0.3.2.0.0-520 [INFO] Parameter: groupId, Value: com.github.paulvid [INFO] Parameter: artifactId, Value: boilerpipe-article-extractor [INFO] Parameter: nifiVersion, Value: 1.7.1 [INFO] Project created from Archetype in dir: /Users/pvidal/Documents/tutorial/boilerpipe-article-extractor [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 02:38 min [INFO] Finished at: 2018-08-23T10:36:04-04:00 [INFO] ------------------------------------------------------------------------ Few important points in the configuration of the archetype: Maven has a ton of archetypes, filter by using "nifi" and then select 1: remote -> org.apache.nifi:nifi-processor-bundle-archetype (-) Choose the version that works with your environment (1.7.1 and 1.7.0 would work in my case) Properties don't really matter for this tutorial but here is what I used: Define value for property 'groupId': com.github.paulvid Define value for property 'artifactId': boilerpipe-article-extractor Define value for property 'version' 1.0-SNAPSHOT: : 1.7.0.3.2.0.0-520 Define value for property 'artifactBaseName': BoilerpipeArticleExtractor Define value for property 'package' com.github.paulvid.processors.BoilerpipeArticleExtractor Step 3: Configure your processor Once the archetype is successfully built, use your favorite IDE and open the folder containing the code. Adding the BoilerPipe dependency Open the file [YOUR_PATH]/tutorial/boilerpipe-article-extractor/nifi-BoilerpipeArticleExtractor-processors/pom.xml Add the following dependency: Rename the processor By default, the processor under [YOUR_PATH]/tutorial/boilerpipe-article-extractor/nifi-BoilerpipeArticleExtractor-processors/src/main/java/com/github/paulvid/processors/BoilerpipeArticleExtractor is called MyProcessor. Using your IDE, rename your Processor and the corresponding test class to BoilerpipeArticleExtractor Add code to processor Modify your BoilerpipeArticleExtractor.java to contain the following package com.github.paulvid.processors.BoilerpipeArticleExtractor; import de.l3s.boilerpipe.BoilerpipeProcessingException; import de.l3s.boilerpipe.extractors.ArticleExtractor; import org.apache.nifi.components.PropertyDescriptor; import org.apache.nifi.flowfile.FlowFile; import org.apache.nifi.annotation.behavior.ReadsAttribute; import org.apache.nifi.annotation.behavior.ReadsAttributes; import org.apache.nifi.annotation.behavior.WritesAttribute; import org.apache.nifi.annotation.behavior.WritesAttributes; import org.apache.nifi.annotation.lifecycle.OnScheduled; import org.apache.nifi.annotation.documentation.CapabilityDescription; import org.apache.nifi.annotation.documentation.SeeAlso; import org.apache.nifi.annotation.documentation.Tags; import org.apache.nifi.processor.exception.ProcessException; import org.apache.nifi.processor.AbstractProcessor; import org.apache.nifi.processor.ProcessContext; import org.apache.nifi.processor.ProcessSession; import org.apache.nifi.processor.ProcessorInitializationContext; import org.apache.nifi.processor.Relationship; import org.apache.nifi.processor.util.StandardValidators; import org.apache.nifi.expression.ExpressionLanguageScope; import java.io.BufferedOutputStream; import java.io.OutputStream; import java.net.MalformedURLException; import java.net.URL; import java.nio.charset.StandardCharsets; import java.util.ArrayList; import java.util.Collections; import java.util.HashSet; import java.util.List; import java.util.Set; @Tags({"example"}) @CapabilityDescription("Provide a description") @SeeAlso({}) @ReadsAttributes({@ReadsAttribute(attribute="", description="")}) @WritesAttributes({@WritesAttribute(attribute="", description="")}) public class BoilerpipeArticleExtractor extends AbstractProcessor { public static final PropertyDescriptor URL_PROPERTY = new PropertyDescriptor .Builder().name("URL") .displayName("URL") .description("URL of the Article to Extract") .required(true) .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES) .addValidator(StandardValidators.NON_EMPTY_VALIDATOR) .build(); public static final Relationship REL_SUCCESS = new Relationship.Builder() .name("success") .description("A FlowFile is routed to this relationship after the article has been successfully extracted") .build(); public static final Relationship REL_FAILURE = new Relationship.Builder() .name("failure") .description("A FlowFile is routed to this relationship if an error occurred during the article extraction") .build(); private List<PropertyDescriptor> descriptors; private Set<Relationship> relationships; @Override protected void init(final ProcessorInitializationContext context) { final List<PropertyDescriptor> descriptors = new ArrayList<PropertyDescriptor>(); descriptors.add(URL_PROPERTY); this.descriptors = Collections.unmodifiableList(descriptors); final Set<Relationship> relationships = new HashSet<Relationship>(); relationships.add(REL_SUCCESS); relationships.add(REL_FAILURE); this.relationships = Collections.unmodifiableSet(relationships); } @Override public Set<Relationship> getRelationships() { return this.relationships; } @Override public final List<PropertyDescriptor> getSupportedPropertyDescriptors() { return descriptors; } @OnScheduled public void onScheduled(final ProcessContext context) { } @Override public void onTrigger(final ProcessContext context, final ProcessSession session) throws ProcessException { FlowFile flowFile = session.get(); if (flowFile == null) { return; } try { java.net.URL articleURL = new URL(String.valueOf(context.getProperty(URL_PROPERTY).evaluateAttributeExpressions(flowFile).getValue())); String text = ArticleExtractor.INSTANCE.getText(articleURL); flowFile = session.write(flowFile, out -> { try (OutputStream outputStream = new BufferedOutputStream(out)) { outputStream.write(text.getBytes(StandardCharsets.UTF_8)); } }); session.transfer(flowFile, REL_SUCCESS); } catch (MalformedURLException mue) { getLogger().error("Failed to get URL {} for {} due to {}", new Object[]{context.getProperty(URL_PROPERTY).evaluateAttributeExpressions(flowFile).getValue(), flowFile, mue}); session.transfer(flowFile, REL_FAILURE); return; } catch (BoilerpipeProcessingException bpe) { getLogger().error("Failed to extract article for {} due to {}", new Object[]{flowFile, bpe}); session.transfer(flowFile, REL_FAILURE); return; } } } Create test case Modify your BoilerpipeArticleExtractor.java to contain the following code: package com.github.paulvid.processors.BoilerpipeArticleExtractor; import org.apache.nifi.util.MockFlowFile; import org.apache.nifi.util.TestRunner; import org.apache.nifi.util.TestRunners; import org.junit.Before; import org.junit.Test; import java.util.List; public class BoilerpipeArticleExtractorTest { private TestRunner testRunner; @Before public void init() { testRunner = TestRunners.newTestRunner(BoilerpipeArticleExtractor.class); } @Test public void testProcessor() { testRunner.setValidateExpressionUsage(false); testRunner.setProperty(BoilerpipeArticleExtractor.URL_PROPERTY,"https://www.nytimes.com/reuters/2018/08/17/business/17reuters-usa-tunnels-china.html"); testRunner.enqueue("Mock FlowFile"); testRunner.run(); testRunner.assertTransferCount(BoilerpipeArticleExtractor.REL_SUCCESS, 1); testRunner.assertTransferCount(BoilerpipeArticleExtractor.REL_FAILURE, 0); List<MockFlowFile> flowFiles = testRunner.getFlowFilesForRelationship(BoilerpipeArticleExtractor.REL_SUCCESS); flowFiles.get(0).assertContentEquals("Musk's Tunnel-Boring Firm Seeks U.S. Tariff Exemption for Chinese Parts\n" + "By Reuters\n" + "Aug. 17, 2018\n" + "WASHINGTON — A company owned by Elon Musk that is trying to lower the cost of building high-speed transit tunnels has asked the Trump administration to exempt it from tariffs for some Chinese-made tunnel boring machine components, warning the tariffs could significantly delay a planned tunnel between New York and Washington.\n" + "In a July 31 letter posted last week on a government website, the Boring Co asked the U.S. Trade Representative to exempt parts like cutterheads, screw conveyors and related machinery. Boring seeks \"limited parts from China in the near-term for use in a small number of tunnel boring machines.\" The letter added those parts are \"readily available only from China.\"\n" + "Privately held Boring added that it is \"working to develop and manufacture our own tunnel boring machines\" and wants to \"restore the now-dormant American tunnel boring machine industry.\"\n" + "The company said for planned tunnels, including a project between Washington and Baltimore, it will \"use machines that are majority-composed of U.S. content.\"\n" + "The tariffs could cause \"severe economic harm\" to the company and U.S. interests and could result in a delay of one to two years in the construction of a proposed Washington-to-Baltimore tunnel that it plans to eventually extend to New York.\n" + "Exempting the parts will not harm U.S. industry, the company said, and noted that tunneling is not one of 10 sectors identified in China's \"Made in 2025\" plan. The company said its business model is \"predicated upon substantially reducing the cost of tunneling.\"\n" + "Musk, who is also chief executive of Tesla Inc, in June proposed building a $1 billion underground transit system in Chicago. The plan would send people from Chicago’s downtown Loop district to O’Hare International Airport at 150 miles (241 km) per hour.\n" + "The Boring Co has been promoting its plans for tunnels that would allow high-speed travel between cities. The company initially plans to ferry passengers between Washington and Baltimore on autonomous electric vehicles carrying 8 to 16 passengers at 125-150 miles per hour, but would not use tracks or railway equipment.\n" + "On Wednesday, the company proposed to build a 3.6-mile tunnel between Dodger Stadium in Los Angeles and the city's subway system.\n" + "The U.S. Trade Representative's Office and Boring did not immediately respond to requests for comment.\n" + "Boring said in an earlier letter it was converting from diesel-powered to electric-powered construction equipment and had innovated in \"concrete mixing, segment production, excavation and hauling practices.\"\n" + "Other companies including General Motors Co have sought exemptions from new U.S. tariffs imposed on Chinese imports.\n" + "(Reporting by David Shepardson; Editing by David Gregorio)\n" + "Advertisement\n"); } } Compile your processor and create a nar In your command line, go to [YOUR_PATH]/tutorial/boilerpipe-article-extractor/ and type the following: $ mvn install You should see an output like this: $ mvn install [INFO] Scanning for projects... [INFO] Inspecting build with total of 3 modules... [INFO] Installing Nexus Staging features: [INFO] ... total of 3 executions of maven-deploy-plugin replaced with nexus-staging-maven-plugin [INFO] ------------------------------------------------------------------------ [INFO] Reactor Build Order: [INFO] [INFO] boilerpipe-article-extractor [pom] [INFO] nifi-BoilerpipeArticleExtractor-processors [jar] [INFO] nifi-BoilerpipeArticleExtractor-nar [nar] [INFO] [INFO] ----------< com.github.paulvid:boilerpipe-article-extractor >----------- [INFO] Building boilerpipe-article-extractor 1.7.0.3.2.0.0-520 [1/3] [INFO] --------------------------------[ pom ]--------------------------------- [INFO] [INFO] --- maven-enforcer-plugin:1.4.1:enforce (enforce-maven-version) @ boilerpipe-article-extractor --- [INFO] [INFO] --- maven-enforcer-plugin:1.4.1:enforce (enforce-maven) @ boilerpipe-article-extractor --- [INFO] [INFO] --- buildnumber-maven-plugin:1.4:create (default) @ boilerpipe-article-extractor --- [INFO] [INFO] --- maven-remote-resources-plugin:1.5:process (process-resource-bundles) @ boilerpipe-article-extractor --- [INFO] [INFO] --- maven-compiler-plugin:3.6.0:testCompile (groovy-tests) @ boilerpipe-article-extractor --- [INFO] No sources to compile [INFO] [INFO] --- maven-site-plugin:3.7:attach-descriptor (attach-descriptor) @ boilerpipe-article-extractor --- [INFO] No site descriptor found: nothing to attach. [INFO] [INFO] --- maven-install-plugin:2.5.2:install (default-install) @ boilerpipe-article-extractor --- [INFO] Installing /Users/pvidal/Documents/tutorial/boilerpipe-article-extractor/pom.xml to /Users/pvidal/.m2/repository/com/github/paulvid/boilerpipe-article-extractor/1.7.0.3.2.0.0-520/boilerpipe-article-extractor-1.7.0.3.2.0.0-520.pom [INFO] [INFO] ---< com.github.paulvid:nifi-BoilerpipeArticleExtractor-processors >---- [INFO] Building nifi-BoilerpipeArticleExtractor-processors 1.7.0.3.2.0.0-520 [2/3] [INFO] --------------------------------[ jar ]--------------------------------- [INFO] [INFO] --- maven-enforcer-plugin:1.4.1:enforce (enforce-maven-version) @ nifi-BoilerpipeArticleExtractor-processors --- [INFO] [INFO] --- maven-enforcer-plugin:1.4.1:enforce (enforce-maven) @ nifi-BoilerpipeArticleExtractor-processors --- [INFO] [INFO] --- buildnumber-maven-plugin:1.4:create (default) @ nifi-BoilerpipeArticleExtractor-processors --- [INFO] [INFO] --- maven-remote-resources-plugin:1.5:process (process-resource-bundles) @ nifi-BoilerpipeArticleExtractor-processors --- [INFO] [INFO] --- maven-resources-plugin:3.0.2:resources (default-resources) @ nifi-BoilerpipeArticleExtractor-processors --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] Copying 1 resource [INFO] Copying 3 resources [INFO] [INFO] --- maven-compiler-plugin:3.6.0:compile (default-compile) @ nifi-BoilerpipeArticleExtractor-processors --- [INFO] Changes detected - recompiling the module! [INFO] Compiling 1 source file to /Users/pvidal/Documents/tutorial/boilerpipe-article-extractor/nifi-BoilerpipeArticleExtractor-processors/target/classes [INFO] [INFO] --- maven-resources-plugin:3.0.2:testResources (default-testResources) @ nifi-BoilerpipeArticleExtractor-processors --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory /Users/pvidal/Documents/tutorial/boilerpipe-article-extractor/nifi-BoilerpipeArticleExtractor-processors/src/test/resources [INFO] Copying 3 resources [INFO] [INFO] --- maven-compiler-plugin:3.6.0:testCompile (default-testCompile) @ nifi-BoilerpipeArticleExtractor-processors --- [INFO] Changes detected - recompiling the module! [INFO] Compiling 1 source file to /Users/pvidal/Documents/tutorial/boilerpipe-article-extractor/nifi-BoilerpipeArticleExtractor-processors/target/test-classes [INFO] [INFO] --- maven-compiler-plugin:3.6.0:testCompile (groovy-tests) @ nifi-BoilerpipeArticleExtractor-processors --- [INFO] Changes detected - recompiling the module! [INFO] Nothing to compile - all classes are up to date [INFO] [INFO] --- maven-surefire-plugin:2.20.1:test (default-test) @ nifi-BoilerpipeArticleExtractor-processors --- [INFO] [INFO] ------------------------------------------------------- [INFO] T E S T S [INFO] ------------------------------------------------------- [INFO] Running com.github.paulvid.processors.BoilerpipeArticleExtractor.BoilerpipeArticleExtractorTest [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.907 s - in com.github.paulvid.processors.BoilerpipeArticleExtractor.BoilerpipeArticleExtractorTest [INFO] [INFO] Results: [INFO] [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0 [INFO] [INFO] [INFO] --- maven-jar-plugin:3.0.2:jar (default-jar) @ nifi-BoilerpipeArticleExtractor-processors --- [INFO] Building jar: /Users/pvidal/Documents/tutorial/boilerpipe-article-extractor/nifi-BoilerpipeArticleExtractor-processors/target/nifi-BoilerpipeArticleExtractor-processors-1.7.0.3.2.0.0-520.jar [INFO] [INFO] --- maven-site-plugin:3.7:attach-descriptor (attach-descriptor) @ nifi-BoilerpipeArticleExtractor-processors --- [INFO] Skipping because packaging 'jar' is not pom. [INFO] [INFO] --- maven-install-plugin:2.5.2:install (default-install) @ nifi-BoilerpipeArticleExtractor-processors --- [INFO] Installing /Users/pvidal/Documents/tutorial/boilerpipe-article-extractor/nifi-BoilerpipeArticleExtractor-processors/target/nifi-BoilerpipeArticleExtractor-processors-1.7.0.3.2.0.0-520.jar to /Users/pvidal/.m2/repository/com/github/paulvid/nifi-BoilerpipeArticleExtractor-processors/1.7.0.3.2.0.0-520/nifi-BoilerpipeArticleExtractor-processors-1.7.0.3.2.0.0-520.jar [INFO] Installing /Users/pvidal/Documents/tutorial/boilerpipe-article-extractor/nifi-BoilerpipeArticleExtractor-processors/pom.xml to /Users/pvidal/.m2/repository/com/github/paulvid/nifi-BoilerpipeArticleExtractor-processors/1.7.0.3.2.0.0-520/nifi-BoilerpipeArticleExtractor-processors-1.7.0.3.2.0.0-520.pom [INFO] [INFO] -------< com.github.paulvid:nifi-BoilerpipeArticleExtractor-nar >------- [INFO] Building nifi-BoilerpipeArticleExtractor-nar 1.7.0.3.2.0.0-520 [3/3] [INFO] --------------------------------[ nar ]--------------------------------- [INFO] [INFO] --- maven-enforcer-plugin:1.4.1:enforce (enforce-maven-version) @ nifi-BoilerpipeArticleExtractor-nar --- [INFO] [INFO] --- maven-enforcer-plugin:1.4.1:enforce (enforce-maven) @ nifi-BoilerpipeArticleExtractor-nar --- [INFO] [INFO] --- buildnumber-maven-plugin:1.4:create (default) @ nifi-BoilerpipeArticleExtractor-nar --- [INFO] [INFO] --- maven-remote-resources-plugin:1.5:process (process-resource-bundles) @ nifi-BoilerpipeArticleExtractor-nar --- [INFO] [INFO] --- maven-resources-plugin:3.0.2:resources (default-resources) @ nifi-BoilerpipeArticleExtractor-nar --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory /Users/pvidal/Documents/tutorial/boilerpipe-article-extractor/nifi-BoilerpipeArticleExtractor-nar/src/main/resources [INFO] Copying 3 resources [INFO] [INFO] --- maven-compiler-plugin:3.6.0:compile (default-compile) @ nifi-BoilerpipeArticleExtractor-nar --- [INFO] No sources to compile [INFO] [INFO] --- maven-resources-plugin:3.0.2:testResources (default-testResources) @ nifi-BoilerpipeArticleExtractor-nar --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory /Users/pvidal/Documents/tutorial/boilerpipe-article-extractor/nifi-BoilerpipeArticleExtractor-nar/src/test/resources [INFO] Copying 3 resources [INFO] [INFO] --- maven-compiler-plugin:3.6.0:testCompile (default-testCompile) @ nifi-BoilerpipeArticleExtractor-nar --- [INFO] No sources to compile [INFO] [INFO] --- maven-compiler-plugin:3.6.0:testCompile (groovy-tests) @ nifi-BoilerpipeArticleExtractor-nar --- [INFO] No sources to compile [INFO] [INFO] --- maven-surefire-plugin:2.20.1:test (default-test) @ nifi-BoilerpipeArticleExtractor-nar --- [INFO] [INFO] --- nifi-nar-maven-plugin:1.2.0:nar (default-nar) @ nifi-BoilerpipeArticleExtractor-nar --- [INFO] Copying nifi-BoilerpipeArticleExtractor-processors-1.7.0.3.2.0.0-520.jar to /Users/pvidal/Documents/tutorial/boilerpipe-article-extractor/nifi-BoilerpipeArticleExtractor-nar/target/classes/META-INF/bundled-dependencies/nifi-BoilerpipeArticleExtractor-processors-1.7.0.3.2.0.0-520.jar [INFO] Copying nekohtml-1.9.21.jar to /Users/pvidal/Documents/tutorial/boilerpipe-article-extractor/nifi-BoilerpipeArticleExtractor-nar/target/classes/META-INF/bundled-dependencies/nekohtml-1.9.21.jar [INFO] Copying xercesImpl-2.11.0.jar to /Users/pvidal/Documents/tutorial/boilerpipe-article-extractor/nifi-BoilerpipeArticleExtractor-nar/target/classes/META-INF/bundled-dependencies/xercesImpl-2.11.0.jar [INFO] Copying xml-apis-1.4.01.jar to /Users/pvidal/Documents/tutorial/boilerpipe-article-extractor/nifi-BoilerpipeArticleExtractor-nar/target/classes/META-INF/bundled-dependencies/xml-apis-1.4.01.jar [INFO] Copying nifi-utils-1.7.1.jar to /Users/pvidal/Documents/tutorial/boilerpipe-article-extractor/nifi-BoilerpipeArticleExtractor-nar/target/classes/META-INF/bundled-dependencies/nifi-utils-1.7.1.jar [INFO] Copying jericho-html-3.3.jar to /Users/pvidal/Documents/tutorial/boilerpipe-article-extractor/nifi-BoilerpipeArticleExtractor-nar/target/classes/META-INF/bundled-dependencies/jericho-html-3.3.jar [INFO] Copying boilerpipe-1.2.3.jar to /Users/pvidal/Documents/tutorial/boilerpipe-article-extractor/nifi-BoilerpipeArticleExtractor-nar/target/classes/META-INF/bundled-dependencies/boilerpipe-1.2.3.jar [INFO] Building jar: /Users/pvidal/Documents/tutorial/boilerpipe-article-extractor/nifi-BoilerpipeArticleExtractor-nar/target/nifi-BoilerpipeArticleExtractor-nar-1.7.0.3.2.0.0-520.nar [INFO] [INFO] --- maven-site-plugin:3.7:attach-descriptor (attach-descriptor) @ nifi-BoilerpipeArticleExtractor-nar --- [INFO] Skipping because packaging 'nar' is not pom. [INFO] [INFO] --- maven-install-plugin:2.5.2:install (default-install) @ nifi-BoilerpipeArticleExtractor-nar --- [INFO] Installing /Users/pvidal/Documents/tutorial/boilerpipe-article-extractor/nifi-BoilerpipeArticleExtractor-nar/target/nifi-BoilerpipeArticleExtractor-nar-1.7.0.3.2.0.0-520.nar to /Users/pvidal/.m2/repository/com/github/paulvid/nifi-BoilerpipeArticleExtractor-nar/1.7.0.3.2.0.0-520/nifi-BoilerpipeArticleExtractor-nar-1.7.0.3.2.0.0-520.nar [INFO] Installing /Users/pvidal/Documents/tutorial/boilerpipe-article-extractor/nifi-BoilerpipeArticleExtractor-nar/pom.xml to /Users/pvidal/.m2/repository/com/github/paulvid/nifi-BoilerpipeArticleExtractor-nar/1.7.0.3.2.0.0-520/nifi-BoilerpipeArticleExtractor-nar-1.7.0.3.2.0.0-520.pom [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary: [INFO] [INFO] boilerpipe-article-extractor 1.7.0.3.2.0.0-520 ..... SUCCESS [ 1.487 s] [INFO] nifi-BoilerpipeArticleExtractor-processors ......... SUCCESS [ 14.513 s] [INFO] nifi-BoilerpipeArticleExtractor-nar 1.7.0.3.2.0.0-520 SUCCESS [ 0.281 s] [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 17.712 s [INFO] Finished at: 2018-08-23T11:18:13-04:00 [INFO] ------------------------------------------------------------------------ Step 4: Upload the generated nar The file nifi-BoilerpipeArticleExtractor-nar-1.7.0.3.2.0.0-520.nar should be generated under [YOUR_PATH]/tutorial/boilerpipe-article-extractor/nifi-BoilerpipeArticleExtractor-nar/target Upload it onto your server under the library folder of nifi (typically /usr/hdf/current/nifi/lib/) Restart NiFi and you should see the BoilerpipeArticleExtractor available Step 5: Configure your processor Configure the processor to use the web_url attribute as a property, as follows: Section 3: Create a custom Mairesse processor This section is a little advanced. I built a custom processor using this personality recognizer (http://farm2.user.srcf.net/research/personality/recognizer). If you want to checkout my github . For this tutorial, we will only concentrate on uploading my compiled code and the dependencies necessary. Step 1: Upload the Mairesse Processor Download the file nifi-MairessePersonalityRecognition-nar-1.5.0.3.1.2.0-7.nar from my github Upload it onto your server under the library folder of nifi (typically /usr/hdf/current/nifi/lib/) Restart NiFi and you should see the MairessePersonalityRecognition available Step 2: Upload the library files Upload the files under the folder to-upload-to-nifi-lib-folder from my github to a location on your server (e.g. /home/nifi/perso-recognition/lib/) Step 3: Configure your processor Configure the processor as follows: Section 4: Create and configure a Kafka topic Step 1: Create a schema in Schema Registry Go to your schema registry and create a new schema called personality-recognition: The schema you will enter should be as follows: { "type": "record", "name": "personalityrecognition", "fields": [ { "name": "web_url", "type": "string", "default": null }, { "name": "snippet", "type": "string", "default": null }, { "name": "byline", "type": "string", "default": null }, { "name": "pub_date", "type": "string", "default": null }, { "name": "headline", "type": "string", "default": null }, { "name": "document_type", "type": "string", "default": null }, { "name": "news_desk", "type": "string", "default": null }, { "name": "timestamp", "type": "string", "default": null }, { "name": "extraversion", "type": "string", "default": null }, { "name": "emotional_stability", "type": "string", "default": null }, { "name": "agreeableness", "type": "string", "default": null }, { "name": "conscientiousness", "type": "string", "default": null }, { "name": "openness_to_experience", "type": "string", "default": null } ] } Step 2: Update attribute to add the schema name Using an UpdateAttribute processor, configure it to use the schema.name personality-recognition, as well as removing double quotes from the headlines and snippet attributes: Step 3: Create Json from attributes Using an ReplaceText processor, configure the replacement value to create the appropriate JSON: The replacement value you will enter should be as follows: { "web_url": "${web_url}", "snippet": "${snippet}", "byline": "${byline}", "pub_date": "${pub_date}", "headline": "${headline}", "document_type": "${document_type}", "news_desk": "${news_desk}", "timestamp": "${now()}", "extraversion": "${mairessepersonalityrecognition.extraversion}", "emotional_stability": "${mairessepersonalityrecognition.emotional_stability}", "agreeableness": "${mairessepersonalityrecognition.agreeableness}", "conscientiousness": "${mairessepersonalityrecognition.conscientiousness}", "openness_to_experience": "${mairessepersonalityrecognition.openness_to_experience}" } Step 4: Create Kafka topic On your server running Kafka, find the shell scripts to create topics (typically under /usr/hdp/current/kafka-broker/bin). Run the following command: $ ./kafka-topics.sh --zookeeper localhost:2181 --topic personality-recognition --create --partitions 1 --replication-factor 1 Step 5: Configure PublishKafkaRecord Configure Hortonworks Schema Registry Controller In your controller configuration add a new Hortonworks Schema Registry controller: The Schema registry URL should look something like this: http://pvidal-personality-recognition0.field.hortonworks.com:7788/api/v1/ Configure JsonPathReader Controller In your controller configuration add a new JsonPathReader Controller to use the schema name and the Hortonworks Schema Registry controller we created, as follows: Add all the fields from the JSON output we created Configure AvroRecordSetWriter Controller In your controller configuration add a new AvroRecordSetWriter Controller to use the schema name and the Hortonworks Schema Registry controller we created, as follows: Configure PublishKafkaRecord Processor Add a new processor to your flow, configured to use the controllers we created, as follows: The Kafka Broker URL should look something like this: pvidal-personality-recognition0.field.hortonworks.com:6667 Step 6: Run flow and consume data via command line On your server running Kafka, find the shell scripts to create topics (typically under /usr/hdp/current/kafka-broker/bin). Run the following command: $ ./kafka-console-consumer.sh --zookeeper localhost:2181 --topic personality-recognition Run your flow in NiFi and you should start seeing data flowing! ... View more

Introduction A few weeks ago, I published an article called Determining the big 5 traits of Personality Psychology of news articles using NiFi, Hive & Zeppelin. Since then, I have worked diligently to improve on this first iteration, with the objective to mock up at the heart of every company today: create an end-to-end platform that uses machine learning to not only generate insights, but keeps improving and feeding consumer applications. While doing this work was a way for me to get into the needy-greedy of the latest Hortonworks tools, I decided to share it with the world in the form of a series of tutorial articles, because I believe it is a great way to get familiar with the stack. Architecture overview Luckily, the Hortonworks platform has all the elements needed to create this end-to-end platform. The figure below gives an overview of this series of articles architecture: As you can see, the goal of this platform is to: Ingest data from news articles (directly from the NYT API at first, then from other RSS feeds) Using Nifi and SAM, read the meta-data of the extracted articles, scrape their content, run personality recognition on their authors, then expose the result via Kafka for Druid consumption, directly pushing to HBase/Phoenix for offline analytics and "micro" services for consumer applications Generate real time insights on this computed data via Druid and Superset Enable Analytics & model training on the data stored in HBase using Zeppelin notebooks & Spark, that would then feed back the personality recognition modes Enable custom application to consumer the data extracted and analyzed Agenda This series of article will be composed of 4 parts: Part 1: Create Nifi Flow with Custom Processors Part 2: Using SAM to consume Kafka, load into Druid and create superset visualization Part 3: Using Spark and Zeppelin to run offline Analytics and train models Part 4: Create Nifi Web Services and a consumer application ... View more

Introduction & Context There is a reason why I spent my life studying and working in computer science: understanding a computer's psychology is usually fairly straight forward. Indeed, when presented with a specific input, computer programs tend to respond in a very predictable way, as opposed to our fellow human beings. Of course, this observation goes out of the window as our algorithms become increasingly more complex and capable of learning. Regardless, as much as I love computer science, I always had a keen interest in human sciences. Personality psychology is a fascinating subject that has seen its ups and downs as any science topic. At the center of personality psychology reside the big five personality traits: Openness to Experience Conscientiousness Extraversion Agreeableness Neuroticism (or Emotional Stability) This taxonomy was determined by applying statistical models to personality surveys, essentially clustering results of surveys of people describing fellow human beings. As such, these traits are meant to categorize common aspect of personality across human beings without moral connotation. The validity of the model and its predictability for real life outcomes is of course controversial, and I wouldn't make it justice here (I most likely already irritated any personality psychologist that read these first few lines). Recently, multiple machine learning algorithms have been designed to determine these 5 personality traits from texts have surfaced, including IBM Watson personality insights. For this article I chose to use the personality recognizer written by Francois Mairesse, and automate personality detection of New York Times articles using HDF 3.1 and HDP 3.0. Solution Overview The solution put in place uses 3 main elements: A NiFi flow to orchestrate data ingestion from API, personality detection and storage to Hive Hive to store the results of the personality detection Zeppelin for visualization of the results The figure below gives an overview of the solution flow: More precisely, the solution can be dissected in 5 main steps, that I'm describing in details below: Step 1: Retrieving data from New York Times API Step 2: Scrape HTML article data Step 3: Run machine learning models for personality detection Step 4: Store results to Hive Step 5: Create simple Zeppelin notebook Step 1: Retrieving data from New York Times API Obtaining an API Key This step is very straight forward. Go to https://developer.nytimes.com/signup and sign-up for a key: Note: The New-York Times API is for non-commercial use only. I could have of course used any news API, but I'm not creative. Configuring InvokeHTTP The InvokeHTTP is used here with all default parameters, except for the URL. Here are some key configuration items and a screenshot of the Processor configuration: HTTP Method: GET Remote URL: http://api.nytimes.com/svc/search/v2/articlesearch.json?fq=source:("The New York Times")&page=0&sort=newest&fl=web_url,snippet,headline,pub_date,document_type,news_desk,byline&api-key=[YOUR_KEY] (This URL selects article from the New York Times as a source, and only selects some of the fields I am interested in: web_url,snippet,headline,pub_date,document_type,news_desk,byline). Content-Type: ${mime.type} Run Schedule: 5 mins (could be set to a little more, I'm not sure the frequency at which new articles are published) Extracting results from Invoke HTTP response The API call parameter page=0 returns results 0-9; for this exercise, I'm only interested in the latest article, so I setup an evaluateJSONpath processor take care of that, as you can see below: A few important points here: The destination is set to flowfile-attribute because we are going to re-use these attributes later in the flow I'm expecting the API to change after some time this article is published. Just to make sure that the JSON paths are good for your version of the API, I recommend JSON paths evaluators online. Massage data to avoid conflicts when inserting to Hive This step is definitely not optimized. The point here is to escape the special characters to avoid errors when inserting into hive. The only thing I am doing here is removing the ' from the snippet as you can see, but it would deserve a second path I think: Step 2: Scrape HTML article data Once we retrieved the meta data of the article, we must obtain the actual text of the article. For this, I'm using boilerpipe, an open source boilerplate removal and fulltext extraction from HTML pages (see reference for details). Create a simple Java class to call boilerplate After downloading the boilerpipe jars (using http://www.java2s.com/Code/Jar/b/Downloadboilerpipe120jar.htm), use your favorite Java IDE and create this simple class: import de.l3s.boilerpipe.BoilerpipeProcessingException; import de.l3s.boilerpipe.extractors.ArticleExtractor; import java.net.MalformedURLException; import java.net.URL; public class extractArticle { public static void main (String args[]) throws MalformedURLException, BoilerpipeProcessingException { if(args.length == 1) { URL url = new URL("" + args[0]); String text = ArticleExtractor.INSTANCE.getText(url); System.out.println(text); } else { System.out.println("Please Specify URL"); } } } Once tested, create an executable jar (in my case extractArticle.jar). Transfer jars to nifi server Connect to your nifi server with your nifi user and create the following directory structure: $ cd /home/nifi $ mkdir extractArticle $ cd extractArticle $ mkdir lib Transfer the following libraries to ~/extractArticle/lib/ : xerces-2.9.1.jar nekohtml-1.9.13.jar boilerpipe-sources-1.2.0.jar boilerpipe-javadoc-1.2.0.jar boilerpipe-demo-1.2.0.jar boilerpipe-1.2.0.jar extractArticle.jar Create a simple Unix script to execute HTML scraping Under ~/extractArticle/ create the script extract_article.sh as follows: #!/bin/bash JDK_PATH=/usr LIB1=./lib/xerces-2.9.1.jar LIB2=./lib/nekohtml-1.9.13.jar LIB3=./lib/boilerpipe-sources-1.2.0.jar LIB4=./lib/boilerpipe-javadoc-1.2.0.jar LIB5=./lib/boilerpipe-demo-1.2.0.jar LIB6=./lib/boilerpipe-1.2.0.jar LIB7=./lib/extractArticle.jar LIBS=$LIB1:$LIB2:$LIB3:$LIB4:$LIB5:$LIB6:$LIB7 $JDK_PATH/bin/java -Xmx512m -classpath $LIBS extractArticle $* Configure ExecuteStreamCommand processor Configure the processor to pass the URL in argument and outputting the output stream to the next processor, as follows: Step 3: Run machine learning models for personality detection Setup PersonalityRecognizer on NiFi server Just as for boilerpipe, we're going to run an ExecuteStream command. To prepare the files, run the following commands: $ cd /home/nifi $ wget http://farm2.user.srcf.net/research/personality/recognizer-1.0.3.tar.gz recognizer-1.0.3.tar.gz $ tar -xvf recognizer-1.0.3.tar.gz $ cd PersonalityRecognizer $ mkdir texts Modify the file PersonalityRecognizer.properties as follows: ################################################## # Configuration File of the Personality Recognizer ################################################## # All variables should be modified according to your # directory structure # Warning: for Windows paths, backslashes need to be # doubled, e.g. c:\Program Files\Recognizer # Root directory of the application appDir = /home/nifi/PersonalityRecognizer # Path to the LIWC dictionary file (LIWC.CAT) liwcCatFile = ./lib/LIWC.CAT # Path to the MRC Psycholinguistic Database file (mrc2.dct) mrcPath = ./ext/mrc2.dct Modify the script PersonalityRecognizer as follows: #! /bin/bash - # ENVIRONMENT VARIABLES JDK_PATH=/usr WEKA=./ext/weka-3-4/weka.jar # ---------------------------------- COMMONS_CLI=./lib/commons-cli-1.0.jar MRC=./lib/jmrc.jar LIBS=.:$WEKA:$COMMONS_CLI:$MRC:bin/ $JDK_PATH/bin/java -Xmx512m -classpath $LIBS recognizer.PersonalityRecognizer $* Finally, create a wrapper script that, using the latest file from the folder texts runs PersonalityRecognizer and outputs only the results in a json format: #!/bin/bash text=`ls -t texts/ | head -1` ./PersonalityRecognizer -i ./texts/$text > tmp.txt extraversion=`cat tmp.txt | grep extraversion | grep -Eo '[+-]?[0-9]+([.][0-9]+)?'` emotional_stability=`cat tmp.txt | grep emotional | grep -Eo '[+-]?[0-9]+([.][0-9]+)?'` agreeableness=`cat tmp.txt | grep agreeableness | grep -Eo '[+-]?[0-9]+([.][0-9]+)?'` conscientiousness=`cat tmp.txt | grep conscientiousness | grep -Eo '[+-]?[0-9]+([.][0-9]+)?'` openness_to_experience=`cat tmp.txt | grep openness | grep -Eo '[+-]?[0-9]+([.][0-9]+)?'` json_output="{\"web_url\" : \"$1\", \"extraversion\" : \"$extraversion\",\"emotional_stability\" : \"$emotional_stability\",\"agreeableness\" : \"$agreeableness\",\"conscientiousness\" : \"$conscientiousness\",\"openness_to_experience\" : \"$openness_to_experience\"}" echo $json_output rm tmp.txt texts/* Configure PutFile processor to create article file This processor takes the output stream of the HTML scraping to create a file, under the appropriate folder, as shown below: Configure the ExecuteStreamCommand Processor Just as for HTML scraping, configure the processor to pass the URL in argument and outputting the output stream to the next processor, as follows: Extract attributes from JSON output Using EvaluateJSONPath, retrieve the results of the PersonalityRecognizer to attributes: Step 4: Store results to Hive Create Hive DB and tables Because we don't control wether we receive the same article twice from the New York Times API, we need to make sure that we don't insert the same data twice into Hive (i.e. upsert data into Hive). Upsert can be implemented by two tables and the merge command. Therefore connect to your hive server and create one database and two tables as follows: CREATE DATABASE personality_detection; use personality_detection; CREATE TABLE text_evaluation ( web_url String, snippet String, byline String, pub_date date, headline String, document_type String, news_desk String, last_updated String, extraversion decimal(10,4), emotional_stability decimal(10,4), agreeableness decimal(10,4), conscientiousness decimal(10,4), openness_to_experience decimal(10,4) ) clustered by (web_url) into 2 buckets stored as orc tblproperties("transactional"="true"); CREATE TABLE all_updates ( web_url String, snippet String, byline String, pub_date date, headline String, document_type String, news_desk String, last_updated String, extraversion decimal(10,4), emotional_stability decimal(10,4), agreeableness decimal(10,4), conscientiousness decimal(10,4), openness_to_experience decimal(10,4) ) STORED AS ORC tblproperties ("orc.compress" = "SNAPPY"); Create HiveQL script Using a ReplaceText processor, create the appropriate HiveQL command to be executed to upsert data into your tables from the data collected in the flow. Code for Replacement Value (note that I remove the timestamp from the pub_date here, because I'm storing it as a date): use personality_detection; insert into all_updates values('${web_url}','${snippet}','${byline}','${pub_date:substring(0,10)}','${headline}','${document_type}','${news_desk}','${now()}','${extraversion}','${emotional_stability}','${agreeableness}','${conscientiousness}','${openness_to_experience}'); merge into text_evaluation using (select distinct web_url, snippet, byline, pub_date, headline, document_type, news_desk, extraversion, emotional_stability, agreeableness, conscientiousness, openness_to_experience from all_updates) all_updates on text_evaluation.web_url = all_updates.web_url when matched then update set snippet=all_updates.snippet, byline=all_updates.byline, pub_date=all_updates.pub_date, headline=all_updates.headline, document_type=all_updates.document_type, news_desk=all_updates.news_desk, last_updated=from_unixtime(unix_timestamp()), extraversion=all_updates.extraversion, emotional_stability=all_updates.emotional_stability, agreeableness=all_updates.agreeableness, conscientiousness=all_updates.conscientiousness, openness_to_experience=all_updates.openness_to_experience when not matched then insert values(all_updates.web_url,all_updates.snippet, all_updates.byline, all_updates.pub_date, all_updates.headline, all_updates.document_type, all_updates.news_desk, from_unixtime(unix_timestamp()), all_updates.extraversion, all_updates.emotional_stability, all_updates.agreeableness, all_updates.conscientiousness, all_updates.openness_to_experience); truncate table all_updates; Processor Overview: Upsert data to hive Finally, configure a simple PutHiveQL processor as follows (make sure you configured your HiveConnectionPool beforehand): Step 5: Create simple Zeppelin notebook Lastly, after running the NiFi flow for a while, create a simple Zeppelin notebook to show your result. This notebook will use the jdbc interpreter for Hive and run the following query: %jdbc(hive) select byline, extraversion, emotional_stability, agreeableness, conscientiousness, openness_to_experience from personality_detection.text_evaluation limit 10 Then, you can play with Zeppelin visualizations to display the average of the big 5 by byline: Conclusion While being a very simple, this exercise is a good starting point for on-the-wire personality recognition. More importantly, in an age of information overload or even misinformation, having the ability to classifying the psychology of a text on the fly can be extremely useful. I do plan on tinkering with this project, improving performance, optimizing models and ingesting more data, so stay tuned! Known possible improvements Better control of data retrieval to avoid duplicate flows (depends on API) Better special character replacement for HiveQL command More elegant way to execute data scraping and run personality recognition java classes Additional scraping from article text to remove title, byline, and other unnecessary information from boilerpipe output More thorough testing of different personality recognizer models (and use other/more recent libraries) References Big Five personality traits: https://en.wikipedia.org/wiki/Big_Five_personality_traits The Big-Five Trait Taxonomy: History, Measurement, and Theoretical Perspectives: http://moityca.com.br/pdfs/bigfive_john.pdf IBM Watson Personality Insights: https://personality-insights-demo.ng.bluemix.net/ Personality Recognizer by Francois Mairesse: http://farm2.user.srcf.net/research/personality/recognizer NYT API sign up: https://developer.nytimes.com/signup NYT API FAQ: https://developer.nytimes.com/faq NYT Article Search readme: https://developer.nytimes.com/article_search_v2.json#/README JSON Path Evaluator: http://jsonpath.com/ Boilerpipe jar download: http://www.java2s.com/Code/Jar/b/Downloadboilerpipe120jar.htm Boilerpipe github: https://github.com/kohlschutter/boilerpipe ... View more

Context Hortonworks tutorials are a great resource and one I use often to explore new topics or re-enforce my current knowledge. Recently, I went through the Predicting Airline Delays using SparkR tutorial. This tutorial is using SparkR and RStudio, running on the HDP Sandbox. For those, like me, who want to run rstudio on an HDP node running on something else than the sandbox, the configuration of rstudio differs a little from the article linked in this tutorial, which is why I thought of writing this quick article. This tutorial will thus detail the installation and unit testing of RStudio Server for the following environment versions: Hortonworks Data Platform: HDP 2.6 OS: Centos 7 64bit Cloud Infrastucture: OpenStack Installing rstudio-server on Centos 7 1. Connect to the server onto which you want to install rstudio-server: $ ssh [your_user]@[your_server] Note: the user with which you connect must have sudo priviledges. 2. Install R: $ sudo yum install R -y 3. Once connected, following the RStudio official site, download the latest version of RStudio: $ wget https://download2.rstudio.org/rstudio-server-rhel-1.1.453-x86_64.rpm $ sudo yum install rstudio-server-rhel-1.1.453-x86_64.rpm 4. Run & Verify service: $ sudo systemctl enable rstudio-server.service $ sudo systemctl start rstudio-server.service $ sudo systemctl status rstudio-server.service You should see something like this: $ sudo systemctl status rstudio-server.service ● rstudio-server.service - RStudio Server Loaded: loaded (/etc/systemd/system/rstudio-server.service; enabled; vendor preset: disabled) Active: active (running) since Mon 2018-07-16 12:31:29 UTC; 22s ago Main PID: 13043 (rserver) CGroup: /system.slice/rstudio-server.service └─13043 /usr/lib/rstudio-server/bin/rserver Jul 16 12:31:29 pvidal-hdp26-9244-compute-2 systemd[1]: Starting RStudio Server... Jul 16 12:31:29 pvidal-hdp26-9244-compute-2 systemd[1]: Started RStudio Server. 5. Check that you can access the server inside of the network: $ wget [your_server]:8787 You should see something like this: $ wget 172.26.252.182:8787 --2018-07-16 12:34:28-- http://172.26.252.182:8787/ Connecting to 172.26.252.182:8787... connected. HTTP request sent, awaiting response... 302 Moved Temporarily Location: http://172.26.252.182:8787/unsupported_browser.htm [following] --2018-07-16 12:34:28-- http://172.26.252.182:8787/unsupported_browser.htm Connecting to 172.26.252.182:8787... connected. HTTP request sent, awaiting response... 200 OK Length: 894 [text/html] Saving to: ‘index.html’ 100%[===============================================================================================================================================================================>] 894 --.-K/s in 0s 2018-07-16 12:34:28 (99.4 MB/s) - ‘index.html’ saved [894/894] Opening rstudio-server port on OpenStack 1. In your OpenStack dashboard, note your stack parameters (found in the overview tab of your stack): 2. Copy your current template to a text file (found in the template tab of your Stack): Note: Make sure that the copy paste from the site did not add text before the "description" parameter. 3. Add the following line to allow TCP connections to 8787 to your security group: - {port_range_max: 8787, port_range_min: 8787, protocol: tcp, remote_ip_prefix: 0.0.0.0/0} 4. Change your template using the OpenStack dashboard: Click on Change Stack Template: Browse to the file you just modified and click Next: Copy paste the parameters you noted previously: Verify access to RStudio 1. Open your browser and go to the following address: http://[your_server]:8787 You should see the following screen: 2. Use any unix user/password to connect. ... View more