Introduction
Continuing my series on Beast Mode Quotient, let's automate the creation and termination of data science ready clusters. As always, since this is step 2 of a series of article, this tutorial depends on my previous article.
Pre-Requisites
In order to run this tutorial you will need to have a CB instance available. There are plenty of good tutorial out there, I recommend this one
Agenda
This tutorial is divided in the following sections:
- Section 1: Create a blueprints & recipes to run a minimal data science ephemeral cluster
- Section 2: Add blueprint and recipes via Cloudbreak interface
- Section 3: Automate cluster launch and terminate clusters
Section 1: Create a blueprints & recipes to run a minimal data science ephemeral cluster
A Cloudbreak blueprint has 3 parts:
- Part 1: Blueprint details
- Part 2: Services Configuration
- Part 3: Host Components configuration
Here are the details of each part for our blueprint.
Blueprint details
This part is fairly simple; we want to run an HDP 3.1 cluster, and I'm naming it bmq-data-science.
"Blueprints": {
"blueprint_name": "bmq-data-science",
"stack_name": "HDP",
"stack_version": "3.1"
}
Host Components configuration
Similarly, I configured one Host with all the components needed for my services
"host_groups": [
{
"name": "master",
"cardinality": "1",
"components": [
{
"name": "ZOOKEEPER_SERVER"
},
{
"name": "NAMENODE"
},
{
"name": "SECONDARY_NAMENODE"
},
{
"name": "RESOURCEMANAGER"
},
{
"name": "HISTORYSERVER"
},
{
"name": "APP_TIMELINE_SERVER"
},
{
"name": "LIVY2_SERVER"
},
{
"name": "SPARK2_CLIENT"
},
{
"name": "SPARK2_JOBHISTORYSERVER"
},
{
"name": "ZEPPELIN_MASTER"
},
{
"name": "METRICS_GRAFANA"
},
{
"name": "METRICS_MONITOR"
},
{
"name": "DATANODE"
},
{
"name": "HIVE_SERVER"
},
{
"name": "HIVE_METASTORE"
},
{
"name": "HIVE_CLIENT"
},
{
"name": "YARN_CLIENT"
},
{
"name": "HDFS_CLIENT"
},
{
"name": "ZOOKEEPER_CLIENT"
},
{
"name": "TEZ_CLIENT"
},
{
"name": "NODEMANAGER"
},
{
"name": "MAPREDUCE2_CLIENT"
}
]
}
]
Services Configuration
For our purposes, I want to create a minimum cluster that will run YARN, HDFS, HIVE, SPARK and ZEPPELIN (plus all the necessary compute engines behind it). I therefore configured these services according to the Cloudbreak examples that are available in the default Cloudbreak blueprints:
"configurations": [
{
"yarn-site": {
"properties": {
"yarn.nodemanager.resource.cpu-vcores": "6",
"yarn.nodemanager.resource.memory-mb": "23296",
"yarn.scheduler.maximum-allocation-mb": "23296"
}
}
},
{
"core-site": {
"properties_attributes": {},
"properties": {
"fs.s3a.threads.max": "1000",
"fs.s3a.threads.core": "500",
"fs.s3a.max.total.tasks": "1000",
"fs.s3a.connection.maximum": "1500"
}
}
},
{
"capacity-scheduler": {
"properties": {
"yarn.scheduler.capacity.root.queues": "default",
"yarn.scheduler.capacity.root.capacity": "100",
"yarn.scheduler.capacity.root.maximum-capacity": "100",
"yarn.scheduler.capacity.root.default.capacity": "100",
"yarn.scheduler.capacity.root.default.maximum-capacity": "100"
}
}
},
{
"spark2-defaults": {
"properties_attributes": {},
"properties": {
"spark.sql.hive.hiveserver2.jdbc.url": "jdbc:hive2://%HOSTGROUP::master%:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2",
"spark.sql.hive.hiveserver2.jdbc.url.principal": "hive/_HOST@EC2.INTERNAL",
"spark.datasource.hive.warehouse.metastoreUri": "thrift://%HOSTGROUP::master%:9083",
"spark.datasource.hive.warehouse.load.staging.dir": "/tmp",
"spark.hadoop.hive.zookeeper.quorum": "%HOSTGROUP::master%:2181"
}
}
},
{
"hive-site": {
"hive.metastore.warehouse.dir": "/apps/hive/warehouse",
"hive.exec.compress.output": "true",
"hive.merge.mapfiles": "true",
"hive.server2.tez.initialize.default.sessions": "true",
"hive.server2.transport.mode": "http",
"hive.metastore.dlm.events": "true",
"hive.metastore.transactional.event.listeners": "org.apache.hive.hcatalog.listener.DbNotificationListener",
"hive.repl.cm.enabled": "true",
"hive.repl.cmrootdir": "/apps/hive/cmroot",
"hive.repl.rootdir": "/apps/hive/repl"
}
},
{
"hdfs-site": {
"properties_attributes": {},
"properties": {
}
}
}
]
You can find the complete blueprint and other recipes on my github, here
Creating a recipe
Recipes in Cloubreak are very useful and allow you to run scripts before a cluster is launched or after. In this example, I created a PRE-AMBARI-START recipe that creates the appropriate Postgres and MySQL services on my master box, as well as recreating the DB for my BMQ analysis:
#!/bin/bash # Cloudbreak-2.7.2 / Ambari-2.7.0 - something is install pgsq95 yum remove -y postgresql95* # Install pgsql96 yum install -y https://download.postgresql.org/pub/repos/yum/9.6/redhat/rhel-7-x86_64/pgdg-redhat96-9.6-3.noarch.rp... yum install -y postgresql96-server yum install -y postgresql96-contrib /usr/pgsql-9.6/bin/postgresql96-setup initdb sed -i 's,#port = 5432,port = 5433,g' /var/lib/pgsql/9.6/data/postgresql.conf systemctl enable postgresql-9.6.service systemctl start postgresql-9.6.service yum remove -y mysql57-community* yum remove -y mysql56-server* yum remove -y mysql-community* rm -Rvf /var/lib/mysql yum install -y epel-release yum install -y libffi-devel.x86_64 ln -s /usr/lib64/libffi.so.6 /usr/lib64/libffi.so.5 yum install -y mysql-connector-java* ambari-server setup --jdbc-db=mysql --jdbc-driver=/usr/share/java/mysql-connector-java.jar if [ $(cat /etc/system-release|grep -Po Amazon) == Amazon ]; then yum install -y mysql56-server service mysqld start else yum localinstall -y https://dev.mysql.com/get/mysql-community-release-el7-5.noarch.rpm yum install -y mysql-community-server systemctl start mysqld.service fi chkconfig --add mysqld chkconfig mysqld on ln -s /usr/share/java/mysql-connector-java.jar /usr/hdp/current/hive-client/lib/mysql-connector-java.jar ln -s /usr/share/java/mysql-connector-java.jar /usr/hdp/current/hive-server2-hive2/lib/mysql-connector-java.jar mysql --execute="CREATE DATABASE beast_mode_db DEFAULT CHARACTER SET utf8" mysql --execute="CREATE USER 'bmq_user'@'localhost' IDENTIFIED BY 'Be@stM0de'" mysql --execute="CREATE USER 'bmq_user'@'%' IDENTIFIED BY 'Be@stM0de'" mysql --execute="GRANT ALL PRIVILEGES ON beast_mode_db.* TO 'bmq_user'@'localhost'" mysql --execute="GRANT ALL PRIVILEGES ON beast_mode_db.* TO 'bmq_user'@'%'" mysql --execute="GRANT ALL PRIVILEGES ON beast_mode_db.* TO 'bmq_user'@'localhost' WITH GRANT OPTION" mysql --execute="GRANT ALL PRIVILEGES ON beast_mode_db.* TO 'bmq_user'@'%' WITH GRANT OPTION" mysql --execute="FLUSH PRIVILEGES" mysql --execute="COMMIT"
Section 2: Add blueprint and recipes via Cloudbreak interface
This part is super simple. Follow the User interface to load the files you just created, as depicted below
Adding a blueprint
Adding a recipe
Section 3: Automate cluster launch and terminate clusters
This is where the fun begins. For this part I created two scripts for launching and terminating the ephemeral cluster (that will then be called by the BMQ app). Both scripts rely on the cb cli that you can download from your CB instance:
Launch Cluster Script
As you will see below, the script is divided in the following part:
- Part 1: Reference the location of the cb-cli for ease of use
- Part 2: Dumps the content of my long lasting cluster to a recipe that will load them
- Part 3: Use CB api to add the recipe to my CB instance
- Part 4: Launch the cluster via cb-cli
#!/bin/bash
###############################
# 0. Initializing environment #
###############################
export PATH=$PATH:/Users/pvidal/Documents/Playground/cb-cli/
###################################################
# 1. Dumping current data and adding it to recipe #
###################################################
rm -rf poci-bmq-data-science.sh >/dev/null 2>&1
echo "mysql -u bmq_user -pBe@stM0de beast_mode_db --execute=\"""$(mysqldump -u bm_user -pHWseftw33# beast_mode_db 2> /dev/null)""\"" >> poci-bmq-data-science.sh
##################################
# 2. Adding recipe to cloudbreak #
##################################
TOKEN=$(curl -k -iX POST -H "accept: application/x-www-form-urlencoded" -d 'credentials={"username":"pvidal@hortonworks.com","password":"HWseftw33#"}' "https://192.168.56.100/identity/oauth/authorize?response_type=token&client_id=cloudbreak_shell&scope.0=openid&source=login&redirect_uri=http://cloudbreak.shell" | grep location | cut -d'=' -f 3 | cut -d'&' -f 1)
echo $TOKEN
ENCODED_RECIPE=$(base64 poci-bmq-data-science.sh)
curl -X DELETE https://192.168.56.100/cb/api/v1/recipes/user/poci-bmq-data-science -H "Authorization: Bearer $TOKEN" -k
curl -X POST https://192.168.56.100/cb/api/v1/recipes/user -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' -H 'cache-control: no-cache' -d " {
\"name\": \"poci-bmq-data-science\",
\"description\": \"Recipe loading BMQ data post BMQ cluster launch\",
\"recipeType\": \"POST_CLUSTER_INSTALL\",
\"content\": \"$POST_CLUSTER_INSTALL\"
}" -k
########################
# 3. Launching cluster #
########################
cb cluster create --cli-input-json tp-bmq-data-science.json --name bmq-data-science-$(date +%s)
Terminate Cluster Script
This script is much simpler; it uses cb cli to list the clusters running and terminate them:
#!/bin/bash
###############################
# 0. Initializing environment #
###############################
export PATH=$PATH:/Users/pvidal/Documents/Playground/cb-cli/
################################################
# 1. Get a list of clusters and terminate them #
################################################
cb cluster list | grep Name | awk -F \" '{print $4}' | while read cluster; do
echo "Terminating ""$cluster""..."
cb cluster delete --name $cluster
done
Conclusion
With this framework, I'm able to launch and terminate clusters in a matter of minutes, as depicted below. Next step will be to model a way to calculate accurate prediction for BMQ!
