Support Questions
Find answers, ask questions, and share your expertise

oozie schedule shell script in parallel like cron jobs in Linux

Highlighted

oozie schedule shell script in parallel like cron jobs in Linux

Explorer

I have a shell script in HDFS. I want to schedule this in oozie

I am able to schedule the script in oozie and getting the desired output.

Now the problem:

I want to schedule this script 10 times in parallel. For this script the argument is table name. So, what I want is when I schedule this workflow, I want the shell script to run in parallel for 10 tables.

I was able to achieve this in Linux by using cron jobs but not in oozie.

What is the best way to do this?

Below is my shell script

#!/bin/bash
LOG_LOCATION=/home/$USER/logs
exec 2>&1
[ $# -ne 1 ] && { echo "Usage : $0 table ";exit 1; }
table=$1
TIMESTAMP=`date "+%Y-%m-%d"`
touch /home/$USER/logs/${TIMESTAMP}.success_log
touch /home/$USER/logs/${TIMESTAMP}.fail_log
success_logs=/home/$USER/logs/${TIMESTAMP}.success_log
failed_logs=/home/$USER/logs/${TIMESTAMP}.fail_log
#Function to get the status of the job creation 

function log_status
{
status=$1
message=$2
if [ "$status" -ne 0 ]; then
echo "`date +\"%Y-%m-%d %H:%M:%S\"` [ERROR] $message [Status] $status : failed" | tee -a "${failed_logs}"
#echo "Please find the attached log file for more details"
exit 1
else
echo "`date +\"%Y-%m-%d %H:%M:%S\"` [INFO] $message [Status] $status : success" | tee -a "${success_logs}"
fi
}


`hive -e "create table testing.${table} as select * from database.${table}"`

g_STATUS=$?
log_status $g_STATUS "Hive create ${table}"
1 REPLY 1

Re: oozie schedule shell script in parallel like cron jobs in Linux

Explorer

You can use fork and join to run the shell script parallel with different hive tables. You need to have ten shell action to run same shell script but pass different parameter.

You can reference this link for the fork and join

https://oozie.apache.org/docs/4.2.0/WorkflowFunctionalSpec.html#a3.1.5_Fork_and_Join_Control_Nodes

Don't have an account?