Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

values in Shell Script send as parameters in a spark Program in Oozie?

Highlighted

values in Shell Script send as parameters in a spark Program in Oozie?

Explorer

I need to pass values in a shell script to Spark as parameters.

 

Any idea how?

3 REPLIES 3
Highlighted

Re: values in Shell Script send as parameters in a spark Program in Oozie?

Champion

@ken328

 

I've explained below some generic idea about how to access hadoop, hive from shell. you may have to tweak it according to your requirement

 

## Shell script

## Arguments
schema_file1=$1
tbl_file1=$2
HDFS_DATA_PATH=$3
LOCAL_TEMP_PATH=$4

## Hadoop/hive binary path
export HADOOP_HOME1=/opt/cloudera/parcels/CDH-5.7.0-1.cdh5.7.0.p1465.1359/bin
export HIVE_HOME=/opt/cloudera/parcels/CDH-5.7.0-1.cdh5.7.0.p1465.1359/bin


## Execution
$HADOOP_HOME1/hadoop fs -ls $HDFS_DATA_PATH  >  $LOCAL_TEMP_PATH/$tmp_file1
$HIVE_HOME/hive -S -e "drop table if exists $schema_file1.$tbl_file1;"
Highlighted

Re: values in Shell Script send as parameters in a spark Program in Oozie?

Explorer
That doesnt help me with the Oozie part of my problem.

Highlighted

Re: values in Shell Script send as parameters in a spark Program in Oozie?

Explorer

I've actually found out that using ${wf:actionData} is the way to go, however once it reaches the next transition it resolves to null. Anyone know how to successfully pass the parameter?

Don't have an account?
Coming from Hortonworks? Activate your account here