Reply
New Contributor
Posts: 3
Registered: ‎03-08-2017

values in Shell Script send as parameters in a spark Program in Oozie?

I need to pass values in a shell script to Spark as parameters.

 

Any idea how?

Posts: 228
Topics: 11
Kudos: 32
Solutions: 20
Registered: ‎09-02-2016

Re: values in Shell Script send as parameters in a spark Program in Oozie?

@ken328

 

I've explained below some generic idea about how to access hadoop, hive from shell. you may have to tweak it according to your requirement

 

## Shell script

## Arguments
schema_file1=$1
tbl_file1=$2
HDFS_DATA_PATH=$3
LOCAL_TEMP_PATH=$4

## Hadoop/hive binary path
export HADOOP_HOME1=/opt/cloudera/parcels/CDH-5.7.0-1.cdh5.7.0.p1465.1359/bin
export HIVE_HOME=/opt/cloudera/parcels/CDH-5.7.0-1.cdh5.7.0.p1465.1359/bin


## Execution
$HADOOP_HOME1/hadoop fs -ls $HDFS_DATA_PATH  >  $LOCAL_TEMP_PATH/$tmp_file1
$HIVE_HOME/hive -S -e "drop table if exists $schema_file1.$tbl_file1;"
New Contributor
Posts: 3
Registered: ‎03-08-2017

Re: values in Shell Script send as parameters in a spark Program in Oozie?

That doesnt help me with the Oozie part of my problem.

Highlighted
New Contributor
Posts: 3
Registered: ‎03-08-2017

Re: values in Shell Script send as parameters in a spark Program in Oozie?

I've actually found out that using ${wf:actionData} is the way to go, however once it reaches the next transition it resolves to null. Anyone know how to successfully pass the parameter?

Announcements