how to execute oozie shell action with script having curl command

[#1]

Hi,

I need to create oozie workflow that exeuctes a shell script. The shell script has curl command which downloads a specific file from the download link.

As commands in shell scripts are only able to recognize hdfs directories, how could i execute the script.?

Lets say below is the Sample code:

curl -o ~/test.jar http://central.maven.org/maven2/commons-lang/commons-lang/2.6/commons-lang-2.6.jar

hdfs dfs -copyFromLocal ~/test.jar /user/sr/test2

How can i execute the script with above two commands using oozie.?

[#2 · Accepted solution]

Hi Srujana,

I have attached sample shell action oozie coordinator code for your reference.

Below are the contents of attached tar file

1. coordinator.xml - contains your scheduling frequency

2. workflow.xml - contains workflow actions

3. myscript.sh - sample shell script ( here your hbase major compaction command will go )

4. job.properties - contains environmental details - you need to modify this file to add your cluster environment details

Once you are done with modifications, upload myscript.sh, workflow.xml & coordinator.xml to HDFS at ${oozie.coord.application.path} location.

Run below command to submit coordinator:

oozie job -oozie http://:11000/oozie>:11000/oozie -config /$path-to-job.properties/job.properties -run

sample-shell-oozie.tar.gz

Note - Add your above commands to myscript.sh

[#3]

@ Kulkarni while running your script I am getting the following error in horton works

sudo oozie job -oozie http://XXXXXXXXXXXXXXXXXXXXXX:11000/oozie -config /home/ubuntu/testing/shell/job.properties –submit

While running I am getting following error

Error: E0803 : E0803: IO error, E0603: SQL error in operation, <openjpa-2.2.2-r422266:1468616 fatal store error> org.apache.openjpa.persistence.RollbackException: The transaction has been rolled back. See the nested exceptions for details on the errors that occurred. FailedObject: org.apache.oozie.CoordinatorJobBean@5ee81cbc

[#4]

I fixed it finally by

As oozie uses derby db - checking the db location to see if there are any lock files

cd /hadoop/oozie/data/oozie-db

ubuntu@ixxxxxxxxx:/hadoop/oozie/data/oozie-db$ ls

dbex.lck db.lck log README_DO_NOT_TOUCH_FILES.txt seg0 service.properties tmp

rm dbex.lck

rm db.lck

Then finally went to Ambari UI hortonworks restarted the oozie, works finally..

Futher information please relay on a single user to avoid problem in future purpose...