question Re: how to execute oozie shell action with script having curl command in Support Questions

how to execute oozie shell action with script having curl command

srujanakuntumal — Wed, 09 Dec 2015 08:44:45 GMT

Hi,

I need to create oozie workflow that exeuctes a shell script. The shell script has curl command which downloads a specific file from the download link.

As commands in shell scripts are only able to recognize hdfs directories, how could i execute the script.?

Lets say below is the Sample code:

curl -o ~/test.jar http://central.maven.org/maven2/commons-lang/commons-lang/2.6/commons-lang-2.6.jar

hdfs dfs -copyFromLocal ~/test.jar /user/sr/test2

How can i execute the script with above two commands using oozie.?

Re: how to execute oozie shell action with script having curl command

KuldeepK — Wed, 09 Dec 2015 22:51:34 GMT

Hi Srujana,

I have attached sample shell action oozie coordinator code for your reference.

Below are the contents of attached tar file

1. coordinator.xml - contains your scheduling frequency

2. workflow.xml - contains workflow actions

3. myscript.sh - sample shell script ( here your hbase major compaction command will go )

4. job.properties - contains environmental details - you need to modify this file to add your cluster environment details

Once you are done with modifications, upload myscript.sh, workflow.xml & coordinator.xml to HDFS at ${oozie.coord.application.path} location.

Run below command to submit coordinator:

oozie job -oozie http://:11000/oozie>:11000/oozie -config /$path-to-job.properties/job.properties -run

sample-shell-oozie.tar.gz

Note - Add your above commands to myscript.sh

Re: how to execute oozie shell action with script having curl command

sathish_mtech01 — Mon, 18 Jan 2016 20:38:04 GMT

@ Kulkarni while running your script I am getting the following error in horton works

sudo oozie job -oozie http://XXXXXXXXXXXXXXXXXXXXXX:11000/oozie -config /home/ubuntu/testing/shell/job.properties –submit

While running I am getting following error

Error: E0803 : E0803: IO error, E0603: SQL error in operation, <openjpa-2.2.2-r422266:1468616 fatal store error> org.apache.openjpa.persistence.RollbackException: The transaction has been rolled back. See the nested exceptions for details on the errors that occurred. FailedObject: org.apache.oozie.CoordinatorJobBean@5ee81cbc

Re: how to execute oozie shell action with script having curl command

sathish_mtech01 — Mon, 18 Jan 2016 21:41:35 GMT

I fixed it finally by

As oozie uses derby db - checking the db location to see if there are any lock files

cd /hadoop/oozie/data/oozie-db

ubuntu@ixxxxxxxxx:/hadoop/oozie/data/oozie-db$ ls

dbex.lck db.lck log README_DO_NOT_TOUCH_FILES.txt seg0 service.properties tmp

rm dbex.lck

rm db.lck

Then finally went to Ambari UI hortonworks restarted the oozie, works finally..

Futher information please relay on a single user to avoid problem in future purpose...