Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Executing Shell script in HDFS

Solved Go to solution

Executing Shell script in HDFS

Hi,

I have a shell script in HDFS echo.sh, which contains

echo Hello

I want to execute the shell script from Unix shell as below.

sh <HDFS_SCRIPT_PATH>/echo.sh fails saying no such file.

Kindly let me know how can i do it.

Regards

Mamta Chawla

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: Executing Shell script in HDFS

@Mamta Chawla

There are 3 ways.

1. hadoop fs -cat /tmp/test.sh|exec sh

2. You can install HDP NFS and mount the hdfs directory on local file system from where you can execute your script.

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.4/bk_hdfs_nfs_gateway/content/user-guide-hdfs...

3. You can write an oozie shell workflow and call your .sh HDFS file inside the workflow.

http://rogerhosto.com/apache-oozie-shell-script-example/

View solution in original post

5 REPLIES 5
Highlighted

Re: Executing Shell script in HDFS

Guru

are you using variables in the absolute path ?

you can test out the location of the file first. I am assuming <HDFS_SCRIPT_PATH> must be a path which is available on the system, something like /home/hdfs/echo.sh. If you want to ensure that it exists, you can try listing it, like "ls /home/hdfs/echo.sh". If it says that there is no such file or directory, you need to have the correct path and locate the actual location of this file.

Highlighted

Re: Executing Shell script in HDFS

yes, we can't submit/run shell scripts directly from local.

we need tp set up NFS mount and give a try.

link for setting up NFS mount

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.4/bk_hdfs_nfs_gateway/content/user-guide-hdfs...

Highlighted

Re: Executing Shell script in HDFS

Guru

Ah, completely missed out the first line, indeed, we cannot execute a file thats inside HDFS.

Highlighted

Re: Executing Shell script in HDFS

@Mamta Chawla

There are 3 ways.

1. hadoop fs -cat /tmp/test.sh|exec sh

2. You can install HDP NFS and mount the hdfs directory on local file system from where you can execute your script.

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.4/bk_hdfs_nfs_gateway/content/user-guide-hdfs...

3. You can write an oozie shell workflow and call your .sh HDFS file inside the workflow.

http://rogerhosto.com/apache-oozie-shell-script-example/

View solution in original post

Highlighted

Re: Executing Shell script in HDFS

Hi @Mamta Chawla,

Please feel free to accept an answer which helped you, so that this thread can be closed. Thanks

Don't have an account?
Coming from Hortonworks? Activate your account here