Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Executing Shell script in HDFS

avatar
Rising Star

Hi,

I have a shell script in HDFS echo.sh, which contains

echo Hello

I want to execute the shell script from Unix shell as below.

sh <HDFS_SCRIPT_PATH>/echo.sh fails saying no such file.

Kindly let me know how can i do it.

Regards

Mamta Chawla

1 ACCEPTED SOLUTION

avatar
Super Guru
@Mamta Chawla

There are 3 ways.

1. hadoop fs -cat /tmp/test.sh|exec sh

2. You can install HDP NFS and mount the hdfs directory on local file system from where you can execute your script.

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.4/bk_hdfs_nfs_gateway/content/user-guide-hdfs...

3. You can write an oozie shell workflow and call your .sh HDFS file inside the workflow.

http://rogerhosto.com/apache-oozie-shell-script-example/

View solution in original post

5 REPLIES 5

avatar
Guru

are you using variables in the absolute path ?

you can test out the location of the file first. I am assuming <HDFS_SCRIPT_PATH> must be a path which is available on the system, something like /home/hdfs/echo.sh. If you want to ensure that it exists, you can try listing it, like "ls /home/hdfs/echo.sh". If it says that there is no such file or directory, you need to have the correct path and locate the actual location of this file.

avatar

yes, we can't submit/run shell scripts directly from local.

we need tp set up NFS mount and give a try.

link for setting up NFS mount

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.4/bk_hdfs_nfs_gateway/content/user-guide-hdfs...

avatar
Guru

Ah, completely missed out the first line, indeed, we cannot execute a file thats inside HDFS.

avatar
Super Guru
@Mamta Chawla

There are 3 ways.

1. hadoop fs -cat /tmp/test.sh|exec sh

2. You can install HDP NFS and mount the hdfs directory on local file system from where you can execute your script.

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.4/bk_hdfs_nfs_gateway/content/user-guide-hdfs...

3. You can write an oozie shell workflow and call your .sh HDFS file inside the workflow.

http://rogerhosto.com/apache-oozie-shell-script-example/

avatar
Super Guru

Hi @Mamta Chawla,

Please feel free to accept an answer which helped you, so that this thread can be closed. Thanks