Executing Shell script in HDFS

[#1]

Hi,

I have a shell script in HDFS echo.sh, which contains

echo Hello

I want to execute the shell script from Unix shell as below.

sh <HDFS_SCRIPT_PATH>/echo.sh fails saying no such file.

Kindly let me know how can i do it.

Regards

Mamta Chawla

[#2]

are you using variables in the absolute path ?

you can test out the location of the file first. I am assuming <HDFS_SCRIPT_PATH> must be a path which is available on the system, something like /home/hdfs/echo.sh. If you want to ensure that it exists, you can try listing it, like "ls /home/hdfs/echo.sh". If it says that there is no such file or directory, you need to have the correct path and locate the actual location of this file.

[#3]

yes, we can't submit/run shell scripts directly from local.

we need tp set up NFS mount and give a try.

link for setting up NFS mount

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.4/bk_hdfs_nfs_gateway/content/user-guide-hdfs...

[#4]

Ah, completely missed out the first line, indeed, we cannot execute a file thats inside HDFS.

[#5 · Accepted solution]

@Mamta Chawla

There are 3 ways.

1. hadoop fs -cat /tmp/test.sh|exec sh

2. You can install HDP NFS and mount the hdfs directory on local file system from where you can execute your script.

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.4/bk_hdfs_nfs_gateway/content/user-guide-hdfs...

3. You can write an oozie shell workflow and call your .sh HDFS file inside the workflow.

http://rogerhosto.com/apache-oozie-shell-script-example/

[#6]

Hi @Mamta Chawla,

Please feel free to accept an answer which helped you, so that this thread can be closed. Thanks