question Executing Shell script in HDFS in Archives of Support Questions (Read Only)

Executing Shell script in HDFS

colorsoflife — Thu, 26 May 2016 02:19:02 GMT

Hi,

I have a shell script in HDFS echo.sh, which contains

echo Hello

I want to execute the shell script from Unix shell as below.

sh <HDFS_SCRIPT_PATH>/echo.sh fails saying no such file.

Kindly let me know how can i do it.

Regards

Mamta Chawla

Re: Executing Shell script in HDFS

srai1 — Thu, 26 May 2016 02:22:36 GMT

are you using variables in the absolute path ?

you can test out the location of the file first. I am assuming <HDFS_SCRIPT_PATH> must be a path which is available on the system, something like /home/hdfs/echo.sh. If you want to ensure that it exists, you can try listing it, like "ls /home/hdfs/echo.sh". If it says that there is no such file or directory, you need to have the correct path and locate the actual location of this file.

Re: Executing Shell script in HDFS

divakarreddy_a — Thu, 26 May 2016 02:24:21 GMT

yes, we can't submit/run shell scripts directly from local.

we need tp set up NFS mount and give a try.

link for setting up NFS mount

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.4/bk_hdfs_nfs_gateway/content/user-guide-hdfs-nfs-instructions.html

Re: Executing Shell script in HDFS

srai1 — Thu, 26 May 2016 02:34:12 GMT

Ah, completely missed out the first line, indeed, we cannot execute a file thats inside HDFS.

Re: Executing Shell script in HDFS

jyadav — Thu, 26 May 2016 02:35:20 GMT

@Mamta Chawla

There are 3 ways.

1. hadoop fs -cat /tmp/test.sh|exec sh

2. You can install HDP NFS and mount the hdfs directory on local file system from where you can execute your script.

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.4/bk_hdfs_nfs_gateway/content/user-guide-hdfs-nfs-instructions.html

3. You can write an oozie shell workflow and call your .sh HDFS file inside the workflow.

http://rogerhosto.com/apache-oozie-shell-script-example/

Re: Executing Shell script in HDFS

jyadav — Thu, 26 May 2016 22:45:45 GMT

Hi @Mamta Chawla,

Please feel free to accept an answer which helped you, so that this thread can be closed. Thanks