question [Closed] : How to store output of shell script in HDFS in Support Questions

[Closed] : How to store output of shell script in HDFS

pateljay — Sun, 18 Aug 2019 00:51:59 GMT

HI All,
I have a shell script on HDFS as well as the locally named script.sh contains echo Hi.

I could execute the script.sh file on locally and store output on locally of course.

But I want to execute script.sh file (wherever on local or on HDFS) and store output on HDFS.

I have done following;

script.sh

#!/bin/bash
echo "`date` hi" > /tmp/output

bash script.sh

above command ran successfully. but if I changed the output path it is giving me an error that ;

script.sh: line 2: hdfs://<host>:<port>/user/oozie/output/shell: No such file or directory

#!/bin/bash 
echo "`date` hi" > hdfs://<HOST>:<PORT>/user/oozie/output/shell

Kindly help me for this.

Thank you,

Jay.

asirna — Fri, 15 Jun 2018 18:58:54 GMT

You cannot directly write the output of echo to hdfs file. Instead you can do like below

echo "`date` hi" > /tmp/output ; hdfs dfs -appendToFile /tmp/output /tmp/abc.txt

-Aditya

pateljay — Sat, 16 Jun 2018 13:28:41 GMT

Thank you Aditya. Your observation worked for me.

NOTE:

echo "`date` hi">/tmp/output ; hdfs dfs -appendToFile <local_directory_path> <hdfs_directory_path>

Regards,

Jay.

whitsonk — Fri, 28 Dec 2018 00:04:31 GMT

This can be done without first writing to local disk...

echo "`date` hi" | hdfs dfs -appendToFile - /tmp/abc.txt

The hyphen "-" reads from stdin

manishsingh2k — Fri, 08 Feb 2019 02:18:07 GMT

Awesome!!!