[Closed] : How to store output of shell script in HDFS

[#1]

HI All,
I have a shell script on HDFS as well as the locally named script.sh contains echo Hi.

I could execute the script.sh file on locally and store output on locally of course.

But I want to execute script.sh file (wherever on local or on HDFS) and store output on HDFS.

I have done following;

script.sh

#!/bin/bash
echo "`date` hi" > /tmp/output

bash script.sh

above command ran successfully. but if I changed the output path it is giving me an error that ;

script.sh: line 2: hdfs://<host>:<port>/user/oozie/output/shell: No such file or directory

#!/bin/bash 
echo "`date` hi" > hdfs://<HOST>:<PORT>/user/oozie/output/shell

Kindly help me for this.

Thank you,

Jay.

[#2 · Accepted solution]

@JAy PaTel,

You cannot directly write the output of echo to hdfs file. Instead you can do like below

echo "`date` hi" > /tmp/output ; hdfs dfs -appendToFile /tmp/output /tmp/abc.txt

.

-Aditya

[#3]

@Aditya Sirna

Thank you Aditya. Your observation worked for me.

NOTE:

echo "`date` hi">/tmp/output ; hdfs dfs -appendToFile <local_directory_path> <hdfs_directory_path>

Regards,

Jay.

[#4]

This can be done without first writing to local disk...

echo "`date` hi" | hdfs dfs -appendToFile - /tmp/abc.txt

The hyphen "-" reads from stdin

[#5]

Awesome!!!