Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

[Closed] : How to store output of shell script in HDFS

avatar
Contributor

HI All,
I have a shell script on HDFS as well as the locally named script.sh contains echo Hi.

I could execute the script.sh file on locally and store output on locally of course.

But I want to execute script.sh file (wherever on local or on HDFS) and store output on HDFS.

I have done following;

script.sh

#!/bin/bash
echo "`date` hi" > /tmp/output

bash script.sh

above command ran successfully. but if I changed the output path it is giving me an error that ;

script.sh: line 2: hdfs://<host>:<port>/user/oozie/output/shell: No such file or directory
#!/bin/bash 
echo "`date` hi" > hdfs://<HOST>:<PORT>/user/oozie/output/shell

78444-shell.jpg

Kindly help me for this.

Thank you,

Jay.

1 ACCEPTED SOLUTION

avatar
Super Guru

@JAy PaTel,

You cannot directly write the output of echo to hdfs file. Instead you can do like below

echo "`date` hi" > /tmp/output ; hdfs dfs -appendToFile /tmp/output /tmp/abc.txt

.

-Aditya

View solution in original post

4 REPLIES 4

avatar
Super Guru

@JAy PaTel,

You cannot directly write the output of echo to hdfs file. Instead you can do like below

echo "`date` hi" > /tmp/output ; hdfs dfs -appendToFile /tmp/output /tmp/abc.txt

.

-Aditya

avatar
Contributor
@Aditya Sirna

Thank you Aditya. Your observation worked for me.

NOTE:

echo "`date` hi">/tmp/output ; hdfs dfs -appendToFile <local_directory_path> <hdfs_directory_path>

Regards,

Jay.

avatar
Explorer

This can be done without first writing to local disk...

echo "`date` hi" | hdfs dfs -appendToFile - /tmp/abc.txt

The hyphen "-" reads from stdin

avatar

Awesome!!!