Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

copyToLocal from shell script doesn't work

Highlighted

copyToLocal from shell script doesn't work

New Contributor

Hi,

I am having a strange issue.

I need to run few hdfs commands from a shell script. Just to test, I created a simple test.bash file, within that I am running the following command

hdfs dfs -copyToLocal hdfs://namenode/dir1/file /tmp/file1

When I execute the shell script and check the status of the command run using $?, I get the status as 0. However, no file is present under /tmp

If I the command directly in a shell prompt, it completes successfully and get the file1 into /tmp

Not sure where its getting wrong. Should be pretty simple, but not able to figure out whats going on here.

Another interesting factor is when I rerun the shell script, I get an error message that "file already exists"

Any pointers?

Thanks

4 REPLIES 4

Re: copyToLocal from shell script doesn't work

Expert Contributor

Not sure either. I just ran a test. It worked for me.

#!/bin/bash

hdfs dfs -copyToLocal hdfs://sandbox.hortonworks.com/user/admin/mydata.csv /tmp/mydata.csv
[root@sandbox ~]# ls -la /tmp/mydata.csv
-rw-r--r-- 1 root root 35 Nov  4 18:42 /tmp/mydata.csv 

Re: copyToLocal from shell script doesn't work

Guru

Maybe try this way

hdfs dfs -copyToLocal /dir1/file /tmp/file1

Re: copyToLocal from shell script doesn't work

@Greenhorn Techie

Ideally it should work, Because I am also using the script for data copy from local to hadoop and hadoop to local.

i.e

cat datacopy.sh

#!/bin/bash

echo "#######################################################"

echo `date +%Y%m%d%c` >> /xyz/datacopy_log.out

echo "#######################################################"

ls -ltr /xyz/ |awk '{print $9}' > /xyz/files.log

for i in `cat /xyz/files.log`; do hadoop fs -copyToLocal /xyz/$i /abc/;done

In this script first I am taking the file name into one file and then copying file from local to hadoop. And this is working script.

If copyToLocal is not working then try with get command.

hadoop fs -get $Local_source $Hadoop_destination.

Re: copyToLocal from shell script doesn't work

Super Guru

can you run an echo to see if the name is right. who is running the script, is there permissions or are you running through cron? do you see your logs?