Am I able to copy a file from my mac (/Users/rev/geolocation.csv) to the HDFS (/tmp/admin/data) via SSH?

[#1]

Hi, I am interested in doing the HDP Certified Developer (HDPCD) Exam. As such I am doing some background work. Within the link (http://hortonworks.com/training/class/hdp-certified-developer-hdpcd-exam/) there are a set of tasks that I was attempting. The first task ‘Input a local file into HDFS using the Hadoop file system shell’ was one I attempted. I can do this via Ambari by going to the HDFS section and simply uploading the file by pressing the browse button and selecting the data file. However I want to do this via command line. Is there anyway I can do this?

My data is stored on my mac /Users/rev/geolocation.csv. I have created a directory in HDFS located: tmp/admin/data. This is where I will copy the geolocation.csv to.

Via SSH (http://127.0.0.1:4200/) I have tried commands like the one below which works fine.

hadoop fs -ls tmp/admin/data 

However, when I type the below I get a no such file or directory message.

hadoop fs -put localfile /Users/rev/geolocation.csv /tmp/admin/data 

Am I able to copy a file from my mac (/Users/rev/geolocation.csv) to the HDFS (/tmp/admin/data)?

[#2 · Accepted solution]

@Revlin Abbi

via SSH, you first have to copy the files to the guest machine, (Sandbox) like so

scp -P 2222 -r file root@127.0.0.1:/home/guest

then when you ssh to the machine, the file will be in /home/guest directory

switch to guest user

su guest

if you do ls on the home directory you will see your file

now you need to upload the file to HDFS

hdfs dfs -put file /user/guest

now you can list the file in hdfs

hdfs dfs -ls /user/guest/

alternatively, you can use the Ambari HDFS Files view to upload directly from your mac to HDFS in Sandbox. The one thing you need to remember is that when you copy data from host to guest you copy to the guest's local filesystem. Loading to HDFS will take another step.

[#3]

Hi Artem, thank you for help.