Unable to copy from local to HDSF

[#1]

Hi

I am a brand new user. I have installed Hortonworks Sandbox on my VM virtual box manager.

I have logged into Ambari 127.0.0.1 using maria_dev user name.

I have installed Putty and set up by using a connection with maria_dev@sandbox-hdp.

I cannot copy a file from my local directory to HDFS. Do I need to set permissions or have I missed a step in the set up process?

Any assistance would be greatly appreciated.

Thanks

[#2]

@ Jay Kumar SenSharma

thank you for your reply.

From Putty I used your command above:

ssh root@127.0.0.1  -p 2222

And received the following error:

ssh: connect to host 127.0.0.1 port 2222: Connection refused

Can you also please explain what you mean by "you will need to SCP your files to Sandbox and then from Sandbox you can put them to HDFS"?

[#3]

@Matt

I am not sure from putty how you entered that command "ssh root@127.0.0.1 -p 2222"

But the best way to connect to Sandbox is described in the following Tutorial which might be really helpful. https://hortonworks.com/tutorial/learning-the-ropes-of-the-hortonworks-sandbox/#terminal-access

.

Usually if you want to put a file from Windows to Linux host then WinScp kind of UI utilities are the best if you want to do SCP.

[#4]

@Matt

Using WInSCP once we are able to copy the file to Sandbox then we should be able to run the following commands to push those files to HDFS from Sandbox.

Suppose in Sandbox host the file is present at "/toot/sample.txt" then it can be pushed to HDFS as following:

.

You will need to do a SSH to the Sandbox first and then run the following commands:

# su - maria_dev
# hadoop fs -copyFromLocal  /root/sample.txt  /user/maria_dev/
(OR)
# hdfs dfs -put  /root/sample.txt  /user/maria_dev/
# hdfs dfs -ls  /user/maria_dev/

NOTE: the user "maria_dev" should have read permission atleast on the file "/root/sample.txt" before it tried to read/push it to HDFS.

.

[#5]

@Matt

If this issue is resolved then it will be also great if you can mark this HCC thread as Answered by clicking on the "Accept" Button on the correct answer. That way other HCC users can quickly find the solution when they encounter the same issue.

[#6]

@ Jay Kiumar SenSharma

ok thank you for your help anyway. I appreciate you replying to me even though I have had no success with it.

[#7]

@Matt

Heya. @Jay Kumar SenSharma's is right on point, though it sounds like there's a bit of confusion as to what Hadoop's "copyFromLocal" command does and what it expects.

You run the Hadoop command with the option "copyFromLocal" from within the sandbox. This command will look on the Sandbox's local filesystem and copy the files you specified into the Hadoop filesystem.

I can definitely see how this may be a bit confusing - it sort of sounds like the command would pull in files from your machine into HDFS on the sandbox. Remember, though, that the sandbox exists on a virtual machine that by default is separate from your own and can't access your filesystem.

So you can definitely use that command, but you'll first have to move your local files into the sandbox. There are a bunch of ways to do that in bulk, check out Jay's post regarding SCP.

Hope that clarifies a bit, and good luck! 🙂

[#8]

@Matt,

Hope you were successful in copying your files to HDFS.

For any other "Getting Started Questions" you might have please check out the Learning the Ropes Hortonworks Tutorial.