Unable to copy from local to HDSF

[#1]

Hi

I am a brand new user. I have installed Hortonworks Sandbox on my VM virtual box manager.

I have logged into Ambari 127.0.0.1 using maria_dev user name.

I have installed Putty and set up by using a connection with maria_dev@sandbox-hdp.

I cannot copy a file from my local directory to HDFS. Do I need to set permissions or have I missed a step in the set up process?

Any assistance would be greatly appreciated.

Thanks

[#2]

@Matthew May

Can you please share the exact command and the error that you are getting while copying a file from local to HDFS?

[#3]

@ Jay Kumar SenSharma

I have a file on my desktop called sample.txt (in location Users/Matt/dir3/sample.txt.

I have tried this:

hadoop fs -copyFromLocal /Users/Matt/dir3/sample.txt /user/maria_dev/ and receive the error:

copyFromLocal: `/Users/Matt/dir3/sample.txt': No such file or directory

[#4]

Based on the error it looks like you are trying to Push the files from your MAC desktop files to HDP Sandbox HDFS cluster. Please correct me if i am wrong.

copyFromLocal: `/Users/Matt/dir3/sample.txt': No such file or directory

.

Please check few things:

1. Is this file "/Users/Matt/dir3/sample.txt" existing and has correct permission ? so that the user who is running "hadoop" command has proper read access to this file? Please share the output of the following file.

# ls -l /Users/Matt/dir3/sample.txt

2. If you just want to Put the local files to HDFS then another simple approach will be to use Ambari File View.

.

.

Additionally if you want to Put a file inside "/user/maria_dev/" HDFS directory then the user who is running this hadoop command must belong to "hadoop" group (OR) should have username "maria_dev" because the HDFS directory has the following permission.

[root@sandbox /]# su - hdfs -c "hadoop fs -ls /user" | grep maria_dev
drwxr-xr-x   - maria_dev hadoop          0 2017-10-21 11:01 /user/maria_dev

.

[#5]

Additionally, In addition to the permissions issue. If you want to run the "hadoop" client commands as you posted from your local machine then you will need to make sue that your local machine is setup as a Hadoop Client machine (means it should have all the hadoop libraries + core-site.xml and hdfs-site.xml) files.

.

However as you are using Hortonworks Sandbox hence it will be really good and easy to Put to files to HDFS from your laptop using Ambari File View. Please see: https://hortonworks.com/tutorial/hadoop-tutorial-getting-started-with-hdp/section/2/

[#6]

@ Jay Kumar SenSharma

you are correct, I am trying to push the file from my (Windows) desktop to Sandbox. The file does exist, however I am unsure if the correct permissions have been set? The output is:

I understand that I can use Ambari File View, thank you, however my desire is to upload multiple files at once from a directory (the sample file is a test).

ls: cannot access /Users/Matt/dir3/sample.txt: No such file or directory

[#7]

@ Jay Kumar SenSharma

Could you please explain how the user who is running hadoop command can belong to "hadoop" group?

[#8]

If you want to run the "hadoop" client commands as you posted from your Windows machine then you will need to make sue that your local machine is setup as a Hadoop Client machine (means it should have all the hadoop libraries + core-site.xml and hdfs-site.xml) files.

.

However as you are using Hortonworks Sandbox hence it will be really good and easy to Put to files to HDFS from your laptop using Ambari File View. Please see: https://hortonworks.com/tutorial/hadoop-tutorial-getting-started-with-hdp/section/2/

.

[#9]

@ Jay Kumar SenSharma

thank you, however I wish to upload multiple files at once from a directory. In Ambari File View, I can only upload a single file if I understand correctly.

[#10]

Regarding your query: "Could you please explain how the user who is running hadoop command can belong to "hadoop" group?"

.

For Windows environment i can not help much. However in Linux based environment you can simple use the to add "testuser" to hadoop group in linux based environment.

# sudo adduser --ingroup hadoop testuser

.

If you really want to test the "copyFromLocal" command then you should do it inside the Sandbox instance as it has "hadoop" group already present in it and all the requierd libraries.

Example:

Login to Snadbox using SSH session on port 2222 (this port must be used instead of default SSH port). Or while using Putty please define the SSH port as 2222

# ssh root@127.0.0.1  -p 2222

.

Once you are inside the Sandbox terminal then you try running the command:

# su - maria_dev
# hadoop fs -copyFromLocal /etc/passwd /user/maria_dev/

.

Instead of file "/etc/passwd" you can push your own files to HDFS. You will need to just SCP your files to Sandbox and then from Sandbox you can put them to HDFS.

Or use Ambari File view to directly post your files to HDFS.