question HDPCD practice exam task 1, how to import data into HDFS? in Archives of Support Questions (Read Only)

HDPCD practice exam task 1, how to import data into HDFS?

senquan_chen — Fri, 16 Sep 2022 13:35:33 GMT

I am having problem to create AWS EC2 because of the limitation and amazon is still processing my request.

So i use hortonworks Sandbox + Virtual box... i tried to upload the "fligthdelays" data into local, and then use " hdfs dfs -copyFromLocal"

But i actually i am confused about "local"....does this local mean... after connecting to maria_dev@localhost? but how can i upload files to home/maria_dev/datasets? even i tried to crate datasets by using ambri, but i can't find files from /home/maria_dev...... i am very confused how to write Task 1 by using sandbox....

Re: HDPCD practice exam task 1, how to import data into HDFS?

stevenmatison — Tue, 14 Aug 2018 23:56:50 GMT

@seninus The Copy To and Copy From HDFS syntax is as follows:

hdfs dfs -copyFromLocal /local/folder/file.txt /hdfs/folder/

hdfs dfs -copyToLocal /hdfs/folder/file2.txt /local/folder/

where /local/ is the local file system and /hdfs/ is the hdfs file system

It is also important to note, you want to execute those commands as hdfs user so:

sudo su - hdfs

sudo su - hdfs -c "hdfs dfs -copyFromLocal /local/folder/file.txt /hdfs/folder/"

If this answer is helpful please choose ACCEPT.

Re: HDPCD practice exam task 1, how to import data into HDFS?

senquan_chen — Wed, 15 Aug 2018 00:25:14 GMT

You mean I have to install local hadoop(hdfs) at my Mac terminal first? and then use sudo su - hdfs at my Mac terminal? I think I installed hadoop( havn't configured, it seems to be a long process), and when I tried "sh - hdfs ", the Mac terminal gave back " su: unknown login: hdfs" to me...

Re: HDPCD practice exam task 1, how to import data into HDFS?

stevenmatison — Wed, 15 Aug 2018 00:38:41 GMT

@seninus those hdfs commands are commands to execute using terminal prompt on the sandbox node, not your mac... copy LOCAL is from sandbox node to hdfs, NOT your local mac

Re: HDPCD practice exam task 1, how to import data into HDFS?

stevenmatison — Wed, 15 Aug 2018 00:39:48 GMT

additionally that sandbox probably has a local (to the mac) folder path mounted on the sandbox file system. You would need to use that path to get files from your mac to the sandbox, then sandbox to hdfs

Re: HDPCD practice exam task 1, how to import data into HDFS?

senquan_chen — Sun, 18 Aug 2019 03:38:29 GMT

Yes, this is what I understood and was trying to do. but I uploaded flightdelays*.csv files to /datasets/flightdelays/ through Ambari. but I can't see local files when I login with maria_dev in terminal...

Re: HDPCD practice exam task 1, how to import data into HDFS?

stevenmatison — Wed, 15 Aug 2018 01:35:15 GMT

now you just need to do the copy from hdfs to local filesystem:

sudo su - hdfs -c "hdfs dfs -copyToLocal /datasets /tmp"

mv /tmp/datasets /home/maria_dev

This last command because hdfs user cant write to /home/maria_dev. So write hdfs to tmp, move from /tmp to /home/maria_dev

Re: HDPCD practice exam task 1, how to import data into HDFS?

senquan_chen — Wed, 15 Aug 2018 03:37:25 GMT

Thanks for the tips. yeah I found the similar questions and solved it. Thanks

https://community.hortonworks.com/questions/9371/am-i-able-to-copy-a-file-from-my-mac-usersrevgeolo.html

Re: HDPCD practice exam task 1, how to import data into HDFS?

stevenmatison — Wed, 15 Aug 2018 21:39:21 GMT

@seninus glad you got it working. Please click accept on the main answer please, it helps close the question and gives me some reputation points. ;O)