question Re: Copy file from Windows to sandbox hosted in Azure in Archives of Support Questions (Read Only)

Copy file from Windows to sandbox hosted in Azure

praskutti — Fri, 16 Sep 2022 11:10:13 GMT

I was successfuly able to copy a file to Azure from Windows using Putty Copy (pscp). I used the following command

pscp - P 22 <file from local folder> tmp\

I could see that file in the sandbox host. Now when i tried to "put" the file from sandbox host to the sandbox itself (docker container), i am not able to do so (throws network not accessible error). The command below

hadoop fs -ls -put \tmp\onecsvfile.csv \tmp\

hadoop fs -ls -put \tmp\onecsvfile.csv root@localhost:2222\\tmp\

Nothing works 😞

Can you tell me how can i move the file from sandbox host to the sandbox (Sandbox 2.5 hosted in Azure)?

Re: Copy file from Windows to sandbox hosted in Azure

darvasip — Tue, 28 Feb 2017 23:01:00 GMT

Hi @Prasanna G!

You have the following options:

1. You can use "docker cp" if you have only one file. Here is the documentation with examples.

2. The most flexible solution is mounting local drives to the sandbox with the -v option of "docker run". This is preferred because all the content in the folder is accessible inside the container.

docker run -v /Users/<path>:/<container path> ...

You can find an example how to configure this for Sandbox here.

Re: Copy file from Windows to sandbox hosted in Azure

drice1 — Tue, 28 Feb 2017 23:14:04 GMT

Hi @Prasanna G

You'll have to first copy the file from the local filesystem into the docker container like below.

First I created a directory on my docker container like so

docker exec sandbox mkdir /dan/ (can then run a docker exec sandbox ls to see that your mkdir worked)

then I copied the file to the directory I just created

docker cp /home/drice/test2.txt sandbox:dan/test2.txt

(where sandbox is the name of the docker container running HDP....you can get a list of containers by running docker ps)

once the file is in the docker container you can then copy to hadoop

docker exec sandbox hadoop fs -put /dan/test2.txt /test2.txt

[root@sandbox drice]# docker exec sandbox hadoop fs -ls /

Found 13 items

drwxrwxrwx - yarn hadoop 0 2016-10-25 08:10 /app-logs

drwxr-xr-x - hdfs hdfs 0 2016-10-25 07:54 /apps

drwxr-xr-x - yarn hadoop 0 2016-10-25 07:48 /ats

drwxr-xr-x - hdfs hdfs 0 2016-10-25 08:01 /demo

drwxr-xr-x - hdfs hdfs 0 2016-10-25 07:48 /hdp

drwxr-xr-x - mapred hdfs 0 2016-10-25 07:48 /mapred

drwxrwxrwx - mapred hadoop 0 2016-10-25 07:48 /mr-history

drwxr-xr-x - hdfs hdfs 0 2016-10-25 07:47 /ranger

drwxrwxrwx - spark hadoop 0 2017-02-28 15:05 /spark-history

drwxrwxrwx - spark hadoop 0 2016-10-25 08:14 /spark2-history

-rw-r--r-- 1 root hdfs 15 2017-02-28 15:04 /test.txt

drwxrwxrwx - hdfs hdfs 0 2016-10-25 08:11 /tmp

drwxr-xr-x - hdfs hdfs 0 2016-10-25 08:11 /user

NOTE: Another way to do this is to just use the ambari file browser view to copy files graphically.

Re: Copy file from Windows to sandbox hosted in Azure

Temidee — Wed, 01 Mar 2017 04:19:26 GMT

@Dan Rice. Thanks dan but please can you explain to me more how to go about this process since am using the virtual machine on azure portal and am just beginning to learn this concept. thanks

Re: Copy file from Windows to sandbox hosted in Azure

Temidee — Wed, 01 Mar 2017 05:09:48 GMT

@Dan Rice

Hi dan thanks for your help just want to give you update about the process when i ran the first line ok commands it work but when i tried this docker cp /home/drice/test2.txt sandbox:dan/test2.txt. it said not such file exist though i used my directory.

Re: Copy file from Windows to sandbox hosted in Azure

drice1 — Wed, 01 Mar 2017 05:10:16 GMT

@Adedayo Adekeye

Ambari is the web ui that is used to administer, monitor, and provision out a hadoop cluster. It also has the concept of VIEWs which allow for browsing the Hadoop Distributed filesystem (HDFS) as well as querying data through Hive, writing pig scripts amongst other things (even extensible to do something custom). Regardless within Ambari (example link - http:// TO AZURE PUBLIC IP>:8080/#/main/views/FILES/1.0.0/AUTO_FILES_INSTANCE )

You can log in as raj_ops (with password as raj_ops) in order to get to the files view. Don't just click the link but you'll have to change the above link to match your Azure sandbox Public IP address. This also assumes you have port 8080 open in Azures Network Security Group setting.

You may want to follow the instructions on how to reset the ambari admin password - https://hortonworks.com/hadoop-tutorial/learning-the-ropes-of-the-hortonworks-sandbox/#setup-ambari-admin-password

Hope this helps

Re: Copy file from Windows to sandbox hosted in Azure

praskutti — Mon, 19 Aug 2019 09:37:18 GMT

Thanks for the response. This dint work in Azure. I get this error:

I was able to ssh into port 2222 though, in Azure. Am i missing something?

However, this worked in my VMWare workstation.

Re: Copy file from Windows to sandbox hosted in Azure

darvasip — Mon, 19 Aug 2019 09:37:10 GMT

@Prasanna G Have you started Sandbox from Azure Marketplace?

If so, you need to manually install Docker, as described here.

You have the option to install Docker extension with VM install, it should work as well:

Re: Copy file from Windows to sandbox hosted in Azure

drice1 — Mon, 19 Aug 2019 09:37:02 GMT

@Prasanna G

You'll have to first sudo su as (or just use sudo docker ps).

See screen shot

Re: Copy file from Windows to sandbox hosted in Azure

jwhitmore — Thu, 02 Mar 2017 04:45:49 GMT

@Prasanna G,

When you ssh to port 2222, you are inside the sandbox container. Is the file there? if you ls -l tmp/ what do you get? which would be different than ls -l /tmp based on the pscp command you ran. you should be able to pscp directly to the container by going to port 2222.

pscp -P 2222 <file_from_local> /tmp

Then, in your shell you should be able to

[root@sandbox ~]# ls -l /tmp
total 548
-rw-r--r-- 1 root       root     7540 Feb 27 10:00 file_from_local

Then I think you will want to copy it from the linux fs to the HDFS using hadoop command:

[root@sandbox ~]# hadoop fs -put /tmp/file_from_local /tmp
[root@sandbox ~]# hadoop fs -ls /tmp
-rw-r--r--   1 root      hdfs          0 2017-03-01 20:43 /tmp/file_from_local

Enjoy!

John

Re: Copy file from Windows to sandbox hosted in Azure

praskutti — Thu, 02 Mar 2017 15:48:41 GMT

Hello @pdarvasi , the docker is installed as part of the Azure Sandbox. I can see the docker service running. And i was also able to run the docker cp command once i did "sudo su"

Re: Copy file from Windows to sandbox hosted in Azure

praskutti — Thu, 02 Mar 2017 15:51:21 GMT

Yes this did it 🙂 I tried the sudo su earlier too, but i dint work. I restarted the docker service and did a sudo su , it all worked fine. guess there was some issue with docker service.

For anyone who wants to try, i did this

After u logged on to the VM via Putty (if ur host OS is Windows) , dont ssh into root@sandbox yet. Stay on the linux VM and do these

services docker restart

sudo su

docker ps

Re: Copy file from Windows to sandbox hosted in Azure

drice1 — Thu, 02 Mar 2017 21:51:09 GMT

Great to hear that you got things working @Prasanna G