Re: RDD take action doesnt work in Zeppelin

praskutti · ‎03-14-2017

@vshukla here are the configs for my machines Azure VM (D12 v2 config) is 4 cores, 28 GB RAM, 200GB HDD; VMWare sandbox has 16GB RAM, 8 cores and 1 TB HDDspace. Will this be not enough for Sandbox? The size of file i am trying to upload - 1 MB. I am able to load 5 MB txt file just fine. Only for CSVs it cribs. I will try out the HDC. Thanks for the link!

praskutti · ‎03-14-2017

Hi.. a gentle bump to the thread..

praskutti · ‎03-13-2017

@vshukla , i did some more digging and found that whenever i run the commands, Yarn-Memory in Ambari dashboard goes upto 90%. What could be wrong?

praskutti · ‎03-08-2017

If it helps: Screenshot from Amabari for Zeppelin Config

praskutti · ‎03-08-2017

@vshukla This is the sandbox from hortonworks. so i suppose it is standalone mode; but how do i verify that? Spark-shell - just launch Putty, ssh'd into root, issued pyspark to get to Spark, and issued command (rdd = sc.textFile(csv)) - Works like a charm Zeppelin - exactly the same - used %pyspark interpreter . - doesnt work My Azure VM (D12 v2) config is 4 cores, 28 GB RAM, 200GB HDD; My local VMWare sandbox has 16GB RAM, 8 cores and 1 TB HDDspace. Will this be not enough for Zeppelin?

praskutti · ‎03-08-2017

One more info : I could the same things on txt file, just now csv's. Any pointers?

praskutti · ‎03-08-2017

@glupu this is exactly what i did, re-imported a new sandbox (and deleted the previous one). The one thing i lost due to that is missed all my Zeppelin notebooks (should have take a bakcup of them).

praskutti · ‎03-08-2017

I am trying to load a csv file into a RDD using textFile function in Zeppelin and then do a take(10). But the take does not produce any result in Zepplin while the same commands outputs rows in SSH (shell) I have attached the file, my zepplin notebook and some screenshots. Can you please suggest me how to resolve this error? (HortonWorks Sandbox HDP2.5 on Microsoft Azure) data : dodgers.zip Zeppelin pras-playground.zip

praskutti · ‎03-08-2017

@Michael Young , I am using imported VMWare Sandbox (not native Docker Sandbox). It was working fine couple of days ago, but failed since I did a restart of my host machine (windows 10).

praskutti · ‎03-07-2017

Hello all, I am unable to start the sandbox docker container on my VMWare (HDP Sandbox 2.5). The sandbox image and docker service are fine, but iimage is not starting. This is the error message i get: Error response from daemon: invalid header field value "oci runtime error: container_linux.go:247: starting container process caused \"process_linux.go:359: container init caused \\rootfs_linux.go:89:

praskutti · ‎03-02-2017

Yes this did it 🙂 I tried the sudo su earlier too, but i dint work. I restarted the docker service and did a sudo su , it all worked fine. guess there was some issue with docker service. For anyone who wants to try, i did this After u logged on to the VM via Putty (if ur host OS is Windows) , dont ssh into root@sandbox yet. Stay on the linux VM and do these services docker restart sudo su docker ps

praskutti · ‎03-02-2017

Hello @pdarvasi , the docker is installed as part of the Azure Sandbox. I can see the docker service running. And i was also able to run the docker cp command once i did "sudo su"

praskutti · ‎03-01-2017

Thanks for the response. This dint work in Azure. I get this error: I was able to ssh into port 2222 though, in Azure. Am i missing something? However, this worked in my VMWare workstation.

praskutti · ‎02-28-2017

I was successfuly able to copy a file to Azure from Windows using Putty Copy (pscp). I used the following command pscp - P 22 <file from local folder> tmp\ I could see that file in the sandbox host. Now when i tried to "put" the file from sandbox host to the sandbox itself (docker container), i am not able to do so (throws network not accessible error). The command below hadoop fs -ls -put \tmp\onecsvfile.csv \tmp\ hadoop fs -ls -put \tmp\onecsvfile.csv root@localhost:2222\\tmp\ Nothing works 😞 Can you tell me how can i move the file from sandbox host to the sandbox (Sandbox 2.5 hosted in Azure)?

praskutti · ‎01-31-2017

In essence, how do i query/distinguish local Sandbox file system and hdfs file system? I put the following queries in PuTTy, but it doesnt return anything hadoop fs -ls /tmp <--doenst show the file hdfs dfs -ls /tmp <-- shows the file Am i missing something?

praskutti · ‎01-31-2017

Thank you qureshi. This solved my issue. Is there any documentation on the HortonWorks Sandbox File Sytem? I understand hadoop/hdfs file system are differenct from your response, but is there a clearer documentation on how to navigate around the file system within the Sandbox?

praskutti · ‎01-30-2017

Hello.. I used PuTTy's pscp command to copy a file (Windows 10) to my Sandbox (HDP 2.5, hosted on VMWare Workstation 12). The copy looked sucessfull But i cannot find the file anywhere in the system - i used the command line (PuTTy) as well as Ambari Web interfac; couldnt find it at all. Praskutti is a folder i created in /user/ and in the /root , but cant find the file in neither. Can you guys help me where//how to find the files?

praskutti · ‎01-27-2017

Hi Brandon Actually, i imported the VMWare image and everything was successful (the import process). What I did not realize was that the image in itself is a VM (docker-contained). So when i tried putty into root@sandbox, it was in fact talking to the docker container and not the HDP sandbox. Essentially, we have 2 VMs after the import is complete - VMware creates a CentOS VM and Docker VM inside it to host the actual HDP Sandbox Once i figured out that i have to ssh root@<sandbox IP shown right after the import process> -p2222 , i was able to follow the tutorials. The bold port number is the key here - to access the sandbox within the docker image via PuTTy. Hope this helps for newbies like me. Also if the HortonWorks team could update their documentation to cleary explain this, it would be helpufl. Thanks for your response again!

praskutti · ‎01-27-2017

Hi John, Thanks for the guidance. I in fact added 8888/8080/9995(for Zepellin) and the likes during my intial attempt (a week ago). But it did not allow me to login. But today, i rebuild the VM from scratch and added them again. I am able to login and I am also able to update ambari password. Thanks once again fro your assistance. <Marking your reply as answer>

praskutti · ‎01-26-2017

I created a VM in Azure (VM spec: D12 v2) with HortonWorks Sandbox 2.4 following the guide here: http://hortonworks.com/hadoop-tutorial/deploying-hortonworks-sandbox-on-microsoft-azure/ On step 13, when i tried to connect using http://<public ip from azure>:<8888/8080/80/22/150> , it doesnt connect at all. i tried all major browsers, from home network, work network and mobile network. doesnt help I checked Azure Network Security group settings and i saw an entry for inbound rules to allow SSH. I also created a new Inbound rule to allow HTTP over port 150, but that doesnt work either. allow-http.png network-security-group.png azure-vm-size.png error.png What am i missing here? Any particular configuration i have to do? Side note: PuTTy doesnt connect to the machine as well. I have been banging my head against the wall for a week with this.

praskutti · ‎01-26-2017

Does Hortonrworks HDP Sandbox 2.5 support VMWare Workstation 12 Pro (host OS - Windows 10/7 ENT) ?

Online	Offline
Last Visited	‎03-14-2017 06:58 PM

Date Registered	‎01-26-2017 01:10 AM
Last Visited	‎03-14-2017 06:58 PM
Total Messages Posted	21
Total Kudos Received	3

Re: RDD take action doesnt work in Zeppelin

Re: RDD take action doesnt work in Zeppelin

Re: RDD take action doesnt work in Zeppelin

Re: RDD take action doesnt work in Zeppelin

Re: RDD take action doesnt work in Zeppelin

Re: RDD take action doesnt work in Zeppelin

Re: sandbox Docker container startup error

RDD take action doesnt work in Zeppelin

Re: sandbox Docker container startup error

sandbox Docker container startup error

Re: Copy file from Windows to sandbox hosted in Az...

Re: Copy file from Windows to sandbox hosted in Az...

Re: Copy file from Windows to sandbox hosted in Az...

Copy file from Windows to sandbox hosted in Azure

Re: Cannot find files copied to Sandbox from Windo...

Re: Cannot find files copied to Sandbox from Windo...

Cannot find files copied to Sandbox from Windows

Re: Sandbox HDP 2.5 support for VMWare Workstation...

Re: sandbox welcome screen does not appear when co...

sandbox welcome screen does not appear when connec...

Sandbox HDP 2.5 support for VMWare Workstation 12 ...