Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Need help with hadoop sandbox file usage

Highlighted

Need help with hadoop sandbox file usage

New Contributor

Hello,

I have downloaded

HDP® 2.5 on Hortonworks Sandbox (Hortonworks Sandbox on a VM) - installed VMware Workstation 12 Player - installed sandbox - logged into Ambari (username - maria_dev, password - maria_dev) and sandbox using (Alt+F5: username - root, password - hadoop). Please help me with the following concerns.

1. I am not able to add any files into HDFS via any command. Common error in sandbox is "-bash: hdfs: command not found"

2. I would like to add a csv file into hdfs and then integrate spark with it.

3. I would then like to integrate R with spark.

What am I missing here? Where am I going wrong? I don't want to use Ambari as of now because I would like to learn to use hadoop, spark and R with codes (not that the UI is not good). I have scala, java (jre and jdk) installed in my system. I have added java as environment variable. What I am unable to figure out is how to use hadoop, add files into hdfs and integrate spark and R on top of each other with hadoop and then finally start building predictive models on R.

I am using a Windows 10 OS and a system with 2TB HDD and 16GB Ram. I am new with Hadoop but I am good with statistics. Kindly help.

Thank you

Anurag

7 REPLIES 7

Re: Need help with hadoop sandbox file usage

New Contributor

Kindly note that I have no knowledge about Java, SQL or databases. I am learning Pig.

Re: Need help with hadoop sandbox file usage

Expert Contributor

Please verify following

1) Verify is all service are up and running like HDFS,YARN others

2) From where you running the command "HDFS Client should be installed"

use below command

#hdfs dfs -put <localsource> ... <destination>

3) If both working fine , then only integrate the spark , search you get help articals

Re: Need help with hadoop sandbox file usage

New Contributor

Hello,

Thanks for the reply. But how do I verify if all the services are up and running?

Also, how do I start hadoop in sandbox?

Re: Need help with hadoop sandbox file usage

Guru

Hello @Anurag Srivastava ,

But how do I verify if all the services are up and running?

1. Login to Sandbox terminal by using username root and its password.

2. Run "ps -ef | grep -i hadoop" - if Hadoop services are running, you should see a lot of processes listed out in the output.

3. Login to Ambari using 'admin' user. (You will need to reset the admin password one time using 'ambari-admin-password-reset' command)

4. In Ambari, all the services should show green status on the left hand column of dashboard.

This is how you verify if the Hadoop services are running or not.

Also, how do I start hadoop in sandbox?

Usually you don't need to start Hadoop services in the Sandbox. They are kick-started during boot operation.

Hope this helps !

Re: Need help with hadoop sandbox file usage

New Contributor

Hello Vipin,

Thanks a lot for the input. This helped me move a step ahead. I need help in integrating spark with hadoop on my local machine. How can I do it? Also how do I integrate R with Spark (which is on Hadoop) for statistical analysis of data?

Please help!

Thanks

Anurag

Re: Need help with hadoop sandbox file usage

New Contributor

Hello Vipin,

Thanks a lot for the input. This helped me move a step ahead. I need help in integrating spark with hadoop on my local machine. How can I do it? Also how do I integrate R with Spark (which is on Hadoop) for statistical analysis of data?

Please help!

Thanks

Anurag

Re: Need help with hadoop sandbox file usage

Guru

@Anurag Srivastava I suppose its because HDP-2.5 sandbox comes with docker instances, (at least last time I checked). See if you can list the instances and login to them.

https://docs.docker.com/engine/reference/commandline/attach/

Also, check this

https://community.hortonworks.com/questions/57757/hdp-25-sandbox-not-starting.html

Don't have an account?
Coming from Hortonworks? Activate your account here