Java MapReduce client connecting to Hortonworks VM

[#1]

I would like to be able to create a simple MapReduce application and then run it locally on my desktop by connecting to the Hortonworks VM (also running on my desktop). Are there any examples or guidelines of what configuration options I need to change in mapred-site.xml, core-site.xml, yarn-site.xml, etc? I have tried a few things from stackoverflow, including setting mapreduce.framework.name as referenced below, but I am seeing the following error when I start the job:

DEBUG [main] (UserGroupInformation.java:1661) - PrivilegedActionException as:bc (auth:SIMPLE) cause:java.io.IOException: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name and the correspond server addresses. Exception in thread "main" java.io.IOException: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name and the correspond server addresses.

I realize I could just scp it over, compile, and submit the job directly on the VM but it would be nice to iterate locally in Eclipse and avoid the churn of copy/compile/jar/submit job every edit.

[#2 · Accepted solution]

you will need the client jars and all the xml files on your local machine. In all honesty, I think it's easier to scp stuff over and run it on Sandbox @Barrett Cervenka. You can also try this tutorial by @Ali Bajwa.

[#3]

Yea, I think I have come to the same conclusion that it would just be easier to copy over to the sandbox. Thanks for the tutorial link though. I had not come across that yet.

[#4]

@Barrett Cervenka

Found this during the search http://stackoverflow.com/questions/12783968/copying-directory-from-local-system-to-hdfs-java-code