question Re: Java MapReduce client connecting to Hortonworks VM in Archives of Support Questions (Read Only)

Java MapReduce client connecting to Hortonworks VM

barrett_cervenk — Sat, 09 Jan 2016 04:12:04 GMT

I would like to be able to create a simple MapReduce application and then run it locally on my desktop by connecting to the Hortonworks VM (also running on my desktop). Are there any examples or guidelines of what configuration options I need to change in mapred-site.xml, core-site.xml, yarn-site.xml, etc? I have tried a few things from stackoverflow, including setting mapreduce.framework.name as referenced below, but I am seeing the following error when I start the job:

DEBUG [main] (UserGroupInformation.java:1661) - PrivilegedActionException as:bc (auth:SIMPLE) cause:java.io.IOException: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name and the correspond server addresses. Exception in thread "main" java.io.IOException: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name and the correspond server addresses.

I realize I could just scp it over, compile, and submit the job directly on the VM but it would be nice to iterate locally in Eclipse and avoid the churn of copy/compile/jar/submit job every edit.

Re: Java MapReduce client connecting to Hortonworks VM

aervits — Sat, 09 Jan 2016 04:18:37 GMT

you will need the client jars and all the xml files on your local machine. In all honesty, I think it's easier to scp stuff over and run it on Sandbox @Barrett Cervenka. You can also try this tutorial by @Ali Bajwa.

Re: Java MapReduce client connecting to Hortonworks VM

barrett_cervenk — Sat, 09 Jan 2016 04:26:46 GMT

Yea, I think I have come to the same conclusion that it would just be easier to copy over to the sandbox. Thanks for the tutorial link though. I had not come across that yet.

Re: Java MapReduce client connecting to Hortonworks VM

nsabharwal — Mon, 15 Feb 2016 04:32:34 GMT

@Barrett Cervenka

Found this during the search http://stackoverflow.com/questions/12783968/copying-directory-from-local-system-to-hdfs-java-code