Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Eclipse to Sandbox

Highlighted

Eclipse to Sandbox

Hi, I am new to HDP. I have a windows 10 machine with Oracle VM running HDP2.5. Since my background is from Java, I am trying to run a simple program (wordcount) written in Java. Following is the setup that I have at this point.

Eclipse running on windows 10 (host machine)

Oracle VM running HDP 2.5

I have tried different posts from this forum but none of those appear to have a simple way to accomplish this task. I need an advise here. If this is not possible then let me know also.

8 REPLIES 8
Highlighted

Re: Eclipse to Sandbox

Super Guru

@Shah din sani

In your eclipse windows environment, provide the following in the class path.

core-site.xml

hdfs-site.xml

For a word count program, this does not need "hive-site.xml" or "hbase-site.xml". Now, in above files, you might have to change your host settings to however you access your Oracle VM from windows environment. If you access using IP address, then just use ip address. If you are able to use dns name, then use that. Whatever works for "pin <oracle vm>" should work.

Highlighted

Re: Eclipse to Sandbox

@mureshi

Thanks for the quick reply here. I do have both files in my path and I don't see any obvious issue also with the url in these files. However, when I run this program, I get following error.

Exception in thread "main" java.io.IOException: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name and the correspond server addresses.

Attached are these two files for your reference.

One more point that I want to mention here is that, since I don't have any local install of hadoop, there is no hadoop_home environment variable etc.

core-site.xmlhdfs-site.xml

Highlighted

Re: Eclipse to Sandbox

Super Guru

on your windows shell, when you ping "sandbox.hortonworks.com" what happens? See, I think your windows doesn't know what url to use to connect to your sandbox. You just need to figure that out. I haven't used a VM in a long time but you basically need to setup your network in a way that you can access it from windows. Do you know how to setup your VM in bridge mode? If you do that, then your windows will be able to ping "sandbox.hortonworks.com" and then it should work. Resolve your p;ing problem, and rest will be resolved.

https://www.virtualbox.org/manual/ch06.html ---> see how to setup in bridge mode.

Highlighted

Re: Eclipse to Sandbox

@mqureshi

Attached is the screen shot that shows results of the pingping-result.png

Highlighted

Re: Eclipse to Sandbox

Super Guru

@Shah din sani

Please add hadoop map reduce client jar to your classpat for your version. Given its HDP 2.5, following is what you are looking for.

http://mvnrepository.com/artifact/org.apache.hadoop/hadoop-mapreduce-client-jobclient/2.7.3

Highlighted

Re: Eclipse to Sandbox

@mqureshi

Thanks again for your help here. I am making progress here. Now I am getting following error when I run word count.

Exception in thread "main" java.io.IOException: (null) entry in command string: null chmod 0700 C:\tmp\hadoop-sdsani\mapred\staging\sdsani1687745498\.staging

Here I am using code from following post.

http://hortonworks.com/hadoop-tutorial/introducing-apache-hadoop-developers/

My main look like this.

public static void main(String[] args) throws Exception { Configuration conf = new Configuration(); Job job = new Job(conf, "wordcount"); job.setOutputKeyClass(Text.class); job.setOutputValueClass(IntWritable.class); job.setMapperClass(Map.class); job.setReducerClass(Reduce.class); job.setInputFormatClass(TextInputFormat.class); job.setOutputFormatClass(TextOutputFormat.class); FileInputFormat.addInputPath(job, new Path("/root/file1.xml")); FileOutputFormat.setOutputPath(job, new Path("/root/file2.xml")); job.waitForCompletion(true); }

I have created two files under root and access is set to 777 for both files.

Highlighted

Re: Eclipse to Sandbox

Super Guru
@Shah din sani

since you are running on windows "chmod" is not available on windows. that's why your program fails. You need cygwin. Please see the following link. search on this page for "chmod". Following instructions here should help resolve your issue.

http://www.thecloudavenue.com/2012/10/debugging-hadoop-mapreduce-program-in.html

Highlighted

Re: Eclipse to Sandbox

@mqureshi

Hi, I tried steps mentioned at below link (as per the post).

http://v-lad.org/Tutorials/Hadoop/00%20-%20Intro.html

Do you feel they are apply to HDP 2.5 (Hadoop 2.7)? Some of them are not working for me on windows 10 machine.

Don't have an account?
Coming from Hortonworks? Activate your account here