Created on 03-23-2019 07:50 PM - edited 09-16-2022 07:14 AM
Hi, I'm following Douglas Eadline's tutorial http://www.informit.com/store/hadoop-and-spark-fundamentals-livelessons-9780134770864 and try to compile the WordCount example, but I fail. Eadline suggests to comile it with hadoop-core.jar, which was renamed in newer versions as I understand.
I tried with javac -classpath /usr/hdp/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core.jar -d wordcount_classes WordCount.java but I still get a load of errors. Any suggestions? Where can I find the correct jar?
WordCount.java:27: error: package org.apache.hadoop.conf does not exist
import org.apache.hadoop.conf.Configuration;
^
WordCount.java:28: error: package org.apache.hadoop.conf does not exist
import org.apache.hadoop.conf.Configured;
^
WordCount.java:29: error: package org.apache.hadoop.fs does not exist
import org.apache.hadoop.fs.Path;
^
WordCount.java:30: error: package org.apache.hadoop.io does not exist
import org.apache.hadoop.io.IntWritable;
^
WordCount.java:31: error: package org.apache.hadoop.io does not exist
import org.apache.hadoop.io.LongWritable;
^
WordCount.java:32: error: package org.apache.hadoop.io does not exist
import org.apache.hadoop.io.Text;
^
WordCount.java:42: error: package org.apache.hadoop.util does not exist
import org.apache.hadoop.util.Tool;
^
WordCount.java:43: error: package org.apache.hadoop.util does not exist
import org.apache.hadoop.util.ToolRunner;
^
WordCount.java:54: error: cannot find symbol
public class WordCount extends Configured implements Tool {
^
symbol: class Configured
WordCount.java:54: error: cannot find symbol
public class WordCount extends Configured implements Tool {
^
symbol: class Tool
WordCount.java:61: error: cannot access Closeable
public static class MapClass extends MapReduceBase
^
class file for org.apache.hadoop.io.Closeable not found
WordCount.java:64: error: cannot find symbol
private final static IntWritable one = new IntWritable(1);
^
symbol: class IntWritable
location: class MapClass
WordCount.java:65: error: cannot find symbol
private Text word = new Text();
^
symbol: class Text
location: class MapClass
WordCount.java:67: error: cannot find symbol
public void map(LongWritable key, Text value,
^
symbol: class LongWritable
location: class MapClass
WordCount.java:67: error: cannot find symbol
public void map(LongWritable key, Text value,
^
symbol: class Text
location: class MapClass
WordCount.java:68: error: cannot find symbol
OutputCollector<Text, IntWritable> output,
^
symbol: class Text
location: class MapClass
WordCount.java:68: error: cannot find symbol
OutputCollector<Text, IntWritable> output,
^
symbol: class IntWritable
location: class MapClass
WordCount.java:83: error: cannot find symbol
implements Reducer<Text, IntWritable, Text, IntWritable> {
^
symbol: class Text
location: class WordCount
WordCount.java:83: error: cannot find symbol
implements Reducer<Text, IntWritable, Text, IntWritable> {
^
symbol: class IntWritable
location: class WordCount
WordCount.java:83: error: cannot find symbol
implements Reducer<Text, IntWritable, Text, IntWritable> {
^
symbol: class Text
location: class WordCount
WordCount.java:83: error: cannot find symbol
implements Reducer<Text, IntWritable, Text, IntWritable> {
^
symbol: class IntWritable
location: class WordCount
WordCount.java:85: error: cannot find symbol
public void reduce(Text key, Iterator<IntWritable> values,
^
symbol: class Text
location: class Reduce
WordCount.java:85: error: cannot find symbol
public void reduce(Text key, Iterator<IntWritable> values,
^
symbol: class IntWritable
location: class Reduce
WordCount.java:86: error: cannot find symbol
OutputCollector<Text, IntWritable> output,
^
symbol: class Text
location: class Reduce
WordCount.java:86: error: cannot find symbol
OutputCollector<Text, IntWritable> output,
^
symbol: class IntWritable
location: class Reduce
Created 03-23-2019 07:54 PM
The hadoop command line utility also provides an option to get the correct classpath produced something like following which you can use:
Example:
# export CLASSPATH=`hadoop classpath`:.: # echo $CLASSPATH # javac -d wordcount_classes WordCount.java
.
Created 03-23-2019 07:54 PM
The hadoop command line utility also provides an option to get the correct classpath produced something like following which you can use:
Example:
# export CLASSPATH=`hadoop classpath`:.: # echo $CLASSPATH # javac -d wordcount_classes WordCount.java
.
Created 03-24-2019 12:19 PM
Thank you, indeed it worked with
javac -cp `hadoop classpath` -d wordcount_classes WordCount.java