Reply
New Contributor
Posts: 2
Registered: ‎11-08-2013

no output in part-00000 when running WordCount v1.0 Example

I'm new to both Cloudera and Hadoop, and the output the Cloudera WordCount 1.0 example (part-00000) is empty. The steps and files I'm using are here. I'm want to provide whatever job log information would help, ditto for versions - I just need some guidance on where to find them. Below is the job output and the source. Of the other parts written (part-00001 through part-00011), the non-empty ones are part-00001 (Bye 1), part-00002 (Hadoop 2), part-00004 (Goodbye 1), part-00005 (World 2), and part-00009 (Hello 2). Any help would be awesome.

 

Here are the commands and output:

 

[me@server ~]$ hadoop fs -cat /user/me/wordcount/input/file0
Hello World Bye World

[me@server ~]$ hadoop fs -cat /user/me/wordcount/input/file1
Hello Hadoop Goodbye Hadoop

[me@server ~]$ hadoop jar wordcount.jar org.myorg.WordCount /user/me/wordcount/input /user/me/wordcount/output
13/11/12 10:39:41 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
13/11/12 10:39:41 INFO mapred.FileInputFormat: Total input paths to process : 2
13/11/12 10:39:42 INFO mapred.JobClient: Running job: job_201311051201_0014
13/11/12 10:39:43 INFO mapred.JobClient:  map 0% reduce 0%
13/11/12 10:39:49 INFO mapred.JobClient:  map 33% reduce 0%
13/11/12 10:39:52 INFO mapred.JobClient:  map 67% reduce 0%
13/11/12 10:39:53 INFO mapred.JobClient:  map 100% reduce 0%
13/11/12 10:39:58 INFO mapred.JobClient:  map 100% reduce 25%
13/11/12 10:40:01 INFO mapred.JobClient:  map 100% reduce 100%
13/11/12 10:40:04 INFO mapred.JobClient: Job complete: job_201311051201_0014
13/11/12 10:40:04 INFO mapred.JobClient: Counters: 33
13/11/12 10:40:04 INFO mapred.JobClient:   File System Counters
13/11/12 10:40:04 INFO mapred.JobClient:     FILE: Number of bytes read=313
13/11/12 10:40:04 INFO mapred.JobClient:     FILE: Number of bytes written=2695420
13/11/12 10:40:04 INFO mapred.JobClient:     FILE: Number of read operations=0
13/11/12 10:40:04 INFO mapred.JobClient:     FILE: Number of large read operations=0
13/11/12 10:40:04 INFO mapred.JobClient:     FILE: Number of write operations=0
13/11/12 10:40:04 INFO mapred.JobClient:     HDFS: Number of bytes read=410
13/11/12 10:40:04 INFO mapred.JobClient:     HDFS: Number of bytes written=41
13/11/12 10:40:04 INFO mapred.JobClient:     HDFS: Number of read operations=18
13/11/12 10:40:04 INFO mapred.JobClient:     HDFS: Number of large read operations=0
13/11/12 10:40:04 INFO mapred.JobClient:     HDFS: Number of write operations=24
13/11/12 10:40:04 INFO mapred.JobClient:   Job Counters
13/11/12 10:40:04 INFO mapred.JobClient:     Launched map tasks=3
13/11/12 10:40:04 INFO mapred.JobClient:     Launched reduce tasks=12
13/11/12 10:40:04 INFO mapred.JobClient:     Data-local map tasks=3
13/11/12 10:40:04 INFO mapred.JobClient:     Total time spent by all maps in occupied slots (ms)=16392
13/11/12 10:40:04 INFO mapred.JobClient:     Total time spent by all reduces in occupied slots (ms)=61486
13/11/12 10:40:04 INFO mapred.JobClient:     Total time spent by all maps waiting after reserving slots (ms)=0
13/11/12 10:40:04 INFO mapred.JobClient:     Total time spent by all reduces waiting after reserving slots (ms)=0
13/11/12 10:40:04 INFO mapred.JobClient:   Map-Reduce Framework
13/11/12 10:40:04 INFO mapred.JobClient:     Map input records=2
13/11/12 10:40:04 INFO mapred.JobClient:     Map output records=8
13/11/12 10:40:04 INFO mapred.JobClient:     Map output bytes=82
13/11/12 10:40:04 INFO mapred.JobClient:     Input split bytes=357
13/11/12 10:40:04 INFO mapred.JobClient:     Combine input records=8
13/11/12 10:40:04 INFO mapred.JobClient:     Combine output records=6
13/11/12 10:40:04 INFO mapred.JobClient:     Reduce input groups=5
13/11/12 10:40:04 INFO mapred.JobClient:     Reduce shuffle bytes=649
13/11/12 10:40:04 INFO mapred.JobClient:     Reduce input records=6
13/11/12 10:40:04 INFO mapred.JobClient:     Reduce output records=5
13/11/12 10:40:04 INFO mapred.JobClient:     Spilled Records=12
13/11/12 10:40:04 INFO mapred.JobClient:     CPU time spent (ms)=15650
13/11/12 10:40:04 INFO mapred.JobClient:     Physical memory (bytes) snapshot=3594293248
13/11/12 10:40:04 INFO mapred.JobClient:     Virtual memory (bytes) snapshot=18375352320
13/11/12 10:40:04 INFO mapred.JobClient:     Total committed heap usage (bytes)=6497697792
13/11/12 10:40:04 INFO mapred.JobClient:   org.apache.hadoop.mapreduce.lib.input.FileInputFormatCounter
13/11/12 10:40:04 INFO mapred.JobClient:     BYTES_READ=50

[me@server ~]$ hadoop fs -cat /user/me/wordcount/output/part-00000

[me@server ~]$ hdfs dfs -ls -R /user/me/wordcount/output
-rw-r--r--   3 me me          0 2013-11-12 10:40 /user/me/wordcount/output/_SUCCESS
drwxr-xr-x   - me me          0 2013-11-12 10:39 /user/me/wordcount/output/_logs
drwxr-xr-x   - me me          0 2013-11-12 10:39 /user/me/wordcount/output/_logs/history
-rw-r--r--   3 me me      67134 2013-11-12 10:40 /user/me/wordcount/output/_logs/history/job_201311051201_0014_1384270782432_me_wordcount
-rw-r--r--   3 me me      81866 2013-11-12 10:39 /user/me/wordcount/output/_logs/history/job_201311051201_0014_conf.xml
-rw-r--r--   3 me me          0 2013-11-12 10:39 /user/me/wordcount/output/part-00000
-rw-r--r--   3 me me          6 2013-11-12 10:39 /user/me/wordcount/output/part-00001
-rw-r--r--   3 me me          9 2013-11-12 10:39 /user/me/wordcount/output/part-00002
-rw-r--r--   3 me me          0 2013-11-12 10:39 /user/me/wordcount/output/part-00003
-rw-r--r--   3 me me         10 2013-11-12 10:39 /user/me/wordcount/output/part-00004
-rw-r--r--   3 me me          8 2013-11-12 10:39 /user/me/wordcount/output/part-00005
-rw-r--r--   3 me me          0 2013-11-12 10:39 /user/me/wordcount/output/part-00006
-rw-r--r--   3 me me          0 2013-11-12 10:39 /user/me/wordcount/output/part-00007
-rw-r--r--   3 me me          0 2013-11-12 10:39 /user/me/wordcount/output/part-00008
-rw-r--r--   3 me me          8 2013-11-12 10:39 /user/me/wordcount/output/part-00009
-rw-r--r--   3 me me          0 2013-11-12 10:39 /user/me/wordcount/output/part-00010
-rw-r--r--   3 me me          0 2013-11-12 10:39 /user/me/wordcount/output/part-00011
[me@server ~]$

 

 

 

Here's the source:

 

package org.myorg;

import java.io.IOException;
import java.util.*;

import org.apache.hadoop.fs.Path;
import org.apache.hadoop.conf.*;
import org.apache.hadoop.io.*;
import org.apache.hadoop.mapred.*;
import org.apache.hadoop.util.*;

public class WordCount {

  public static class Map extends MapReduceBase implements Mapper<LongWritable, Text, Text, IntWritable> {
    private final static IntWritable one = new IntWritable(1);
    private Text word = new Text();

    public void map(LongWritable key, Text value, OutputCollector<Text, IntWritable> output, Reporter reporter) throws IOException {
      String line = value.toString();
      StringTokenizer tokenizer = new StringTokenizer(line);
      while (tokenizer.hasMoreTokens()) {
        word.set(tokenizer.nextToken());
        output.collect(word, one);
      }
    }
  }

  public static class Reduce extends MapReduceBase implements Reducer<Text, IntWritable, Text, IntWritable> {
    public void reduce(Text key, Iterator<IntWritable> values, OutputCollector<Text, IntWritable> output, Reporter reporter) throws IOException {
      int sum = 0;
      while (values.hasNext()) {
        sum += values.next().get();
      }
      output.collect(key, new IntWritable(sum));
    }
  }

  public static void main(String[] args) throws Exception {
    JobConf conf = new JobConf(WordCount.class);
    conf.setJobName("wordcount");

    conf.setOutputKeyClass(Text.class);
    conf.setOutputValueClass(IntWritable.class);

    conf.setMapperClass(Map.class);
    conf.setCombinerClass(Reduce.class);
    conf.setReducerClass(Reduce.class);

    conf.setInputFormat(TextInputFormat.class);
    conf.setOutputFormat(TextOutputFormat.class);

    FileInputFormat.setInputPaths(conf, new Path(args[0]));
    FileOutputFormat.setOutputPath(conf, new Path(args[1]));

    JobClient.runJob(conf);
  }
}

 

Cloudera Employee
Posts: 31
Registered: ‎10-16-2013

Re: no output in part-00000 when running WordCount v1.0 Example

If you'll look at the job output, you'll see:
Launched reduce tasks=12

You'll see that this corresponds to the number of output files - each reducer wrote a single file.

Since you have 12 reducers and only 5 different words, some files will be empty.

You can do the following:

1. Try to find out where you configured a default of 12 reducers per task. Look for mapred.reduce.tasks in Cloudera-Manager or in mapred-site.xml

2. Try to force this job to run with 1 reducer using:
-D mapred.reduce.tasks=1
parameter