Support Questions
Find answers, ask questions, and share your expertise

How to remove r-00000 extention from reducer output in mapreduce

How to remove r-00000 extention from reducer output in mapreduce

Rising Star

I am able to rename my reducer output file correctly but r-00000 is still persisting . I have used MultipleOutputs in my reducer class . Here is details of the that .Not sure what am i missing or what extra i have to do?

  public class MyReducer extends Reducer<NullWritable, Text, NullWritable, Text> {
     private Logger logger = Logger.getLogger(MyReducer.class);
     private MultipleOutputs<NullWritable, Text> multipleOutputs;
     String strName = "";
     public void setup(Context context) {
       logger.info("Inside Reducer.");
       multipleOutputs = new MultipleOutputs<NullWritable, Text>(context);
     }
     @Override
     public void reduce(NullWritable Key, Iterable<Text> values, Context context)
         throws IOException, InterruptedException {
       for (Text value : values) {
         final String valueStr = value.toString();
         StringBuilder sb = new StringBuilder();
         sb.append(strArrvalueStr[0] + "|!|");
         multipleOutputs.write(NullWritable.get(), new Text(sb.toString()),strName);
       }
     }
     public void cleanup(Context context) throws IOException,
         InterruptedException {
       multipleOutputs.close();
     }
  }
3 REPLIES 3

Re: How to remove r-00000 extention from reducer output in mapreduce

Super Mentor
@sudarshan kumar

Have you already tried the following approach?

 job.getConfiguration().set("mapreduce.output.basename", "abcd");

Above should will result in your files being called "abcd-r-00000".

Re: How to remove r-00000 extention from reducer output in mapreduce

Rising Star

Yes but in this way also r-00000 will not be removed .

Re: How to remove r-00000 extention from reducer output in mapreduce

Rising Star

I was able to do it explicitly after my job finishes and thats ok for me.No delay in the job

  if (b){
         DateFormat dateFormat = new SimpleDateFormat("yyyy-MM-dd-HHmm");
         Calendar cal = Calendar.getInstance();
         String strDate=dateFormat.format(cal.getTime());
        FileSystem hdfs = FileSystem.get(getConf());
        FileStatus fs[] = hdfs.listStatus(new Path(args[1]));
        if (fs != null){ 
        for (FileStatus aFile : fs) {
        if (!aFile.isDir()) {
        hdfs.rename(aFile.getPath(), new Path(aFile.getPath().toString()+".txt"));
        }
        }
        }
       }