Reply
Contributor
Posts: 55
Registered: ‎09-17-2013

How to test MapReduce program with multipleoutputs with MR Unit

I am trying to set up mr unit for our project.

 

My Mapper program uses multiple outputs to write the output.

If i tried testing with MR Unit, i am getting null pointer exception at multiple outputs line, as below.

 

java.lang.NullPointerException
    at org.apache.hadoop.fs.Path.<init>(Path.java:105)
    at org.apache.hadoop.fs.Path.<init>(Path.java:94)
    at org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.getDefaultWorkFile(FileOutputFormat.java:286)
    at org.apache.hadoop.mapreduce.lib.output.TextOutputFormat.getRecordWriter(TextOutputFormat.java:129)
    at org.apache.hadoop.mapreduce.lib.output.MultipleOutputs.getRecordWriter(MultipleOutputs.java:475)

 

 

Could you let me know, how could i perform testing for this case.?

 

Cloudera Employee
Posts: 314
Registered: ‎01-16-2014

Re: How to test MapReduce program with multipleoutputs with MR Unit

You will have to share some of your code to allow us to help you.

The mapper and unit test that you wrote would be the start.

 

Wilfred

Contributor
Posts: 55
Registered: ‎09-17-2013

Re: How to test MapReduce program with multipleoutputs with MR Unit

This is my code.

 

Each map method will get file name as key and desired name in subfolder as value.

Takes the file and reads it and writes each record from the file.

 

 

public class MyMapperTest {
      MapDriver<Text, Text, Text, NullWritable> mapDriver;
      MapReduceDriver<Text, Text, Text, NullWritable, Text, NullWritable> mapReduceDriver;
      
      @Before
      public void setUp() {
        DecryptionMapper mapper = new MyMapper();
        mapDriver = MapDriver.newMapDriver(mapper);
        mapReduceDriver = MapReduceDriver.newMapReduceDriver();
        mapReduceDriver.setMapper(mapper);
      }
    
      @Test
      public void testMapper() throws IOException {
        mapDriver.getConfiguration().set("input.delimiter", ",");
        mapDriver.withInput(new Text("C:/Users/Srini/Desktop/2015/07/23/input.txt"),new Text("/2015-07-03/decrypted-file"));
        mapDriver.withOutput(new Text("12345"), NullWritable.get());
        mapDriver.runTest();
      }
    
    }

Cloudera Employee
Posts: 314
Registered: ‎01-16-2014

Re: How to test MapReduce program with multipleoutputs with MR Unit

The code looks OK at a first glance. What does it do does it throw exceptios, does it just fail the test. You will need to describe the issue you are seeing.

 

Wilfred

Announcements

Our community is getting a little larger. And a lot better.


Learn More about the Cloudera and Hortonworks community merger planned for late July and early August.