Support Questions

Find answers, ask questions, and share your expertise

HBase MapReduce job

avatar
Rising Star

Here is my code for the driver for mapreduce :

public int run(String[] args) throws Exception {
// TODO Auto-generated method stub
if (args.length!=1){
System.out.println("usage: [input]");
System.exit(-1);
}
Job job = Job.getInstance(getConf());
job.setJarByClass(HBaseDriver.class);
FileInputFormat.addInputPath(job
, new Path(args[0]));
job.getConfiguration().set(TableOutputFormat.OUTPUT_TABLE, "students");
job.setInputFormatClass(StudentInputFormat.class);
job.setMapperClass(HBaseStudentsMapper.class);
job.setOutputFormatClass(TableOutputFormat.class);
job.setNumReduceTasks(0);
returnjob.waitForCompletion(true)? 0 :1;
}

Looks like the system fails on this line :

job.setOutputFormatClass(TableOutputFormat.class);

the message is:

Exception in thread "main" java.lang.NoClassDefFoundError:

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/protobuf/generated/MasterProtos$MasterService$BlockingInterface

The interesting thing is that if I comment this line and do some Hbase stuff in the mapper it works.

More over I have he hbase-protocol library and I supply it with my libjars command

here is how I run the map reduce:

hadoop jar mo.jar HBaseDriver -libjars hbase-client.jar,hbase-common.jar,hbase-protocol.jar,htrace-core-2.00.jar,hbase-server.jar input
1 ACCEPTED SOLUTION

avatar
Rising Star

Hi,

I managed to solve it

There are 2 things need to be done for this to work:

First add few jars in the libjars and second add hbase lib to hadoop class path and then it should work.

Here are the libjars needed

hbase-client.jar,hbase-server.jar,hbase-protocol.jar,hbase-common.jar,htrace-core-2.00.jar

pay attention to htrace-core-2.00.jar which is Cloudera jar and I am using hortonworks sandbox 2.2.4.2-2

and need to add the hbase lib to hadoop-env.sh otherwise you wont be able to run the main thread

Thank you for your help hope it helps!!

View solution in original post

10 REPLIES 10

avatar

Actually if you are using HDP the jar you would use is

htrace-core-3.1.0-incubating.jar found in /usr/hdp/current/hbase-master/lib instead of htrace-core-2.0.0.jar