Created 02-01-2016 01:51 PM
Here is my code for the driver for mapreduce :
public int run(String[] args) throws Exception {
// TODO Auto-generated method stub
if (args.length!=1){
System.out.println("usage: [input]");
System.exit(-1);
}
Job job = Job.getInstance(getConf());
job.setJarByClass(HBaseDriver.class);
FileInputFormat.addInputPath(job
, new Path(args[0]));
job.getConfiguration().set(TableOutputFormat.OUTPUT_TABLE, "students");
job.setInputFormatClass(StudentInputFormat.class);
job.setMapperClass(HBaseStudentsMapper.class);
job.setOutputFormatClass(TableOutputFormat.class);
job.setNumReduceTasks(0);
returnjob.waitForCompletion(true)? 0 :1;
}
Looks like the system fails on this line :
job.setOutputFormatClass(TableOutputFormat.class);
the message is:
Exception in thread "main" java.lang.NoClassDefFoundError:
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/protobuf/generated/MasterProtos$MasterService$BlockingInterface
The interesting thing is that if I comment this line and do some Hbase stuff in the mapper it works.
More over I have he hbase-protocol library and I supply it with my libjars command
here is how I run the map reduce:
hadoop jar mo.jar HBaseDriver -libjars hbase-client.jar,hbase-common.jar,hbase-protocol.jar,htrace-core-2.00.jar,hbase-server.jar input
Created 02-02-2016 08:34 AM
Hi,
I managed to solve it
There are 2 things need to be done for this to work:
First add few jars in the libjars and second add hbase lib to hadoop class path and then it should work.
Here are the libjars needed
hbase-client.jar,hbase-server.jar,hbase-protocol.jar,hbase-common.jar,htrace-core-2.00.jar
pay attention to htrace-core-2.00.jar which is Cloudera jar and I am using hortonworks sandbox 2.2.4.2-2
and need to add the hbase lib to hadoop-env.sh otherwise you wont be able to run the main thread
Thank you for your help hope it helps!!
Created 02-02-2016 09:06 AM
Actually if you are using HDP the jar you would use is
htrace-core-3.1.0-incubating.jar found in /usr/hdp/current/hbase-master/lib instead of htrace-core-2.0.0.jar