Created 02-01-2016 01:51 PM
Here is my code for the driver for mapreduce :
public int run(String[] args) throws Exception { // TODO Auto-generated method stub if (args.length!=1){ System.out.println("usage: [input]"); System.exit(-1); } Job job = Job.getInstance(getConf()); job.setJarByClass(HBaseDriver.class); FileInputFormat.addInputPath(job , new Path(args[0])); job.getConfiguration().set(TableOutputFormat.OUTPUT_TABLE, "students"); job.setInputFormatClass(StudentInputFormat.class); job.setMapperClass(HBaseStudentsMapper.class); job.setOutputFormatClass(TableOutputFormat.class); job.setNumReduceTasks(0); returnjob.waitForCompletion(true)? 0 :1; }
Looks like the system fails on this line :
job.setOutputFormatClass(TableOutputFormat.class);
the message is:
Exception in thread "main" java.lang.NoClassDefFoundError:
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/protobuf/generated/MasterProtos$MasterService$BlockingInterface
The interesting thing is that if I comment this line and do some Hbase stuff in the mapper it works.
More over I have he hbase-protocol library and I supply it with my libjars command
here is how I run the map reduce:
hadoop jar mo.jar HBaseDriver -libjars hbase-client.jar,hbase-common.jar,hbase-protocol.jar,htrace-core-2.00.jar,hbase-server.jar input
Created 02-02-2016 08:34 AM
Hi,
I managed to solve it
There are 2 things need to be done for this to work:
First add few jars in the libjars and second add hbase lib to hadoop class path and then it should work.
Here are the libjars needed
hbase-client.jar,hbase-server.jar,hbase-protocol.jar,hbase-common.jar,htrace-core-2.00.jar
pay attention to htrace-core-2.00.jar which is Cloudera jar and I am using hortonworks sandbox 2.2.4.2-2
and need to add the hbase lib to hadoop-env.sh otherwise you wont be able to run the main thread
Thank you for your help hope it helps!!
Created 02-02-2016 09:06 AM
Actually if you are using HDP the jar you would use is
htrace-core-3.1.0-incubating.jar found in /usr/hdp/current/hbase-master/lib instead of htrace-core-2.0.0.jar