About koert

koert · ‎08-14-2016

i am beggining to think these shaded jars cannot be mixed with regular hadoop jars. but of course to run a job on yarn i need yarn and mapreduce jars on my classpath as well. the issue is that when you mix jars things get quickly confusing about shaded versus unshaded. like this: Caused by: java.lang.ClassCastException: org.apache.hadoop.yarn.proto.YarnServiceProtos$GetNewApplicationRequestProto cannot be cast to org.apache.hadoop.hbase.shaded.com.google.protobuf.Message so if i cannot uses shaded hbase jars in a mapreduce job (due to shaded vs unshaded protobuf issues), and i cannot use normal hbase jars in a mapreduce job (due to netty conflicts), what is the way forward? does cdh 5.7+ simply not support mapreduce jobs over hbase?

koert · ‎08-14-2016

i am trying to use the cdh 5.8.0 hbase shaded jars to put hbase on my hadoop classpath. i am resorting to shaded jars since hbase 1.2.0-cdh5.8.0 has different netty dependency and so just putting normal hbase jars on hadoop classpath leads to jar-hell. however there seems to be something weird about the hbase shaded client and server jars. first i notice classes in the jar that are not hbase related and not shaded, such as hamcrest. for example in the jar is: org/hamcrest/BaseDescription.class as far as i know a shaded jar should not include unshaded classes for dependencies. it should either be shaded or declared a (transitive) dependency and not be included. second as soon as i put these shaded jars on my classpath for hadoop job, the jobs runs in local mode. it refuses to run on yarn. so something in these jars is messing with my settings. any idea what that could be and why?

Online	Offline
Last Visited	‎08-17-2016 02:24 PM

Member Since	‎08-13-2016 10:54 AM
Last Visited	‎08-17-2016 02:24 PM
Posts	3

Cloudera Community

Cloudera Community

Re: hbase shaded jars

hbase shaded jars