hbase shaded jars

[#1]

i am trying to use the cdh 5.8.0 hbase shaded jars to put hbase on my hadoop classpath. i am resorting to shaded jars since hbase 1.2.0-cdh5.8.0 has different netty dependency and so just putting normal hbase jars on hadoop classpath leads to jar-hell.

however there seems to be something weird about the hbase shaded client and server jars.

first i notice classes in the jar that are not hbase related and not shaded, such as hamcrest. for example in the jar is:

org/hamcrest/BaseDescription.class

as far as i know a shaded jar should not include unshaded classes for dependencies. it should either be shaded or declared a (transitive) dependency and not be included.

second as soon as i put these shaded jars on my classpath for hadoop job, the jobs runs in local mode. it refuses to run on yarn. so something in these jars is messing with my settings. any idea what that could be and why?

[#2]

i am beggining to think these shaded jars cannot be mixed with regular hadoop jars. but of course to run a job on yarn i need yarn and mapreduce jars on my classpath as well.

the issue is that when you mix jars things get quickly confusing about shaded versus unshaded. like this:

Caused by: java.lang.ClassCastException: org.apache.hadoop.yarn.proto.YarnServiceProtos$GetNewApplicationRequestProto cannot be cast to org.apache.hadoop.hbase.shaded.com.google.protobuf.Message

so if i cannot uses shaded hbase jars in a mapreduce job (due to shaded vs unshaded protobuf issues), and i cannot use normal hbase jars in a mapreduce job (due to netty conflicts), what is the way forward? does cdh 5.7+ simply not support mapreduce jobs over hbase?

[#3]

I am also getting the same issue with Hadoop 3.2.1, did you know any workaround ? 

[#4]

@Dhiru, as this is an older post, you would have a better chance of receiving a resolution by starting a new thread. This will also be an opportunity to provide details specific to your environment that could aid others in assisting you with a more accurate answer to your question. You can link this thread as a reference in your new post.

[#5]

@VidyaSargur Thanks for your hint. I will keep this points in my mind during upcoming discussions.