Support Questions
Find answers, ask questions, and share your expertise

hbase shaded jars

Highlighted

hbase shaded jars

New Contributor

i am trying to use the cdh 5.8.0 hbase shaded jars to put hbase on my hadoop classpath. i am resorting to shaded jars since hbase 1.2.0-cdh5.8.0 has different netty dependency and so just putting normal hbase jars on hadoop classpath leads to jar-hell.

 

however there seems to be something weird about the hbase shaded client and server jars.

 

first i notice classes in the jar that are not hbase related and not shaded, such as hamcrest. for example in the jar is:

org/hamcrest/BaseDescription.class

 

as far as i know a shaded jar should not include unshaded classes for dependencies. it should either be shaded or declared a (transitive) dependency and not be included.

 

second as soon as i put these shaded jars on my classpath for hadoop job, the jobs runs in local mode. it refuses to run on yarn. so something in these jars is messing with my settings. any idea what that could be and why?

4 REPLIES 4
Highlighted

Re: hbase shaded jars

New Contributor

i am beggining to think these shaded jars cannot be mixed with regular hadoop jars. but of course to run a job on yarn i need yarn and mapreduce jars on my classpath as well.

 

the issue is that when you mix jars things get quickly confusing about shaded versus unshaded. like this:

Caused by: java.lang.ClassCastException: org.apache.hadoop.yarn.proto.YarnServiceProtos$GetNewApplicationRequestProto cannot be cast to org.apache.hadoop.hbase.shaded.com.google.protobuf.Message

 

so if i cannot uses shaded hbase jars in a mapreduce job (due to shaded vs unshaded protobuf issues), and i cannot use normal hbase jars in a mapreduce job (due to netty conflicts), what is the way forward? does cdh 5.7+ simply not support mapreduce jobs over hbase?

Highlighted

Re: hbase shaded jars

New Contributor

I am also getting the same issue with Hadoop 3.2.1, did you know any workaround ? 

Re: hbase shaded jars

Community Manager

@Dhiru, as this is an older post, you would have a better chance of receiving a resolution by starting a new thread. This will also be an opportunity to provide details specific to your environment that could aid others in assisting you with a more accurate answer to your question. You can link this thread as a reference in your new post.


Regards,

Vidya Sargur,
Community Manager


Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

Learn more about the Cloudera Community:

Highlighted

Re: hbase shaded jars

New Contributor

@VidyaSargur Thanks for your hint. I will keep this points in my mind during upcoming discussions.