Hbase on HDP 2.5: java.lang.NoClassDefFoundError: Could not initialize class org.apache.hadoop.hbase.util.ByteStringer

[#1]

I'm getting the following error with HDP 2.5 client libraries for Hbase. I use them via Gradle, having a line like that:

compile group: 'org.apache.hbase', name: 'hbase-client', version: '1.1.2.2.5.0.0-1245'

Then, upon running the app, again via Gradle, I do see hbase-protocols explicitly referred in the Zookeeper client classpath, , i.e. the line I see starting with

6879 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.class.path=

contains, among other JARs, a reference to the Gradle cache, i.e. C:\Users\testUser\.gradle\caches\modules-2\files-2.1\org.apache.hbase\hbase-protocol\1.1.2.2.5.0.0-1245\b037b1aff481f7dfc40f918a38005a6dce72fe49\hbase-protocol-1.1.2.2.5.0.0-1245.jar

So basically org.apache.hadoop.hbase.util.ByteStringer should be found. But it is not:

13640 [main-SendThread(hadoop-dev:2181)] DEBUG org.apache.zookeeper.ClientCnxn  - Reading reply sessionid:0x15853747d08002b, packet:: clientPath:null serverPath:null finished:false header:: 12,4  replyHeader:: 12,1530,0  request:: '/hbase-unsecure/meta-region-server,F  response:: #ffffffff0001a726567696f6e7365727665723a3136303230ffffffff7653ffffffe44971ffffffa3ffffffcf50425546a23a1773616e64626f782e686f72746f6e776f726b732e636f6d10ffffff947d18ffffffc2ffffffaeffffff93ffffff9cffffff852b100183,s{1378,1378,1478870023688,1478870023688,0,0,0,0,76,0,1378}
14143 [main] DEBUG org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation  - locateRegionInMeta parentTable=hbase:meta, metaLocation=, attempt=4 of 35 failed; retrying after sleep of 1005 because: java.lang.NoClassDefFoundError: Could not initialize class org.apache.hadoop.hbase.util.ByteStringer
14146 [main-SendThread(hadoop-dev:2181)] DEBUG org.apache.zookeeper.ClientCnxn  - Reading reply sessionid:0x15853747d08002b, packet:: clientPath:null serverPath:null finished:false header:: 13,8  replyHeader:: 13,1530,0  request:: '/hbase-unsecure,F  response:: v{'meta-region-server,'rolllog-proc,'backup-masters,'switch,'table,'draining,'region-in-transition,'table-lock,'running,'master,'namespace,'hbaseid,'online-snapshot,'replication,'splitWAL,'recovering-regions,'rs,'flush-table-proc}

As soon as I change my Gradle hbase-client reference to:

compile group: 'org.apache.hbase', name: 'hbase-client', version: '1.1.2'

i.e. the library becomes the default, non-recommended HBase libs, the problem goes away. But, of course, I do not want to use the non-recommended HBase client libraries - I had another issue that actually got resolved by using the Hortonworks libs.

[#2]

Can you please verify that the JAR file in your Gradle cache is not malformed? e.g. `jar tf C:\Users\testUser\.gradle\caches\modules-2\files-2.1\org.apache.hbase\hbase-protocol\1.1.2.2.5.0.0-1245\b037b1aff481f7dfc40f918a38005a6dce72fe49\hbase-protocol-1.1.2.2.5.0.0-1245.jar` and look for ByteStringer.class in the output.

[#3]

Yes, it's there: here's the actual command executed:

C:\Users\testUser\.gradle\caches\modules-2\files-2.1\org.apache.hbase\hbase-protocol\1.1.2.2.5.0.0-1245\b037b1aff481f7dfc40f918a38005a6dce72fe49>jar tf hbase-protocol-1.1.2.2.5.0.0-1245.jar| findstr ByteStringer
org/apache/hadoop/hbase/util/ByteStringer.class

I also had deleted the entire .gradle cache earlier, and everything got downloaded again, and still the same.

And yes, I'm using gradle run, not some other external way which relies on a different classpath.

[#4]

Josh, I discovered something. Because I had the following gRPC references in the Gradle:

compile 'io.grpc:grpc-netty:1.0.1'
compile 'io.grpc:grpc-protobuf:1.0.1'
compile 'io.grpc:grpc-stub:1.0.1'

and now when I commented them out, the whole issue disappears (i.e. is solved).

So it seems to be a clash of something, but it's strange -- at first sight, of course gRPC can't be containing references to Hadoop/Hbase classes.

[#5]

Aha, it's a protocol buffers issue :). gRPC uses protobuf-java-3.x whereas that version of HBase is on protobuf-java-2.5 (if I recall correctly).

[#6]

ah, so hbase-protocol actually has to do with protobuf stuff... didn't realize that.

And yes, you recall correctly -- https://www.versioneye.com/java/org.apache.hbase:hbase-protocol/1.1.2 shows that protobuf-java is version 2.5 for HBase 1.1.2.

What can be done about it? Given that I can control neither HDP libs, nor gRPC libs, I guess shading or a similar approach won't work, correct?

[#7]

There is a shaded HBase client which you can try to use. I'm not sure if that will work around conflicting protocol buffer versions though.

[#8]

Thanks for the response -- even before I saw your message, I happily found out myself that there is a shaded version -- but *unfortunately*, yes, it does *not* work around protobuf versions. Can I raise a HDP bug about that? I mean, this is strange to me -- I mean -- does not anyone else use HBase java client with gRPC yet? Sorry for being a bit bothersome on that, it's really strange to me, honestly, as both projects are not something new.

[#9]

There is https://issues.apache.org/jira/browse/HBASE-15638 which is slated for Apache HBase 2.0 which will shade protobuf as well, but this has not yet been released.

[#10]

Hey, Josh, thanks a lot for the responses -- I went through http://apache-hbase.679495.n3.nabble.com/DISCUSS-More-Shading-td4083025.html and ended up at exactly HBASE-15638 which you give. Looks it will take a while -- quoting from the first link:

The HBASE-15638 (protobuf shading) approach has us reference the relocated artifact explicitly. This makes for an ugly ripple across the codebase as we declare which protobuf Message is intended; either com.google.protobuf.Message or org.apache.hadoop.hbase.shaded.com.google.protobuf.Message. It is a pain making all the changes but the intent is clear.

So that means that having the gRPC and HBase stuff in the same process is a no-go for now... definitely strange.

Thank you once again, I have upvoted all your answers.