Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

map-red over hbase in cdh 5.7+

map-red over hbase in cdh 5.7+

New Contributor

when hbase 1.0.0 was introduced in cdh 5.4.0 i think netty was kept at version 3 for compatibility.

 

but with hbase 1.2.0 in cdh 5.7.0 this doesn't seem to be the case? i see netty 4

 

since hadoop still uses netty 3 it is not clear to me how i can run map-red jobs over hbase anymore, since depending on hadoop and hbase (both provided) now causes trouble.

 

our approach so far has been to compile agains apache hadoop 2.6.0 and apache hbase 1.0.0, and then run the jobs on cdh 5.4.x with some hbase jars added to the hadoop classpath. i think the list of extra jars looked something like this:

HADOOP_CLASSPATH="$HADOOP_CLASSPATH:/usr/lib/hbase/conf:/usr/lib/hbase/hbase-client.jar:/usr/lib/hbase/hbase-common.jar:/usr/lib/hbase/hbase-protocol.jar:/usr/lib/hbase/hbase-server.jar:/usr/lib/hbase/lib/zookeeper.jar:/usr/lib/hbase/lib/htrace-core.jar:/usr/lib/hbase/hbase-hadoop-compat.jar:/usr/lib/hbase/hbase-hadoop2-compat.jar"

 

this no longer works. the first error is now:

Exception in thread "main" java.io.IOException: java.lang.reflect.InvocationTargetException
	at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:240)
	at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:218)
	at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:119)
	...
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
	at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:238)
	... 25 more
Caused by: java.lang.NoClassDefFoundError: io/netty/channel/EventLoopGroup
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:274)
	at org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:2138)
	at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2103)
	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2197)
	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2223)
	at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.<init>(ConnectionManager.java:665)
	... 30 more
Caused by: java.lang.ClassNotFoundException: io.netty.channel.EventLoopGroup
	at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
	... 37 more

 the missing netty jar is from netty 4 it seems. modifying the HADOOP_CLASSPATH to include /usr/lib/hbase/lib/netty-all-4.0.23.Final.jar makes this go away, but causes the job to hang with:

 

3 13:04:14 INFO client.RpcRetryingCaller: Call exception, tries=10, retries=35, started=68250 ms ago, cancelled=false, msg=row '' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=node04,60020,1471036308906, seqNum=0
16/08/13 13:04:34 INFO client.RpcRetryingCaller: Call exception, tries=11, retries=35, started=88434 ms ago, cancelled=false, msg=row '' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=node04,60020,1471036308906, seqNum=0
16/08/13 13:04:54 INFO client.RpcRetryingCaller: Call exception, tries=12, retries=35, started=108630 ms ago, cancelled=false, msg=row '' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=node04,60020,1471036308906, seqNum=0

i tried instead putting hbase client shaded jar on classpath at runtime, but this gives:

 

Exception in thread "main" java.lang.ClassCastException: org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$GetFileInfoRequestProto cannot be cast to org.apache.hadoop.hbase.shaded.com.google.protobuf.Message
    at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:227)
    at com.sun.proxy.$Proxy12.getFileInfo(Unknown Source)
    at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:757)

 

any ideas how to proceed?

1 REPLY 1

Re: map-red over hbase in cdh 5.7+

New Contributor

Hi, did you (or anyone?) happen to solve this issue ? I am having something similar.

hadoop 2.6.0 cdh 5.7.5 hbase 1.2.0 , 

and I'm having this error: 

 

Caused by: java.lang.RuntimeException: Could not connect to HBase with zookeeper quorum [causata-perf-dan02,causata-perf-dan03]
        at com.causata.sdal.SharedDataAccessLayerFactory$HBaseSDALBuilder$1.call(SharedDataAccessLayerFactory.java:152) ~[infrastructure-7.2.0022-SNAPSHOT.jar:]
        at com.causata.sdal.FakeUserUtil$2.run(FakeUserUtil.java:27) ~[infrastructure-7.2.0022-SNAPSHOT.jar:]
        ... 47 common frames omitted
Caused by: java.io.IOException: java.lang.reflect.InvocationTargetException
        at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:240) ~[hbase-client-1.2.0-cdh5.7.5.jar:na]
        at org.apache.hadoop.hbase.client.ConnectionManager.createConnection(ConnectionManager.java:433) ~[hbase-client-1.2.0-cdh5.7.5.jar:na]
        at org.apache.hadoop.hbase.client.ConnectionManager.createConnection(ConnectionManager.java:426) ~[hbase-client-1.2.0-cdh5.7.5.jar:na]
        at org.apache.hadoop.hbase.client.ConnectionManager.getConnectionInternal(ConnectionManager.java:304) ~[hbase-client-1.2.0-cdh5.7.5.jar:na]
        at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:231) ~[hbase-client-1.2.0-cdh5.7.5.jar:na]
        at com.causata.sdal.SharedDataAccessLayerFactory$HBaseSDALBuilder$1.call(SharedDataAccessLayerFactory.java:149) ~[infrastructure-7.2.0022-SNAPSHOT.jar:]
        ... 48 common frames omitted
Caused by: java.lang.reflect.InvocationTargetException: null
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:1.7.0_75]
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) ~[na:1.7.0_75]
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[na:1.7.0_75]
        at java.lang.reflect.Constructor.newInstance(Constructor.java:526) ~[na:1.7.0_75]
        at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:238) ~[hbase-client-1.2.0-cdh5.7.5.jar:na]
        ... 53 common frames omitted
Caused by: java.lang.NoClassDefFoundError: io/netty/channel/EventLoopGroup
        at java.lang.Class.forName0(Native Method) ~[na:1.7.0_75]
        at java.lang.Class.forName(Class.java:274) ~[na:1.7.0_75]
        at org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:2138) ~[hadoop-common-2.6.0-cdh5.7.5.jar:na]
        at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2103) ~[hadoop-common-2.6.0-cdh5.7.5.jar:na]
        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2197) ~[hadoop-common-2.6.0-cdh5.7.5.jar:na]
        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2223) ~[hadoop-common-2.6.0-cdh5.7.5.jar:na]
        at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.<init>(ConnectionManager.java:665) ~[hbase-client-1.2.0-cdh5.7.5.jar:na]
        ... 58 common frames omitted
Caused by: java.lang.ClassNotFoundException: io.netty.channel.EventLoopGroup
        at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1714) ~[catalina.jar:7.0.34]
        at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1559) ~[catalina.jar:7.0.34]

 

 

Don't have an account?
Coming from Hortonworks? Activate your account here