Hi everyone,
we have a strange issue with an actual HDP setup for our dev environment. We installed - like many times before - ranger and enabled the ranger plugins. We created the polices for our services in Ranger and Hive works, HDFS works, Knox works, Yarn works fine.. but testing the connection to hbase doesn't work and we have no idea why. That's our first installation with HDP 2.6, but we installed and using many clusters of the previous versions (2.0 - 2.5).
The failure:
Testing connection for habse in ranger - the popup error:
org.apache.ranger.plugin.client.HadoopException: getHBaseStatus:
Unable to check availability of Hbase environment
[multinode-hdpqa_hbase]..
getHBaseStatus: Unable to check availability of Hbase environment [multinode-hdpqa_hbase]..
java.lang.reflect.InvocationTargetException.
Could not initialize class org.apache.hadoop.hbase.protobuf.ProtobufUtil.
Extract of the xa_portal.log:
2017-05-10 08:17:02,788 [timed-executor-pool-0] ERROR apache.ranger.services.hbase.client.HBaseClient$1 (HBaseClient.java:178) - getHBaseStatus: Unable to check availability of Hbase environment [multinode-h
dpqa_hbase].java.io.IOException: java.lang.reflect.InvocationTargetException
2017-05-10 08:17:02,789 [timed-executor-pool-0] ERROR apache.ranger.services.hbase.client.HBaseClient (HBaseClient.java:102) - <== HBaseClient.testConnection(): Unable to retrieve any databases using given p
arameters
org.apache.ranger.plugin.client.HadoopException: getHBaseStatus: Unable to check availability of Hbase environment [multinode-hdpqa_hbase].
at org.apache.ranger.services.hbase.client.HBaseClient$1.run(HBaseClient.java:175)
at org.apache.ranger.services.hbase.client.HBaseClient$1.run(HBaseClient.java:128)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:360)
at org.apache.ranger.services.hbase.client.HBaseClient.getHBaseStatus(HBaseClient.java:128)
at org.apache.ranger.services.hbase.client.HBaseClient.connectionTest(HBaseClient.java:100)
at org.apache.ranger.services.hbase.client.HBaseResourceMgr.connectionTest(HBaseResourceMgr.java:48)
at org.apache.ranger.services.hbase.RangerServiceHBase.validateConfig(RangerServiceHBase.java:59)
at org.apache.ranger.biz.ServiceMgr$ValidateCallable.actualCall(ServiceMgr.java:560)
at org.apache.ranger.biz.ServiceMgr$ValidateCallable.actualCall(ServiceMgr.java:547)
at org.apache.ranger.biz.ServiceMgr$TimedCallable.call(ServiceMgr.java:508)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: java.lang.reflect.InvocationTargetException
at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:240)
at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:218)
at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:119)
at org.apache.hadoop.hbase.client.HBaseAdmin.checkHBaseAvailable(HBaseAdmin.java:3230)
at org.apache.ranger.services.hbase.client.HBaseClient$1.run(HBaseClient.java:138)
... 14 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.GeneratedConstructorAccessor104.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:238)
... 18 more
Caused by: java.lang.NoClassDefFoundError: Could not initialize class org.apache.hadoop.hbase.protobuf.ProtobufUtil
at org.apache.hadoop.hbase.ClusterId.parseFrom(ClusterId.java:64)
at org.apache.hadoop.hbase.zookeeper.ZKClusterId.readClusterIdZNode(ZKClusterId.java:75)
at org.apache.hadoop.hbase.client.ZooKeeperRegistry.getClusterId(ZooKeeperRegistry.java:105)
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.retrieveClusterId(ConnectionManager.java:889)
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.<init>(ConnectionManager.java:645)
... 22 more
2017-05-10 08:17:02,789 [timed-executor-pool-0] ERROR apache.ranger.services.hbase.client.HBaseResourceMgr (HBaseResourceMgr.java:50) - <== HBaseResourceMgr.connectionTest() Error: org.apache.ranger.plugin.client.HadoopException: getHBaseStatus: Unable to check availability of Hbase environment [multinode-hdpqa_hbase].
2017-05-10 08:17:02,789 [timed-executor-pool-0] ERROR org.apache.ranger.services.hbase.RangerServiceHBase (RangerServiceHBase.java:61) - <== RangerServiceHBase.validateConfig() Error:org.apache.ranger.plugin.client.HadoopException: getHBaseStatus: Unable to check availability of Hbase environment [multinode-hdpqa_hbase].
2017-05-10 08:17:02,789 [timed-executor-pool-0] ERROR org.apache.ranger.biz.ServiceMgr$TimedCallable (ServiceMgr.java:510) - TimedCallable.call: Error:org.apache.ranger.plugin.client.HadoopException: getHBaseStatus: Unable to check availability of Hbase environment [multinode-hdpqa_hbase].
2017-05-10 08:17:02,790 [http-bio-6080-exec-60] ERROR org.apache.ranger.biz.ServiceMgr (ServiceMgr.java:188) - ==> ServiceMgr.validateConfig Error:org.apache.ranger.plugin.client.HadoopException: org.apache.ranger.plugin.client.HadoopException: getHBaseStatus: Unable to check availability of Hbase environment [multinode-hdpqa_hbase].
2017-05-10 08:18:24,904 [http-bio-6080-exec-80] INFO org.apache.ranger.security.handler.RangerAuthenticationProvider (RangerAuthenticationProvider.java:147) - Authentication with SHA-256 failed. Now trying with MD5.
There are no errors in the hdfs, hbase, zookeeper, phoenix or yarn logs...
The facts:
The credentials are correct, hbase works fine - also with the users and the audit in ranger,
HDP-Version: HDP-2.6.0.3
Cluster: 6 Nodes - each node with 64 GB RAM, 8 Cores and a bunch of storage
Environment: DEV (testing cluster)
Services installed:
HDFS, YARN, MapReduce2, Tez, Hive, HBase, Pig, Sqoop, ZooKeeper, Ambari Infra, Ambari Metrics, Kafka, Knox, Ranger, Spark2, Zeppelin Notebook, Slider
Security:
Simple (SSL, Knox with LDAP Demo, Unix...)
Any ideas?
Best regards,
Jan
