question Re: Atlas Metadata server not starting in Support Questions

Atlas Metadata server not starting

bkandalkar — Fri, 25 Jan 2019 19:29:47 GMT

Hello Team,

After enabling Kerberos on HDP-2.6 cluster, Atlas metadata server is not starting. Getting following error:

java exception
ERROR Java::OrgApacheHadoopHbaseIpc::RemoteWithExtrasException: org.apache.hadoop.hbase.coprocessor.CoprocessorException: HTTP 503 Error: HTTP 503
    at org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor.grant(RangerAuthorizationCoprocessor.java:1236)
    at org.apache.hadoop.hbase.protobuf.generated.AccessControlProtos$AccessControlService$1.grant(AccessControlProtos.java:9933)
    at org.apache.hadoop.hbase.protobuf.generated.AccessControlProtos$AccessControlService.callMethod(AccessControlProtos.java:10097)
    at org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:7857)
    at org.apache.hadoop.hbase.regionserver.RSRpcServices.execServiceOnRegion(RSRpcServices.java:1999)
    at org.apache.hadoop.hbase.regionserver.RSRpcServices.execService(RSRpcServices.java:1981)
    at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32389)
    at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2150)
    at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
    at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:187)
    at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:167)

How to solve this? Please suggest.

Thanks,

Bhushan

Re: Atlas Metadata server not starting

Shelton — Sat, 26 Jan 2019 00:44:07 GMT

@Bhushan Kandalkar

This looks like an authorization issue in ranger, Atlas need the hbase DB for the backend ensure all hbase components are running, Check the in Ranger that Atlas has the correct permissions to 2 important tables Atlas_titan and ATLAS_ENTITY_AUDIT_EVENTS see attached screenshots

Log on the Zookeeper CLI and check the ACL's you should have a similar output

[zk: localhost:2181(CONNECTED) 0] getAcl  /hbase-unsecure/table/ATLAS_ENTITY_AUDIT_EVENTS
'world,'anyone
: cdrwa
[zk: localhost:2181(CONNECTED) 1] getAcl  /hbase-unsecure/table/atlas_titan
'world,'anyone
: cdrwa
[zk: localhost:2181(CONNECTED)2] getAcl  /hbase-unsecure/table/hbase:meta
'world,'anyone
: cdrwa
[zk: localhost:2181(CONNECTED) 3] getAcl  /hbase-unsecure/table/hbase:namespace
'world,'anyone
: cdrwa

Please revert

Re: Atlas Metadata server not starting

bkandalkar — Mon, 28 Jan 2019 16:41:06 GMT

This issue occurred as we have used Load balancer while implementing Ranger HA. Breaking HA and keeping only one Ranger Admin resolved this issue for us.

Re: Atlas Metadata server not starting

Shelton — Mon, 28 Jan 2019 16:50:47 GMT

@Bhushan Kandalkar

Good it worked out but you shouldn't have omitted the information about the architecture ie Load balancer such info is critical in the analysis ....:-)

Happy hadooping