Skip to main content
bkandalkar
bkandalkar
Gold ⭐️
January 25, 2019
Solved

Atlas Metadata server not starting

Hello Team,

After enabling Kerberos on HDP-2.6 cluster, Atlas metadata server is not starting. Getting following error:

java exception
ERROR Java::OrgApacheHadoopHbaseIpc::RemoteWithExtrasException: org.apache.hadoop.hbase.coprocessor.CoprocessorException: HTTP 503 Error: HTTP 503
at org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor.grant(RangerAuthorizationCoprocessor.java:1236)
at org.apache.hadoop.hbase.protobuf.generated.AccessControlProtos$AccessControlService$1.grant(AccessControlProtos.java:9933)
at org.apache.hadoop.hbase.protobuf.generated.AccessControlProtos$AccessControlService.callMethod(AccessControlProtos.java:10097)
at org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:7857)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.execServiceOnRegion(RSRpcServices.java:1999)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.execService(RSRpcServices.java:1981)
at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32389)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2150)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:187)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:167)

How to solve this? Please suggest.

Thanks,

Bhushan

bkandalkar
bkandalkarAnswer
Gold ⭐️
January 28, 2019

This issue occurred as we have used Load balancer while implementing Ranger HA. Breaking HA and keeping only one Ranger Admin resolved this issue for us.

View in thread

3 replies

Shelton
Shelton
Gold ⭐️⭐️⭐️
January 26, 2019

@Bhushan Kandalkar

This looks like an authorization issue in ranger, Atlas need the hbase DB for the backend ensure all hbase components are running, Check the in Ranger that Atlas has the correct permissions to 2 important tables Atlas_titan and ATLAS_ENTITY_AUDIT_EVENTS see attached screenshots

Log on the Zookeeper CLI and check the ACL's you should have a similar output

[zk: localhost:2181(CONNECTED) 0] getAcl /hbase-unsecure/table/ATLAS_ENTITY_AUDIT_EVENTS
'world,'anyone
: cdrwa
[zk: localhost:2181(CONNECTED) 1] getAcl /hbase-unsecure/table/atlas_titan
'world,'anyone
: cdrwa
[zk: localhost:2181(CONNECTED)2] getAcl /hbase-unsecure/table/hbase:meta
'world,'anyone
: cdrwa
[zk: localhost:2181(CONNECTED) 3] getAcl /hbase-unsecure/table/hbase:namespace
'world,'anyone
: cdrwa

Please revert


atlas-titan.pngatlas-entity-audit.png
Translate
bkandalkar
bkandalkarAuthorAnswer
Gold ⭐️
January 28, 2019

This issue occurred as we have used Load balancer while implementing Ranger HA. Breaking HA and keeping only one Ranger Admin resolved this issue for us.

Translate
Shelton
Shelton
Gold ⭐️⭐️⭐️
January 28, 2019

@Bhushan Kandalkar

Good it worked out but you shouldn't have omitted the information about the architecture ie Load balancer such info is critical in the analysis ....:-)

Happy hadooping

Translate
Terms & ConditionsAccessibility statement