I enabled Kerberos on an HDP2.5.3 cluster via Ambari. I am trying run HBase on Slider, but it is failing with an error related to YarnRegistry in ZooKeeper.
2017-02-01 08:20:16,968 [main] INFO appmaster.SliderAppMaster - Starting Yarn registry
2017-02-01 08:20:17,183 [main] INFO appmaster.SliderAppMaster - Service YarnRegistry in state YarnRegistry: STARTED
Connection="fixed ZK quorum "los90hdpc4m2.EXAMPLE.COM:2181,los90hdpc4m3.EXAMPLE.COM:2181,los90hdpc4m4.EXAMPLE.COM:2181"
" root="/registry" secure cluster; secure registry; Curator service access policy: anon; System ACLs:
0x01: 'world,'anyone
0x1f: 'sasl,'yarn@LAKEE.EXAMPLE.COM
0x1f: 'sasl,'mapred@LAKEE.EXAMPLE.COM
0x1f: 'sasl,'hdfs@LAKEE.EXAMPLE.COM
0x1f: 'sasl,'hadoop@LAKEE.EXAMPLE.COM
User: hbase: hbase (auth:KERBEROS) hasKerberosCredentials=false isFromKeytab=false kerberos is enabled in Hadoop =true;
Kerberos Realm: LAKEE.EXAMPLE.COM;
java.security.auth.login.config=(undefined);
zookeeper.sasl.client=false;
zookeeper.allowSaslFailedClients=(undefined but defaults to true);
zookeeper.maintain_connection_despite_sasl_failure=(undefined)
2017-02-01 08:20:23,071 [main] INFO appmaster.SliderAppMaster - Slider AM Security Mode: KEYTAB
2017-02-01 08:20:23,108 [main] INFO appmaster.SliderAppMaster - Kind: RM_DELEGATION_TOKEN, Service: 10.0.0.111:8032,10.0.0.112:8032, Ident: (owner=hbase@LAKEE.EXAMPLE.COM, renewer=yarn, realUser=, issueDate=1485958371269, maxDate=1486563171269, sequenceNumber=3341, masterKeyId=684); owner=hbase@LAKEE.EXAMPLE.COM, renewer=yarn, realUser=, issueDate=1485958371269, maxDate=1486563171269, sequenceNumber=3341, masterKeyId=684; Renewer: yarn; Issued: 2/1/17 8:12 AM; Max Date: 2/8/17 8:12 AM
2017-02-01 08:20:23,271 [main] INFO security.SecurityConfiguration - No host keytab file path specified. Will attempt to retrieve keytab file hbase-slider.headless.keytab as a local resource for the container
2017-02-01 08:20:23,271 [main] INFO appmaster.SliderAppMaster - Logging in as hbase with keytab keytabs/hbase-slider.headless.keytab
2017-02-01 08:20:23,333 [main] INFO security.UserGroupInformation - Login successful for user hbase using keytab file /mnt/hdfs5/yarn/local/usercache/hbase/appcache/application_1485859889689_0043/container_e340_1485859889689_0043_01_000001/keytabs/hbase-slider.headless.keytab
2017-02-01 08:20:29,326 [AmExecutor-006] ERROR actions.QueueExecutor - Exception processing org.apache.slider.server.appmaster.actions.ActionRegisterServiceInstance@14479ca8 name='ActionRegisterServiceInstance', delay=0, attrs=0, sequenceNumber=5}: org.apache.hadoop.registry.client.exceptions.NoPathPermissionsException: `/registry/users/hbase/services/org-apache-slider/hbase95': Not authorized to access path; ACLs: [
0x01: 'world,'anyone
0x1f: 'sasl,'yarn@LAKEE.EXAMPLE.COM
0x1f: 'sasl,'mapred@LAKEE.EXAMPLE.COM
0x1f: 'sasl,'hdfs@LAKEE.EXAMPLE.COM
0x1f: 'sasl,'hadoop@LAKEE.EXAMPLE.COM
]: KeeperErrorCode = NoAuth for /registry/users/hbase/services/org-apache-slider/hbase95
org.apache.hadoop.registry.client.exceptions.NoPathPermissionsException: `/registry/users/hbase/services/org-apache-slider/hbase95': Not authorized to access path; ACLs: [
0x01: 'world,'anyone
0x1f: 'sasl,'yarn@LAKEE.EXAMPLE.COM
0x1f: 'sasl,'mapred@LAKEE.EXAMPLE.COM
0x1f: 'sasl,'hdfs@LAKEE.EXAMPLE.COM
0x1f: 'sasl,'hadoop@LAKEE.EXAMPLE.COM
]: KeeperErrorCode = NoAuth for /registry/users/
at org.apache.hadoop.registry.client.impl.zk.CuratorService.operationFailure(CuratorService.java:381)
at org.apache.hadoop.registry.client.impl.zk.CuratorService.zkCreate(CuratorService.java:598)
at org.apache.hadoop.registry.client.impl.zk.CuratorService.zkSet(CuratorService.java:638)
at org.apache.hadoop.registry.client.impl.zk.RegistryOperationsService.bind(RegistryOperationsService.java:114)
at org.apache.slider.server.services.yarnregistry.YarnRegistryViewForProviders.putService(YarnRegistryViewForProviders.java:189)
at org.apache.slider.server.services.yarnregistry.YarnRegistryViewForProviders.registerSelf(YarnRegistryViewForProviders.java:224)
at org.apache.slider.server.appmaster.SliderAppMaster.registerServiceInstance(SliderAppMaster.java:1343)
at org.apache.slider.server.appmaster.actions.ActionRegisterServiceInstance.execute(ActionRegisterServiceInstance.java:57)
at org.apache.slider.server.appmaster.actions.QueueExecutor.run(QueueExecutor.java:73)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /registry/users/
at org.apache.zookeeper.KeeperException.create(KeeperException.java:113)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
In reading, I thought my understanding was that YARN was supposed to create this registry structure in ZooKeeper. However, it seems that the Application Manager for my HBase instance is using Curator to try and create this (and getting the permission denied). Any advice on what one does to get past this error?
