Support Questions
Find answers, ask questions, and share your expertise
Announcements
Check out our newest addition to the community, the Cloudera Innovation Accelerator group hub.

HBase on Slider - YARN Registry KeeperErrorCode = NoAuth

I enabled Kerberos on an HDP2.5.3 cluster via Ambari. I am trying run HBase on Slider, but it is failing with an error related to YarnRegistry in ZooKeeper.

2017-02-01 08:20:16,968 [main] INFO  appmaster.SliderAppMaster - Starting Yarn registry
2017-02-01 08:20:17,183 [main] INFO  appmaster.SliderAppMaster - Service YarnRegistry in state YarnRegistry: STARTED 
Connection="fixed ZK quorum "los90hdpc4m2.EXAMPLE.COM:2181,los90hdpc4m3.EXAMPLE.COM:2181,los90hdpc4m4.EXAMPLE.COM:2181" 
" root="/registry" secure cluster; secure registry; Curator service access policy: anon; System ACLs:
0x01: 'world,'anyone
 0x1f: 'sasl,'yarn@LAKEE.EXAMPLE.COM
 0x1f: 'sasl,'mapred@LAKEE.EXAMPLE.COM
 0x1f: 'sasl,'hdfs@LAKEE.EXAMPLE.COM
 0x1f: 'sasl,'hadoop@LAKEE.EXAMPLE.COM
User: hbase: hbase (auth:KERBEROS) hasKerberosCredentials=false isFromKeytab=false kerberos is enabled in Hadoop =true; 
 Kerberos Realm: LAKEE.EXAMPLE.COM; 
 java.security.auth.login.config=(undefined); 
 zookeeper.sasl.client=false; 
 zookeeper.allowSaslFailedClients=(undefined but defaults to true); 
 zookeeper.maintain_connection_despite_sasl_failure=(undefined)

2017-02-01 08:20:23,071 [main] INFO  appmaster.SliderAppMaster - Slider AM Security Mode: KEYTAB
2017-02-01 08:20:23,108 [main] INFO  appmaster.SliderAppMaster - Kind: RM_DELEGATION_TOKEN, Service: 10.0.0.111:8032,10.0.0.112:8032, Ident: (owner=hbase@LAKEE.EXAMPLE.COM, renewer=yarn, realUser=, issueDate=1485958371269, maxDate=1486563171269, sequenceNumber=3341, masterKeyId=684); owner=hbase@LAKEE.EXAMPLE.COM, renewer=yarn, realUser=, issueDate=1485958371269, maxDate=1486563171269, sequenceNumber=3341, masterKeyId=684; Renewer: yarn; Issued: 2/1/17 8:12 AM; Max Date: 2/8/17 8:12 AM

2017-02-01 08:20:23,271 [main] INFO  security.SecurityConfiguration - No host keytab file path specified. Will attempt to retrieve keytab file hbase-slider.headless.keytab as a local resource for the container
2017-02-01 08:20:23,271 [main] INFO  appmaster.SliderAppMaster - Logging in as hbase with keytab keytabs/hbase-slider.headless.keytab
2017-02-01 08:20:23,333 [main] INFO  security.UserGroupInformation - Login successful for user hbase using keytab file /mnt/hdfs5/yarn/local/usercache/hbase/appcache/application_1485859889689_0043/container_e340_1485859889689_0043_01_000001/keytabs/hbase-slider.headless.keytab

2017-02-01 08:20:29,326 [AmExecutor-006] ERROR actions.QueueExecutor - Exception processing org.apache.slider.server.appmaster.actions.ActionRegisterServiceInstance@14479ca8 name='ActionRegisterServiceInstance', delay=0, attrs=0, sequenceNumber=5}: org.apache.hadoop.registry.client.exceptions.NoPathPermissionsException: `/registry/users/hbase/services/org-apache-slider/hbase95': Not authorized to access path; ACLs: [
0x01: 'world,'anyone
 0x1f: 'sasl,'yarn@LAKEE.EXAMPLE.COM
 0x1f: 'sasl,'mapred@LAKEE.EXAMPLE.COM
 0x1f: 'sasl,'hdfs@LAKEE.EXAMPLE.COM
 0x1f: 'sasl,'hadoop@LAKEE.EXAMPLE.COM
 ]: KeeperErrorCode = NoAuth for /registry/users/hbase/services/org-apache-slider/hbase95
org.apache.hadoop.registry.client.exceptions.NoPathPermissionsException: `/registry/users/hbase/services/org-apache-slider/hbase95': Not authorized to access path; ACLs: [
0x01: 'world,'anyone
 0x1f: 'sasl,'yarn@LAKEE.EXAMPLE.COM
 0x1f: 'sasl,'mapred@LAKEE.EXAMPLE.COM
 0x1f: 'sasl,'hdfs@LAKEE.EXAMPLE.COM
 0x1f: 'sasl,'hadoop@LAKEE.EXAMPLE.COM
 ]: KeeperErrorCode = NoAuth for /registry/users/
        at org.apache.hadoop.registry.client.impl.zk.CuratorService.operationFailure(CuratorService.java:381)
        at org.apache.hadoop.registry.client.impl.zk.CuratorService.zkCreate(CuratorService.java:598)
        at org.apache.hadoop.registry.client.impl.zk.CuratorService.zkSet(CuratorService.java:638)
        at org.apache.hadoop.registry.client.impl.zk.RegistryOperationsService.bind(RegistryOperationsService.java:114)
        at org.apache.slider.server.services.yarnregistry.YarnRegistryViewForProviders.putService(YarnRegistryViewForProviders.java:189)
        at org.apache.slider.server.services.yarnregistry.YarnRegistryViewForProviders.registerSelf(YarnRegistryViewForProviders.java:224)
        at org.apache.slider.server.appmaster.SliderAppMaster.registerServiceInstance(SliderAppMaster.java:1343)
        at org.apache.slider.server.appmaster.actions.ActionRegisterServiceInstance.execute(ActionRegisterServiceInstance.java:57)
        at org.apache.slider.server.appmaster.actions.QueueExecutor.run(QueueExecutor.java:73)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /registry/users/
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:113)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
        
        

In reading, I thought my understanding was that YARN was supposed to create this registry structure in ZooKeeper. However, it seems that the Application Manager for my HBase instance is using Curator to try and create this (and getting the permission denied). Any advice on what one does to get past this error?

1 ACCEPTED SOLUTION

Cloudera Employee

Is hadoop.registry.secure set to true in yarn-site? If yes, then the following properties should be set as well:

hadoop.registry.client.auth=kerberos
hadoop.registry.jaas.context=Client

But this may not work because of this issue: https://issues.apache.org/jira/browse/YARN-6104

This will be fixed in HDP2.6.

View solution in original post

1 REPLY 1

Cloudera Employee

Is hadoop.registry.secure set to true in yarn-site? If yes, then the following properties should be set as well:

hadoop.registry.client.auth=kerberos
hadoop.registry.jaas.context=Client

But this may not work because of this issue: https://issues.apache.org/jira/browse/YARN-6104

This will be fixed in HDP2.6.