Support Questions

mpaul · ‎11-30-2017

Hi,

I get this error when running Hive Interactive Server :

2017-11-30 12:22:49,298 [main] INFO  tools.SliderUtils - JVM initialized into secure mode with kerberos realm DOMAIN
2017-11-30 12:22:50,202 [main] WARN  shortcircuit.DomainSocketFactory - The short-circuit local reads feature cannot be used because libhadoop cannot be loaded.
2017-11-30 12:22:50,345 [main] INFO  client.AHSProxy - Connecting to Application History server at host/10.121.206.118:10200
2017-11-30 12:22:50,705 [main] INFO  client.RequestHedgingRMFailoverProxyProvider - Looking for the active RM in [rm1, rm2]...
2017-11-30 12:22:50,840 [main] INFO  client.RequestHedgingRMFailoverProxyProvider - Found active RM [rm1]
2017-11-30 12:22:50,850 [main] INFO  client.SliderClient - Cluster llap0 is in a terminated state FAILED
2017-11-30 12:22:50,852 [main] INFO  util.ExitUtil - Exiting with status 0
2017-11-30 12:22:53,280 [main] INFO  tools.SliderUtils - JVM initialized into secure mode with kerberos realm DOMAIN
2017-11-30 12:22:54,061 [main] WARN  shortcircuit.DomainSocketFactory - The short-circuit local reads feature cannot be used because libhadoop cannot be loaded.
2017-11-30 12:22:54,168 [main] INFO  client.AHSProxy - Connecting to Application History server at host/10.121.206.118:10200
2017-11-30 12:22:54,205 [main] INFO  client.RequestHedgingRMFailoverProxyProvider - Looking for the active RM in [rm1, rm2]...
2017-11-30 12:22:54,463 [main] INFO  client.RequestHedgingRMFailoverProxyProvider - Found active RM [rm1]
2017-11-30 12:22:54,546 [main] INFO  zk.ZKIntegration - Binding ZK client to host:2181,host:2181,host:2181
2017-11-30 12:22:54,566 [main] INFO  zk.BlockingZKWatcher - waiting for ZK event
2017-11-30 12:22:54,600 [main-EventThread] INFO  zk.BlockingZKWatcher - ZK binding callback received
2017-11-30 12:22:54,631 [main] INFO  zk.RegistrySecurity - Enabling ZK sasl client: jaasClientEntry = Client, principal = null, keytab = null
2017-11-30 12:22:54,655 [main] INFO  imps.CuratorFrameworkImpl - Starting
2017-11-30 12:22:54,663 [main-SendThread(host:2181)] WARN  zookeeper.ClientCnxn - SASL configuration failed: javax.security.auth.login.LoginException: No key to store Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
2017-11-30 12:22:54,666 [main-EventThread] ERROR curator.ConnectionState - Authentication failed
2017-11-30 12:22:54,671 [main-EventThread] INFO  state.ConnectionStateManager - State change: CONNECTED
2017-11-30 12:22:54,682 [main] WARN  client.SliderClient - Error deleting registry entry /users/hive/services/org-apache-slider/llap0: org.apache.hadoop.registry.client.exceptions.NoPathPermissionsException: `/registry/users/hive/services/org-apache-slider/llap0': Not authorized to access path; ACLs: [null ACL]: KeeperErrorCode = NoAuth for /registry/users/hive/services/org-apache-slider/llap0 
org.apache.hadoop.registry.client.exceptions.NoPathPermissionsException: `/registry/users/hive/services/org-apache-slider/llap0': Not authorized to access path; ACLs: [null ACL]: KeeperErrorCode = NoAuth for /registry/users/hive/services/org-apache-slider/llap0
    at org.apache.hadoop.registry.client.impl.zk.CuratorService.operationFailure(CuratorService.java:385)
    at org.apache.hadoop.registry.client.impl.zk.CuratorService.operationFailure(CuratorService.java:364)
    at org.apache.hadoop.registry.client.impl.zk.CuratorService.zkDelete(CuratorService.java:684)
    at org.apache.hadoop.registry.client.impl.zk.RegistryOperationsService.delete(RegistryOperationsService.java:160)
    at org.apache.slider.client.SliderClient.actionDestroy(SliderClient.java:677)
    at org.apache.slider.client.SliderClient.exec(SliderClient.java:379)
    at org.apache.slider.client.SliderClient.runService(SliderClient.java:333)
    at org.apache.slider.core.main.ServiceLauncher.launchService(ServiceLauncher.java:188)
    at org.apache.slider.core.main.ServiceLauncher.launchServiceRobustly(ServiceLauncher.java:475)
    at org.apache.slider.core.main.ServiceLauncher.launchServiceAndExit(ServiceLauncher.java:403)
    at org.apache.slider.core.main.ServiceLauncher.serviceMain(ServiceLauncher.java:630)
    at org.apache.slider.Slider.main(Slider.java:49)
Caused by: org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /registry/users/hive/services/org-apache-slider/llap0
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:113)
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
    at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873)
    at org.apache.curator.framework.imps.DeleteBuilderImpl$5.call(DeleteBuilderImpl.java:238)
    at org.apache.curator.framework.imps.DeleteBuilderImpl$5.call(DeleteBuilderImpl.java:233)
    at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107)
    at org.apache.curator.framework.imps.DeleteBuilderImpl.pathInForeground(DeleteBuilderImpl.java:230)
    at org.apache.curator.framework.imps.DeleteBuilderImpl.forPath(DeleteBuilderImpl.java:214)
    at org.apache.curator.framework.imps.DeleteBuilderImpl.forPath(DeleteBuilderImpl.java:41)
    at org.apache.hadoop.registry.client.impl.zk.CuratorService.zkDelete(CuratorService.java:680)
    ... 9 more
2017-11-30 12:22:54,690 [main] INFO  client.SliderClient - Destroyed cluster llap0

Looks like some Kerberos properties were not properly set ? Has anyone already seen this error ?

Thanks

Manfred

matt_andruff · ‎12-03-2017

I had a weird issue when I used LDAP. The group mapping was not right. But it was a slightly different error. Does this folder exist?

dheer_vijji_rag · ‎12-04-2017

@Manfred PAUL

you can refer these:

https://community.hortonworks.com/questions/86100/llap-error.html

https://community.hortonworks.com/articles/90705/hive-llap-fails-with-invalidacl-for-llap-sasluser.h...

mpaul · ‎12-05-2017

@Venkata Sudheer Kumar M

I already looked at these topics, but everything is correct. But here is what I thinks is different in my case :

2017-12-05 12:05:47,992 [main] INFO  zk.RegistrySecurity - Enabling ZK sasl client: jaasClientEntry = Client, principal = null, keytab = null
2017-12-05 12:05:48,022 [main] INFO  imps.CuratorFrameworkImpl - Starting
2017-12-05 12:05:48,033 [main-SendThread(zer332su.distribution.edf.fr:2181)] WARN  zookeeper.ClientCnxn - SASL configuration failed: javax.security.auth.login.LoginException: No key to store Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
2017-12-05 12:05:48,035 [main-EventThread] ERROR curator.ConnectionState - Authentication failed
2017-12-05 12:05:48,045 [main-EventThread] INFO  state.ConnectionStateManager - State change: CONNECTED
2017-12-05 12:05:48,058 [main] WARN  client.SliderClient - Error deleting registry entry /users/hive/services/org-apache-slider/llap0: org.apache.hadoop.registry.client.exceptions.NoPathPermissionsException: `/registry/users/hive/services/org-apache-slider/llap0': Not authorized to access path; ACLs: [null ACL]: KeeperErrorCode = NoAuth for /registry/users/hive/services/org-apache-slider/llap0 
org.apache.hadoop.registry.client.exceptions.NoPathPermissionsException: `/registry/users/hive/services/org-apache-slider/llap0': Not authorized to access path; ACLs: [null ACL]: KeeperErrorCode = NoAuth for /registry/users/hive/services/org-apache-slider/llap0
	at org.apache.hadoop.registry.client.impl.zk.CuratorService.operationFailure(CuratorService.java:385)
	at org.apache.hadoop.registry.client.impl.zk.CuratorService.operationFailure(CuratorService.java:364)
	at org.apache.hadoop.registry.client.impl.zk.CuratorService.zkDelete(CuratorService.java:684)
	at org.apache.hadoop.registry.client.impl.zk.RegistryOperationsService.delete(RegistryOperationsService.java:160)
	at org.apache.slider.client.SliderClient.actionDestroy(SliderClient.java:677)
	at org.apache.slider.client.SliderClient.exec(SliderClient.java:379)
	at org.apache.slider.client.SliderClient.runService(SliderClient.java:333)
	at org.apache.slider.core.main.ServiceLauncher.launchService(ServiceLauncher.java:188)
	at org.apache.slider.core.main.ServiceLauncher.launchServiceRobustly(ServiceLauncher.java:475)
	at org.apache.slider.core.main.ServiceLauncher.launchServiceAndExit(ServiceLauncher.java:403)
	at org.apache.slider.core.main.ServiceLauncher.serviceMain(ServiceLauncher.java:630)
	at org.apache.slider.Slider.main(Slider.java:49)

Slider is not using kerberos correctly (no principal / keytab) ! Like it is written here :

2017-12-05 12:05:47,992 [main] INFO  zk.RegistrySecurity - Enabling ZK sasl client: jaasClientEntry = Client, principal = null, keytab = null

When I connect myself as the hive user here is what I get (without a principal / keytab to test):

sudo su - hive

klist
klist: No credentials cache found (ticket cache FILE:/tmp/krb5cc_xxxx)

/usr/hdp/current/zookeeper-client/bin/zkCli.sh -server $hostname

Connecting to $hostname
Welcome to ZooKeeper!
JLine support is enabled

WATCHER::

WatchedEvent state:AuthFailed type:None path:null



WATCHER::

WatchedEvent state:SyncConnected type:None path:null
[zk: $hostname(CONNECTED) 0]

Looking for the acl of /registry/users/hive/services/org-apache-slider/llap0

[zk: $hostname(CONNECTED) 0] getAcl /registry/users/hive/services/org-apache-slider/llap0
'world,'anyone
: r
'sasl,'yarn
: cdrwa
'sasl,'jhs
: cdrwa
'sasl,'hdfs
: cdrwa
'sasl,'rm
: cdrwa
'sasl,'hive
: cdrwa
'sasl,'hive/hostname@REALM
: cdrwa

When the slider application is started, shoudn't this one use the keytab of the hive user ?

@Matt Andruff

I currently test with kerberos only. At the end I will use a custom authentication (like I used before)

Sould I check slider ?

Thanks

dheer_vijji_rag · ‎12-07-2017

@Manfred PAUL

In relation to this:

When I connect myself as the hive user here is what I get (without a principal / keytab to test):

sudo su - hive

klist
klist:No credentials cache found (ticket cache FILE:/tmp/krb5cc_xxxx)

are you missing kinit?

Can you please provide how you are generating the ticket for hive user? did you configure it in .bashrc?

mpaul · ‎12-08-2017

@Venkata Sudheer Kumar M

Hi,

well I did this on purpose (not generating a kerberos ticket). Shouldn't this be done by Hive / Slider themselves before starting the LLAP application ? I have not read anything about this in the documentation.

My goal was to show that anonymous connections are allowed (read) but not for writing, so I cannot explain this exception :

2017-12-05 12:05:48,058 [main] WARN  client.SliderClient - Error deleting registry entry /users/hive/services/org-apache-slider/llap0: org.apache.hadoop.registry.client.exceptions.NoPathPermissionsException: `/registry/users/hive/services/org-apache-slider/llap0': Not authorized to access path; ACLs: [null ACL]: KeeperErrorCode = NoAuth for /registry/users/hive/services/org-apache-slider/llap0

Why does Slider run without Kerberos credentials ?

2017-12-05 12:05:47,992 [main] INFO  zk.RegistrySecurity - Enabling ZK sasl client: jaasClientEntry = Client, principal = null, keytab = null

Just to remember : the original problem is that LLAP does not start.

I'll post the YARN log later

Cheers

matt_andruff · ‎12-08-2017

@Manfred PAUL

Absolutely check the slider log.

Also check the yarn log for any hints it gives... like what it used to pass in as user group to the job that was being launched. This is how I found my similar issue.

You can check to see if the keytab works for llap. ("su - llap", kinit using the keytab) see if it works correctly. I seem llap to recall it needing to be fixed after install.

Also:

If you aren't using a central authorization system and you are only using kerberos. You created the user on all machines?

dheer_vijji_rag · ‎12-11-2017

@Manfred PAUL

Yes, i was looking at only the current session.

Can you please check whether you have all the keytabs generated properly for all the services?

matt_andruff · ‎12-11-2017

@Manfred PAUL

Im my last reply I actually meant to include a link to this article:

https://community.hortonworks.com/articles/110104/fixing-hiveserver-interactive-llap-failures-with-e...

Cloudera Community

Support Questions

hive / llap / slider / kerberos : error while launching llap