Member since
10-31-2017
21
Posts
0
Kudos Received
0
Solutions
03-02-2018
02:29 PM
Hi, But what about the service configuration? Even when you re-add your host Ambari won't recognize the old service configuration. This has to be done when "reinstalling" the cluster ? Manfred
... View more
12-08-2017
12:52 PM
@Venkata Sudheer Kumar M Hi, well
I did this on purpose (not generating a kerberos ticket). Shouldn't
this be done by Hive / Slider themselves before starting the LLAP
application ? I have not read anything about this in the documentation. My goal was to show that anonymous connections are allowed (read) but not for writing, so I cannot explain this exception : 2017-12-05 12:05:48,058 [main] WARN client.SliderClient - Error deleting registry entry /users/hive/services/org-apache-slider/llap0: org.apache.hadoop.registry.client.exceptions.NoPathPermissionsException: `/registry/users/hive/services/org-apache-slider/llap0': Not authorized to access path; ACLs: [null ACL]: KeeperErrorCode = NoAuth for /registry/users/hive/services/org-apache-slider/llap0 Why does Slider run without Kerberos credentials ? 2017-12-05 12:05:47,992 [main] INFO zk.RegistrySecurity - Enabling ZK sasl client: jaasClientEntry = Client, principal = null, keytab = null
Just to remember : the original problem is that LLAP does not start. I'll post the YARN log later Cheers
... View more
12-05-2017
11:40 AM
@Venkata Sudheer Kumar M I already looked at these topics, but everything is correct. But here is what I thinks is different in my case : 2017-12-05 12:05:47,992 [main] INFO zk.RegistrySecurity - Enabling ZK sasl client: jaasClientEntry = Client, principal = null, keytab = null
2017-12-05 12:05:48,022 [main] INFO imps.CuratorFrameworkImpl - Starting
2017-12-05 12:05:48,033 [main-SendThread(zer332su.distribution.edf.fr:2181)] WARN zookeeper.ClientCnxn - SASL configuration failed: javax.security.auth.login.LoginException: No key to store Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
2017-12-05 12:05:48,035 [main-EventThread] ERROR curator.ConnectionState - Authentication failed
2017-12-05 12:05:48,045 [main-EventThread] INFO state.ConnectionStateManager - State change: CONNECTED
2017-12-05 12:05:48,058 [main] WARN client.SliderClient - Error deleting registry entry /users/hive/services/org-apache-slider/llap0: org.apache.hadoop.registry.client.exceptions.NoPathPermissionsException: `/registry/users/hive/services/org-apache-slider/llap0': Not authorized to access path; ACLs: [null ACL]: KeeperErrorCode = NoAuth for /registry/users/hive/services/org-apache-slider/llap0
org.apache.hadoop.registry.client.exceptions.NoPathPermissionsException: `/registry/users/hive/services/org-apache-slider/llap0': Not authorized to access path; ACLs: [null ACL]: KeeperErrorCode = NoAuth for /registry/users/hive/services/org-apache-slider/llap0
at org.apache.hadoop.registry.client.impl.zk.CuratorService.operationFailure(CuratorService.java:385)
at org.apache.hadoop.registry.client.impl.zk.CuratorService.operationFailure(CuratorService.java:364)
at org.apache.hadoop.registry.client.impl.zk.CuratorService.zkDelete(CuratorService.java:684)
at org.apache.hadoop.registry.client.impl.zk.RegistryOperationsService.delete(RegistryOperationsService.java:160)
at org.apache.slider.client.SliderClient.actionDestroy(SliderClient.java:677)
at org.apache.slider.client.SliderClient.exec(SliderClient.java:379)
at org.apache.slider.client.SliderClient.runService(SliderClient.java:333)
at org.apache.slider.core.main.ServiceLauncher.launchService(ServiceLauncher.java:188)
at org.apache.slider.core.main.ServiceLauncher.launchServiceRobustly(ServiceLauncher.java:475)
at org.apache.slider.core.main.ServiceLauncher.launchServiceAndExit(ServiceLauncher.java:403)
at org.apache.slider.core.main.ServiceLauncher.serviceMain(ServiceLauncher.java:630)
at org.apache.slider.Slider.main(Slider.java:49)
Slider is not using kerberos correctly (no principal / keytab) ! Like it is written here : 2017-12-05 12:05:47,992 [main] INFO zk.RegistrySecurity - Enabling ZK sasl client: jaasClientEntry = Client, principal = null, keytab = null When I connect myself as the hive user here is what I get (without a principal / keytab to test): sudo su - hive klist
klist: No credentials cache found (ticket cache FILE:/tmp/krb5cc_xxxx) /usr/hdp/current/zookeeper-client/bin/zkCli.sh -server $hostname
Connecting to $hostname
Welcome to ZooKeeper!
JLine support is enabled
WATCHER::
WatchedEvent state:AuthFailed type:None path:null
WATCHER::
WatchedEvent state:SyncConnected type:None path:null
[zk: $hostname(CONNECTED) 0] Looking for the acl of /registry/users/hive/services/org-apache-slider/llap0 [zk: $hostname(CONNECTED) 0] getAcl /registry/users/hive/services/org-apache-slider/llap0
'world,'anyone
: r
'sasl,'yarn
: cdrwa
'sasl,'jhs
: cdrwa
'sasl,'hdfs
: cdrwa
'sasl,'rm
: cdrwa
'sasl,'hive
: cdrwa
'sasl,'hive/hostname@REALM
: cdrwa
When the slider application is started, shoudn't this one use the keytab of the hive user ? @Matt Andruff I currently test with kerberos only. At the end I will use a custom authentication (like I used before) Sould I check slider ? Thanks
... View more
11-30-2017
01:44 PM
Hi, I get this error when running Hive Interactive Server : 2017-11-30 12:22:49,298 [main] INFO tools.SliderUtils - JVM initialized into secure mode with kerberos realm DOMAIN
2017-11-30 12:22:50,202 [main] WARN shortcircuit.DomainSocketFactory - The short-circuit local reads feature cannot be used because libhadoop cannot be loaded.
2017-11-30 12:22:50,345 [main] INFO client.AHSProxy - Connecting to Application History server at host/10.121.206.118:10200
2017-11-30 12:22:50,705 [main] INFO client.RequestHedgingRMFailoverProxyProvider - Looking for the active RM in [rm1, rm2]...
2017-11-30 12:22:50,840 [main] INFO client.RequestHedgingRMFailoverProxyProvider - Found active RM [rm1]
2017-11-30 12:22:50,850 [main] INFO client.SliderClient - Cluster llap0 is in a terminated state FAILED
2017-11-30 12:22:50,852 [main] INFO util.ExitUtil - Exiting with status 0
2017-11-30 12:22:53,280 [main] INFO tools.SliderUtils - JVM initialized into secure mode with kerberos realm DOMAIN
2017-11-30 12:22:54,061 [main] WARN shortcircuit.DomainSocketFactory - The short-circuit local reads feature cannot be used because libhadoop cannot be loaded.
2017-11-30 12:22:54,168 [main] INFO client.AHSProxy - Connecting to Application History server at host/10.121.206.118:10200
2017-11-30 12:22:54,205 [main] INFO client.RequestHedgingRMFailoverProxyProvider - Looking for the active RM in [rm1, rm2]...
2017-11-30 12:22:54,463 [main] INFO client.RequestHedgingRMFailoverProxyProvider - Found active RM [rm1]
2017-11-30 12:22:54,546 [main] INFO zk.ZKIntegration - Binding ZK client to host:2181,host:2181,host:2181
2017-11-30 12:22:54,566 [main] INFO zk.BlockingZKWatcher - waiting for ZK event
2017-11-30 12:22:54,600 [main-EventThread] INFO zk.BlockingZKWatcher - ZK binding callback received
2017-11-30 12:22:54,631 [main] INFO zk.RegistrySecurity - Enabling ZK sasl client: jaasClientEntry = Client, principal = null, keytab = null
2017-11-30 12:22:54,655 [main] INFO imps.CuratorFrameworkImpl - Starting
2017-11-30 12:22:54,663 [main-SendThread(host:2181)] WARN zookeeper.ClientCnxn - SASL configuration failed: javax.security.auth.login.LoginException: No key to store Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
2017-11-30 12:22:54,666 [main-EventThread] ERROR curator.ConnectionState - Authentication failed
2017-11-30 12:22:54,671 [main-EventThread] INFO state.ConnectionStateManager - State change: CONNECTED
2017-11-30 12:22:54,682 [main] WARN client.SliderClient - Error deleting registry entry /users/hive/services/org-apache-slider/llap0: org.apache.hadoop.registry.client.exceptions.NoPathPermissionsException: `/registry/users/hive/services/org-apache-slider/llap0': Not authorized to access path; ACLs: [null ACL]: KeeperErrorCode = NoAuth for /registry/users/hive/services/org-apache-slider/llap0
org.apache.hadoop.registry.client.exceptions.NoPathPermissionsException: `/registry/users/hive/services/org-apache-slider/llap0': Not authorized to access path; ACLs: [null ACL]: KeeperErrorCode = NoAuth for /registry/users/hive/services/org-apache-slider/llap0
at org.apache.hadoop.registry.client.impl.zk.CuratorService.operationFailure(CuratorService.java:385)
at org.apache.hadoop.registry.client.impl.zk.CuratorService.operationFailure(CuratorService.java:364)
at org.apache.hadoop.registry.client.impl.zk.CuratorService.zkDelete(CuratorService.java:684)
at org.apache.hadoop.registry.client.impl.zk.RegistryOperationsService.delete(RegistryOperationsService.java:160)
at org.apache.slider.client.SliderClient.actionDestroy(SliderClient.java:677)
at org.apache.slider.client.SliderClient.exec(SliderClient.java:379)
at org.apache.slider.client.SliderClient.runService(SliderClient.java:333)
at org.apache.slider.core.main.ServiceLauncher.launchService(ServiceLauncher.java:188)
at org.apache.slider.core.main.ServiceLauncher.launchServiceRobustly(ServiceLauncher.java:475)
at org.apache.slider.core.main.ServiceLauncher.launchServiceAndExit(ServiceLauncher.java:403)
at org.apache.slider.core.main.ServiceLauncher.serviceMain(ServiceLauncher.java:630)
at org.apache.slider.Slider.main(Slider.java:49)
Caused by: org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /registry/users/hive/services/org-apache-slider/llap0
at org.apache.zookeeper.KeeperException.create(KeeperException.java:113)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873)
at org.apache.curator.framework.imps.DeleteBuilderImpl$5.call(DeleteBuilderImpl.java:238)
at org.apache.curator.framework.imps.DeleteBuilderImpl$5.call(DeleteBuilderImpl.java:233)
at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107)
at org.apache.curator.framework.imps.DeleteBuilderImpl.pathInForeground(DeleteBuilderImpl.java:230)
at org.apache.curator.framework.imps.DeleteBuilderImpl.forPath(DeleteBuilderImpl.java:214)
at org.apache.curator.framework.imps.DeleteBuilderImpl.forPath(DeleteBuilderImpl.java:41)
at org.apache.hadoop.registry.client.impl.zk.CuratorService.zkDelete(CuratorService.java:680)
... 9 more
2017-11-30 12:22:54,690 [main] INFO client.SliderClient - Destroyed cluster llap0 Looks like some Kerberos properties were not properly set ? Has anyone already seen this error ? Thanks Manfred
... View more
Labels:
- Labels:
-
Apache Hive