Try to switch on Kerberos with manual creation of keytabs and principals.
Principal and keytabs has been created. Keytabs distributed to all nodes. Almost all services started.
Hive, HBase and Kafk don't.
Now I try to solve Hive issue.
hiveserver2.log
2016-10-24 16:57:10,382 ERROR [Thread-15-SendThread(DN2.REALM.COM:2181)]: client.ZooKeeperSaslClient (ZooKeeperSaslClient.java:createSaslToken(384)) - An error: (java.security.PrivilegedActionException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Server not found in Kerberos database (7))]) occurred when evaluating Zookeeper Quorum Member's received SASL token. Zookeeper Client will go to AUTH_FAILED state.
2016-10-24 16:57:10,382 ERROR [Thread-15-SendThread(DN2.REALM.COM:2181)]: zookeeper.ClientCnxn (ClientCnxn.java:run(1059)) - SASL authentication with Zookeeper Quorum member failed: javax.security.sasl.SaslException: An error: (java.security.PrivilegedActionException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Server not found in Kerberos database (7))]) occurred when evaluating Zookeeper Quorum Member's received SASL token. Zookeeper Client will go to AUTH_FAILED state.
2016-10-24 16:57:10,383 ERROR [Thread-15-EventThread]: curator.ConnectionState (ConnectionState.java:checkState(245)) - Authentication failed
2016-10-24 16:57:10,390 FATAL [Thread-15]: thrift.ThriftCLIService (ThriftBinaryCLIService.java:run(101)) - Error starting HiveServer2: could not start ThriftBinaryCLIService
org.apache.hadoop.hive.thrift.DelegationTokenStore$TokenStoreException: Error creating path /hive/cluster/delegation/HIVESERVER2/keys
at org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.ensurePath(ZooKeeperTokenStore.java:166)
at org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.initClientAndPaths(ZooKeeperTokenStore.java:236)
at org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.init(ZooKeeperTokenStore.java:469)
at org.apache.hadoop.hive.thrift.HiveDelegationTokenManager.startDelegationTokenSecretManager(HiveDelegationTokenManager.java:92)
at org.apache.hive.service.auth.HiveAuthFactory.<init>(HiveAuthFactory.java:128)
at org.apache.hive.service.cli.thrift.ThriftBinaryCLIService.run(ThriftBinaryCLIService.java:57)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.zookeeper.KeeperException$AuthFailedException: KeeperErrorCode = AuthFailed for /hive/cluster/delegation/HIVESERVER2/keys
at org.apache.zookeeper.KeeperException.create(KeeperException.java:123)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
at org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:688)
at org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:672)
at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107)
at org.apache.curator.framework.imps.CreateBuilderImpl.pathInForeground(CreateBuilderImpl.java:668)
at org.apache.curator.framework.imps.CreateBuilderImpl.protectedPathInForeground(CreateBuilderImpl.java:453)
at org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:443)
at org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:423)
at org.apache.curator.framework.imps.CreateBuilderImpl$3.forPath(CreateBuilderImpl.java:257)
at org.apache.curator.framework.imps.CreateBuilderImpl$3.forPath(CreateBuilderImpl.java:205)
at org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.ensurePath(ZooKeeperTokenStore.java:160)
... 6 more
2016-10-24 16:57:10,399 INFO [Thread-4]: server.HiveServer2 (HiveStringUtils.java:run(711)) - SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down HiveServer2 at DN1.REALM.COM/172.26.140.200
************************************************************/
2016-10-24 16:57:10,419 INFO [main]: server.HiveServer2 (HiveServer2.java:addServerInstanceToZooKeeper(249)) - Created a znode on ZooKeeper for HiveServer2 uri: DN1.REALM.COM:10000
2016-10-24 16:57:10,420 INFO [Thread-13]: server.HiveServer2 (HiveServer2.java:stop(397)) - Shutting down HiveServer2
2016-10-24 16:57:10,420 INFO [Thread-13]: service.AbstractService (AbstractService.java:stop(125)) - Service:ThriftBinaryCLIService is stopped.
2016-10-24 16:57:10,420 INFO [Thread-13]: service.AbstractService (AbstractService.java:stop(125)) - Service:OperationManager is stopped.
2016-10-24 16:57:10,420 INFO [Thread-13]: service.AbstractService (AbstractService.java:stop(125)) - Service:SessionManager is stopped.
