I have attached the master, zk and rs logs. hbase-ams-master-m2-trunkatedlog.txt hbase-ams-zookeeper-m2log.txt hbase-ams-regionserver-m2log.txt There seems to be authentication issue in the RS: 2016-07-06 11:42:43,688 ERROR [main] regionserver.HRegionServerCommandLine: Region server exiting java.lang.RuntimeException: Failed construction of Regionserver: class org.apache.hadoop.hbase.regionserver.HRegionServer at org.apache.hadoop.hbase.regionserver.HRegionServer.constructRegionServer(HRegionServer.java:2636) at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.start(HRegionServerCommandLine.java:64) at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.run(HRegionServerCommandLine.java:87) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126) at org.apache.hadoop.hbase.regionserver.HRegionServer.main(HRegionServer.java:2651) Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:422) at org.apache.hadoop.hbase.regionserver.HRegionServer.constructRegionServer(HRegionServer.java:2634) ... 5 more Caused by: java.io.IOException: Login failure for amshbasemaster/m2.DOMAIN@DOMAIN from keytab /etc/security/keytabs/ams-hbase.regionserver.keytab: javax.security.auth.login.LoginException: Unable to obtain password from user at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:962) at org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:246) at org.apache.hadoop.hbase.security.User$SecureHadoopUser.login(User.java:386) at org.apache.hadoop.hbase.security.User.login(User.java:253) at org.apache.hadoop.hbase.security.UserProvider.login(UserProvider.java:115) at org.apache.hadoop.hbase.regionserver.HRegionServer.login(HRegionServer.java:612) at org.apache.hadoop.hbase.regionserver.HRegionServer.<init>(HRegionServer.java:550) ... 10 more Caused by: javax.security.auth.login.LoginException: Unable to obtain password from user at com.sun.security.auth.module.Krb5LoginModule.promptForPass(Krb5LoginModule.java:897) at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:760) at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:617) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755) at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195) at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682) at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680) at javax.security.auth.login.LoginContext.login(LoginContext.java:587) at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:953) ... 16 more ZOOKEEPER: ERROR [main] quorum.QuorumPeerConfig: Invalid configuration, only one server specified (ignoring) MASTER: 2016-07-06 10:48:18,075 WARN [main-SendThread(localhost:61181)] zookeeper.ClientCnxn: Session 0x155bfd1eb150003 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081) 2016-07-06 10:48:20,103 INFO [main-SendThread(localhost:61181)] zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.0.1:61181. Will not attempt to authenticate using SASL (unknown error) 2016-07-06 10:48:20,104 WARN [main-SendThread(localhost:61181)] zookeeper.ClientCnxn: Session 0x155bfd1eb150003 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081) 2016-07-06 10:48:21,724 INFO [main-SendThread(localhost:61181)] zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.0.1:61181. Will not attempt to authenticate using SASL (unknown error) 2016-07-06 10:48:21,725 WARN [main-SendThread(localhost:61181)] zookeeper.ClientCnxn: Session 0x155bfd1eb150003 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081) 2016-07-06 10:48:21,825 WARN [RS:0;m2:49385] zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper, quorum=localhost:61181, exception=org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/rs/m2.tmaut.tlabsdata.com,49385,1467802057410 2016-07-06 10:48:21,826 ERROR [RS:0;m2:49385] zookeeper.RecoverableZooKeeper: ZooKeeper delete failed after 4 attempts 2016-07-06 10:48:21,826 WARN [RS:0;m2:49385] regionserver.HRegionServer: Failed deleting my ephemeral node org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/rs/m2.tmaut.tlabsdata.com,49385,1467802057410 at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.delete(RecoverableZooKeeper.java:178) at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:1345) at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:1334) at org.apache.hadoop.hbase.regionserver.HRegionServer.deleteMyEphemeralNode(HRegionServer.java:1403) at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1079) at java.lang.Thread.run(Thread.java:745) ... View more

hbase-ams-master-m2-trunkatedlog.txt hbase-ams-zookeeper-m2log.txt hbase-ams-regionserver-m2log.txt sorry for the delay ... View more

Hi Geoffrey, In HBase config I have: zookeeper.znode.parent=/hbase-secure In Ambari metrics: zookeeper.znode.parent=/ams-hbase-secure I am not sure if those must be the same. ... View more

Hi swagle, the cluster is kerberized. hbase.zookeeper.quorum looks ok ... View more

Hi, I changed it, but now I am getting 2016-07-06 17:55:20,187 ERROR org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation: The node /ams-hbase-secure is not in ZooKeeper. It should have been written by the master. Check the value configured in 'zookeeper.znode.parent'. There could be a mismatch with the one configured in the master. ... View more

Hi Orivier, It is Version2.1.2.1 ... View more

I am having the same issue. I cannot sent the ACLs and also cannot delete the ACLs because they were created by the kerberized environment. Is there any way to work around this? ... View more

Looks like it was a syntactical error in the file, not sure what exactly. By recreating the contents it works, which seems like a whitespace or some other character to be the reason for the error. ... View more

Hi, I have been trying to configure Nifi to work with kerberized Kafka, but to no avail. Here my zookeeper-jaas.conf: Client { com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true keyTab="./conf/user.keytab" storeKey=true useTicketCache=false principal="user@REALM.COM”; }; KafkaClient { com.sun.security.auth.module.Krb5LoginModule required useTicketCache=true renewTicket=true serviceName="kafka" useKeyTab=true keyTab="./conf/user.keytab" principal="user@REALM.COM"; }; I am getting errors when starting the PutKafka processor: Caused by: java.io.IOException: Configuration Error: Line 8: expected [option key] at sun.security.provider.ConfigFile$Spi.ioException(ConfigFile.java:666) ~[na:1.8.0_66] at sun.security.provider.ConfigFile$Spi.match(ConfigFile.java:562) ~[na:1.8.0_66] at sun.security.provider.ConfigFile$Spi.parseLoginEntry(ConfigFile.java:477) ~[na:1.8.0_66] at sun.security.provider.ConfigFile$Spi.readConfig(ConfigFile.java:427) ~[na:1.8.0_66] at sun.security.provider.ConfigFile$Spi.init(ConfigFile.java:329) ~[na:1.8.0_66] at sun.security.provider.ConfigFile$Spi.init(ConfigFile.java:227) ~[na:1.8.0_66] at sun.security.provider.ConfigFile$Spi.access$000(ConfigFile.java:115) ~[na:1.8.0_66] at sun.security.provider.ConfigFile$Spi$1.run(ConfigFile.java:180) ~[na:1.8.0_66] at sun.security.provider.ConfigFile$Spi$1.run(ConfigFile.java:169) ~[na:1.8.0_66] at java.security.AccessController.doPrivileged(Native Method) ~[na:1.8.0_66] at sun.security.provider.ConfigFile$Spi.<init>(ConfigFile.java:169) ~[na:1.8.0_66] at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:1.8.0_66] at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[na:1.8.0_66] at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[na:1.8.0_66] at java.lang.reflect.Constructor.newInstance(Constructor.java:422) ~[na:1.8.0_66] at java.security.Provider$Service.newInstance(Provider.java:1609) ~[na:1.8.0_66] ... 26 common frames omitted I have checked the zookeeper-jaas.conf many times and cannot identify the error. Any pointers will be appreciated. ... View more