Thank you @wengelbrecht - But how does it manage all services who are using older keytab? Does it restart them? ... View more

Sure Josh, thanks a lot for taking time to look at these logs. ... View more

Hi @Josh Elser - I hope you are able to spot a problem in data that I uploaded. Is there any hint in that which corroborates with my assessment above or points to a different problem altogether? Any input is appreciated. Thanks, Sumit ... View more

"GSS initiate failed" without any detailed error message is like a catch-all in the class com.sun.security.sasl.gsskerb.GssKrb5Server in the method evaluateResponse(byte[] responseData). There are many lines in that big try-catch that can throw this exception. So, after I debugged the running hbase master process through a debugger I am able to clearly see that the stack trace leading to this GSSException is actually coming from method acceptSecContext(InputStream is, int mechTokenSize) in the class Krb5Context. The actual exception is KrbException being thrown as GSSException. This points to issues in keytab in master node vs details in service ticket that region server is presenting, assuming JCE and encryption types on both nodes are appropriate. ... View more

@Josh Elser - I've uploaded the kerberos out logs from Hbase master and region servers - hbase-kerberos-logs.zip For security reasons, I have had to mask the host names, etc. ... View more

Hi @Josh Elser - I hope the logs I attached show some issue. I am not able to find anything amiss. Plus, with a simple socket server test program I notice that we are able to successfully get service ticket and also send data back and forth. So, I'd assume that from kerberos side the issue is not there. Then, something from hbase side is messing things up? Or maybe some permissions of user / user groups in HDFS, LDAP? ... View more

Hi @Josh Elser - I have attached 2 logs (after kerberos debug flag) for the HMaster and HRegion server. Overall logs are quite huge so I have tried to remove some portions. Like I removed the hex dump of tickets. However, do let me know if I need to attach full logs. Some comments in general: The error "GSS initiate failed" shows up even when master and region server come up on same host. After enabling debug logs for HBase, HMaster shows: 2017-01-20 18:17:11,699 DEBUG [main-EventThread] zookeeper.RegionServerTracker: Added tracking of RS /srvuser/hbase/rs/a1.example.com,52412,1484889430172 2017-01-20 18:17:11,823 DEBUG [RpcServer.listener,port=42263] ipc.RpcServer: RpcServer.listener,port=42263: connection from 10.64.130.53:46270; # active connections: 1 2017-01-20 18:17:11,856 DEBUG [RpcServer.reader=2,bindAddress=a1.example.com,port=42263] ipc.RpcServer: Kerberos principal name is srvuser/a1.example.com@ADC.EXAMPLE.COM 2017-01-20 18:17:11,857 DEBUG [RpcServer.reader=2,bindAddress=a1.example.com,port=42263] ipc.RpcServer: Created SASL server with mechanism = GSSAPI 2017-01-20 18:17:11,857 DEBUG [RpcServer.reader=2,bindAddress=a1.example.com,port=42263] ipc.RpcServer: Have read input token of size 1824 for processing by saslServer.evaluateResponse() 2017-01-20 18:17:11,857 DEBUG [RpcServer.reader=2,bindAddress=a1.example.com,port=42263] ipc.RpcServer: RpcServer.listener,port=42263: Caught exception while reading:GSS initiate failed 2017-01-20 18:17:11,857 DEBUG [RpcServer.reader=2,bindAddress=a1.example.com,port=42263] ipc.RpcServer: RpcServer.listener,port=42263: DISCONNECTING client 10.64.130.53:46270 because read count=-1. Number of active connections: 1hbase-regionsvr-kerberos-output.txthmaster-kerberos-flag-output.txt ... View more

@Zhao Chaofeng - I am having exact problem with Hbase (1.1.2) - GSS Initiate failed even with a valid kerberos service ticket. Can you please let me know what version of kerberos libraries did you re-install? I am using Kerberos 5 version 1.10.3 Thanks! Sumit ... View more

@Sergey Soldatov - Thanks for the suggestion. I had added that flag already and that only showed me that I have a valid service ticket (as mentioned above). The SecurityAuth.audit log of hbase master shows the following error: 2017-01-20 18:17:08,221 WARN SecurityLogger.org.apache.hadoop.hbase.Server: Auth failed for x.y.z.q:55872:null 2017-01-20 18:17:11,857 WARN SecurityLogger.org.apache.hadoop.hbase.Server: Auth failed for x.y.z.q:46270:null Not sure if this can point to any problems. The IP for which auth shows as failed above is where HRegion server is running. ... View more