Member since
04-13-2016
36
Posts
4
Kudos Received
0
Solutions
06-28-2017
08:08 AM
Thank you @wengelbrecht - But how does it manage all services who are using older keytab? Does it restart them?
... View more
06-28-2017
05:17 AM
Ambari creates keytabs internally because it has details of the AD it is connecting to. However, how does ambari regenerate keytabs once the passwords expire at AD end? How does it ensure that the services dependent on those keytabs do not go down? Or all services have to be shut down when a new keytab
is provisioned?
... View more
Labels:
- Labels:
-
Apache Ambari
02-10-2017
10:07 AM
Sure Josh, thanks a lot for taking time to look at these logs.
... View more
02-06-2017
06:50 AM
Hi @Josh Elser - I hope you are able to spot a problem in data that I uploaded. Is there any hint in that which corroborates with my assessment above or points to a different problem altogether? Any input is appreciated. Thanks,
Sumit
... View more
02-02-2017
11:03 AM
"GSS initiate failed" without any detailed error message is like a catch-all in the class com.sun.security.sasl.gsskerb.GssKrb5Server in the method evaluateResponse(byte[] responseData). There are many lines in that big try-catch that can throw this exception. So, after I debugged the running hbase master process through a debugger I am able to clearly see that the stack trace leading to this GSSException is actually coming from method acceptSecContext(InputStream is, int mechTokenSize) in the class Krb5Context. The actual exception is KrbException being thrown as GSSException. This points to issues in keytab in master node vs details in service ticket that region server is presenting, assuming JCE and encryption types on both nodes are appropriate.
... View more
02-01-2017
05:16 PM
@Josh Elser - I've uploaded the kerberos out logs from Hbase master and region servers - hbase-kerberos-logs.zip For security reasons, I have had to mask the host names, etc.
... View more
01-30-2017
03:47 AM
Hi @Josh Elser - I hope the logs I attached show some issue. I am not able to find anything amiss. Plus, with a simple socket server test program I notice that we are able to successfully get service ticket and also send data back and forth. So, I'd assume that from kerberos side the issue is not there. Then, something from hbase side is messing things up? Or maybe some permissions of user / user groups in HDFS, LDAP?
... View more
01-25-2017
06:08 AM
Hi @Josh Elser - I have attached 2 logs (after kerberos debug flag) for the HMaster and HRegion server. Overall logs are quite huge so I have tried to remove some portions. Like I removed the hex dump of tickets. However, do let me know if I need to attach full logs. Some comments in general:
The error "GSS initiate failed" shows up even when master and region server come up on same host. After enabling debug logs for HBase, HMaster shows:
2017-01-20 18:17:11,699 DEBUG [main-EventThread] zookeeper.RegionServerTracker: Added tracking of RS /srvuser/hbase/rs/a1.example.com,52412,1484889430172 2017-01-20 18:17:11,823 DEBUG [RpcServer.listener,port=42263] ipc.RpcServer: RpcServer.listener,port=42263: connection from 10.64.130.53:46270; # active connections: 1 2017-01-20 18:17:11,856 DEBUG [RpcServer.reader=2,bindAddress=a1.example.com,port=42263] ipc.RpcServer: Kerberos principal name is srvuser/a1.example.com@ADC.EXAMPLE.COM 2017-01-20 18:17:11,857 DEBUG [RpcServer.reader=2,bindAddress=a1.example.com,port=42263] ipc.RpcServer: Created SASL server with mechanism = GSSAPI 2017-01-20 18:17:11,857 DEBUG [RpcServer.reader=2,bindAddress=a1.example.com,port=42263] ipc.RpcServer: Have read input token of size 1824 for processing by saslServer.evaluateResponse() 2017-01-20 18:17:11,857 DEBUG [RpcServer.reader=2,bindAddress=a1.example.com,port=42263] ipc.RpcServer: RpcServer.listener,port=42263: Caught exception while reading:GSS initiate failed 2017-01-20 18:17:11,857 DEBUG [RpcServer.reader=2,bindAddress=a1.example.com,port=42263] ipc.RpcServer: RpcServer.listener,port=42263: DISCONNECTING client 10.64.130.53:46270 because read count=-1. Number of active connections: 1hbase-regionsvr-kerberos-output.txthmaster-kerberos-flag-output.txt
... View more
01-23-2017
09:26 AM
@Zhao Chaofeng - I am having exact problem with Hbase (1.1.2) - GSS Initiate failed even with a valid kerberos service ticket. Can you please let me know what version of kerberos libraries did you re-install? I am using Kerberos 5 version 1.10.3
Thanks!
Sumit
... View more
01-23-2017
03:19 AM
@Sergey Soldatov - Thanks for the suggestion. I had added that flag already and that only showed me that I have a valid service ticket (as mentioned above). The SecurityAuth.audit log of hbase master shows the following error: 2017-01-20 18:17:08,221 WARN SecurityLogger.org.apache.hadoop.hbase.Server: Auth failed for x.y.z.q:55872:null 2017-01-20 18:17:11,857 WARN SecurityLogger.org.apache.hadoop.hbase.Server: Auth failed for x.y.z.q:46270:null Not sure if this can point to any problems. The IP for which auth shows as failed above is where HRegion server is running.
... View more