region server kerberos debug flag enabled logs: ----------------------------------------------- 2017-01-20 18:20:23,404 DEBUG [regionserver/xxxx] security.HBaseSaslRpcClient: Have sent token of size 1824 from initSASLContext. 2017-01-20 18:20:23,406 DEBUG [regionserver/xxxx] ipc.AbstractRpcClient: Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException): GSS initiate failed Native config name: /etc/krb5.conf Loaded from native config >>>KinitOptions cache name is /tmp/krb5cc_1443726158 >>> KeyTabInputStream, readName(): a1.example.com >>> KeyTab: load() entry length: 93; type: 23 Added key: 23version: 1 Ordering keys wrt default_tkt_enctypes list default etypes for default_tkt_enctypes: 23. >>>DEBUG: TCPClient reading 207 bytes >>> KrbKdcReq send: #bytes read=207 >>>Pre-Authentication Data: PA-DATA type = 11 PA-ETYPE-INFO etype = 23, salt = >>>Pre-Authentication Data: PA-DATA type = 19 PA-ETYPE-INFO2 etype = 23, salt = null, s2kparams = null >>>Pre-Authentication Data: PA-DATA type = 2 PA-ENC-TIMESTAMP >>>Pre-Authentication Data: PA-DATA type = 16 >>>Pre-Authentication Data: PA-DATA type = 15 >>> KdcAccessibility: remove swlgdcp1.example.com.:88 >>> KDCRep: init() encoding tag is 126 req type is 11 >>>KRBError: sTime is Fri Jan 20 18:17:10 NZDT 2017 1484889430000 suSec is 668608 error code is 25 error Message is Additional pre-authentication required realm is ADC.EXAMPLE.COM sname is krbtgt/ADC.EXAMPLE.COM eData provided. msgType is 30 >>>Pre-Authentication Data: PA-DATA type = 11 PA-ETYPE-INFO etype = 23, salt = Added key: 23version: 1 Ordering keys wrt default_tkt_enctypes list default etypes for default_tkt_enctypes: 23. Debug is true storeKey false useTicketCache false useKeyTab true doNotPrompt false ticketCache is null isInitiator true KeyTab is /data/yarn/nm/usercache/srvuser/appcache/application_1484884580211_0007/container_1484884580211_0007_01_000002/keytabs/hbase.keytab refreshKrb5Config is false principal is srvuser/a1.example.com@ADC.EXAMPLE.COM tryFirstPass is false useFirstPass is false storePass is false clearPass is false >>> EType: sun.security.krb5.internal.crypto.ArcFourHmacEType >>> KrbAsRep cons in KrbAsReq.getReply srvuser/a1.example.com principal is srvuser/a1.example.com@xxx Will use keytab Commit Succeeded Found ticket for srvuser/a1.example.com@ADC.EXAMPLE.COM to go to krbtgt/ADC.EXAMPLE.COM@ADC.EXAMPLE.COM expiring on Sat Jan 21 04:17:10 NZDT 2017 Entered Krb5Context.initSecContext with state=STATE_NEW Found ticket for srvuser/a1.example.com@ADC.EXAMPLE.COM to go to krbtgt/ADC.EXAMPLE.COM@ADC.EXAMPLE.COM expiring on Sat Jan 21 04:17:10 NZDT 2017 Service ticket not found in the subject >>> Credentials acquireServiceCreds: same realm Krb5Context.unwrap: token=[60 30 06 09 2a 86 48 86 f7 12 01 02 02 02 01 11 00 ff ff ff ff 23 58 c9 46 b4 4c b6 d6 8f 53 e5 07 3e 36 72 67 df ba eb 14 c6 d7 aa d9 01 01 00 00 01 ] Krb5Context.unwrap: data=[01 01 00 00 ] Krb5Context.wrap: data=[01 01 00 00 73 72 76 64 65 76 61 6e 61 6c 79 74 69 63 73 69 6e 66 61 2f 64 78 6c 68 75 62 30 32 2e 6e 7a 2e 74 68 65 6e 61 74 69 6f 6e 61 6c 2e 63 6f 6d 40 42 4e 5a 4e 41 47 2e 4e 5a 2e 54 48 45 4e 41 54 49 4f 4e 41 4c 2e 43 4f 4d ] Krb5Context.wrap: token=[60 79 06 09 2a 86 48 86 f7 12 01 02 02 02 01 11 00 ff ff ff ff 8c 39 cb 18 d9 f8 27 4d b7 f7 d7 3f 20 64 60 f8 c5 0d d0 fe e4 b1 3e 85 01 01 00 00 73 72 76 64 65 76 61 6e 61 6c 79 74 69 63 73 69 6e 66 61 2f 64 78 6c 68 75 62 30 32 2e 6e 7a 2e 74 68 65 6e 61 74 69 6f 6e 61 6c 2e 63 6f 6d 40 42 4e 5a 4e 41 47 2e 4e 5a 2e 54 48 45 4e 41 54 49 4f 4e 41 4c 2e 43 4f 4d 01 ] Found ticket for srvuser/a1.example.com@ADC.EXAMPLE.COM to go to krbtgt/a1.example.com@ADC.EXAMPLE.COM expiring on Sat Jan 21 04:17:10 NZDT 2017 Entered Krb5Context.initSecContext with state=STATE_NEW Found ticket for srvuser/a1.example.com@ADC.EXAMPLE.COM to go to krbtgt/a1.example.com@ADC.EXAMPLE.COM expiring on Sat Jan 21 04:17:10 NZDT 2017 Found ticket for srvuser/a1.example.com@ADC.EXAMPLE.COM to go to zookeeper/a1.example.com@ADC.EXAMPLE.COM expiring on Sat Jan 21 04:17:10 NZDT 2017 Found service ticket in the subjectTicket (hex) = Found ticket for srvuser/a1.example.com@ADC.EXAMPLE.COM to go to krbtgt/a1.example.com@ADC.EXAMPLE.COM expiring on Sat Jan 21 04:17:10 NZDT 2017 Entered Krb5Context.initSecContext with state=STATE_NEW Found ticket for srvuser/a1.example.com@ADC.EXAMPLE.COM to go to krbtgt/a1.example.com@ADC.EXAMPLE.COM expiring on Sat Jan 21 04:17:10 NZDT 2017 Found ticket for srvuser/a1.example.com@ADC.EXAMPLE.COM to go to srvuser/a1.example.com@ADC.EXAMPLE.COM expiring on Sat Jan 21 04:17:10 NZDT 2017 Client Principal = srvuser/a1.example.com@ADC.EXAMPLE.COM Server Principal = srvuser/a1.example.com@ADC.EXAMPLE.COM Session Key = EncryptionKey: keyType=23 keyBytes (hex dump)= 0000: 95 8E 0F A5 BE 3D BA A4 9E FE C6 06 32 96 64 6E .....=......2.dn Forwardable Ticket true Forwarded Ticket false Proxiable Ticket false Proxy Ticket false Postdated Ticket false Renewable Ticket false Initial Ticket false Auth Time = Fri Jan 20 18:17:10 NZDT 2017 Start Time = Fri Jan 20 18:17:11 NZDT 2017 End Time = Sat Jan 21 04:17:10 NZDT 2017 Renew Till = null Client Addresses Null >>> KrbApReq: APOptions are 00100000 00000000 00000000 00000000 >>> EType: sun.security.krb5.internal.crypto.ArcFourHmacEType Krb5Context setting mySeqNumber to: 982345056