Reply
Highlighted
Explorer
Posts: 6
Registered: ‎01-25-2017

Unable to do Hadoop command after enable cross realm trust with Windows AD

Hi, 

I have enabled kerberos, Local MIT KDC  and Cross-Realm trust with Windows AD, on my Cloudera manager. 

with the following properties

 

<property>
<name>hadoop.security.auth_to_local</name>
<value>
RULE:[1:$1@$0](^.*@HADOOP\.COM$)s/^(.*)@HADOOP\.COM$/$1/g
RULE:[2:$1@$0](^.*@HADOOP\.COM$)s/^(.*)@HADOOP\.COM$/$1/g
DEFAULT
</value>
</property>

 

I am able to access and create files through HUE file browser using User Accounts from AD. But I am not able to run Hadoop command

hadoop fs -ls /

17/01/26 14:48:02 WARN security.UserGroupInformation: PriviledgedActionException as:root (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
17/01/26 14:48:02 WARN ipc.Client: Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
17/01/26 14:48:02 WARN security.UserGroupInformation: PriviledgedActionException as:root (auth:KERBEROS) cause:java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
ls: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host is: "quickstart.hadoop.com.sg/192.168.56.101"; destination host is: "quickstart.hadoop.com.sg":8020;

 

This is my Ticket:

kinit dino@hadoop.com
klist

Default principal: dino@HADOOP.COM Valid starting Expires Service principal 01/26/2017 15:29:30 01/27/2017 01:29:30 krbtgt/HADOOP.COM@HADOOP.COM renew until 02/02/2017 15:29:26


Can anyone advice me what should I do next? 
Thanks :)

 

Posts: 642
Topics: 3
Kudos: 121
Solutions: 67
Registered: ‎08-16-2016

Re: Unable to do Hadoop command after enable cross realm trust with Windows AD

Can you share anything else like the krb5.conf or kdc.conf? What do you have set for the trusted realmn?

The hostname seem to be in the realm hadoop.com.sg. I want to say that the krbtgt principal should be krbtgt/HADOOP.COM.SG/HADOOP.COM.

Also try adding -Djava.security.debug=gssloginconfig,configfile,configparser,logincontext to the command and posting the output. It will be a lot but should help nailed down where it is going wrong.
Explorer
Posts: 6
Registered: ‎01-25-2017

Re: Unable to do Hadoop command after enable cross realm trust with Windows AD

kdc.conf

 

[realms]
 EXAMPLE.COM = {
  master_key_type = aes256-cts
  acl_file = /var/kerberos/krb5kdc/kadm5.acl
  dict_file = /usr/share/dict/words
  admin_keytab = /var/kerberos/krb5kdc/kadm5.keytab
  supported_enctypes = aes256-cts:normal aes128-cts:normal des3-hmac-sha1:normal arcfour-hmac:normal camellia256-cts:normal camellia128-cts:normal des-hmac-sha1:normal des-cbc-md5:normal des-cbc-crc:normal
#default_tkt_enctypes = aes256-cts aes256-cts-hmac-sha1-96 aes128-cts-hmac-sha1-96 aes128-cts rc4-hmac des3-cbc-sha1 des-cbc-md5 des-cbc-crc arcfour-hmac arcfour-hmac-md5
#    default_tgs_enctypes = aes256-cts aes256-cts-hmac-sha1-96 aes128-cts-hmac-sha1-96 aes128-cts rc4-hmac des3-cbc-sha1 des-cbc-md5 des-cbc-crc arcfour-hmac arcfour-hmac-md5

 

 

krb5.conf

HADOOP.COM is my AD  realm

[logging]
 default = FILE:/var/log/krb5libs.log
 kdc = FILE:/var/log/krb5kdc.log
 admin_server = FILE:/var/log/kadmind.log

[libdefaults]
 dns_lookup_realm = false
 ticket_lifetime = 24h
 renew_lifetime = 7d
 forwardable = true
 rdns = false
 default_realm = EXAMPLE.COM
 default_ccache_name = KEYRING:persistent:%{uid}

[realms]
 EXAMPLE.COM = {
  kdc = 192.168.56.101
  admin_server = 192.168.56.101
 }
 HADOOP.COM = {
kdc=192.168.56.102
admin_server= 192.168.56.102
default_domain =hadoop.com
 }

[domain_realm]
 .example.com = EXAMPLE.COM
 example.com = EXAMPLE.COM
 .hadoop.com = HADOOP.COM
 hadoop.com = HADOOP.COM

hadoop fs -ls /  -Djava.security.debug=gssloginconfig,configfile,configparser,logincontext
(is this what you mean?)

17/01/26 19:32:45 WARN security.UserGroupInformation: PriviledgedActionException as:ncshadoop (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
17/01/26 19:32:45 WARN ipc.Client: Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
17/01/26 19:32:45 WARN security.UserGroupInformation: PriviledgedActionException as:ncshadoop (auth:KERBEROS) cause:java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
ls: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host is: "quickstart.hadoop.ncs.com.sg/192.168.56.101"; destination host is: "quickstart.hadoop.ncs.com.sg":8020;

list_principals

kadmin.local:  list_principals
HTTP/quickstart.hadoop.com.sg@EXAMPLE.COM
K/M@EXAMPLE.COM
cloudera-manager/admin@EXAMPLE.COM
hdfs/quickstart.hadoop.com.sg@EXAMPLE.COM
hdfs@EXAMPLE.COM
hive/quickstart.hadoop.com.sg@EXAMPLE.COM
hue/quickstart.hadoop.com.sg@EXAMPLE.COM
kadmin/admin@EXAMPLE.COM
kadmin/changepw@EXAMPLE.COM
kadmin/quickstart.hadoop.com.sg@EXAMPLE.COM
kiprop/quickstart.hadoop.com.sg@EXAMPLE.COM
krbtgt/EXAMPLE.COM@EXAMPLE.COM
krbtgt/EXAMPLE.COM@HADOOP.COM
krbtgt/HADOOP.COM@EXAMPLE.COM
krbtgt/HADOOP.COM@HADOOP.COM
mapred/quickstart.hadoop.com.sg@EXAMPLE.COM
oozie/quickstart.hadoop.com.sg@EXAMPLE.COM
test@EXAMPLE.COM
yarn/quickstart.hadoop.com.sg@EXAMPLE.COM
zookeeper/quickstart.hadoop.com.sg@EXAMPLE.COM