Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

klist: no credentials cache found

avatar
Expert Contributor

I have "klist" written in front of all hdfs commands in my script. When the job starts, it says the credentials are present and valid for next few days. But immediately once the next hdfs command starts it says as follows:

"klist: No credentials cache found (ticket cache FILE:/tmp/krb5cc_603)"

[2017-09-20 08:24:57,335] {bash_operator.py:74} INFO - javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
[2017-09-20 08:24:57,335] {bash_operator.py:74} INFO - at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212)
[2017-09-20 08:24:57,335] {bash_operator.py:74} INFO - at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:413)
[2017-09-20 08:24:57,335] {bash_operator.py:74} INFO - at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:563)
[2017-09-20 08:24:57,335] {bash_operator.py:74} INFO - at org.apache.hadoop.ipc.Client$Connection.access$1900(Client.java:378)
[2017-09-20 08:24:57,335] {bash_operator.py:74} INFO - at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:732)
[2017-09-20 08:24:57,336] {bash_operator.py:74} INFO - at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:728)
[2017-09-20 08:24:57,336] {bash_operator.py:74} INFO - at java.security.AccessController.doPrivileged(Native Method)
[2017-09-20 08:24:57,336] {bash_operator.py:74} INFO - at javax.security.auth.Subject.doAs(Subject.java:415)

I am really not sure if any other process is corrupting the cache.

Has anyone faced the same issue ?

1 ACCEPTED SOLUTION

avatar
Expert Contributor
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login
6 REPLIES 6

avatar

@Kumar Veerappan,

This is a tough question to answer since there is no indication of what your script is doing. The cache file, /tmp/krb5cc_603, is owned by the user with the uid of 603. Is this the user that was used to issue the kinit? Assuming a kinit was executed at some point before or during the script execution.

avatar
Expert Contributor

@Robert Levas forgot to mention that this is sporadic. the script does bunch of HDFS operations.

sudo -u user1 bash -c "kinit -R || kinit -k -t ~user1/user1.headless.keytab

avatar

@Kumar Veerappan,

From https://web.mit.edu/kerberos/krb5-1.12/doc/user/user_commands/kinit.html

-R	
requests renewal of the ticket-granting ticket. Note that an expired ticket cannot be renewed, even if the ticket is still within its renewable life.

Maybe sometime during the script execution the ticket expires and kinit -K is not failing as you expect?

avatar
Expert Contributor

Is there a way to find if any other process is destroying the ticket..may be some logs ?

avatar
New Contributor

How did you find that some process is destroying the ticket?

I am also facing same issue.

avatar
Expert Contributor
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login