Reply
Explorer
Posts: 36
Registered: ‎07-15-2015

Kerberos tickets failing to renew through java code for long running jobs

Hi,

 

We are unable to renew kerberos user tickets from keytab using java code, while it's working with "kinit -R"

code:

 

UserGroupInformation loginUser = UserGroupInformation.getLoginUser(); loginUser.checkTGTAndReloginFromKeytab();

 

 Please help me out.

Posts: 519
Topics: 14
Kudos: 90
Solutions: 45
Registered: ‎09-02-2016

Re: Kerberos tickets failing to renew through java code for long running jobs

@RakeshE

 

you don't need to renew kerberos ticket from java code instead you can create a shell script with kinit and call it from a cron job. you can also schedule this cron once in 24 hrs or 23.59 hrs (based on your ticket validity period). so that your long running job will run without any issue

Explorer
Posts: 36
Registered: ‎07-15-2015

Re: Kerberos tickets failing to renew through java code for long running jobs

[ Edited ]

Thanks for your reply @saranvisa. But we are running some jobs through Java code and we should not get the Kerberos login tickets from os level as the different Kerberos users will log into the severs os level.So we are trying to login Kerberos user with keytab through Java for a specific job

Posts: 519
Topics: 14
Kudos: 90
Solutions: 45
Registered: ‎09-02-2016

Re: Kerberos tickets failing to renew through java code for long running jobs

@RakeshE

 

In general the option that i've mentioned is the default method. You have to work with your hadoop/unix admin to setup a cron job for your key tab.

 

I don't think controlling kerberos ticket from java code is a good option. Because you have to do this for all your jobs... it is not recommended.  forget about different kerbeors users as long as your team (or) batchid has a dedicated keytab file but if the different kerbeors user is also using your keytab then the ticket validity is common for both of you

New Contributor
Posts: 4
Registered: ‎01-17-2018

Re: Kerberos tickets failing to renew through java code for long running jobs

I'm running into a similar problem, but only in regards to Data At Rest Encryption (DARE).  All other HDFS operations work perpetually and tickets renew as needed.

 

With DARE, everything seems to be set up correctly and works transparently through our app for about an hour, then all we get are "Execution of 'abc.csv' failed. Error details: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)" errors.

 

I thought this might be related to HADOOP-12559 and/or HADOOP-10786 but we upgraded our test environment to CDH 5.8.5 and the problem persists.

 

Manual kinit does not seem to help (and I see valid tickets for our app and for hdfs).

 

Restarting our app seems to reset everything, but I can find no explicit kerberos login that would account for that.

 

My best guess is that there is some principal (possibly HTTP/ourserver.com@REALM.com ?) that needs to renew so that it can validate against the KMS, but doesn't.  I tried manually kinit-ing the HTTP principal on the cm server, but to no avail.

 

An alternate possibility is that something else is failing and the tgt error is a red herring, but the timeout aspect inclines me to think it's a kerberos issue.

 

Any help appreciated!!

Highlighted
New Contributor
Posts: 4
Registered: ‎01-17-2018

Re: Kerberos tickets failing to renew through java code for long running jobs

See my update here

Announcements

Currently incubating in Cloudera Labs:

Envelope
HTrace
Ibis
Impyla
Livy
Oryx
Phoenix
Spark Runner for Beam SDK
Time Series for Spark
YCSB