Sqoop with kerberos security not working in cron , throw GSSException

[#1]

When i run sqoop job from edge node , its working fine and able to extract data from oracle, But when i schedule the same job in crontab, it throw kerberos security error.

I has valid kerberos ticket before cron job start and its valid for 10 hours, But when i schedule cron job in 5 minutes but cron Sqoop job throw kerberos error.

Please suggest, what are the steps to be followed before start sqoop cron job.


Below is the detailed error message :

>17/11/23 11:24:17 WARN ipc.Client: Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
>17/11/23 11:24:17 ERROR tool.ImportTool: Encountered IOException running import job: java.io.IOException: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host is: "ps2pr028380.express-scripts.com/10.221.171.22"; destination host is: "ps2pr028377.express-scripts.com":8020;
> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:782)
> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1558)
> at org.apache.hadoop.ipc.Client.call(Client.java:1498)
> at org.apache.hadoop.ipc.Client.call(Client.java:1398)
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
> at com.sun.proxy.$Proxy11.getDelegationToken(Unknown Source)
> at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getDelegationToken(ClientNamenodeProtocolTranslatorPB.java:980)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:291)
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:203)
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:185)
> at com.sun.proxy.$Proxy12.getDelegationToken(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.getDelegationToken(DFSClient.java:1041)
> at org.apache.hadoop.hdfs.DistributedFileSystem.getDelegationToken(DistributedFileSystem.java:1688)
> at org.apache.hadoop.fs.FileSystem.collectDelegationTokens(FileSystem.java:549)
> at org.apache.hadoop.fs.FileSystem.addDelegationTokens(FileSystem.java:527)
> at org.apache.hadoop.hdfs.DistributedFileSystem.addDelegationTokens(DistributedFileSystem.java:2400)
> at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:140)
> at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:100)
> at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:80)
> at org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:142)
> at org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:266)
> at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:139)
> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290)
> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1287)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1287)
> at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1308)
> at org.apache.sqoop.mapreduce.ImportJobBase.doSubmitJob(ImportJobBase.java:200)
> at org.apache.sqoop.mapreduce.ImportJobBase.runJob(ImportJobBase.java:173)
> at org.apache.sqoop.mapreduce.ImportJobBase.runImport(ImportJobBase.java:270)
> at org.apache.sqoop.manager.SqlManager.importQuery(SqlManager.java:748)
> at org.apache.sqoop.manager.OracleManager.importQuery(OracleManager.java:454)
> at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:509)
> at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:615)
> at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:225)
> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234)
> at org.apache.sqoop.Sqoop.main(Sqoop.java:243)
>Caused by: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
> at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:720)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
> at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:683)
> at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:770)
> at org.apache.hadoop.ipc.Client$Connection.access$3200(Client.java:397)
> at org.apache.hadoop.ipc.Client.getConnection(Client.java:1620)
> at org.apache.hadoop.ipc.Client.call(Client.java:1451)
> ... 43 more
>Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
> at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
> at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:413)
> at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:595)
> at org.apache.hadoop.ipc.Client$Connection.access$2000(Client.java:397)
> at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:762)
> at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:758)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
> at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:757)
> ... 46 more
>Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
> at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
> at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122)
> at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
> at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:224)
> at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
> at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
> at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192)
> ... 55 more

[#2]

Hi @Saravanan Nagarajan,

this is because there is no TTY, Profile setup for the user environment, implies that it cant know where the krb cache located.

you can handle the scenario by changing the cron command with remote execution command with tty, so your cron command will turn out to be (to mimic exactly what you have done in command line):

<schedule> ssh -t -i <private-keyfile> <username>@<servername> "<your_command>" >cronlog.log 2>& 

as an alternative, you can source the profile and open an TTY by specifying nohup and redirect the log on background in along with sourcing the bashrc/profile.

one more last thing is that, you can set the KRB5CACHE environment variable to grab the cache. the value for the cache can be obtained from klist command by removing last userID extension, for example if cache : /tmp/ticketcache_someUserID then the export KRB5CACHE=/tmp/ticketcache

hope one of this will helps!!

[#3]

Thanks Raju, I willl try your solution.

I solved this issue, just by adding below comment in my script before sqoop start.

kinit -kt /home/userName/userName.keytab userName@PROD.DATAHUB.LOCAL