Reply
Explorer
Posts: 14
Registered: ‎08-14-2015

MR Job killed.

Hi,

 

Can some one help me out please.

 

I am newbie to Hadoop and I am learning Hadoop.

As a part of my learning I am trying to understand kerberos security impact on MR jobs.

I have changed the default configuration present in krb5.conf and restarted kerberos services on all servers present in the cluster.

 

I did execute an MR job and below is the output:

 

Open Question:

 

1) Does MR job fail if the job continues execution even after life time? How does life time in kerberos configuration file effect MR job?

In this particular case, I am not sure but I set life time to few minutes / hours.

 

 

 

 

[vishwa828@hdp-poc2 hadoop-mapreduce]$ time hadoop jar hadoop-mapreduce-examples.jar teragen 1000000000 /user/vishwa828/terasort-input99

 

15/08/19 14:51:41 INFO mapreduce.Job: map 10% reduce 0%

15/08/19 14:52:19 INFO mapreduce.Job: map 11% reduce 0%

15/08/19 14:52:50 WARN security.UserGroupInformation: Exception encountered while running the renewal command. Aborting renew thread. ExitCodeException exitCode=1: kinit: Ticket expired while renewing credentials

 

15/08/19 14:52:58 INFO mapreduce.Job: map 12% reduce 0%

15/08/19 14:53:17 INFO mapreduce.Job: map 13% reduce 0%

 

15/08/19 15:20:27 WARN security.UserGroupInformation: PriviledgedActionException as:vishwa828@hydNEXTGEN.COM (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]

15/08/19 15:20:28 WARN security.UserGroupInformation: PriviledgedActionException as:vishwa828@hydNEXTGEN.COM (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]

15/08/19 15:20:28 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.

15/08/19 15:20:31 WARN security.UserGroupInformation: PriviledgedActionException as:vishwa828@hydNEXTGEN.COM (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]

15/08/19 15:20:31 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.

15/08/19 15:20:33 WARN security.UserGroupInformation: PriviledgedActionException as:vishwa828@hydNEXTGEN.COM (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]

15/08/19 15:20:33 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.

15/08/19 15:20:34 WARN security.UserGroupInformation: PriviledgedActionException as:vishwa828@hydNEXTGEN.COM (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]

15/08/19 15:20:34 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.

15/08/19 15:20:37 WARN security.UserGroupInformation: PriviledgedActionException as:vishwa828@hydNEXTGEN.COM (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]

15/08/19 15:20:37 WARN ipc.Client: Couldn't setup connection for vishwa828@hydNEXTGEN.COM to hdp-poc2.hydnextgen.com/10.138.90.72:8032

javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]

at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212)

at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:413)

at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:553)

at org.apache.hadoop.ipc.Client$Connection.access$1800(Client.java:368)

at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:722)

at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:718)

at java.security.AccessController.doPrivileged(Native Method)

at javax.security.auth.Subject.doAs(Subject.java:415)

at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)

at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:717)

at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:368)

at org.apache.hadoop.ipc.Client.getConnection(Client.java:1521)

at org.apache.hadoop.ipc.Client.call(Client.java:1438)

at org.apache.hadoop.ipc.Client.call(Client.java:1399)

at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)

at com.sun.proxy.$Proxy19.getApplicationReport(Unknown Source)

at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getApplicationReport(ApplicationClientProtocolPBClientImpl.java:187)

at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:601)

at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)

at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)

at com.sun.proxy.$Proxy20.getApplicationReport(Unknown Source)

at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getApplicationReport(YarnClientImpl.java:399)

at org.apache.hadoop.mapred.ResourceMgrDelegate.getApplicationReport(ResourceMgrDelegate.java:302)

at org.apache.hadoop.mapred.ClientServiceDelegate.getProxy(ClientServiceDelegate.java:153)

at org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:322)

at org.apache.hadoop.mapred.ClientServiceDelegate.getTaskCompletionEvents(ClientServiceDelegate.java:389)

at org.apache.hadoop.mapred.YARNRunner.getTaskCompletionEvents(YARNRunner.java:582)

at org.apache.hadoop.mapreduce.Job$5.run(Job.java:689)

at org.apache.hadoop.mapreduce.Job$5.run(Job.java:686)

at java.security.AccessController.doPrivileged(Native Method)

at javax.security.auth.Subject.doAs(Subject.java:415)

at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)

at org.apache.hadoop.mapreduce.Job.getTaskCompletionEvents(Job.java:686)

at org.apache.hadoop.mapreduce.Job.monitorAndPrintJob(Job.java:1387)

at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1327)

at org.apache.hadoop.examples.terasort.TeraGen.run(TeraGen.java:305)

at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)

at org.apache.hadoop.examples.terasort.TeraGen.main(TeraGen.java:309)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:601)

at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)

at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144)

at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:601)

at org.apache.hadoop.util.RunJar.run(RunJar.java:221)

at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)

at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)

at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121)

at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)

at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:223)

at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)

at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)

at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:193)

... 52 more

15/08/19 15:20:37 WARN security.UserGroupInformation: PriviledgedActionException as:vishwa828@hydNEXTGEN.COM (auth:KERBEROS) cause:java.io.IOException: Couldn't setup connection for vishwa828@hydNEXTGEN.COM to hdp-poc2.hydnextgen.com/10.138.90.72:8032

15/08/19 15:20:37 WARN security.UserGroupInformation: PriviledgedActionException as:vishwa828@hydNEXTGEN.COM (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]

15/08/19 15:20:37 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.

15/08/19 15:20:38 WARN security.UserGroupInformation: PriviledgedActionException as:vishwa828@hydNEXTGEN.COM (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]

15/08/19 15:20:38 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.

15/08/19 15:20:40 WARN security.UserGroupInformation: PriviledgedActionException as:vishwa828@hydNEXTGEN.COM (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]

15/08/19 15:20:40 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.

15/08/19 15:20:43 WARN security.UserGroupInformation: PriviledgedActionException as:vishwa828@hydNEXTGEN.COM (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]

15/08/19 15:20:43 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.

15/08/19 15:20:45 WARN security.UserGroupInformation: PriviledgedActionException as:vishwa828@hydNEXTGEN.COM (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]

15/08/19 15:20:45 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.

15/08/19 15:20:45 WARN security.UserGroupInformation: PriviledgedActionException as:vishwa828@hydNEXTGEN.COM (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]

15/08/19 15:20:45 WARN ipc.Client: Couldn't setup connection for vishwa828@hydNEXTGEN.COM to hdp-poc2.hydnextgen.com/10.138.90.72:8032

javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]

at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212)

at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:413)

at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:553)

at org.apache.hadoop.ipc.Client$Connection.access$1800(Client.java:368)

at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:722)

at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:718)

at java.security.AccessController.doPrivileged(Native Method)

at javax.security.auth.Subject.doAs(Subject.java:415)

at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)

at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:717)

at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:368)

at org.apache.hadoop.ipc.Client.getConnection(Client.java:1521)

at org.apache.hadoop.ipc.Client.call(Client.java:1438)

at org.apache.hadoop.ipc.Client.call(Client.java:1399)

at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)

at com.sun.proxy.$Proxy19.getApplicationReport(Unknown Source)

at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getApplicationReport(ApplicationClientProtocolPBClientImpl.java:187)

at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:601)

at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)

at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)

at com.sun.proxy.$Proxy20.getApplicationReport(Unknown Source)

at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getApplicationReport(YarnClientImpl.java:399)

at org.apache.hadoop.mapred.ResourceMgrDelegate.getApplicationReport(ResourceMgrDelegate.java:302)

at org.apache.hadoop.mapred.ClientServiceDelegate.getProxy(ClientServiceDelegate.java:153)

at org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:322)

at org.apache.hadoop.mapred.ClientServiceDelegate.getTaskCompletionEvents(ClientServiceDelegate.java:389)

at org.apache.hadoop.mapred.YARNRunner.getTaskCompletionEvents(YARNRunner.java:582)

at org.apache.hadoop.mapreduce.Job$5.run(Job.java:689)

at org.apache.hadoop.mapreduce.Job$5.run(Job.java:686)

at java.security.AccessController.doPrivileged(Native Method)

at javax.security.auth.Subject.doAs(Subject.java:415)

at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)

at org.apache.hadoop.mapreduce.Job.getTaskCompletionEvents(Job.java:686)

at org.apache.hadoop.mapreduce.Job.monitorAndPrintJob(Job.java:1387)

at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1327)

at org.apache.hadoop.examples.terasort.TeraGen.run(TeraGen.java:305)

at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)

at org.apache.hadoop.examples.terasort.TeraGen.main(TeraGen.java:309)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:601)

at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)

at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144)

at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:601)

at org.apache.hadoop.util.RunJar.run(RunJar.java:221)

at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)

at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)

at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121)

at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)

at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:223)

at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)

at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)

at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:193)

... 52 more

15/08/19 15:20:45 WARN security.UserGroupInformation: PriviledgedActionException as:vishwa828@hydNEXTGEN.COM (auth:KERBEROS) cause:java.io.IOException: Couldn't setup connection for vishwa828@hydNEXTGEN.COM to hdp-poc2.hydnextgen.com/10.138.90.72:8032

15/08/19 15:20:45 WARN security.UserGroupInformation: PriviledgedActionException as:vishwa828@hydNEXTGEN.COM (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]

15/08/19 15:20:45 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.

15/08/19 15:20:47 WARN security.UserGroupInformation: PriviledgedActionException as:vishwa828@hydNEXTGEN.COM (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]

15/08/19 15:20:47 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.

15/08/19 15:20:52 WARN security.UserGroupInformation: PriviledgedActionException as:vishwa828@hydNEXTGEN.COM (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]

15/08/19 15:20:52 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.

15/08/19 15:20:53 WARN security.UserGroupInformation: PriviledgedActionException as:vishwa828@hydNEXTGEN.COM (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]

15/08/19 15:20:53 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.

15/08/19 15:20:56 WARN security.UserGroupInformation: PriviledgedActionException as:vishwa828@hydNEXTGEN.COM (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]

15/08/19 15:20:56 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.

15/08/19 15:21:00 WARN security.UserGroupInformation: PriviledgedActionException as:vishwa828@hydNEXTGEN.COM (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]

15/08/19 15:21:00 WARN ipc.Client: Couldn't setup connection for vishwa828@hydNEXTGEN.COM to hdp-poc2.hydnextgen.com/10.138.90.72:8032

javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]

at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212)

at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:413)

at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:553)

at org.apache.hadoop.ipc.Client$Connection.access$1800(Client.java:368)

at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:722)

at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:718)

at java.security.AccessController.doPrivileged(Native Method)

at javax.security.auth.Subject.doAs(Subject.java:415)

at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)

at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:717)

at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:368)

at org.apache.hadoop.ipc.Client.getConnection(Client.java:1521)

at org.apache.hadoop.ipc.Client.call(Client.java:1438)

at org.apache.hadoop.ipc.Client.call(Client.java:1399)

at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)

at com.sun.proxy.$Proxy19.getApplicationReport(Unknown Source)

at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getApplicationReport(ApplicationClientProtocolPBClientImpl.java:187)

at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:601)

at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)

at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)

at com.sun.proxy.$Proxy20.getApplicationReport(Unknown Source)

at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getApplicationReport(YarnClientImpl.java:399)

at org.apache.hadoop.mapred.ResourceMgrDelegate.getApplicationReport(ResourceMgrDelegate.java:302)

at org.apache.hadoop.mapred.ClientServiceDelegate.getProxy(ClientServiceDelegate.java:153)

at org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:322)

at org.apache.hadoop.mapred.ClientServiceDelegate.getTaskCompletionEvents(ClientServiceDelegate.java:389)

at org.apache.hadoop.mapred.YARNRunner.getTaskCompletionEvents(YARNRunner.java:582)

at org.apache.hadoop.mapreduce.Job$5.run(Job.java:689)

at org.apache.hadoop.mapreduce.Job$5.run(Job.java:686)

at java.security.AccessController.doPrivileged(Native Method)

at javax.security.auth.Subject.doAs(Subject.java:415)

at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)

at org.apache.hadoop.mapreduce.Job.getTaskCompletionEvents(Job.java:686)

at org.apache.hadoop.mapreduce.Job.monitorAndPrintJob(Job.java:1387)

at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1327)

at org.apache.hadoop.examples.terasort.TeraGen.run(TeraGen.java:305)

at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)

at org.apache.hadoop.examples.terasort.TeraGen.main(TeraGen.java:309)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:601)

at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)

at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144)

at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:601)

at org.apache.hadoop.util.RunJar.run(RunJar.java:221)

at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)

at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)

at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121)

at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)

at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:223)

at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)

at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)

at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:193)

... 52 more

15/08/19 15:21:00 WARN security.UserGroupInformation: PriviledgedActionException as:vishwa828@hydNEXTGEN.COM (auth:KERBEROS) cause:java.io.IOException: Couldn't setup connection for vishwa828@hydNEXTGEN.COM to hdp-poc2.hydnextgen.com/10.138.90.72:8032

15/08/19 15:21:00 WARN security.UserGroupInformation: PriviledgedActionException as:vishwa828@hydNEXTGEN.COM (auth:KERBEROS) cause:java.io.IOException: Failed on local exception: java.io.IOException: Couldn't setup connection for vishwa828@hydNEXTGEN.COM to hdp-poc2.hydnextgen.com/10.138.90.72:8032; Host Details : local host is: "hdp-poc2.hydnextgen.com/10.138.90.72"; destination host is: "hdp-poc2.hydnextgen.com":8032;

java.io.IOException: Failed on local exception: java.io.IOException: Couldn't setup connection for vishwa828@hydNEXTGEN.COM to hdp-poc2.hydnextgen.com/10.138.90.72:8032; Host Details : local host is: "hdp-poc2.hydnextgen.com/10.138.90.72"; destination host is: "hdp-poc2.hydnextgen.com":8032;

at org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:351)

at org.apache.hadoop.mapred.ClientServiceDelegate.getTaskCompletionEvents(ClientServiceDelegate.java:389)

at org.apache.hadoop.mapred.YARNRunner.getTaskCompletionEvents(YARNRunner.java:582)

at org.apache.hadoop.mapreduce.Job$5.run(Job.java:689)

at org.apache.hadoop.mapreduce.Job$5.run(Job.java:686)

at java.security.AccessController.doPrivileged(Native Method)

at javax.security.auth.Subject.doAs(Subject.java:415)

at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)

at org.apache.hadoop.mapreduce.Job.getTaskCompletionEvents(Job.java:686)

at org.apache.hadoop.mapreduce.Job.monitorAndPrintJob(Job.java:1387)

at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1327)

at org.apache.hadoop.examples.terasort.TeraGen.run(TeraGen.java:305)

at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)

at org.apache.hadoop.examples.terasort.TeraGen.main(TeraGen.java:309)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:601)

at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)

at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144)

at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:601)

at org.apache.hadoop.util.RunJar.run(RunJar.java:221)

at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

 

 

Posts: 1,886
Kudos: 425
Solutions: 300
Registered: ‎07-31-2013

Re: MR Job killed.

 Open Question:

 

1) Does MR job fail if the job continues execution even after life time? How does life time in kerberos configuration file effect MR job?

 


No, the job itself (on the cluster) will not fail cause it uses a delegation token (obtained from the services in the cluster, against the original kerberos tgt pre-submission) instead. The client in your case fails at polling the job cause the local tgt has expired (and the job was run with synchronous monitoring at the client), but this does not impact the actual, running job which you can continue to monitor via a 'mapred job' command subsequently, or via the Web UI.

Highlighted
Explorer
Posts: 14
Registered: ‎08-14-2015

Re: MR Job killed.

Hi,

 

THanks for your time on this query.

 

MR job is killed and I am not sure on how to resume the job!

Could you please provide more details on the explanation provided by you, I could not understand it as I am a newbie to Hadoop.

Announcements