Skip to main content
Kolli
Kolli
Silver ⭐️⭐️
September 5, 2023
Question

Client can not authenticate via ( token, kerberos)

I am getting client can not communicate via ( token, kerberos) exception when I am trying to access yarn resource manager from java application. 

As part of my requirement in my spark java application I need to check some other applications running or not. For this I am using YarnClient class when I try to access this method getting above mentioned exception. 

Same working for me in my local system with kerberos authentication. 

I am using Hadoop library classes only to authenticate. UsergroupInformation class

 

IMG_20230905_203420.jpg

 

IMG_20230905_203620.jpg

 

 

7 replies

DianaTorres
Community Manager
DianaTorres
Community Manager
September 5, 2023

@Kolli Welcome to the Cloudera Community!

To help you get the best possible solution, I have tagged our YARN expert @Bharati  who may be able to assist you further.

Please keep us updated on your post, and we hope you find a satisfactory solution to your query.

Translate
Regards, Diana Torres, Senior Community Moderator
Kolli
KolliAuthor
Silver ⭐️⭐️
September 5, 2023

Hi Diana, 

Thank you. 

Attaching more logs for reference

2023/09/04 09:30:05.522 INFO o.a.h.s.UserGroupInformation
: Login successful for user abc using keytab file abc. Keytab auto renewal enabled : false
2023/09/04 09:30:05.522 INFO c.s.f.c.s.c.SchedulingConfig After login
2023/09/04 09:30:05.730 WARN o.a.h.i.Client
Exception encountered while connecting to the server : org.apache.hadoop. security.AccessControlException: Client cannot authenticate via: [TOKEN, KERBEROS] 2023/09/04 09:30:05.734 ERROR o.a.s.d.y.ApplicationMaster
: User class threw exception: java.io. IOException: DestHost: destPort uklvadhdp123 : 8032 , LocalHost: localPort ukhdp/133.0Failed on local exception: java.io. IOException: org.apache. hadoop. security. AccessControlException: Client cannot authenticate via: [TOKEN, KERBEROS]

java.io. IOException: DestHost: destPort uklvad, LocalHost: localPort Failed on local exception: java.io. IOException: org. apache. hadoop. security. AccessControlException: Client cannot authenticate via: [TOKEN, KERBEROS]
at sun.reflect.NativeConstructorAccessorImpl.newInstance0 (Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance (NativeConstructorAccessorImpl. java : 62) at sun. reflect.DelegatingConstructorAccessorImpl.newInstance (DelegatingConstructorAccessorImpl. java : 45) at java.lang.reflect. Constructor.newInstance (Constructor. java : 423) at org.apache.hadoop.net.NetUtils. wrapWithMessage (NetUtils.java : 892) at org.apache.hadoop.net.NetUtils. wrapException (NetUtils.java : 867) at org. apache.hadoop.ipc.Client. getRpcResponse (Client. java: 1566) at org. apache.hadoop.ipc.Client.call (Client. java: 1508) at org.apache.hadoop.ipc.Client.call (Client.java: 1405) at org. apache.hadoop. ipc. ProtobufRpcEngine$Invoker.invoke (ProtobufRpcEngine. java :233) at org. apache. hadoop.ipc. ProtobufRpcEngine$Invoker.invoke (ProtobufRpcEngine. java :118) at com. sun. proxy. $Proxy79.getApplications (Unknown Source) at org. apache. hadoop.yarn. api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getApplications (ApplicationClientProtocolPBClientImpl. java: 316) at sun. reflect.NativeMethodAccessorImpl.invoke0 (Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl. java: 62) at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl. java : 43) at java.lang. reflect. Method. invoke (Method. java : 498) at org. apache.hadoop.io. retry.RetryInvocationHandler. invokeMethod (RetryInvocationHandler. java : 431) at org. apache.hadoop.io. retry. RetryInvocationHandler$Call.invokeMethod (RetryInvocationHandler. java:166) at org. apache.hadoop.io. retry.RetryInvocationHandler$Call.invoke (RetryInvocationHandler. java: 158) at org.apache.hadoop.io. retry. RetryInvocationHandler$Call. invokeOnce (RetryInvocationHandler. java : 96) at org. apache.hadoop.io. retry. RetryInvocationHandler. invoke (RetryInvocationHandler. java : 362) at com. sun.proxy.$Proxy80.getApplications (Unknown Source) at org.apache.hadoop.yarn.client.api. impl. YarnClientImpl.getApplications (YarnClientImpl. java:651) at com. scheduling. config. SchedulingConfig.renameFolders (SchedulingConfig. java: 901) at com.scheduling. SchedulingApp.main (SchedulingApp. java : 30)
>

 

I have modified destination host and source host for security reasons. 

Please let me know if any details are required to identify the root cause

 

Translate
Asok
Gold ⭐️
Asok
Gold ⭐️
September 6, 2023

Hello @Kolli 
Do you have a valid Kerberos ticket? Check using klist and it should display a message  like below.

Ticket cache: FILE:/tmp/krb5cc_12345
Default principal: user@example.com

Valid starting Expires Service principal
09/06/23 13:30:00 09/07/23 13:30:00 krbtgt/example.com@example.com
renew until 09/08/23 13:30:00

Translate
Kolli
KolliAuthor
Silver ⭐️⭐️
September 6, 2023

Hi Asok, 

Yes, we have valid kerberos ticket. We are using keytab to authenticate. With same keytab I am able to access from local. When I deploy to cluster then I am facing this issue. 

Thanks. 

Translate
Kolli
KolliAuthor
Silver ⭐️⭐️
September 6, 2023

2023/09/04 09:30:05.522 INFO o.a.h.s.UserGroupInformation
: Login successful for user abc using keytab file abc. Keytab auto renewal enabled : false
2023/09/04 09:30:05.522 INFO c.s.f.c.s.c.SchedulingConfig After login
2023/09/04 09:30:05.730 WARN o.a.h.i.Client
Exception encountered while connecting to the server : org.apache.hadoop. security.AccessControlException: Client cannot authenticate via: [TOKEN, KERBEROS] 2023/09/04 09:30:05.734 ERROR o.a.s.d.y.ApplicationMaster
: User class threw exception: java.io. IOException: DestHost: destPort uklvadhdp123 : 8032 , LocalHost: localPort ukhdp/133.0Failed on local exception: java.io. IOException: org.apache. hadoop. security. AccessControlException: Client cannot authenticate via: [TOKEN, KERBEROS]

In above error message first I am getting login successful. After that if I try to access yarn resource manager getting this issue

Translate
lou-201
lou-201
Silver ⭐️⭐️
June 21, 2024

Hi I'm also facing this same exact issue.

I can submit the distcp job to yarn which runs and completes but I'm not able to check the progress of the job while it's in running.

The strange thing is that there's no issue when I use the hadoop2 library. This issue came up as I upgrade to hadoop version > 3.2.3

Where you able to resolve your issue?

 

Translate
    lou-201
    lou-201
    Silver ⭐️⭐️
    June 25, 2024

    Despite extensive efforts, I was unable to directly resolve the issue, but I devised a workaround.

    Rather than directly accessing the Hadoop Job object for status updates, I extracted the job ID after submitting the job. Using this job ID, I created an ApplicationID object, which I then used to instantiate my YarnClient. This approach enabled me to effectively monitor the status and completion rate of the running job.

    Interestingly, both my distcp job and YarnClient are utilizing the same HadoopConf object (YarnClient is instantiated right after the DistCP Job is executed) and is within the same scope of the UserGroupInformation. The exact reason why the YarnClient can access the necessary information while the Job object cannot remains unclear. Nevertheless, this workaround has successfully unblocked me.

    Additional context: I am using Java 8 and running on an Ubuntu Xenial image.

    Translate
    Terms & ConditionsAccessibility statement