Member since
09-05-2023
5
Posts
0
Kudos Received
0
Solutions
09-12-2023
07:46 AM
Log on driver : 2021/02/18 18:10:09.612 DEBUG UserGroupInformation: [] hadoop login 2021/02/18 18:10:09.614 DEBUG UserGroupInformation: [] hadoop login commit 2021/02/18 18:10:09.617 DEBUG UserGroupInformation: [] using local user:UnixPrincipal: tss 2021/02/18 18:10:09.618 DEBUG UserGroupInformation: [] Using user: "UnixPrincipal: tss" with name tss 2021/02/18 18:10:09.618 DEBUG UserGroupInformation: [] User entry: "tss" 2021/02/18 18:10:09.622 DEBUG UserGroupInformation: [] Reading credentials from location set in HADOOP_TOKEN_FILE_LOCATION: /srv/BigData/hadoop/data1/nm/localdir/usercache/tss/appcache/application_1611987488531_0743/container_e13_1611987488531_0743_02_000001/container_tokens 2021/02/18 18:10:09.635 DEBUG UserGroupInformation: [] Loaded 2 tokens 2021/02/18 18:10:09.636 DEBUG UserGroupInformation: [] UGI loginUser:tss (auth:SIMPLE) 2021/02/18 18:10:09.637 INFO AMCredentialRenewer: [] Attempting to login to KDC using principal: tss@HADOOP.COM 2021/02/18 18:10:09.727 DEBUG UserGroupInformation: [] hadoop login 2021/02/18 18:10:09.728 DEBUG UserGroupInformation: [] hadoop login commit 2021/02/18 18:10:09.728 DEBUG UserGroupInformation: [] using kerberos user:tss@HADOOP.COM 2021/02/18 18:10:09.729 DEBUG UserGroupInformation: [] Using user: "tss@HADOOP.COM" with name tss@HADOOP.COM 2021/02/18 18:10:09.729 DEBUG UserGroupInformation: [] User entry: "tss@HADOOP.COM" 2021/02/18 18:10:09.730 INFO AMCredentialRenewer: [] Successfully logged into KDC. 2021/02/18 18:10:09.733 DEBUG UserGroupInformation: [] PrivilegedAction as:tss@HADOOP.COM (auth:KERBEROS) Log on executor: 2021/02/18 18:10:19.426 DEBUG UserGroupInformation: [] hadoop login 2021/02/18 18:10:19.427 DEBUG UserGroupInformation: [] hadoop login commit 2021/02/18 18:10:19.428 DEBUG UserGroupInformation: [] using local user:UnixPrincipal: tss 2021/02/18 18:10:19.429 DEBUG UserGroupInformation: [] Using user: "UnixPrincipal: tss" with name tss 2021/02/18 18:10:19.429 DEBUG UserGroupInformation: [] User entry: "tss" 2021/02/18 18:10:19.432 DEBUG UserGroupInformation: [] Reading credentials from location set in HADOOP_TOKEN_FILE_LOCATION: /srv/BigData/hadoop/data1/nm/localdir/usercache/tss/appcache/application_1611987488531_0743/container_e13_1611987488531_0743_02_000002/container_tokens 2021/02/18 18:10:19.448 DEBUG UserGroupInformation: [] Loaded 2 tokens 2021/02/18 18:10:19.449 DEBUG UserGroupInformation: [] UGI loginUser:tss (auth:SIMPLE) 2021/02/18 18:10:19.449 DEBUG UserGroupInformation: [] PrivilegedAction as:tss (auth:SIMPLE) from:org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:64) In driver ,am would renew the token ,ugi can login as kerberos. While in executor ,the ugi login simple that ugi method hasKerberosCredentials() return a false. As as result , a exception has been throwed which reports "Could not locate Kerberos Principal on currently logged in user." spark-submit --class cn.zwy.SparkWordCount \ --master yarn --deploy-mode cluster --conf 'spark.es.nodes=fusioninsight02:24100' \ --jars ~/elasticsearch-hadoop-7.10.2.jar \ --conf 'spark.es.ssl.enabled=true' \ --conf 'spark.es.net.ssl=true' \ --conf 'spark.es.resource=22lastk_eoi_2020_02_05' \ --conf 'es.net.ssl=true' \ --conf 'spark.es.security.authentication=kerberos' \ --conf 'es.security.authentication=kerberos' \ --conf 'spark.es.net.spnego.auth.elasticsearch.principal=elasticsearch/hadoop.hadoop.com@HADOOP.COM' \ --conf spark.yarn.submit.waitAppCompletion=false \ --conf spark.driver.extraJavaOptions="-Djava.security.krb5.conf=/opt/huawei/Bigdata/FusionInsight_BASE_6.5.1/3_49_KerberosClient/etc/kdc.conf" \ --conf spark.executor.extraJavaOptions="-Djava.security.krb5.conf=/opt/huawei/Bigdata/FusionInsight_BASE_6.5.1/3_49_KerberosClient/etc/kdc.conf" \ --conf spark.yarn.credentials.file=/tmp \ --principal tss@HADOOP.COM \ --keytab /data01/jax1.0/jax/user.keytab \ --conf 'hadoop.security.authentication=kerberos' \ ~/spark-test-1.0-SNAPSHOT.jar Anyone can help resolve this issue
... View more
Labels:
- Labels:
-
Kerberos
09-06-2023
07:51 AM
2023/09/04 09:30:05.522 INFO o.a.h.s.UserGroupInformation : Login successful for user abc using keytab file abc. Keytab auto renewal enabled : false 2023/09/04 09:30:05.522 INFO c.s.f.c.s.c.SchedulingConfig After login 2023/09/04 09:30:05.730 WARN o.a.h.i.Client Exception encountered while connecting to the server : org.apache.hadoop. security.AccessControlException: Client cannot authenticate via: [TOKEN, KERBEROS] 2023/09/04 09:30:05.734 ERROR o.a.s.d.y.ApplicationMaster : User class threw exception: java.io. IOException: DestHost: destPort uklvadhdp123 : 8032 , LocalHost: localPort ukhdp/133.0Failed on local exception: java.io. IOException: org.apache. hadoop. security. AccessControlException: Client cannot authenticate via: [TOKEN, KERBEROS] In above error message first I am getting login successful. After that if I try to access yarn resource manager getting this issue
... View more
09-06-2023
07:48 AM
Hi Asok, Yes, we have valid kerberos ticket. We are using keytab to authenticate. With same keytab I am able to access from local. When I deploy to cluster then I am facing this issue. Thanks.
... View more
09-05-2023
09:02 AM
Hi Diana, Thank you. Attaching more logs for reference 2023/09/04 09:30:05.522 INFO o.a.h.s.UserGroupInformation : Login successful for user abc using keytab file abc. Keytab auto renewal enabled : false 2023/09/04 09:30:05.522 INFO c.s.f.c.s.c.SchedulingConfig After login 2023/09/04 09:30:05.730 WARN o.a.h.i.Client Exception encountered while connecting to the server : org.apache.hadoop. security.AccessControlException: Client cannot authenticate via: [TOKEN, KERBEROS] 2023/09/04 09:30:05.734 ERROR o.a.s.d.y.ApplicationMaster : User class threw exception: java.io. IOException: DestHost: destPort uklvadhdp123 : 8032 , LocalHost: localPort ukhdp/133.0Failed on local exception: java.io. IOException: org.apache. hadoop. security. AccessControlException: Client cannot authenticate via: [TOKEN, KERBEROS] java.io. IOException: DestHost: destPort uklvad, LocalHost: localPort Failed on local exception: java.io. IOException: org. apache. hadoop. security. AccessControlException: Client cannot authenticate via: [TOKEN, KERBEROS] at sun.reflect.NativeConstructorAccessorImpl.newInstance0 (Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance (NativeConstructorAccessorImpl. java : 62) at sun. reflect.DelegatingConstructorAccessorImpl.newInstance (DelegatingConstructorAccessorImpl. java : 45) at java.lang.reflect. Constructor.newInstance (Constructor. java : 423) at org.apache.hadoop.net.NetUtils. wrapWithMessage (NetUtils.java : 892) at org.apache.hadoop.net.NetUtils. wrapException (NetUtils.java : 867) at org. apache.hadoop.ipc.Client. getRpcResponse (Client. java: 1566) at org. apache.hadoop.ipc.Client.call (Client. java: 1508) at org.apache.hadoop.ipc.Client.call (Client.java: 1405) at org. apache.hadoop. ipc. ProtobufRpcEngine$Invoker.invoke (ProtobufRpcEngine. java :233) at org. apache. hadoop.ipc. ProtobufRpcEngine$Invoker.invoke (ProtobufRpcEngine. java :118) at com. sun. proxy. $Proxy79.getApplications (Unknown Source) at org. apache. hadoop.yarn. api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getApplications (ApplicationClientProtocolPBClientImpl. java: 316) at sun. reflect.NativeMethodAccessorImpl.invoke0 (Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl. java: 62) at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl. java : 43) at java.lang. reflect. Method. invoke (Method. java : 498) at org. apache.hadoop.io. retry.RetryInvocationHandler. invokeMethod (RetryInvocationHandler. java : 431) at org. apache.hadoop.io. retry. RetryInvocationHandler$Call.invokeMethod (RetryInvocationHandler. java:166) at org. apache.hadoop.io. retry.RetryInvocationHandler$Call.invoke (RetryInvocationHandler. java: 158) at org.apache.hadoop.io. retry. RetryInvocationHandler$Call. invokeOnce (RetryInvocationHandler. java : 96) at org. apache.hadoop.io. retry. RetryInvocationHandler. invoke (RetryInvocationHandler. java : 362) at com. sun.proxy.$Proxy80.getApplications (Unknown Source) at org.apache.hadoop.yarn.client.api. impl. YarnClientImpl.getApplications (YarnClientImpl. java:651) at com. scheduling. config. SchedulingConfig.renameFolders (SchedulingConfig. java: 901) at com.scheduling. SchedulingApp.main (SchedulingApp. java : 30) > I have modified destination host and source host for security reasons. Please let me know if any details are required to identify the root cause
... View more
09-05-2023
08:07 AM
I am getting client can not communicate via ( token, kerberos) exception when I am trying to access yarn resource manager from java application. As part of my requirement in my spark java application I need to check some other applications running or not. For this I am using YarnClient class when I try to access this method getting above mentioned exception. Same working for me in my local system with kerberos authentication. I am using Hadoop library classes only to authenticate. UsergroupInformation class
... View more
Labels:
- Labels:
-
Apache YARN