Hello 🙂 I have the same issue! I have integrated the edge node with Active directory users could connect and submit theirs jobs to yarn before enabling Kerberos on the cluster. Actually I have used samba on edge node to create users folders and get information about users    Now I configured the Kerberos and so I am getting the same error user1 not found , user1 is in AD  should I now add this user with normal command add user on alll nodes ? How could it be as AD user and not local one ? I did not configured samba on others nodes may I do it ?  thanks a lot in advance  ... View more

Hello   Please how did u add users? Actually i am using the active directory users and I just add them into Edge node using samba + kerberos   Now I have enabled kerberos on the hadoop hortonworks cluster => I got the same issue as yours So may I add the same  user to all nodes? adduser? which group? how could it be resolved as an AD user?   Thanks   ... View more

should i create principle for each user in the AD ? We are using active directory users? If yes how so?   Many thanks Asma ... View more

Actually, for more details: In my ambari server machine I have this ticket: [root@ambariserver ~]# klist Ticket cache: FILE:/tmp/krb5cc_0 Default principal: spark-analytics_hadoop@REALM.COM Valid starting Expires Service principal 02/03/2020 13:31:21 02/03/2020 23:31:21 krbtgt/REALM.COM@REALM.COM renew until 02/10/2020 13:31:21   When i connect with spark user : HADOOP_ROOT_LOGGER=DEBUG,console /usr/hdp/3.1.4.0-315/spark2/bin/spark-submit --class org.apache.spark.examples.SparkPi --master spark://Edgenode:7077 --num-executors 4 --driver-memory 512m --executor-memory 512m --executor-cores 1 /usr/hdp/3.1.4.0-315/spark2/examples/jars/spark-examples_2.11-2.3.2.3.1.4.0-315.jar   => OK   Now if I connect from the Edge Node [root@EdgeNode~]# klist Ticket cache: FILE:/tmp/krb5cc_0 Default principal: spark/EdgeNode@REALM.COM Valid starting Expires Service principal 02/03/2020 16:52:12 02/04/2020 02:52:12 krbtgt/REALM.COM@REALM.COM renew until 02/10/2020 16:52:12 But when I connect with user spark   HADOOP_ROOT_LOGGER=DEBUG,console /usr/hdp/3.1.4.0-315/spark2/bin/spark-submit --class org.apache.spark.examples.SparkPi --master spark://Edgenode:7077 --num-executors 4 --driver-memory 512m --executor-memory 512m --executor-cores 1 /usr/hdp/3.1.4.0-315/spark2/examples/jars/spark-examples_2.11-2.3.2.3.1.4.0-315.jar   => I got error :   20/02/03 17:53:01 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@69cac930{/metrics/json,null,AVAILABLE,@Spark} 20/02/03 17:53:01 WARN Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS] 20/02/03 17:53:01 ERROR SparkContext: Error initializing SparkContext. java.io.IOException: DestHost:destPort NameNode:8020 , LocalHost:localPort EdgeNode/10.48.142.32:0. Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS] at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:4     Did I miss something please?   Users from their laptop launch this commands   cluster = RxSpark(sshHostname = " EdgeNode ", sshUsername = "username") rxSetComputeContext(cluster) source = c("~/AirlineDemoSmall.csv") dest_file = "/share" rxHadoopMakeDir(dest_file) They are getting thr same issue On all node cluster hdfs dfs -ls / is working well   Please advise Thanks Asma   ... View more

Thanks a lot   Now the problem for hdfs is fixed  however when i try to launch a script from an edge node , i am getting the same issue /usr/hdp/3.1.4.0-315/spark2/bin/spark-submit --class org.apache.spark.examples.SparkPi --master spark://edgenode.servername:7077 --num-executors 4 --driver-memory 512m --executor-memory 512m --executor-cores 1 /usr/hdp/3.1.4.0-315/spark2/examples/jars/spark-examples_2.11-2.3.2.3.1.4.0-315.jar   Results : 20/02/03 15:13:41 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20200203151341-0000/79 is now RUNNING 20/02/03 15:13:41 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@69cac930{/metrics/json,null,AVAILABLE,@Spark} 20/02/03 15:13:42 WARN Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS] 20/02/03 15:13:42 ERROR SparkContext: Error initializing SparkContext. java.io.IOException: DestHost:destPort namenode.servername:8020 , LocalHost:localPort edgenodeaddress:0. Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS] at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:831) at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:806) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1502) at org.apache.hadoop.ipc.Client.call(Client.java:1444) at org.apache.hadoop.ipc.Client.call(Client.java:1354) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116) at com.sun.proxy.$Proxy11.getFileInfo(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:900) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) at com.sun.proxy.$Proxy12.getFileInfo(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1660) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1577) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1574) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1589) at org.apache.spark.scheduler.EventLoggingListener.start(EventLoggingListener.scala:100) at org.apache.spark.SparkContext.<init>(SparkContext.scala:522) at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2498) at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:934) at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:925) at scala.Option.getOrElse(Option.scala:121) at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:925) at org.apache.spark.examples.SparkPi$.main(SparkPi.scala:31) at org.apache.spark.examples.SparkPi.main(SparkPi.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:904) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:198) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:228) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:137) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) Caused by: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS] at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:758) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730) at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:721) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:814) at org.apache.hadoop.ipc.Client$Connection.access$3600(Client.java:411) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1559) at org.apache.hadoop.ipc.Client.call(Client.java:1390) ... View more

Thanks a lot 🙂   I have configured the Cluster with Kerberos using Active Directory but i got some issues when connecting    [root@server keytabs]# hdfs dfs -ls / 20/01/31 16:31:19 WARN ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS] ls: DestHost:destPort namenode:8020 , LocalHost:localPort ambari/ip:0. Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]   Any idea please? looks like  the 8020 ports is also blocked   Thanks Asma ... View more

thak you for your help I tried to restart th ambari server but in vain . I got this error 2020-01-30 18:20:21,866 INFO [main] KerberosChecker:64 - Checking Ambari Server Kerberos credentials. 2020-01-30 18:20:22,052 ERROR [main] KerberosChecker:120 - Client not found in Kerberos database (6) 2020-01-30 18:20:22,052 ERROR [main] AmbariServer:1119 - Failed to run the Ambari Server org.apache.ambari.server.AmbariException: Ambari Server Kerberos credentials check failed. Check KDC availability and JAAS configuration in /etc/ambari-server/conf/krb5JAASLogin.conf at org.apache.ambari.server.controller.utilities.KerberosChecker.checkJaasConfiguration(KerberosChecker.java:121) at org.apache.ambari.server.controller.AmbariServer.main(AmbariServer.java:1110)   ht JAASLogin is configured like this   com.sun.security.jgss.krb5.initiate { com.sun.security.auth.module.Krb5LoginModule required renewTGT=false doNotPrompt=true useKeyTab=true keyTab="/etc/security/ambariservername.keytab" principal="ambariservername@REALM.COM" storeKey=true useTicketCache=false; };   I tried to follow  these links  https://docs.cloudera.com/HDPDocuments/HDP3/HDP-3.1.5/authentication-with-kerberos/content/kerberos_optional_install_a_new_mit_kdc.html https://docs.cloudera.com/HDPDocuments/HDP3/HDP-3.1.5/authentication-with-kerberos/content/set_up_kerberos_for_ambari_server.html   Any suggestion please? 😞 Thanks ... View more

Thank you again!!   1) I have installed these client on my edge node 🙂   2) For instance, we decide that the cluster is not configured with kerberos   3) Actually, I want that end users could submit spark-shell 🙂   4) For this , I created a local user called "sparkuser" on the edge node   5) with which tool could client use the spark-shell? API? application? All I want that users can use the sparkuser that I have created on edge nôde to submit their scripts but I don't not want that these users access the edge node server directly (like remotelx or via an API? or else? I hope that I explained more 😄   Many thanks for your help and patience Asma   ... View more

In a nutshell, you don't need a Master process the edge node but Client to initiate communication with the Cluster   => Actually this is my question 🙂 Which client? Should I create a local user like spark and then ask the end users to use it in order to launch this command for example from their machines??   spark-sumbit  --master spark://edgenode:7077  calculPi.jar?   All that I need for now that the end  users could execute their scripts spark or Python ... from their side, how could we do this? do they need to access the edge server?   Thanks Asma ... View more

Thank you for your reply 🙂   I will explain my actual situation. I have installed Hadoop Cluster using ambari HortonWorks. One of nodes is an Edge node, I defined it as a spark master and I could then run spark-sumbit from this linux server (edge node) with spark user (spark) on 4 workers ( datanodes)   My question now, I will have developers and end users who should execute scripts from their local machine on the edge node. These users should not access directly the linux server (Edge node)  but they will have to launch scripts (spark-sumbit). How can I create accounts for them? How could they access the edge node ? Thanks Asma ... View more