Unable to execute job on Yarn after Cluster Hadoop Enabling Kerberos

[#1]

Dear Community,

After enabling Kerberos with Active Directory on HortonWorks Hadoop Cluster (Ambari), users are unable to submit jobs on yarn

Error

Error in rxRemoteExecute(computeContext, shellCmd, schedulerJobInstance) :
  /var/RevoShare/aduser/cluster-7B39D0A894BC4F73ABC73D192697AFC3/start-job.sh: line 97: export: `-Xrs': not a valid identifier
/var/RevoShare/aduser/cluster-7B39D0A894BC4F73ABC73D192697AFC3/start-job.sh: line 97: export: `-Xss4m': not a valid identifier
ERROR: Fail to execute spark-submit. Last 20 lines' log:
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)
Caused by: org.apache.hadoop.security.AccessControlException: User aduser does not have permission to submit application_1580742122197_0003 to queue default
at org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.createAndPopulateNewRMApp(RMAppManager.java:429)
... 12 more
at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1498)
at org.apache.hadoop.ipc.Client.call(Client.java:1444)
at org.ap

I set these properties like 

yarn.scheduler.capacity.root.default.acl_submit_applications=yarn,yarn-ats,*

yarn.scheduler.capacity.root.acl_submit_applications=yarn,ambari-qa,*

Please advice

Asma

[#2]

when i updated the yarn conf

yarn.scheduler.capacity.root.default.acl_administer_jobs=yarn,*,user1
yarn.scheduler.capacity.root.default.acl_administer_queue=yarn,*,user1
yarn.scheduler.capacity.root.default.acl_submit_applications=yarn,yarn-ats,*,user1

=> user1 is able to authenticate

However he is getting this error now

Error in rxRemoteExecute(computeContext, shellCmd, schedulerJobInstance) :
  /var/RevoShare/user1/cluster-127006E48F49439EA1A090A78C9851C9/start-job.sh: line 97: export: `-Xrs': not a valid identifier
/var/RevoShare/fcuni001/cluster-127006E48F49439EA1A090A78C9851C9/start-job.sh: line 97: export: `-Xss4m': not a valid identifier
ERROR: Fail to execute spark-submit. Last 20 lines' log:
at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:934)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:925)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:925)
at com.microsoft.scaler.spark.api.SparkApp$.main(SparkApp.scala:28)
at com.microsoft.scaler.spark.api.SparkApp.main(SparkApp.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.re

On the namenode web interface

Application application_1580827205892_0001 failed 2 times due to AM Container for appattempt_1580827205892_0001_000002 exited with exitCode: -1000 Failing this attempt.Diagnostics: [2020-02-04 15:43:08.042]Application application_1580827205892_0001 initialization failed (exitCode=255) with output: main : command provided 0 main : run as user is fcuni001 main : requested yarn user is user1 User user1 not found For more detailed output, check the application tracking page: http://namenode:8088/cluster/app/application_1580827205892_0001 Then click on links to logs of each attempt. . Failing the application.

Any idea please?

Thanks a lot

[#3]

Were you able to figure out the issue to your "not a valid identifier" issue?

[#4]

Hi,

do you have apache ranger installed ? if yes, check that the right policies are added under yarn service and the ranger user sync service is configured and  syncing AD users and groups.

Best Regards