When I am running Spark Applications in YARN Cluster mode on HDP 2.4.0 using spark-submit (App runs fine without any errors) ,but I do not see application logs when I try to check in the History Server - Resource manager UI or through yarn logs command. ( Also checked that log directories are empty)
When I do kdestroy and then get TGT for Spark user using
kinit -kt /keytabs/spark-headless-keytab.. spark
and then run the Spark application using spark-submit and run Spark Application (For example SparkPI ) and see the logs through History Server UI or using yarn logs command I see the logs.
I am not sure if I am missing some configurations here. Or Am I supposed to run all the Spark applications as Spark user?
I think that the reason for your issue is the permission on the folder you have configured as location for Spark logs. For instance, if the folder is owned by the user spark and the permissions are drwxr--r--, nobody but the spark user will be able to write in that folder, so only the jobs launched with the spark user will have the logs.