Spark yarn log location on Hdfs

[#1]

Hi ,

I have to locate the spark application yarn log on hdfs. When I go to hdfs://app-logs/user/logs-ifile then it shows container wise logs but I need consolidated log in file (like we get the consolidated log on CMD when we run spark-submit manually on CMD).So, from where can I locate the consolidated log of an application on hdfs?

[#2]

@ Muhammad Umar If you have run the spark job in yarn cluster mode ,then you can get both driver and executor container logs through yarn logs -applicationId <appId>, if it was a case of yarn client mode executor container logs would be available through yarn logs -applicationId <appId>, but driver logs are available on console. You can collect the driver logs to a file by configuring the log4j and passing to driver options, explained @ https://community.hortonworks.com/articles/138849/how-to-capture-spark-driver-and-executor-logs-in-y...