Support Questions
Find answers, ask questions, and share your expertise

Spark yarn log location on Hdfs

Spark yarn log location on Hdfs

Hi ,

I have to locate the spark application yarn log on hdfs. When I go to hdfs://app-logs/user/logs-ifile then it shows container wise logs but I need consolidated log in file (like we get the consolidated log on CMD when we run spark-submit manually on CMD).So, from where can I locate the consolidated log of an application on hdfs?


Re: Spark yarn log location on Hdfs

Cloudera Employee

@ Muhammad Umar If you have run the spark job in yarn cluster mode ,then you can get both driver and executor container logs through yarn logs -applicationId <appId>, if it was a case of yarn client mode executor container logs would be available through yarn logs -applicationId <appId>, but driver logs are available on console. You can collect the driver logs to a file by configuring the log4j and passing to driver options, explained @