Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Spark yarn log location on Hdfs

Spark yarn log location on Hdfs

Hi ,

I have to locate the spark application yarn log on hdfs. When I go to hdfs://app-logs/user/logs-ifile then it shows container wise logs but I need consolidated log in file (like we get the consolidated log on CMD when we run spark-submit manually on CMD).So, from where can I locate the consolidated log of an application on hdfs?

1 REPLY 1
Highlighted

Re: Spark yarn log location on Hdfs

Cloudera Employee

@ Muhammad Umar If you have run the spark job in yarn cluster mode ,then you can get both driver and executor container logs through yarn logs -applicationId <appId>, if it was a case of yarn client mode executor container logs would be available through yarn logs -applicationId <appId>, but driver logs are available on console. You can collect the driver logs to a file by configuring the log4j and passing to driver options, explained @ https://community.hortonworks.com/articles/138849/how-to-capture-spark-driver-and-executor-logs-in-y...

Don't have an account?
Coming from Hortonworks? Activate your account here