Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Logs from spark executors

avatar
Explorer

Where can I find the logs from spark functions? I can find the logs of the spark application, but if I need to debug a map function that I wrote, I cannot find these logs.

thanks!

2 REPLIES 2

avatar
Master Collaborator

If you print or log to stdout, it goes to the stdout of the executor process, wherever that is running. In YARN-based deployment, you can use "yarn logs ..." to find the executor logs, I believe. Or dig through from the resource manager and find the executor process and its logs from the UI.

avatar
Explorer

Thanks for your reply.

I found in the UI the path to /var/log/spark/spark-worker. That is the spark executor's own log. I am actually looking for the logs that I added to functions that I wrote to map RDDs.

Please forgive me if I am using the wrong terminology here.