Support Questions

Find answers, ask questions, and share your expertise

Spark job Slowness and spark-submit log file

Hello,

We are running spark jobs using spark-submit and it is taking longer to run. Where is the log file I can track the status?

How do I find out the status ?

1 REPLY 1

@Anpan K

Depends on what master and deploy mode you are using.

For master yarn and deploy mode client the driver logs will be printed as part of spark-submit output. To fetch Application Master and executor logs you need to check RM UI or run yarn logs -applicationId <appId>

For master yarn and deploy mode cluster you need to check RM UI or run yarn logs -applicationId <appId> since everything is running on the yarn cluster.

For master local everything is printed as part of the stdout of the spark-submit command.

HTH

*** If you found this answer addressed your question, please take a moment to login and click the "accept" link on the answer.

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.