Support Questions
Find answers, ask questions, and share your expertise

Java Spark driver and executor logs in cluster mode

New Contributor
--driver-java-options %%JAVA_OPTIONS%% --driver-memory 6g \
--master dse://?workpool=default \
--deploy-mode cluster \
--supervise \
--conf spark.executor.extraJavaOptions=%%EXECUTOR_JAVA_OPTIONS%% --total-executor-cores 6 --executor-memory 6g --num-executors 6 \
--conf spark.driver.extraClassPath=${CLASSPATH} --conf spark.executor.extraClassPath=${CLASSPATH} \
%%ADDITIONAL_SPARK_SUBMIT_CONFIGS%% \

using the above configuration.

I am trying to consolidate driver and worker node logs in console or single file and new to spark application.
Can you please provide your adivse


3 REPLIES 3

Expert Contributor

Hi @Sugumar 

 

When you run application in client mode, you can see driver logs in your console (where you have submitted application) and executor logs in respected container.

 

But in the case of cluster mode, spark driver will be launched in one of the container. So you will not see driver logs in console. 

 

To get the logs in yarn,

yarn logs -applicationId <Application_ID> > application_id.log

 

New Contributor

I am running spark job using DSE not yarn client/cluster mode, 

herewith i am including spark submit command for your reference,

-- spark-submit \
--driver-java-options %%JAVA_OPTIONS%% --driver-memory 6g \
--master dse://?workpool=default \
--deploy-mode cluster \
--supervise \
--conf spark.executor.extraJavaOptions=%%EXECUTOR_JAVA_OPTIONS%% --total-executor-cores 6 --executor-memory 6g --num-executors 6 \
--conf spark.driver.extraClassPath=${CLASSPATH} --conf spark.executor.extraClassPath=${CLASSPATH} \
%%ADDITIONAL_SPARK_SUBMIT_CONFIGS%% \
--class com.test.start.RptProcessor file:%%JAR_PATH%%

Expert Contributor

Hi @Sugumar 

 

I don't have much idea on DSE cluster. Please check the following link maybe it will help

 

https://docs.datastax.com/en/dse/6.7/dse-admin/datastax_enterprise/spark/sparkLogging.html

 

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.