Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Java Spark driver and executor logs in cluster mode

avatar
New Contributor
--driver-java-options %%JAVA_OPTIONS%% --driver-memory 6g \
--master dse://?workpool=default \
--deploy-mode cluster \
--supervise \
--conf spark.executor.extraJavaOptions=%%EXECUTOR_JAVA_OPTIONS%% --total-executor-cores 6 --executor-memory 6g --num-executors 6 \
--conf spark.driver.extraClassPath=${CLASSPATH} --conf spark.executor.extraClassPath=${CLASSPATH} \
%%ADDITIONAL_SPARK_SUBMIT_CONFIGS%% \

using the above configuration.

I am trying to consolidate driver and worker node logs in console or single file and new to spark application.
Can you please provide your adivse


3 REPLIES 3

avatar
Master Collaborator

Hi @Sugumar 

 

When you run application in client mode, you can see driver logs in your console (where you have submitted application) and executor logs in respected container.

 

But in the case of cluster mode, spark driver will be launched in one of the container. So you will not see driver logs in console. 

 

To get the logs in yarn,

yarn logs -applicationId <Application_ID> > application_id.log

 

avatar
New Contributor

I am running spark job using DSE not yarn client/cluster mode, 

herewith i am including spark submit command for your reference,

-- spark-submit \
--driver-java-options %%JAVA_OPTIONS%% --driver-memory 6g \
--master dse://?workpool=default \
--deploy-mode cluster \
--supervise \
--conf spark.executor.extraJavaOptions=%%EXECUTOR_JAVA_OPTIONS%% --total-executor-cores 6 --executor-memory 6g --num-executors 6 \
--conf spark.driver.extraClassPath=${CLASSPATH} --conf spark.executor.extraClassPath=${CLASSPATH} \
%%ADDITIONAL_SPARK_SUBMIT_CONFIGS%% \
--class com.test.start.RptProcessor file:%%JAR_PATH%%

avatar
Master Collaborator

Hi @Sugumar 

 

I don't have much idea on DSE cluster. Please check the following link maybe it will help

 

https://docs.datastax.com/en/dse/6.7/dse-admin/datastax_enterprise/spark/sparkLogging.html