question Java Spark driver and executor logs in cluster mode in Support Questions

Java Spark driver and executor logs in cluster mode

Sugumar — Tue, 27 Apr 2021 17:56:49 GMT

--driver-java-options %%JAVA_OPTIONS%% --driver-memory 6g \
--master dse://?workpool=default \
--deploy-mode cluster \
--supervise \
--conf spark.executor.extraJavaOptions=%%EXECUTOR_JAVA_OPTIONS%% --total-executor-cores 6 --executor-memory 6g --num-executors 6 \
--conf spark.driver.extraClassPath=${CLASSPATH} --conf spark.executor.extraClassPath=${CLASSPATH} \
%%ADDITIONAL_SPARK_SUBMIT_CONFIGS%% \

using the above configuration.

I am trying to consolidate driver and worker node logs in console or single file and new to spark application.
Can you please provide your adivse

Re: Java Spark driver and executor logs in cluster mode

RangaReddy — Tue, 04 May 2021 06:56:58 GMT

Hi @Sugumar

When you run application in client mode, you can see driver logs in your console (where you have submitted application) and executor logs in respected container.

But in the case of cluster mode, spark driver will be launched in one of the container. So you will not see driver logs in console.

To get the logs in yarn,

yarn logs -applicationId <Application_ID> > application_id.log

Re: Java Spark driver and executor logs in cluster mode

Sugumar — Tue, 04 May 2021 07:17:25 GMT

I am running spark job using DSE not yarn client/cluster mode,

herewith i am including spark submit command for your reference,

-- spark-submit \
--driver-java-options %%JAVA_OPTIONS%% --driver-memory 6g \
--master dse://?workpool=default \
--deploy-mode cluster \
--supervise \
--conf spark.executor.extraJavaOptions=%%EXECUTOR_JAVA_OPTIONS%% --total-executor-cores 6 --executor-memory 6g --num-executors 6 \
--conf spark.driver.extraClassPath=${CLASSPATH} --conf spark.executor.extraClassPath=${CLASSPATH} \
%%ADDITIONAL_SPARK_SUBMIT_CONFIGS%% \
--class com.test.start.RptProcessor file:%%JAR_PATH%%

Re: Java Spark driver and executor logs in cluster mode

RangaReddy — Tue, 04 May 2021 08:48:11 GMT

Hi @Sugumar

I don't have much idea on DSE cluster. Please check the following link maybe it will help

https://docs.datastax.com/en/dse/6.7/dse-admin/datastax_enterprise/spark/sparkLogging.html