About Sugumar

Sugumar · ‎05-04-2021

I am running spark job using DSE not yarn client/cluster mode, herewith i am including spark submit command for your reference, -- spark-submit \ --driver-java-options %%JAVA_OPTIONS%% --driver-memory 6g \ --master dse://?workpool=default \ --deploy-mode cluster \ --supervise \ --conf spark.executor.extraJavaOptions=%%EXECUTOR_JAVA_OPTIONS%% --total-executor-cores 6 --executor-memory 6g --num-executors 6 \ --conf spark.driver.extraClassPath=${CLASSPATH} --conf spark.executor.extraClassPath=${CLASSPATH} \ %%ADDITIONAL_SPARK_SUBMIT_CONFIGS%% \ --class com.test.start.RptProcessor file:%%JAR_PATH%%

Sugumar · ‎05-03-2021

Hello Team, I am running spark job in cluster mode using java, the jobs are running without issue but not able to see the logs through console both driver and executor log, but spark UI able to see the executor logs. I want to see the driver and executor logs in console while running the spark job. Could you please assist someone to help this issue asap.

Sugumar · ‎04-27-2021

--driver-java-options %%JAVA_OPTIONS%% --driver-memory 6g \ --master dse://?workpool=default \ --deploy-mode cluster \ --supervise \ --conf spark.executor.extraJavaOptions=%%EXECUTOR_JAVA_OPTIONS%% --total-executor-cores 6 --executor-memory 6g --num-executors 6 \ --conf spark.driver.extraClassPath=${CLASSPATH} --conf spark.executor.extraClassPath=${CLASSPATH} \ %%ADDITIONAL_SPARK_SUBMIT_CONFIGS%% \ using the above configuration. I am trying to consolidate driver and worker node logs in console or single file and new to spark application. Can you please provide your adivse

Online	Offline
Last Visited	‎05-28-2021 03:30 AM

Member Since	‎04-27-2021 10:52 AM
Last Visited	‎05-28-2021 03:30 AM
Posts	3

Cloudera Community

Cloudera Community

Re: Java Spark driver and executor logs in cluster...

Spark Log4j - Driver and executor log in console

Java Spark driver and executor logs in cluster mod...