Support Questions
Find answers, ask questions, and share your expertise
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Spark UI to use Marathon assigned port

Spark UI to use Marathon assigned port

New Contributor

I am running PySpark Job (v2.0.2) with checkpoint enabled in Mesos cluster and am using Marathon for orchestration. When the job is restarted using Marathon, Spark UI is not getting started at the port specified by Marathon. Instead, it is picking port from the checkpoint. Is there a way we can make spark job to use the port assigned by Marathon instead of Spark job picking the configuration from the Checkpoint?

Any pointers would be greatly appreciated.

Thank you

Don't have an account?
Coming from Hortonworks? Activate your account here