Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Why spark taking time to allocate executors and memory before submiting job

Why spark taking time to allocate executors and memory before submiting job

New Contributor

I am running spark job total time taking around 18s. in that 8s for actual logic processing and remaining 10s for spark context initialization and allocation of executors and memory.

Any idea how can reduce 10s to 5s or 4s.

2 REPLIES 2

Re: Why spark taking time to allocate executors and memory before submiting job

Super Collaborator

Hi @Munikrishna Nalipireddy,

Two ways you can allocate the resources upfront(not 100% but almost) to reduce the duration

  1. Using the Livy For submitting the job where the context is already initiated and ready to take the request via the REST. (I believe best suited in your case)
  2. Using the streaming Context (applicable for cyclic schedules) which is already initiate the context and pickup your request in next cycle.

Hope this helps !!

Re: Why spark taking time to allocate executors and memory before submiting job

New Contributor

Thanks a lot bkosaraju....

Don't have an account?
Coming from Hortonworks? Activate your account here