Support Questions
Find answers, ask questions, and share your expertise

spark submit take time untill the process starts

Explorer

for the past few month we used the ambari with few servers.

I am trying to improve the time it take from submiting the job, to the time the jobs actualy starts\

In the current setup we have 4 computers with total of 64 executors

when the job is accepted in the server it take aprox 35 seconds to initialize all exectutors and start our process, where i can find the reason and is it posibble to improve\optimize the flow ?

can i keep the executor alive ? most of them time (99.99 % of the cases ) it the cpu/ram is not changed upon submited job

93029-executors-time.png

we based on HDP-2.6.4.0