Support Questions
Find answers, ask questions, and share your expertise

spark job take over 30 seconds to start after submit

spark job take over 30 seconds to start after submit

Explorer

Hi all,

we started to test our spark grid, currently it small and contains 5 servers,

After the initial submit there is one long java service that take approx 15-20 seconds,

and then allot of small java services (one for each executor ) each takes another 15-20 seconds

and only now my process is launched it takes 36 seconds on average

When running large heavy jobs the grid work well, but when running small jobs that take few seconds each it does not work as we would like it to ( it take 35 seconds to init and then 2-3 seconds to run...)

any idea what can be done ?

1 REPLY 1

Re: spark job take over 30 seconds to start after submit