I am running spark job total time taking around 18s. in that 8s for actual logic processing and remaining 10s for spark context initialization and allocation of executors and memory.
Any idea how can reduce 10s to 5s or 4s.
Two ways you can allocate the resources upfront(not 100% but almost) to reduce the duration
Hope this helps !!