Hi Sudee,   How do you identify that driver kills all the allocated executors when the new stage gets started? Did you have application logs and event logs? Did you find any information in the application logs when the driver kills the executor? In which mode are you submitting the spark (Client (or). Cluster) .Do you have the driver logs if it is submitted in client mode Can you share to us?   Thanks AKR   ... View more

Reading through this blog helped me a lot: http://blog.cloudera.com/blog/2014/11/guidelines-for-installing-cdh-packages-on-unsupported-operating-systems/   Specially understanding that this can be a package mix/match issue because of trying to run YARN on a unsupported version (Ubuntu 16.04).   I removed package and reinstalled like this:   $> sudo apt-get remove zookeeper    this will remove a bunch of stuff along with zookeeper   $> sudo apt-get install hadoop-yarn-resourcemanager   this will reinstall resource manager for you.   Hope this helps! ... View more