Hi Sudee,   How do you identify that driver kills all the allocated executors when the new stage gets started? Did you have application logs and event logs? Did you find any information in the application logs when the driver kills the executor? In which mode are you submitting the spark (Client (or). Cluster) .Do you have the driver logs if it is submitted in client mode Can you share to us?   Thanks AKR   ... View more

Hi,   From the above Error "java.lang.OutOfMemoryError: Java heap space" this seems to be memory issue.Could you please increase the memory settings in your job and rerun the job again?   Thanks AKR ... View more

Hi Abdul,   Could you please share the entire Yarn application logs for our analysis (or) the entire stack trace?   Thanks AKR ... View more

Hi,   Please refer to the below forum. https://community.cloudera.com/t5/Support-Questions/cant-use-hive-on-spark-engine-cannot-create-client-erorr/td-p/240504   Hope this will help.   Thanks AKR ... View more

Hi Suresh,   With respect to the Error "java.lang.NoClassDefFoundError" seems to be a jar file missing.  Could you please make sure all the relevant jar files are placed in the relevant classpath? Are you running this pyspark program first time? Is it possible to share the command you are using to run the pyspark program?   Thanls AKR ... View more

Hi Wilson,   Exitcode 1 means that it is Failing to initialize the container localizer.  Could you please try uploading the application logs if it is available now, else Please share the Resource manager logs and Nodemanager logs to understand what happenned during the container creation.     Thanks AKR ... View more

Hi,    The Error means some Jar files is missing. Please make sure that the related hbase jars are added and try it again.   Thanks Arun ... View more

Hi,   With respect to checking the resources availability, Please check the Resource manager WebUI for available / used memory and vcores. Also in the scheduler page check whether the Queue has the enough resources available. Try submitting the job to some other queue and let us know the updates,   Please also share the RM logs / application logs to check for any errors.   Thanks AKR ... View more

Hi,   Does it happens for only one job (or) is it happens for all other jobs too? Could you please share the Entire application logs for our analysis purpose?    Thanks AKR ... View more

Hi,   Is this a new Cluster / Configuration setup? how do you identify it was taking 6 mins to load spark shell?   Thanks AKR ... View more