Fixed:   This is what i infered, while running spark the mode is made as client as you see below:   Parsed arguments: master local[*] deployMode null executorMemory null executorCores null totalExecutorCores null propertiesFile /usr/hdp/current/spark2-client/conf/spark-defaults.conf driverMemory 4g driverCores null driverExtraClassPath null driverExtraLibraryPath /usr/hdp/current/hadoop-client/lib/native:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64 driverExtraJavaOptions null supervise false queue default numExecutors null files null pyFiles null archives null mainClass null primaryResource pyspark-shell name PySparkShell childArgs [] jars null packages null packagesExclusions null repositories null verbose true         When we use --master yarn this gets success !! . ... View more

The following values are used based on ZooKeeper settings:   minSessionTimeout = ticktime * 2,  maxSessionTimeout=ticktime * 20   If no Session Timeouts are set in the client, the server will use tickTime*2. Hence, any session which establishes to ZooKeeper, will have this value as a minimum timeout.  Likewise, if there are no SessionTimout set, the maximum value will be tickTime*20. Hence, any session which establishes to ZooKeeper will have this value as a maximum timeout.  The sessionTimeout cannot be less than minSessionTimeout (ticktime * 2) or greater than maxSessionTimeout (ticktime * 20).  ... View more