@Lakshu : Seems to be configuration issue, your code doesnt find the correct region as per the configration.   Go through hbase site xml files for the below configuration parameters, if not found add:   hbase.thrift.support.proxyuser --> true hbase.regionserver.thrift.http --> true   Add these configurations and restart the hbase and let me know how this works.  ... View more

Fixed:   This is what i infered, while running spark the mode is made as client as you see below:   Parsed arguments: master local[*] deployMode null executorMemory null executorCores null totalExecutorCores null propertiesFile /usr/hdp/current/spark2-client/conf/spark-defaults.conf driverMemory 4g driverCores null driverExtraClassPath null driverExtraLibraryPath /usr/hdp/current/hadoop-client/lib/native:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64 driverExtraJavaOptions null supervise false queue default numExecutors null files null pyFiles null archives null mainClass null primaryResource pyspark-shell name PySparkShell childArgs [] jars null packages null packagesExclusions null repositories null verbose true         When we use --master yarn this gets success !! . ... View more