Bit late to reply, but if the cluster is secure, try pointing the hbase configuration to the spark driver and executor classpath explicitly using 'spark.executor.extraClassPath' and 'spark.driver.extraClassPath'. Also make sure that the host from where you are running the spark command has the gateway role added. Example: $ pyspark --jars /opt/cloudera/parcels/CDH/jars/spark-examples-1.6.0-cdh5.7.3-hadoop2.6.0-cdh5.7.3.jar,/opt/cloudera/parcels/CDH/jars/hbase-examples-1.2.0-cdh5.7.3.jar --conf "spark.executor.extraClassPath=/etc/hbase/conf/" --conf "spark.driver.extraClassPath=/etc/hbase/conf/"
... View more
