You will have problems starting spark-shell with any master because there are permissioning issues with the hdfs folders that spark uses.   To fix it do the following:   sudo -u hdfs hadoop fs -chmod 777 /user/spark sudo -u spark hadoop fs -chmod 777 /user/spark/applicationHistory   That should do it 🙂 ... View more