I had a similar issue starting up pyspark shell, spark 1.6 and turned out that my program is writing log info to /user/spark/applhistorylogs and it does not have sufficient permissions to write to this path on hdfs. Changing permissions to 777 helped. Any idea why this issue popped up all of a sudden. I have been using same environment for last 2 months ... View more