Working through the tutorial, got this error when i ran the spark-shell comming in the tutorial:
spark-shell --master yarn-client
16/08/15 17:40:10 ERROR spark.SparkContext: Error initializing SparkContext.
org.apache.hadoop.security.AccessControlException: Permission denied: user=cloudera, access=WRITE, inode="/user/spark/applicationHistory":spark:supergroup:drwxr-xr-x
stack trace ....
Ending with these:
<console>:16: error: not found: value sqlContext
import sqlContext.implicits._
<console>:16: error: not found: value sqlContext
import sqlContext.sql
scala>
Fixed it by giving the cloudera access to the hadoop filesystem folder with this command:
sudo -u hdfs hadoop fs -chmod 777 /user/spark/applicationHistory
Then the rest of the exercise worked.
Joe
