For me it was solved after login using hdfs user. You can typically then run the spark-shell command without errors ... View more