The Spark session could not be created in the cluster for CDP 7

[#1]

I have set up Cloudera Manager (Trial) 7.4.4  in Ubuntu 20.04.4 LTS (Focal Fossa) as a single node cluster in AWS EC2.

All the services HDFS, Hue , Hive , Spark , Livy , Solr , ZooKeeper , Yarn are in good health and running.

I can login to Hue editor and open spark notebook. But I'm getting the issue as The Spark session could not be created in the cluster when trying to execute notebook. 

I

The Spark session could not be created in the cluster: at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1898) at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:144) at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184) at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86) at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:922) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:931) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) 22/06/01 06:32:29 INFO util.ShutdownHookManager: Shutdown hook called 22/06/01 06:32:29 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-84563e48-b47b-4c3e-ab3d-4f25bb31a036 YARN Diagnostics:

Any help is appreciated.

Thanks,

[#2]

Hi @sss123, this seems to be a bug. Please refer to https://issues.cloudera.org/browse/LIVY-3. Kindly note that Spark Notebook is not currently supported. Also please review the discussion in https://github.com/cloudera/hue/issues/254

[#3]

create a directory for the user in HDFS......

hadoop create a staging directory for storing job execution files.

[#4]

@sss123  Are you able to run spark commands via spark-shell spark-submit?