Support Questions

Find answers, ask questions, and share your expertise
Check out our newest addition to the community, the Cloudera Data Analytics (CDA) group hub.

The Spark session could not be created in the cluster for CDP 7

New Contributor

I have set up Cloudera Manager (Trial) 7.4.4  in Ubuntu 20.04.4 LTS (Focal Fossa) as a single node cluster in AWS EC2.


All the services HDFS, Hue , Hive , Spark , Livy , Solr , ZooKeeper , Yarn are in good health and running.


I can login to Hue editor and open spark notebook. But I'm getting the issue as The Spark session could not be created in the cluster when trying to execute notebook. 




The Spark session could not be created in the cluster: at at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:144) at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184) at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86) at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:922) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:931) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) 22/06/01 06:32:29 INFO util.ShutdownHookManager: Shutdown hook called 22/06/01 06:32:29 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-84563e48-b47b-4c3e-ab3d-4f25bb31a036 YARN Diagnostics:



Any help is appreciated.



Expert Contributor

Hi @sss123, this seems to be a bug. Please refer to Kindly note that Spark Notebook is not currently supported. Also please review the discussion in

New Contributor

create a directory for the user in HDFS......

hadoop create a staging directory for storing job execution files.

Rising Star

@sss123  Are you able to run spark commands via spark-shell spark-submit? 

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.