question The Spark session could not be created in the cluster for CDP 7 in Support Questions

The Spark session could not be created in the cluster for CDP 7

sss123 — Tue, 21 Apr 2026 07:55:37 GMT

I have set up Cloudera Manager (Trial) 7.4.4 in Ubuntu 20.04.4 LTS (Focal Fossa) as a single node cluster in AWS EC2.

All the services HDFS, Hue , Hive , Spark , Livy , Solr , ZooKeeper , Yarn are in good health and running.

I can login to Hue editor and open spark notebook. But I'm getting the issue as The Spark session could not be created in the cluster when trying to execute notebook.

The Spark session could not be created in the cluster: at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1898) at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:144) at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184) at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86) at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:922) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:931) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) 22/06/01 06:32:29 INFO util.ShutdownHookManager: Shutdown hook called 22/06/01 06:32:29 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-84563e48-b47b-4c3e-ab3d-4f25bb31a036 YARN Diagnostics:

Any help is appreciated.

Thanks,

Re: The Spark session could not be created in the cluster for CDP 7

jagadeesan — Mon, 27 Jun 2022 22:16:20 GMT

Hi @sss123, this seems to be a bug. Please refer to https://issues.cloudera.org/browse/LIVY-3. Kindly note that Spark Notebook is not currently supported. Also please review the discussion in https://github.com/cloudera/hue/issues/254

Re: The Spark session could not be created in the cluster for CDP 7

m5a — Mon, 28 Nov 2022 11:20:33 GMT

create a directory for the user in HDFS......

hadoop create a staging directory for storing job execution files.

Re: The Spark session could not be created in the cluster for CDP 7

AsimShaikh — Fri, 02 Dec 2022 17:44:50 GMT

@sss123 Are you able to run spark commands via spark-shell spark-submit?