Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

The Spark session could not be created in the cluster for CDP 7

avatar
New Contributor

I have set up Cloudera Manager (Trial) 7.4.4  in Ubuntu 20.04.4 LTS (Focal Fossa) as a single node cluster in AWS EC2.

 

All the services HDFS, Hue , Hive , Spark , Livy , Solr , ZooKeeper , Yarn are in good health and running.

 

I can login to Hue editor and open spark notebook. But I'm getting the issue as The Spark session could not be created in the cluster when trying to execute notebook. 

sss123_2-1654067083258.png

 

I

The Spark session could not be created in the cluster: at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1898) at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:144) at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184) at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86) at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:922) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:931) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) 22/06/01 06:32:29 INFO util.ShutdownHookManager: Shutdown hook called 22/06/01 06:32:29 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-84563e48-b47b-4c3e-ab3d-4f25bb31a036 YARN Diagnostics:

 

 

Any help is appreciated.

Thanks,

3 REPLIES 3

avatar
Master Collaborator

Hi @sss123, this seems to be a bug. Please refer to https://issues.cloudera.org/browse/LIVY-3. Kindly note that Spark Notebook is not currently supported. Also please review the discussion in https://github.com/cloudera/hue/issues/254

avatar
New Contributor

create a directory for the user in HDFS......

hadoop create a staging directory for storing job execution files.

avatar
Expert Contributor

@sss123  Are you able to run spark commands via spark-shell spark-submit?