Skip to main content
sss123
Bronze ⭐️⭐️⭐️
Jun 1, 2022 7:24 AM UTC
Question

The Spark session could not be created in the cluster for CDP 7

I have set up Cloudera Manager (Trial) 7.4.4  in Ubuntu 20.04.4 LTS (Focal Fossa) as a single node cluster in AWS EC2.

 

All the services HDFS, Hue , Hive , Spark , Livy , Solr , ZooKeeper , Yarn are in good health and running.

 

I can login to Hue editor and open spark notebook. But I'm getting the issue as The Spark session could not be created in the cluster when trying to execute notebook. 

sss123_2-1654067083258.png

 

I

The Spark session could not be created in the cluster: at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1898) at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:144) at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184) at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86) at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:922) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:931) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) 22/06/01 06:32:29 INFO util.ShutdownHookManager: Shutdown hook called 22/06/01 06:32:29 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-84563e48-b47b-4c3e-ab3d-4f25bb31a036 YARN Diagnostics:

 

 

Any help is appreciated.

Thanks,

3 replies

jagadeesan
Platinum ⭐️
Platinum ⭐️
Jun 27, 2022 10:16 PM UTC

Hi @sss123, this seems to be a bug. Please refer to https://issues.cloudera.org/browse/LIVY-3. Kindly note that Spark Notebook is not currently supported. Also please review the discussion in https://github.com/cloudera/hue/issues/254

Translate
m5a
Silver ⭐️
Nov 28, 2022 11:20 AM UTC

create a directory for the user in HDFS......

hadoop create a staging directory for storing job execution files.

Translate
AsimShaikh
Gold ⭐️
Gold ⭐️
Dec 2, 2022 5:44 PM UTC

@sss123  Are you able to run spark commands via spark-shell spark-submit? 

Translate