Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

unable to access spark web

unable to access spark web

Explorer

Dear Community

 

I have installed a cluster with spark server and spark client nodes

When trying to access the spark server webi UI

 

 
tory:
 hdfs:///spark2-history/

Last updated: 2019-11-21 12:50:57

Client local time zone: Europe/Berlin

No completed applications found!

Did you specify the correct logging directory? Please verify your setting of spark.history.fs.logDirectory listed above and whether you have the permissions to access it.
It is also possible that your application did not run to completion or did not stop the SparkContext.

Show incomplete applications
What should I do please?
 
I have installed the Spark Server on the same ambari node , should this cause a problem ?
 
Thanks and Regards
Asma
1 REPLY 1
Highlighted

Re: unable to access spark web

Super Mentor

@asmarz

One of the reason might be that the "hdfs://spark2-history" directory size has grown too much due to some old applications data might not be cleared.

# su - hdfs -c "hdfs dfs -du -s -h  hdfs:///spark2-history/"
# su - hdfs -c "hdfs dfs -ls  hdfs:///spark2-history/" | wc -l
# su - hdfs -c "hdfs dfs -ls  hdfs:///spark2-history/" 

 .

Do you see too many inprogress logs?

# su - hdfs -c "hdfs dfs -ls  hdfs:///spark2-history/"  | grep 'inprogress'


Do you see any repeated warning / errors in spark logs?

What is the value set for the following properties set in your spark configs?

spark.history.fs.cleaner.enabled
spark.history.fs.cleaner.interval
spark.history.fs.cleaner.maxAge

Fro more details on these properties please refer to spark documentation: https://spark.apache.org/docs/latest/monitoring.html

.

 

Don't have an account?
Coming from Hortonworks? Activate your account here