CDE Spark UI fails to load

[#1]

Hi

I've got a CDE 1.22.0-h1-b3 cluster that fails to load the Spark UI after the job has completed. I can see the Spark UI during the running of a job but it fails to load/open once the job has completed.

I see an http GET to the /jobs endpoint fails with a 404 Not Found.

Have you come across this error and resolved it please?

Cheers

Craig

[#2]

@Craig-C Welcome to the Cloudera Community!

To help you get the best possible solution, I have tagged our CDE expert @Azhar_Shaikh  who may be able to assist you further.

Please keep us updated on your post, and we hope you find a satisfactory solution to your query.

[#3]

hi @Craig-C,

analyze the logs of the CDE pods, monitor the logs during the execution of the job and after the job ends, you will possibly find the problem.

[#4]

@Craig-C Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future. Thanks.

[#5]

Hello @Craig-C 

Greetings from Cloudera Community. Based on the Post, Spark UI is available for Running Jobs yet the Spark UI isn't available for Completed Jobs. 

Note that Spark UI for Running Jobs is Serviced via Driver yet the Spark UI for Completed Jobs is Serviced via Spark-History-Server. We have observed few Scenarios wherein Spark UI isn't available for Completed Jobs during High-Concurrency & have a Solution until a Long-Term Fix at CDE.

The Solution is a bit Verbose & as such, We shall recommend your Team to file a Support Case with Cloudera Support & We shall help your Team confirming the Issue reported above while applying the Mitigation Solution until a Long-Term Fix. 

Regards, Smarak