question Re: Are the YARN Job History Server and Spark History Server required for the availability of a HA Cluster? in Support Questions

Are the YARN Job History Server and Spark History Server required for the availability of a HA Cluster?

MrPoseido — Fri, 28 Jul 2023 08:10:44 GMT

Hello everyone,

we are currently working on automaticlly patching the operating system of our CDP Private Cloud Base cluster. We enabled/configured our cluster to be HA, but for two components (YARN Job History Server and Spark History Server) we couldn't find any solution. We designed our automation to be split over the timespan of three days. Starting at 8 PM each day we update one of the master nodes, one of the edge nodes (e.g. know) and one third of the worker nodes in sequential order. Our problem is how to handle an unexpected failure. Is it possible to wait until the next day or is the availabilty in danger if one server (with the mentioned compontens) isn't booting up etc.

How important are the YARN Job History Server and Spark History Server for the successful execution. As far as I understand both are only required for storing/providing logs for executed applications. Do I miss anything here?

Re: Are the YARN Job History Server and Spark History Server required for the availability of a HA Cluster?

VidyaSargur — Fri, 28 Jul 2023 08:39:44 GMT

@MrPoseido, Welcome to our community! To help you get the best possible answer, I have tagged in our Yarn/Spark experts @Babasaheb @RangaReddy who may be able to assist you further.

Please feel free to provide any additional information or details about your query, and we hope that you will find a satisfactory solution to your question.

Re: Are the YARN Job History Server and Spark History Server required for the availability of a HA Cluster?

RangaReddy — Fri, 28 Jul 2023 09:12:19 GMT

Hi @MrPoseido

By using SHS, just we can the see the list of applications and its status. If SHS is down still you can copy the files to another cluster where SHS service is running and you can visualize.

Cloudera team is working on new feature to enable Spark History Server HA feature in future releases.

Re: Are the YARN Job History Server and Spark History Server required for the availability of a HA Cluster?

MrPoseido — Mon, 31 Jul 2023 07:24:32 GMT

Hi @RangaReddy

Thanks for your information about the SHS HA feature in future release. Let me rephrase your answer SHS is not required for a sucessful execution of any spark application. What I am missing now is the information if this states also for Yarn JHS or not?

Re: Are the YARN Job History Server and Spark History Server required for the availability of a HA Cluster?

RangaReddy — Mon, 31 Jul 2023 12:16:45 GMT

Hi @MrPoseido

By enabling the YARN HA you can access the Job History Server any time simply you need to assign to the new host.

* https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/ResourceManagerHA.html

* https://docs.cloudera.com/cdp-private-cloud-base/7.1.6/yarn-high-availability/topics/yarn-resourcemanager-ha-overview.html

Re: Are the YARN Job History Server and Spark History Server required for the availability of a HA Cluster?

VidyaSargur — Wed, 02 Aug 2023 06:43:41 GMT

@MrPoseido, Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future.