question Re: Pyspark yarn-client application lifetime in Support Questions

Pyspark yarn-client application lifetime

Jaaaaay — Wed, 11 Dec 2024 09:46:54 GMT

My pyspark yarn-client application got killed by cluster because of this setting, yarn.scheduler.capacity.root.default-application-lifetime. What config should I use to declare my application lifetime and avoid getting killed?

Re: Pyspark yarn-client application lifetime

DianaTorres — Wed, 11 Dec 2024 18:25:18 GMT

@Jaaaaay Welcome to the Cloudera Community!

To help you get the best possible solution, I have tagged our Spark experts @Bharati @jagadeesan who may be able to assist you further.

Please keep us updated on your post, and we hope you find a satisfactory solution to your query.

Re: Pyspark yarn-client application lifetime

Bharati — Thu, 12 Dec 2024 17:32:10 GMT

Admins can enforce application lifetime SLAs at a service level by setting yarn.scheduler.capacity.<queue-path>.maximum-application-lifetime ” and “yarn.scheduler.capacity.root.<queue-path>.default-application-lifetime” in
capacity-scheduler.xml.
CM > Yarn > Configuration > Capacity Scheduler Configuration Advanced Configuration Snippet (Safety Valve)

reference link is https://blog.cloudera.com/enforcing-application-lifetime-slas-yarn/

How are you setting this parameter? Can you share the entire stacktrace of the failure?

Re: Pyspark yarn-client application lifetime

Jaaaaay — Fri, 13 Dec 2024 02:25:52 GMT

Admin only set a value of 86400 for yarn.scheduler.capacity.root.<queue-path>.default-application-lifetime. Which should be a soft enforcement in a sense. Value for yarn.scheduler.capacity.<queue-path>.maximum-application-lifetime is not set. My question is how to configure my pyspark application to bypass this soft enforcement.

Re: Pyspark yarn-client application lifetime

Jaaaaay — Thu, 19 Dec 2024 01:29:33 GMT

@Bharati Any updates? I have tried the REST API method. Getting my app’s expiry time works just fine. But put request get a 401 unauthorized error. It’s a shared cluster and I don’t have the admin-level authorization.