My Spark cluster runs via YARN. End-user scripts connect via ssh to a grid node to submit jobs via spark-submit using a custom keytab that has been created for each user, to avoid the need to run kinit. Subsequently the scripts keep track of the status of the Yarn application by sending periodic "yarn application -status ..." commands. I'm looking for a way to use the same keytabs to invoke yarn.
Is there an equivalent for yarn of the --keytab switch in spark-submit? Or can I allow application statuses to be queried without requiring any authentitcation?
Alternatively, how can I obtain the same information programatically without using the Spark client, and without resorting to custom Java code, e.g. from a REST API?
PS - Note that since I'm running Spark via Yarn, spark-submit --status would not work.
The information I need is indeed availale via a web-service API, by sending HTTP GET to URLs like:
See: this article.