Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Query status of yarn application using a custom Kerberos keytab

Query status of yarn application using a custom Kerberos keytab

New Contributor

My Spark cluster runs via YARN. End-user scripts connect via ssh to a grid node to submit jobs via spark-submit using a custom keytab that has been created for each user, to avoid the need to run kinit. Subsequently the scripts keep track of the status of the Yarn application by sending periodic "yarn application -status ..." commands. I'm looking for a way to use the same keytabs to invoke yarn.
Is there an equivalent for yarn of the --keytab switch in spark-submit? Or can I allow application statuses to be queried without requiring any authentitcation?
Alternatively, how can I obtain the same information programatically without using the Spark client, and without resorting to custom Java code, e.g. from a REST API?
PS - Note that since I'm running Spark via Yarn, spark-submit --status would not work.

1 REPLY 1

Re: Query status of yarn application using a custom Kerberos keytab

New Contributor

The information I need is indeed availale via a web-service API, by sending HTTP GET to URLs like:
"http://<<resource-manager>>:8088/ws/v1/cluster/apps/application_1460195242962_0054"
See: this article.

Don't have an account?
Coming from Hortonworks? Activate your account here