Reply
Highlighted
New Contributor
Posts: 2
Registered: ‎09-20-2017

View resources allocated to particular application in Cloudera Manager

 

 

Is there a way to view the resources allocated to a particular yarn application while it is running?

 

The issue I am trying to determine is why my spark application decides to shut down the spark context.

 

I've greped the yarn logs for exceptions and I didn't find any memory related issues.

Posts: 642
Topics: 3
Kudos: 121
Solutions: 67
Registered: ‎08-16-2016

Re: View resources allocated to particular application in Cloudera Manager

The RM UI and the Spark History are what you are looking for.  The yarn command can also be used in a pinch.

New Contributor
Posts: 2
Registered: ‎09-20-2017

Re: View resources allocated to particular application in Cloudera Manager

Can you point me to particular documentation?

Ideally it would the same output at Ganglia where you can view I/O, memory, disk for each machine in use.
Posts: 642
Topics: 3
Kudos: 121
Solutions: 67
Registered: ‎08-16-2016

Re: View resources allocated to particular application in Cloudera Manager

https://spark.apache.org/docs/1.6.2/monitoring.html

This is old but still useful.

http://hadooptutorial.info/yarn-web-ui/

I know you can get the info from the RM UI. You should be able to get it from the Spark History UI through curl commands as well. The information could then be fed into Ganglia.