@Abizer A There is no documents for Resource Manager Heap. RM store some application states to render the UI which is controlled by yarn.resourcemanager.max-completed-applications, the default value for this is 10000, so at any time RM need somewhere ~1G memory to store these applications in memory.
You can set this around 4GB in your cluster which should be enough to store the Job status.
@Tamil Selvan K Its obviously not enough hence the question, Cluster in question RSS stays around 24 GB (Xmx) and bumps up to 26 GB once in a while and then down . Perhaps i might need to take heap dump and investigate whats stored in the heap and figure out if some leak is causing this behaviour .
What i was hoping to know is from someone who is running 1000 node busy production cluster whats Xmx set and recommendations on this if any.