The comment on the setting in CM should have explained it for you:

Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx.

You can not set that in a configuration file since the JVM is started before the configuration file is read and you need to specify the heap size on startup. CM passes the value to the agent and the agent then creates the java cmd line. The only place they are stored is in CM.

Under normal circumstances in a small cluster, lets say 10 nodes, 2 GB should be enough. In an average size cluster, 50 nodes, a RM should not need more than 4GB. In a large cluster,hundred or more NM's, you need to increase that. For a NM you can normally leave that at 1GB on small nodes or 2-4GB on large nodes. The number of containers you can run on a node is the size of the node.

Wilfred