question Re: Please help: JAVA HEAP settings for HDFS components lost effects after YARN is enabled. in Archives of Support Questions (Read Only)

Please help: JAVA HEAP settings for HDFS components lost effects after YARN is enabled.

chenggl — Fri, 16 Sep 2022 09:38:35 GMT

Before adding YARN services, my JAVA HEAP tunning for datanode, secondarynamenode, and namenode (through cloudera manager WEB UI) are working perfectly.

I can see the processes with -Xmx set to my specified value. But after YARN is turned on and cluster is restarted, HDFS node processes' -Xmx value is turned back to 1GB again

I checked the cloudera manager Web GUI again and again, and the parameter are still my specified value, say, for namenode it is set at 6GB, but why the when cloudera bring up namenode process, the process comes up with '-Xmx 1GiB', I got completely confused as I am a newbie. Any one see similar issue?

I am running Cloudera 5.2.1. Thanks a lot.

Re: Please help: JAVA HEAP settings for HDFS components lost effects after YARN is enabled.

chenggl — Fri, 21 Aug 2015 05:08:26 GMT

Figured it out. Thanks.

Re: Please help: JAVA HEAP settings for HDFS components lost effects after YARN is enabled.

cjervis — Fri, 21 Aug 2015 12:55:39 GMT

I am happy to see you solved your issue chenggl. Do you mind sharing how you corrected it in case someone else encounters a similar issue? 🙂

Re: Please help: JAVA HEAP settings for HDFS components lost effects after YARN is enabled.

chenggl — Mon, 24 Aug 2015 07:46:59 GMT

The problem is in fact, a confusion instead of a problem.

HDFS components, say, DataNode, NameNode, etc, runs Java with two -Xmx options. The first one is Hadoop Default, 1GiB,

and the second one is user's custom settings, say 3GiB.

When I list the java processes with ps, it has been so difficult to see the second -Xmx option b/c it is buried deep inside the long

Java commands, while the first -Xmx is very easy to browse as it is on the first line (of ps output).

I reported the problem and then realized that there was a second -Xmx, and it was supposed to overwrite the first one according to Java

options specifications.

In Short, it is not a problem, just a confusion to Cloudera Hadoop newbies. Thanks.

Re: Please help: JAVA HEAP settings for HDFS components lost effects after YARN is enabled.

cjervis — Mon, 24 Aug 2015 13:13:30 GMT

Some of the most usefull solutions result from newbie confusion. Yours may help others more than you imagine. Please mark it as the solution so that others may find it easier.