question Re: Map and Reduce Error: Java heap space in Archives of Support Questions (Read Only)

Map and Reduce Error: Java heap space

Triffids — Fri, 16 Sep 2022 10:43:01 GMT

I'm using QuickStart VM with CHD5.3, trying to run modified sample from MR-parquet read. It is worked OK on 10M rows parquet table, but I've got "Java heap space" error on table having 40M rows:

[cloudera@quickstart sep]$ yarn jar testmr-1.0-SNAPSHOT.jar TestReadParquet /user/hive/warehouse/parquet_table out_file18 -Dmapreduce.reduce.memory.mb=5120 -Dmapreduce.reduce.java.opts=-Xmx4608m -Dmapreduce.map.memory.mb=5120 -Dmapreduce.map.java.opts=-Xmx4608m
16/10/03 12:19:30 INFO client.RMProxy: Connecting to ResourceManager at quickstart.cloudera/127.0.0.1:8032
16/10/03 12:19:31 INFO input.FileInputFormat: Total input paths to process : 1
Oct 03, 2016 12:19:31 PM parquet.Log info
INFO: Total input paths to process : 1
Oct 03, 2016 12:19:31 PM parquet.Log info
INFO: Initiating action with parallelism: 5
Oct 03, 2016 12:19:31 PM parquet.Log info
INFO: reading another 1 footers
Oct 03, 2016 12:19:31 PM parquet.Log info
INFO: Initiating action with parallelism: 5
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
16/10/03 12:19:31 INFO Configuration.deprecation: mapred.max.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.maxsize
16/10/03 12:19:31 INFO Configuration.deprecation: mapred.min.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize
Oct 03, 2016 12:19:31 PM parquet.Log info
INFO: There were no row groups that could be dropped due to filter predicates
16/10/03 12:19:32 INFO mapreduce.JobSubmitter: number of splits:1
16/10/03 12:19:32 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1475517800829_0009
16/10/03 12:19:33 INFO impl.YarnClientImpl: Submitted application application_1475517800829_0009
16/10/03 12:19:33 INFO mapreduce.Job: The url to track the job: http://quickstart.cloudera:8088/proxy/application_1475517800829_0009/
16/10/03 12:19:33 INFO mapreduce.Job: Running job: job_1475517800829_0009
16/10/03 12:19:47 INFO mapreduce.Job: Job job_1475517800829_0009 running in uber mode : false
16/10/03 12:19:47 INFO mapreduce.Job: map 0% reduce 0%
16/10/03 12:20:57 INFO mapreduce.Job: map 100% reduce 0%
16/10/03 12:20:57 INFO mapreduce.Job: Task Id : attempt_1475517800829_0009_m_000000_0, Status : FAILED
Error: Java heap space
Container killed by the ApplicationMaster.
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143

Also I've tryed to edit /etc/hadoop/conf/mapred-site.xml, tryed via cloudera manager GUI (clusters->hdfs-> ... Java Heap Size of DataNode in Bytes )

[cloudera@quickstart sep]$ free -m
total used free shared buffers cached
Mem: 13598 13150 447 0 23 206
-/+ buffers/cache: 12920 677
Swap: 6015 2187 3828

Mapper class:

public static class MyMap extends
        Mapper<LongWritable, Group, NullWritable, Text> {

    @Override
    public void map(LongWritable key, Group value, Context context) throws IOException, InterruptedException {
        NullWritable outKey = NullWritable.get();
        String outputRecord = "";
        // Get the schema and field values of the record
        // String inputRecord = value.toString();
        // Process the value, create an output record
        // ...
        int field1 = value.getInteger("x", 0);

        if (field1 < 3) {
            context.write(outKey, new Text(outputRecord));
        }
    }
}

Re: Map and Reduce Error: Java heap space

Triffids — Thu, 06 Oct 2016 11:59:47 GMT

hadoop-cmf-yarn-NODEMANAGER-quickstart.cloudera.log.out:

2016-10-03 12:22:14,533 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 18309 for container-id container_1475517800829_0009_01_000005: 130.2 MB of 3 GB physical memory used; 859.9 MB of 6.3 GB virtual memory used
2016-10-03 12:22:28,045 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 16676 for container-id container_1475517800829_0009_01_000001: 178.8 MB of 1 GB physical memory used; 931.1 MB of 2.1 GB virtual memory used
2016-10-03 12:22:31,303 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 18309 for container-id container_1475517800829_0009_01_000005: 128.8 MB of 3 GB physical memory used; 859.9 MB of 6.3 GB virtual memory used
2016-10-03 12:22:46,965 WARN org.apache.hadoop.yarn.util.ProcfsBasedProcessTree: Error reading the stream java.io.IOException: No such process
2016-10-03 12:22:46,966 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 16676 for container-id container_1475517800829_0009_01_000001: 179.0 MB of 1 GB physical memory used; 931.1 MB of 2.1 GB virtual memory used
2016-10-03 12:22:47,122 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Stopping container with container Id: container_1475517800829_0009_01_000005

Re: Map and Reduce Error: Java heap space

csguna — Thu, 06 Oct 2016 12:10:25 GMT

Please add some more memory by editing the mapred-site.xml

<property>
    <name>mapred.child.java.opts</name>
    <value>-Xmx4096m</value>
</property>

The above tag i have used 5gb.

Let me know if that helped you

alternatively you can also edit the hadoop-env.sh file

add

export HADOOP_OPTS="-Xmx5096m"

Re: Map and Reduce Error: Java heap space

Triffids — Fri, 07 Oct 2016 06:05:54 GMT

Thanks ! mapred.child.java.opts in mapred-site.xml solved the issue

Re: Map and Reduce Error: Java heap space

csguna — Fri, 07 Oct 2016 06:37:24 GMT

Sounds good mate

Re: Map and Reduce Error: Java heap space

saranvisa — Fri, 04 Nov 2016 01:20:50 GMT

If you need more details, pls refer below

mapred.map.child.java.opts is for Hadoop 1.x

Those who are using Hadoop 2.x, pls use the below parameters instead

mapreduce.map.java.opts=-Xmx4g # Note: 4 GB

mapreduce.reduce.java.opts=-Xmx4g # Note: 4 GB

Also when you set java.opts, you need to note two important points

1. It has dependency on memory.mb, so always try to set java.opts upto 80% of memory.mb

2. Follow the "-Xmx4g" format for opt but numerical value for memory.mb

mapreduce.map.memory.mb = 5012 # Note: 5 GB

mapreduce.reduce.memory.mb = 5012 # Note: 5 GB

Finally, some organization will not allow you to alter mapred-site.xml directly or via CM. Also we need thease kind of setup only to handle very big tables, so it is not recommanded to alter the configuration only for few tables..so you can do this setup temporarly by following below steps:

1. From HDFS:

HDFS> export HIVE_OPTS="-hiveconf mapreduce.map.memory.mb=5120 -hiveconf mapreduce.reduce.memory.mb=5120 -hiveconf mapreduce.map.java.opts=-Xmx4g -hiveconf mapreduce.reduce.java.opts=-Xmx4g"

2. From Hive:

hive> set mapreduce.map.memory.mb=5120;

hive> set mapreduce.reduce.memory.mb=5120;

hive> set mapreduce.map.java.opts=-Xmx4g;

hive> set mapreduce.reduce.java.opts=-Xmx4g;

Note: HIVE_OPTS is to handle only HIVE, if you need similar setup for HADOOP then use HADOOP_OPTS

Thanks

Kumar

Re: Map and Reduce Error: Java heap space

onsbt — Wed, 19 Apr 2017 13:01:54 GMT

i would like to know the location of this file , because i found many files of mapred-site;thx again

Re: Map and Reduce Error: Java heap space

saranvisa — Wed, 19 Apr 2017 13:51:34 GMT

@onsbt

In general, the path is /etc/hadoop/conf

But I would recommend you to not update this file directly, instead update via Cloudera manager -> Yarn -> Configuration. if you are not using CM ask your admin

Also an another recommendation is, you can set those values 'temporarily' & directly in HDFS/Hive and test to find the suitable value for your environment before you make the permanent change in configuration file

Re: Map and Reduce Error: Java heap space

onsbt — Wed, 19 Apr 2017 13:59:03 GMT

thanks for replying , actually i want to decrase the size of heap mmemory of HDFS and kafka do you have any propositions ?i mofied the /opt/cloudera/parcels/KAFKA-2.1.1-1.2.1.1.p0.18/lib/kafka/bin/kafka-run-class.sh file but this does'nt give me any result

any help plz ?

Re: Map and Reduce Error: Java heap space

saranvisa — Wed, 19 Apr 2017 14:02:26 GMT

@onsbt

In general a service restart required after any configuration change

Again as I mentioned, it is recommended to update any configuration change via CM

Re: Map and Reduce Error: Java heap space

onsbt — Thu, 20 Apr 2017 09:41:34 GMT

thanks for replying ,the problem was solved i would like to ask another question , because after research i didn't found the solution , so after installing cloudera manager i got a problem with HDFS "hdfs: Problèmes d'état d'intégrité
HDFS
Blocs sous-répliqués " do you have an idea about the solution ?

Re: Map and Reduce Error: Java heap space

saranvisa — Thu, 20 Apr 2017 13:47:47 GMT

@onsbt

Can you translate your issue in english? also if it is not related to Java heap space, I would recommend you to create a new thread instead so that it is easy to track and others to contribute as well

Re: Map and Reduce Error: Java heap space

onsbt — Fri, 21 Apr 2017 07:29:01 GMT

thanks for replying

i updated this file directly, instead via Cloudera manager, and i resolve my problem now 🙂 thank you so much , but i have another question i Iam running cloudera with default configuration with one-node cluster, and would like to find where HDFS stores files locally.i create a file in hdfs with hue but when i see /dfs/nn it's empty i can't find the file that i have already created

Re: Map and Reduce Error: Java heap space

saranvisa — Fri, 21 Apr 2017 14:14:04 GMT

@onsbt

The default path is /opt/hadoop/dfs/nn

You can confirm this by Cloudera manager -> HDFS -> Configuration -> search for "dfs.namenode.name.dir"

Re: Map and Reduce Error: Java heap space

onsbt — Fri, 21 Apr 2017 14:24:53 GMT

the path /opt/hadoop/dfs/nn does not exist ,
and when i look for the file that i already created i can't find it in the path

Re: Map and Reduce Error: Java heap space

saranvisa — Fri, 21 Apr 2017 14:28:24 GMT

@onsbt

As mentioned already, please create a new topic for new issue as it may mislead others

Also please check the full answer and reply, so that you will get desired answer

Re: Map and Reduce Error: Java heap space

desind — Tue, 26 Sep 2017 13:42:45 GMT

@saranvisa

The last reducer of my mapreduce job fails with the below error.

2017-09-20 16:23:23,732 FATAL [main] org.apache.hadoop.mapred.YarnChild: Error running child : java.lang.OutOfMemoryError: GC overhead limit exceeded
	at java.util.regex.Matcher.<init>(Matcher.java:224)
	at java.util.regex.Pattern.matcher(Pattern.java:1088)
	at java.lang.String.replaceAll(String.java:2162)
	at com.sas.ci.acs.extract.CXAService$myReduce.parseEvent(CXAService.java:1612)
	at com.sas.ci.acs.extract.CXAService$myReduce.reduce(CXAService.java:919)
	at com.sas.ci.acs.extract.CXAService$myReduce.reduce(CXAService.java:237)
	at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:444)
	at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392)
	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)

2017-09-20 16:23:23,834 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping ReduceTask metrics system...
2017-09-20 16:23:23,834 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: ReduceTask metrics system stopped.
2017-09-20 16:23:23,834 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: ReduceTask metrics system shutdown complete.

Current settings:

mapreduce.map.java.opts

-Djava.net.preferIPv4Stack=true -Xmx3865051136

mapreduce.reduce.java.opts

-Djava.net.preferIPv4Stack=true -Xmx6144067296

1) do you recommend increasing the following properties to the below values ?

"mapreduce.map.java.opts","-Xmx4g"
"mapreduce.reduce.java.opts","-Xmx8g"

2) These are my map and reduce memory current settings. Do i also need to bump up my reduce memory to 10240m ?

mapreduce.reduce.memory.mb 8192
mapreduce.reduce.memory.mb 8192

Re: Map and Reduce Error: Java heap space

saranvisa — Tue, 26 Sep 2017 19:07:15 GMT

@desind

I will not recommend to change your settings, instead you can pass the memory & java Opt when you execute your Jar.

Ex: Below are some sample value, you can change it as needed.

hadoop jar ${JAR_PATH} ${CONFIG_PATH}/filename.xml ${ENV} ${ODATE} mapMem=12288 mapJavaOpts=Xmx9830 redurMem=12288 redurJavaOpts=Xmx9830

Note:

mapJavaopts = mapMem * 0.8

redurJavaOpts = redurMem * 0.8

Re: Map and Reduce Error: Java heap space

desind — Thu, 05 Oct 2017 19:31:31 GMT

@saranvisa

What are the implications for increasing mapreduce/reduce.memory.mb and mapreduce.reduce.java.opts to a higher value in the cluster itself ?

One of them would be that jobs that do not need this additional memory will get it. which is of no use

Other jobs during that time may be impacted

Anything else ?

Re: Map and Reduce Error: Java heap space

saranvisa — Fri, 06 Oct 2017 16:11:21 GMT

@desind

To add on to your point, the cluster setup is applicable to all the mapreduce job, so it may impact other non-mapreduce jobs.

In fact I am not against setup higher value in cluster itself, but you can do that based on how many jobs requires higher values and performance, etc