Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded

sandyws — Fri, 16 Sep 2022 09:25:00 GMT

I have around 3 lacs files stored in directory on hdfs. When i tried to access that folder using following command

hadoop fs -copyToLocal /user/docsearch/data/DiscardedAttachments /opt/

I am getting following issue :

Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.lang.AbstractStringBuilder.<init>(AbstractStringBuilder.java:64)
at java.lang.StringBuffer.<init>(StringBuffer.java:108)
at java.net.URI.decode(URI.java:2756)
at java.net.URI.getPath(URI.java:1318)
at org.apache.hadoop.fs.Path.isUriPathAbsolute(Path.java:210)
at org.apache.hadoop.fs.Path.isAbsolute(Path.java:223)
at org.apache.hadoop.fs.Path.makeQualified(Path.java:335)
at org.apache.hadoop.hdfs.DistributedFileSystem.makeQualified(DistributedFileSystem.java:373)
at org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:445)
at org.apache.hadoop.fs.shell.PathData.getDirectoryContents(PathData.java:213)
at org.apache.hadoop.fs.shell.Command.recursePath(Command.java:337)
at org.apache.hadoop.fs.shell.CommandWithDestination.recursePath(CommandWithDestination.java:193)
at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:308)
at org.apache.hadoop.fs.shell.Command.processPathArgument(Command.java:278)
at org.apache.hadoop.fs.shell.CommandWithDestination.processPathArgument(CommandWithDestination.java:147)
at org.apache.hadoop.fs.shell.Command.processArgument(Command.java:260)
at org.apache.hadoop.fs.shell.Command.processArguments(Command.java:244)
at org.apache.hadoop.fs.shell.CommandWithDestination.processArguments(CommandWithDestination.java:124)
at org.apache.hadoop.fs.shell.Command.processRawArguments(Command.java:190)
at org.apache.hadoop.fs.shell.Command.run(Command.java:154)
at org.apache.hadoop.fs.FsShell.run(FsShell.java:254)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at org.apache.hadoop.fs.FsShell.main(FsShell.java:304)

I have also did some changes in heap size of hdfs daemons, assigned 4GB to namenode and all datanodes. Still facing same issue. Need urgent help

Re: Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded

Harsh J — Wed, 24 Jun 2015 17:25:46 GMT

You need to raise the client heap size. For a one-off change, you can do the below:

~> export HADOOP_CLIENT_OPTS="$HADOOP_CLIENT_OPTS -Xmx5g"
~> hadoop fs -copyToLocal /user/docsearch/data/DiscardedAttachments /opt/

For a more permanent change, locate Gateway Client Java Heap configs in the relevant service (HDFS, YARN or Hive) in CM, raise the value and redeploy cluster-wide configs [1].

[1] - https://www.youtube.com/watch?v=4S9H3wftM_0

Re: Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded

htailor — Thu, 16 Jul 2015 08:29:23 GMT

That fixed it for me! Thanks Harsh

Re: Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded

Harsh J — Mon, 27 Jul 2015 10:25:38 GMT

Glad to hear; Please consider marking the topic as resolved so others with similar issues can find it easily.

question Re: Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded in Archives of Support Questions (Read Only)

Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded

Re: Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded

Re: Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded

Re: Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded