Reply
Explorer
Posts: 13
Registered: ‎05-23-2016
Accepted Solution

Catalogd OutOfMemoryError: Java heap space

Dear Impala users,

 

We are seeing the following messages in our catalogd logs:

 

E0523 05:28:25.921648 15916 catalog-server.cc:279] OutOfMemoryError: Java heap space

 

Impala version: Impala Shell v2.2.0-cdh5 (2ffd73a) built on Tue Apr 21 12:09:21 PDT 2015

 

I have tried to increase catalogd memory via CM's catalogd safety valves:

 

"Catalog Server Command Line Argument Advanced Configuration Snippet (Safety Valve)" set to 

-rm_default_memory=8G

 

And through 

"Impala Catalog Server Environment Advanced Configuration Snippet (Safety Valve)" set to

JAVA_TOOLS_OPTIONS="-Xmx8GB"

 

I have verified through jmap that neither of the above settings have increased JVM max heap available to catalogd process.

 

I have also read on a post (https://groups.google.com/a/cloudera.org/forum/#!topic/impala-user/1QKAzAHVIO0) that catalogd supports jvm_args or jvm-args, however our version of impala/catalogd does not seem to support such an option, as below command produces no result:  

catalogd --help | grep jvm 

 

I was wondering if anyone has any suggestions or if anyone can provide insight into how to go about troubleshooting this issue.

 

Thank you in advance for your time.

 

 

 

 

 

Cloudera Employee
Posts: 266
Registered: ‎10-16-2013

Re: Catalogd OutOfMemoryError: Java heap space

Setting the JAVA_TOOL_OPTIONS like you did via the CM safety valve is the preferred way and should work.

Did you perhaps have a typo?

 

JAVA_TOOLS_OPTIONS="-Xmx8GB"

 

should be

 

JAVA_TOOLS_OPTIONS="-Xmx8g"

 

 

Explorer
Posts: 13
Registered: ‎05-23-2016

Re: Catalogd OutOfMemoryError: Java heap space

Hey Alex,

 

thanks for your response. I just tried copy pasting your version of the line and restarting the catalog server. Jmap still shows 4gb max heap. Is it possible that my version of impala does not support "JAVA_TOOLS_OPTIONS" environment variable? (Impala Shell v2.2.0-cdh5 (2ffd73a) built on Tue Apr 21 12:09:21 PDT 2015 )

 

# /usr/java/jdk1.7.0_80/bin/jmap -heap 30062

Attaching to process ID 30062, please wait...

Debugger attached successfully.

Server compiler detected.

JVM version is 24.80-b11

 

using thread-local object allocation.

Parallel GC with 4 thread(s)

 

Heap Configuration:

   MinHeapFreeRatio = 0

   MaxHeapFreeRatio = 100

   MaxHeapSize      = 3917479936 (3736.0MB)

   NewSize          = 1310720 (1.25MB)

   MaxNewSize       = 17592186044415 MB

   OldSize          = 5439488 (5.1875MB)

   NewRatio         = 2

   SurvivorRatio    = 8

   PermSize         = 21757952 (20.75MB)

   MaxPermSize      = 85983232 (82.0MB)

   G1HeapRegionSize = 0 (0.0MB)

 

 

 

Explorer
Posts: 13
Registered: ‎05-23-2016

Re: Catalogd OutOfMemoryError: Java heap space

i thought I can look through the catalogd startup script, it looks like the only JAVA option available there is JAVA_HOME:

 

# cat /usr/bin/catalogd | grep "JAVA"

# Autodetect JAVA_HOME if not defined

if [ -z "${JAVA_HOME}" ]; then

  echo "Unable to find Java. JAVA_HOME should be set in /etc/default/bigtop-utils"

    library_file=`find ${JAVA_HOME}/ -name $library | head -1`

 

 

Cloudera Employee
Posts: 266
Registered: ‎10-16-2013

Re: Catalogd OutOfMemoryError: Java heap space

Thanks for following up with more details. We're investigation and will get back to you as soon as we have more info.

Cloudera Employee
Posts: 266
Registered: ‎10-16-2013

Re: Catalogd OutOfMemoryError: Java heap space

Mystery solved, I had a typo as well :)

 

The option should be:

 

JAVA_TOOL_OPTIONS="-Xmx8g"

 

(TOOL instead of TOOLS)

Explorer
Posts: 13
Registered: ‎05-23-2016

Re: Catalogd OutOfMemoryError: Java heap space

Adding JAVA_TOOL_OPTIONS=-Xmx8g to "Impala Catalog Server Environment Advanced Configuration Snippet (Safety Valve)" worked. Thanks!

 

New Contributor
Posts: 5
Registered: ‎07-03-2017

Re: Catalogd OutOfMemoryError: Java heap space

iam using the cdh 5.11 and my catalog server is not starting even i keeping the Impala Catalog Server Environment Advanced Configuration Snippet (Safety Valve) as JAVA_TOOL_OPTIONS=-Xmx8g or JAVA_TOOL_OPTIONS=-Xss2m
Announcements