Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Catalogd OutOfMemoryError: Java heap space

Solved Go to solution

Catalogd OutOfMemoryError: Java heap space

Explorer

Dear Impala users,

 

We are seeing the following messages in our catalogd logs:

 

E0523 05:28:25.921648 15916 catalog-server.cc:279] OutOfMemoryError: Java heap space

 

Impala version: Impala Shell v2.2.0-cdh5 (2ffd73a) built on Tue Apr 21 12:09:21 PDT 2015

 

I have tried to increase catalogd memory via CM's catalogd safety valves:

 

"Catalog Server Command Line Argument Advanced Configuration Snippet (Safety Valve)" set to 

-rm_default_memory=8G

 

And through 

"Impala Catalog Server Environment Advanced Configuration Snippet (Safety Valve)" set to

JAVA_TOOLS_OPTIONS="-Xmx8GB"

 

I have verified through jmap that neither of the above settings have increased JVM max heap available to catalogd process.

 

I have also read on a post (https://groups.google.com/a/cloudera.org/forum/#!topic/impala-user/1QKAzAHVIO0) that catalogd supports jvm_args or jvm-args, however our version of impala/catalogd does not seem to support such an option, as below command produces no result:  

catalogd --help | grep jvm 

 

I was wondering if anyone has any suggestions or if anyone can provide insight into how to go about troubleshooting this issue.

 

Thank you in advance for your time.

 

 

 

 

 

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Catalogd OutOfMemoryError: Java heap space

Master Collaborator

Mystery solved, I had a typo as well :)

 

The option should be:

 

JAVA_TOOL_OPTIONS="-Xmx8g"

 

(TOOL instead of TOOLS)

7 REPLIES 7

Re: Catalogd OutOfMemoryError: Java heap space

Master Collaborator

Setting the JAVA_TOOL_OPTIONS like you did via the CM safety valve is the preferred way and should work.

Did you perhaps have a typo?

 

JAVA_TOOLS_OPTIONS="-Xmx8GB"

 

should be

 

JAVA_TOOLS_OPTIONS="-Xmx8g"

 

 

Highlighted

Re: Catalogd OutOfMemoryError: Java heap space

Explorer

Hey Alex,

 

thanks for your response. I just tried copy pasting your version of the line and restarting the catalog server. Jmap still shows 4gb max heap. Is it possible that my version of impala does not support "JAVA_TOOLS_OPTIONS" environment variable? (Impala Shell v2.2.0-cdh5 (2ffd73a) built on Tue Apr 21 12:09:21 PDT 2015 )

 

# /usr/java/jdk1.7.0_80/bin/jmap -heap 30062

Attaching to process ID 30062, please wait...

Debugger attached successfully.

Server compiler detected.

JVM version is 24.80-b11

 

using thread-local object allocation.

Parallel GC with 4 thread(s)

 

Heap Configuration:

   MinHeapFreeRatio = 0

   MaxHeapFreeRatio = 100

   MaxHeapSize      = 3917479936 (3736.0MB)

   NewSize          = 1310720 (1.25MB)

   MaxNewSize       = 17592186044415 MB

   OldSize          = 5439488 (5.1875MB)

   NewRatio         = 2

   SurvivorRatio    = 8

   PermSize         = 21757952 (20.75MB)

   MaxPermSize      = 85983232 (82.0MB)

   G1HeapRegionSize = 0 (0.0MB)

 

 

 

Re: Catalogd OutOfMemoryError: Java heap space

Explorer

i thought I can look through the catalogd startup script, it looks like the only JAVA option available there is JAVA_HOME:

 

# cat /usr/bin/catalogd | grep "JAVA"

# Autodetect JAVA_HOME if not defined

if [ -z "${JAVA_HOME}" ]; then

  echo "Unable to find Java. JAVA_HOME should be set in /etc/default/bigtop-utils"

    library_file=`find ${JAVA_HOME}/ -name $library | head -1`

 

 

Re: Catalogd OutOfMemoryError: Java heap space

Master Collaborator

Thanks for following up with more details. We're investigation and will get back to you as soon as we have more info.

Re: Catalogd OutOfMemoryError: Java heap space

Master Collaborator

Mystery solved, I had a typo as well :)

 

The option should be:

 

JAVA_TOOL_OPTIONS="-Xmx8g"

 

(TOOL instead of TOOLS)

Re: Catalogd OutOfMemoryError: Java heap space

Explorer

Adding JAVA_TOOL_OPTIONS=-Xmx8g to "Impala Catalog Server Environment Advanced Configuration Snippet (Safety Valve)" worked. Thanks!

 

Re: Catalogd OutOfMemoryError: Java heap space

New Contributor
iam using the cdh 5.11 and my catalog server is not starting even i keeping the Impala Catalog Server Environment Advanced Configuration Snippet (Safety Valve) as JAVA_TOOL_OPTIONS=-Xmx8g or JAVA_TOOL_OPTIONS=-Xss2m
Don't have an account?
Coming from Hortonworks? Activate your account here