Catalogd OutOfMemoryError: Java heap space

[#1]

Dear Impala users,

We are seeing the following messages in our catalogd logs:

E0523 05:28:25.921648 15916 catalog-server.cc:279] OutOfMemoryError: Java heap space

Impala version: Impala Shell v2.2.0-cdh5 (2ffd73a) built on Tue Apr 21 12:09:21 PDT 2015

I have tried to increase catalogd memory via CM's catalogd safety valves:

"Catalog Server Command Line Argument Advanced Configuration Snippet (Safety Valve)" set to 

-rm_default_memory=8G

And through 

"Impala Catalog Server Environment Advanced Configuration Snippet (Safety Valve)" set to

JAVA_TOOLS_OPTIONS="-Xmx8GB"

I have verified through jmap that neither of the above settings have increased JVM max heap available to catalogd process.

I have also read on a post (https://groups.google.com/a/cloudera.org/forum/#!topic/impala-user/1QKAzAHVIO0) that catalogd supports jvm_args or jvm-args, however our version of impala/catalogd does not seem to support such an option, as below command produces no result:  

catalogd --help | grep jvm 

I was wondering if anyone has any suggestions or if anyone can provide insight into how to go about troubleshooting this issue.

Thank you in advance for your time.

[#2]

Setting the JAVA_TOOL_OPTIONS like you did via the CM safety valve is the preferred way and should work.

Did you perhaps have a typo?

JAVA_TOOLS_OPTIONS="-Xmx8GB"

should be

JAVA_TOOLS_OPTIONS="-Xmx8g"

[#3]

Hey Alex,

thanks for your response. I just tried copy pasting your version of the line and restarting the catalog server. Jmap still shows 4gb max heap. Is it possible that my version of impala does not support "JAVA_TOOLS_OPTIONS" environment variable? (Impala Shell v2.2.0-cdh5 (2ffd73a) built on Tue Apr 21 12:09:21 PDT 2015 )

# /usr/java/jdk1.7.0_80/bin/jmap -heap 30062

Attaching to process ID 30062, please wait...

Debugger attached successfully.

Server compiler detected.

JVM version is 24.80-b11

using thread-local object allocation.

Parallel GC with 4 thread(s)

Heap Configuration:

   MinHeapFreeRatio = 0

   MaxHeapFreeRatio = 100

   MaxHeapSize      = 3917479936 (3736.0MB)

   NewSize          = 1310720 (1.25MB)

   MaxNewSize       = 17592186044415 MB

   OldSize          = 5439488 (5.1875MB)

   NewRatio         = 2

   SurvivorRatio    = 8

   PermSize         = 21757952 (20.75MB)

   MaxPermSize      = 85983232 (82.0MB)

   G1HeapRegionSize = 0 (0.0MB)

[#4]

i thought I can look through the catalogd startup script, it looks like the only JAVA option available there is JAVA_HOME:

# cat /usr/bin/catalogd | grep "JAVA"

# Autodetect JAVA_HOME if not defined

if [ -z "${JAVA_HOME}" ]; then

  echo "Unable to find Java. JAVA_HOME should be set in /etc/default/bigtop-utils"

    library_file=`find ${JAVA_HOME}/ -name $library | head -1`

[#5]

Thanks for following up with more details. We're investigation and will get back to you as soon as we have more info.

[#6 · Accepted solution]

Mystery solved, I had a typo as well 🙂

The option should be:

JAVA_TOOL_OPTIONS="-Xmx8g"

(TOOL instead of TOOLS)

[#7]

Adding JAVA_TOOL_OPTIONS=-Xmx8g to "Impala Catalog Server Environment Advanced Configuration Snippet (Safety Valve)" worked. Thanks!

[#8]

iam using the cdh 5.11 and my catalog server is not starting even i keeping the Impala Catalog Server Environment Advanced Configuration Snippet (Safety Valve) as JAVA_TOOL_OPTIONS=-Xmx8g or JAVA_TOOL_OPTIONS=-Xss2m