question Re: Catalogd OutOfMemoryError: Java heap space in Archives of Support Questions (Read Only)

Catalogd OutOfMemoryError: Java heap space

arov — Fri, 16 Sep 2022 10:21:08 GMT

Dear Impala users,

We are seeing the following messages in our catalogd logs:

E0523 05:28:25.921648 15916 catalog-server.cc:279] OutOfMemoryError: Java heap space

Impala version: Impala Shell v2.2.0-cdh5 (2ffd73a) built on Tue Apr 21 12:09:21 PDT 2015

I have tried to increase catalogd memory via CM's catalogd safety valves:

"Catalog Server Command Line Argument Advanced Configuration Snippet (Safety Valve)" set to

-rm_default_memory=8G

And through

"Impala Catalog Server Environment Advanced Configuration Snippet (Safety Valve)" set to

JAVA_TOOLS_OPTIONS="-Xmx8GB"

I have verified through jmap that neither of the above settings have increased JVM max heap available to catalogd process.

I have also read on a post (https://groups.google.com/a/cloudera.org/forum/#!topic/impala-user/1QKAzAHVIO0) that catalogd supports jvm_args or jvm-args, however our version of impala/catalogd does not seem to support such an option, as below command produces no result:

catalogd --help | grep jvm

I was wondering if anyone has any suggestions or if anyone can provide insight into how to go about troubleshooting this issue.

Thank you in advance for your time.

Re: Catalogd OutOfMemoryError: Java heap space

alex.behm — Wed, 25 May 2016 05:19:47 GMT

Setting the JAVA_TOOL_OPTIONS like you did via the CM safety valve is the preferred way and should work.

Did you perhaps have a typo?

JAVA_TOOLS_OPTIONS="-Xmx8GB"

should be

JAVA_TOOLS_OPTIONS="-Xmx8g"

Re: Catalogd OutOfMemoryError: Java heap space

arov — Wed, 25 May 2016 12:33:45 GMT

Hey Alex,

thanks for your response. I just tried copy pasting your version of the line and restarting the catalog server. Jmap still shows 4gb max heap. Is it possible that my version of impala does not support "JAVA_TOOLS_OPTIONS" environment variable? (Impala Shell v2.2.0-cdh5 (2ffd73a) built on Tue Apr 21 12:09:21 PDT 2015 )

# /usr/java/jdk1.7.0_80/bin/jmap -heap 30062

Attaching to process ID 30062, please wait...

Debugger attached successfully.

Server compiler detected.

JVM version is 24.80-b11

using thread-local object allocation.

Parallel GC with 4 thread(s)

Heap Configuration:

MinHeapFreeRatio = 0

MaxHeapFreeRatio = 100

MaxHeapSize = 3917479936 (3736.0MB)

NewSize = 1310720 (1.25MB)

MaxNewSize = 17592186044415 MB

OldSize = 5439488 (5.1875MB)

NewRatio = 2

SurvivorRatio = 8

PermSize = 21757952 (20.75MB)

MaxPermSize = 85983232 (82.0MB)

G1HeapRegionSize = 0 (0.0MB)

Re: Catalogd OutOfMemoryError: Java heap space

arov — Wed, 25 May 2016 12:37:04 GMT

i thought I can look through the catalogd startup script, it looks like the only JAVA option available there is JAVA_HOME:

# cat /usr/bin/catalogd | grep "JAVA"

# Autodetect JAVA_HOME if not defined

if [ -z "${JAVA_HOME}" ]; then

echo "Unable to find Java. JAVA_HOME should be set in /etc/default/bigtop-utils"

library_file=`find ${JAVA_HOME}/ -name $library | head -1`

Re: Catalogd OutOfMemoryError: Java heap space

alex.behm — Wed, 25 May 2016 17:06:33 GMT

Thanks for following up with more details. We're investigation and will get back to you as soon as we have more info.

Re: Catalogd OutOfMemoryError: Java heap space

alex.behm — Wed, 25 May 2016 17:10:03 GMT

Mystery solved, I had a typo as well 🙂

The option should be:

JAVA_TOOL_OPTIONS="-Xmx8g"

(TOOL instead of TOOLS)

Re: Catalogd OutOfMemoryError: Java heap space

arov — Wed, 25 May 2016 17:28:29 GMT

Adding JAVA_TOOL_OPTIONS=-Xmx8g to "Impala Catalog Server Environment Advanced Configuration Snippet (Safety Valve)" worked. Thanks!

Re: Catalogd OutOfMemoryError: Java heap space

sri1993 — Wed, 05 Jul 2017 13:32:02 GMT

iam using the cdh 5.11 and my catalog server is not starting even i keeping the Impala Catalog Server Environment Advanced Configuration Snippet (Safety Valve) as JAVA_TOOL_OPTIONS=-Xmx8g or JAVA_TOOL_OPTIONS=-Xss2m