Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Cloudera manager crashing on start with OutOfMemoryError: unable to create new native thread

Highlighted

Cloudera manager crashing on start with OutOfMemoryError: unable to create new native thread

After a new Installation of cloudera-scm-server on a SLES 12 SP4 system I got following error when starting CM:

 

cloudera-scm-server.service - Cloudera CM Server Service
 Loaded: loaded (/usr/lib/systemd/system/cloudera-scm-server.service; enabled; vendor preset: disabled)
 Active: active (running) since Tue 2019-11-12 13:26:52 CET; 5min ago
  Process: 2339 ExecStartPre=/opt/cloudera/cm/bin/cm-server-pre (code=exited, status=0/SUCCESS)
 Main PID: 2383 (java)
    Tasks: 498 (limit: 512)
   CGroup: /system.slice/cloudera-scm-server.service
           └─2383 /usr/lib64/jvm/java/bin/java -cp .:/usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/java/postgresql-connector-java.jar:lib/* -server -Dlog4j.configuration=file:/etc/cloudera-scm-server/log4j.properties -Dfile.encoding=UTF-8 -Dcmf.root.logger=INFO,LOGFILE -Dcmf.log.dir=/var/log/cloudera-scm-server -Dcmf.log.file=cloudera-scm-server.log -Dcmf.jetty.threshhold=WARN -Dcmf.schema.dir=/opt/cloudera/cm/schema -Djava.awt.headless=true -Djava.net.preferIPv4Stack=true -Dpython.home=/opt/cloudera/cm/python -XX:+HeapDumpOnOutOfMemoryError -Xmx2G -XX:MaxPermSize=256m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:OnOutOfMemoryError=kill -9 %p com.cloudera.server.cmf.Main
Nov 12 13:26:52 heshadoopmp51 systemd[1]: Started Cloudera CM Server Service.
Nov 12 13:26:52 heshadoopmp51 cm-server[2383]: JAVA_HOME=/usr/lib64/jvm/java
Nov 12 13:26:53 heshadoopmp51 cm-server[2383]: OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=256m; support was removed in 8.0
Nov 12 13:26:57 heshadoopmp51 cm-server[2383]: ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console. Set system property 'org.apache.logging.log4j.simplelog.StatusLogger.level' to TRACE to show Log4j2 internal initialization logging.
Nov 12 13:27:28 heshadoopmp51 cm-server[2383]: Exception in thread "MainThread" java.lang.OutOfMemoryError: unable to create new native thread
Nov 12 13:27:28 heshadoopmp51 cm-server[2383]:         at java.lang.Thread.start0(Native Method)
Nov 12 13:27:28 heshadoopmp51 cm-server[2383]:         at java.lang.Thread.start(Thread.java:717)
Nov 12 13:27:28 heshadoopmp51 cm-server[2383]:         at com.cloudera.enterprise.PeriodicEnterpriseService.startService(PeriodicEnterpriseService.java:139)
Nov 12 13:27:28 heshadoopmp51 cm-server[2383]:         at com.cloudera.server.cmf.Main.run(Main.java:606)
Nov 12 13:27:28 heshadoopmp51 cm-server[2383]:         at com.cloudera.server.cmf.Main.main(Main.java:237)

All limits on the system were properly set. The problem was the maximum number of  tasks set to 512. Were is this limit coming from?

 

The problem was solved in the meantime by calling  

systemctl edit cloudera-scm-server

and adding this to the opened file:

[Service]
TasksMax=1024

 I have a similar machine where this problem did not come up. The affected machine has 256GB of RAM and 48 CPUs. Can thisbe the reason?

1 REPLY 1

Re: Cloudera manager crashing on start with OutOfMemoryError: unable to create new native thread

Expert Contributor

@webtransactor  you might also need to check the process limit for user using which you are starting the service/running the service.

 

/etc/security/limits.conf -- is the default file to set the limits

 

you can also check doing -

su - <username>

ulimit -a

 

You can increase ulimit accordingly.

 

Don't have an account?
Coming from Hortonworks? Activate your account here